Remote sensing image semantic segmentation method and system based on diffusion model and knowledge distillation

Abstract

The application discloses a kind of based on diffusion model and knowledge distillation's remote sensing image semantic segmentation method and system, the method includes: sample data set is divided into support set and query set;Using diffusion model generates unlabeled image set;Teacher encoder and student encoder are jointly pre-trained;Teacher encoder extracts features to support set, and prototype is obtained;Teacher encoder extracts features to unlabeled set, realizes prediction;The distance of support set and prototype is calculated as clustering radius, and unlabeled sample is screened;Prototype of unlabeled set is calculated, reverse prototype generation is carried out, and loss is calculated;Pseudo-label is used to update the sample in clustering cluster, realizes prototype update;Student encoder carries out multi-scale feature embedding to query set, and the distance with prototype is calculated, realizes class division;Comprehensive loss, iterative training.The present application combines multi-scale feature extraction, diffusion model, knowledge distillation, positive and negative prototype and other technologies, and can effectively improve model precision and robustness.

Description

TECHNICAL FIELD

[0001] The present application relates to a kind of based on diffusion model and knowledge distillation's semi-supervised multiscale remote sensing image semantic segmentation method and system, belong to land resources remote sensing field and computer vision field. BACKGROUND

[0002] Remote sensing interpretation refers to the characteristics of various identification targets provided by remote sensing images are analyzed, inferred and judged, and ultimately the purpose of identifying targets or phenomena is achieved.Semantic segmentation refers to dividing an image into several groups of pixel regions with specific semantic meanings, and identifying the class of each region, and ultimately obtaining an image with pixel-level semantic annotation.Semantic segmentation technology is widely used in urban planning, disaster assessment and prediction, traffic navigation and other fields.These fields play an important role in their respective fields, so the research on remote sensing image semantic segmentation has certain practical significance.

[0003] At present, most of the remote sensing image processing is still completed by human experts through manual annotation, which is a time-consuming and laborious process.In recent years, with the development of deep learning algorithms, more and more deep learning-based algorithm models have been applied to image semantic segmentation tasks.However, deep learning still needs a certain amount of data set as training samples, but due to the difficulty of manual annotation task and small sample size, the prediction performance of the model is greatly limited, resulting in weak generalization ability of the model, which cannot accurately predict remote sensing images, which faces the small sample problem of deep learning.

[0004] At the same time, compared with natural images, the background of remote sensing images is more complex and variable.In addition, the size difference between different objects in remote sensing images is extremely large, if the receptive field is set too small, the features of large objects cannot be completely extracted;If the receptive field is set too large, the features of small objects are easily ignored. SUMMARY

[0005] The present application aims to provide a kind of based on diffusion model and knowledge distillation's semi-supervised multiscale remote sensing image semantic segmentation method and system, to train multiscale remote sensing image semantic segmentation model, to predict the class of each pixel in new land resources remote sensing image.

[0006] Technical scheme: in order to achieve the above-mentioned application purpose, the present application provides a kind of based on diffusion model and knowledge distillation's remote sensing image semantic segmentation method, including the following steps:

[0007] Step 1: divide the remote sensing image data set into support set and query set, each sample contains original image and mask image;

[0008] Step 2: input all image data samples into the diffusion model, generate images using the forward noise adding process and the backward noise removing process of the diffusion model, and obtain the unlabeled set;

[0009] Step 3: jointly pre-train the teacher encoder for multi-scale feature extraction and the lightweight student encoder;

[0010] Step 4: use the teacher encoder to extract multi-scale features from the support set samples, scale and splice the multi-scale features to obtain the overall features, perform mask average pooling on the features of each scale to obtain sub-prototypes, splice the sub-prototypes of different scales to obtain the support set prototypes, and aggregate the support set prototypes to obtain the clustering factor;

[0011] Step 5: use the teacher encoder to extract multi-scale features from the unlabeled set, measure the distance between the features and each support set cluster, obtain the pseudo label, and predict the mask;

[0012] Step 6: calculate the average distance between the cluster center and the support set samples belonging to the class as the clustering radius, and use it to screen the unlabeled samples;

[0013] Step 7: calculate the unlabeled set prototype and the clustering factor using the predicted mask of the unlabeled set, measure the distance between the support set samples and the unlabeled cluster, and predict the mask of the support set samples, and calculate the loss with the real mask;

[0014] Step 8: update the sample set in the corresponding cluster by the pseudo label of the unlabeled sample screened in step 6, and recalculate the support set prototype;

[0015] Step 9: use the student encoder to extract multi-scale features from the query set samples, measure the distance between the query set samples and each support set cluster, obtain the pseudo label, and predict the mask, and calculate the loss with the real mask of the query set samples;

[0016] Step 10: combine the mask loss of the support set samples in step 7 and the mask loss of the query set samples in step 9 as the total loss, update the network parameters by back propagation, continue the iteration of the training task, and stop until the training endpoint is reached.

[0017] As preferred, in step 1, N classes are selected from all classes of the image data set, and K samples are selected from each class as the support set

[0018] S={(s 1,1 ,M 1,1 ),(s 2,1 ,M 2,1 ),...,(s j,c ,M j,c ),...,(s K,B ,MK,N ), where c = {1, 2,..., N} represents the image category, and j = {1, 2,..., K} represents the order of the sample in the sample set of its corresponding category; and then select one image from the samples not selected from these categories as the query set and represent the original images of the support set and the query set samples, respectively, M j,c and M k represent the respective mask images, k = {1, 2,..., N}, C, H, and W are the channel number, height, and width of the original image, respectively.

[0019] Preferably, the teacher encoder HRNet and the student encoder light-HRNet in step 3 perform feature extraction at four scales of down-sampling 1 / 4, 1 / 8, 1 / 16, and 1 / 32, respectively; wherein the student encoder light-HRNet is a structured pruning of HRNet to reduce the number of convolutional layers to achieve lightweight development.

[0020] Preferably, step 4 includes the following steps:

[0021] Step 4.1: embedding the support set original image into multiple scales of features through the teacher encoder, for the jth image I j,c of the cth category, generating a set of feature embeddings at multiple scales where is the embedding feature of the image I j,c at the ith scale, and are the channel number, height, and width of the feature at the ith scale, respectively;

[0022] scaling the features at each scale to have a dimension of and connecting them as the overall feature embedding belonging to the category c, the calculation process is as follows:

[0023]

[0024] where, α i is a series of learnable scale-specific scalar parameters, FSev j,c is the overall feature of the jth image of the cth category;

[0025] Step 4.2: performing mask average pooling on the embedding feature at a certain scale and the mask after scale adjustment to obtain the sub-prototype at the corresponding scale

[0026] scaling the sub-prototypes at different scales to have a dimension of And then the prototype of the final proportion balance of the cluster category is obtained by splicing, and the calculation process is as follows:

[0027]

[0028] Step 4.3: Aggregating the support set samples and class prototypes in the cluster to obtain the cluster factor RS c , which is used as a feature dictionary, and the calculation process is as follows:

[0029] RS c = [s 1,c ,s 2,c ,...,s K,c ,psev c ]

[0030] As preferred, the step 5 comprises the following steps:

[0031] Step 5.1: Embedding the original image in the unlabeled set through the teacher encoder to obtain a plurality of scale feature embeddings, and for the lth unlabeled image I l , a set of feature embeddings at multiple scales is generated Wherein is the embedding feature of the image I l at the i-th scale, respectively, the channel number, height and width of the feature at the i-th scale;

[0032] Scale it and connect it to the final proportion balance total feature FUev l of the lth unlabeled image belonging to the class c.

[0033] Step 5.2: Scale adjusting the total feature embedding FUev l of the lth image in the unlabeled set to Calculate the feature coefficient vector between it and a support set cluster The calculation process is as follows:

[0034]

[0035] Wherein, ε is a constant value; l = 1, 2,..., L; L is the total number of image samples in the unlabeled set, is the unlabeled image sample; is the cluster factor of the class; is the total feature of the unlabeled image sample; is the sub-coefficient vector related to the cluster of class c of the unlabeled image sample u l .

[0036] The reconstruction error is used to calculate the unlabeled image sample u land the similarity distance du between the support clusters l,c The calculation process is as follows:

[0037]

[0038] where Similarity() represents the calculation of cosine similarity;

[0039] Calculate the pixel-level segmentation probability score mapping of the unlabeled sample belonging to a cluster The calculation process is as follows:

[0040]

[0041] where is the set of label classes , and N represents the number of classes;

[0042] Upsample the pixel-level segmentation probability score mapping to predict the final pseudo-segmentation mask of the pixel-level label annotation The calculation process is as follows:

[0043]

[0044] where the upsampling process is realized by bilinear interpolation.

[0045] As a preferred, the step 6 includes the following steps:

[0046] Step 6.1: For the overall feature embedding of the jth image sample belonging to class c in the support set, calculate the feature coefficient vector between it and the class c, and use the reconstruction error to obtain the similarity distance ds between the support set sample and the cluster j,c ;

[0047] Step 6.2: According to the maximum similarity and the minimum similarity between the support set sample belonging to class c and the prototype, the class radius is obtained, and the calculation process is as follows:

[0048]

[0049] Step 6.3: Use the similarity measure function allclose(du l,c ,r c ) to compare the similarity du of the unlabeled sample to the cluster l,c and the closeness of the class radius r c , and only keep the unlabeled samples whose value is true.

[0050] As a preferred, the step 7 includes the following steps:

[0051] Step 7.1: embedding features of the i-th scale of the l-th image in the unlabeled set are obtained without label with the mask adjusted in scale Mask average pooling is performed to obtain the sub-prototype of the corresponding scale The sub-prototypes of different scales are scaled and spliced to obtain the final proportion-balanced prototype of the cluster class in the unlabeled set.

[0052] Step 7.2: The cluster factor is obtained from the unlabeled set samples in the cluster and the class prototype, and is used as a feature dictionary.

[0053] Step 7.3: For the multi-scale feature embedding FSev j,c of the j-th image in the support set, the feature coefficient vector between it and a certain cluster class is calculated, and the similarity ds j,c between the sample and the unlabeled cluster is obtained using the reconstruction error; the pixel-level segmentation probability score mapping of the support set sample belonging to a certain cluster is calculated based on the similarity , and is up-sampled to predict the pseudo-segmentation mask of the support set sample

[0054] Step 7.4: Cross-entropy loss is calculated using the support set prediction mask and the real image mask.

[0055] As a preferred, the step 9 comprises the following steps:

[0056] Step 9.1: The original image of the query set is embedded in multi-scale features by the student encoder, and for the k-th image I k , a set of feature embeddings in multiple scales is generated wherein is the embedding feature of the image I k in the i-th scale, respectively, the number of channels, height and width of the feature in the i-th scale; the multi-scale feature embedding of the query set is scaled and connected to obtain the overall feature FQev k ;

[0057] Step 9.2: For the overall feature embedding FQev k , the feature coefficient vector between it and a certain cluster class is calculated; the similarity between the sample and the cluster is calculated using the reconstruction error, and the pixel-level segmentation probability score mapping of the sample belonging to a certain cluster is calculated based on the similarity , and is up-sampled to predict the final pseudo-segmentation mask

[0058] Step 9.3: Cross-entropy loss is calculated using the query set prediction mask and the real mask.

[0059] Based on the same inventive concept, the application provides a remote sensing image semantic segmentation system based on a diffusion model and knowledge distillation, which comprises:

[0060] A preprocessing module is configured to divide a remote sensing image dataset into a support set and a query set, and each sample contains an original image and a mask image.

[0061] A knowledge distillation model pre-training module is configured to jointly pre-train a multi-scale feature extraction teacher encoder and a lightweight student encoder thereof.

[0062] A diffusion model generated image module is configured to input all image data samples into a diffusion model, generate a label-free image by using a forward noise adding process and a backward noise removing process of the diffusion model, and obtain a label-free set for semi-supervised learning.

[0063] A semi-supervised learning training module is configured to use the teacher encoder to perform multi-scale feature extraction on the support set samples, splice the multi-scale features after scale reduction to obtain overall features, perform mask average pooling on the features of each scale to obtain sub-prototypes, splice the sub-prototypes of different scales to obtain a support set prototype, and aggregate the support set prototype with the support set samples to obtain clustering factors; use the teacher encoder to perform multi-scale feature extraction on the label-free set, measure the distance between the features and each support set cluster, obtain pseudo labels, and predict masks; at the same time, calculate the average distance between the clustering center and the support set samples belonging to the class as the clustering radius, and use the clustering radius to screen the label-free samples; calculate the label-free set prototype and the clustering factors by using the predicted masks of the label-free set, measure the distance between the support set samples and the label-free cluster, and predict the masks of the support set samples; calculate the loss by using the real masks of the support set samples; update the sample set in the corresponding clustering cluster by using the pseudo labels of the screened label-free samples, and recalculate the support set prototype; use the student encoder to perform multi-scale feature extraction on the query set samples, measure the distance between the query set samples and each support set cluster, obtain pseudo labels, and predict masks; calculate the loss by using the real masks of the query set samples; combine the mask loss of the support set samples and the mask loss of the query set samples as a total loss, update network parameters by back propagation, continue to perform iteration of the training task, and stop until the training endpoint is reached.

[0064] Based on the same inventive concept, the application provides a computer system, which comprises a memory, a processor, and a computer program stored on the memory and capable of running on the processor, wherein the computer program is loaded into the processor to implement the steps of the remote sensing image semantic segmentation method based on the diffusion model and the knowledge distillation.

[0065] Beneficial effects: The half-supervised multi-scale remote sensing image semantic segmentation method based on a diffusion model and knowledge distillation provided by the application has the following advantages compared with the prior art: 1, the method of knowledge distillation is used to realize knowledge transfer, the teacher encoder and the lightweight student encoder are pre-trained to realize weight setting, and the reduction of network model delay and the compression of network parameters can be realized. 2, in the application, the noise adding and noise removing operation of the diffusion model is used to realize image generation, which helps the model to realize half-supervised learning. 3, in the application, the average distance from the labeled sample to the class prototype is used as the radius to filter the unlabeled samples generated by the diffusion model, so as to prevent the interference of inaccurate unlabeled images on the model training. 4, compared with traditional images, the feature information of remote sensing images is richer, and the multi-scale feature extractor used in the application can maintain high-resolution feature representation in the whole process of image feature extraction compared with the traditional feature extractor, so that the output feature map with richer feature information can be obtained, and the segmentation precision is improved. 5, in the past, the distance between the features of the unlabeled sample and the prototype is directly clustered, but in this way, the data distribution information and feature information of the labeled sample in the class are not considered, and the multi-factor clustering method is used in the application, which can greatly enhance the robustness of the prototype network classification prediction ability. 6, the application introduces the reverse prototype generation, which is expected to find the most representative prototype of the same target area from the unlabeled data, and the labeled data will guide it more strictly and directly. 7, the application combines the training label and the pseudo label through the self-training mode of half-supervised learning of the unlabeled image, realizes the fine-tuning update of the prototype, makes the prototype more representative of the class center, and thus improves the generalization ability and training performance of the model. 8, the application uses the lightweight student encoder to predict the query set sample to realize the optimization of the loss and improve the prediction accuracy of the model. BRIEF DESCRIPTION OF DRAWINGS

[0066] Figure 1 is the overall flowchart of the embodiment of the application;

[0067] Figure 2 is the half-supervised prototype network flowchart based on a diffusion model and knowledge distillation in the embodiment of the application;

[0068] Figure 3 is the diffusion model flowchart in the embodiment of the application;

[0069] Figure 4 is the network structure diagram of the teacher encoder and the student encoder in the embodiment of the application;

[0070] Figure 5 is the MFG (multi-scale feature embedding and overall feature embedding generation) module structure diagram in the embodiment of the application;

[0071] Figure 6This is a structural diagram of the PFD (Prototype and Feature Dictionary Generation) module in an embodiment of the present invention;

[0072] Figure 7 This is a structural diagram of the LM (predictive label generation and mask generation) module in an embodiment of the present invention. Detailed Implementation

[0073] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading the present invention, any modifications of the present invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0074] like Figure 1 As shown in the figure, the semi-supervised multi-scale remote sensing image semantic segmentation method based on diffusion model and knowledge distillation disclosed in this invention mainly includes the following steps:

[0075] Step 1: Divide the remote sensing image dataset into a support set and a query set. Each sample contains both the original image and the masked image. In this embodiment, the data is divided into support and query sets according to N-way K-shot; that is, N classes are selected from all categories of the image dataset, and K samples are selected from each class as the support set S = {(s...} 1,1 M 1,1 ),(s 2,1 M 2,1 ),...,(s j,c M j,c ),...,(s K,N M K,N )}, where c =

[0076] {1,2,...,N} represents the image categories, and j = {1,2,...,K} represents the order of the sample in the sample set of its corresponding category. Then, an image is selected from each of the unselected samples in these categories as the query set. and M represents the original graphs of the support set and query set samples, respectively. j,c and M k These represent their respective mask images, k = {1, 2, ..., N}, and C, H, and W are the number of channels, height, and width of the original image, respectively.

[0077] Step 2: Input all image data samples into the diffusion model, and use the forward noise addition and backward noise reduction processes of the diffusion model to generate an unlabeled set. This specifically includes the following steps:

[0078] Step 2.1: input all data samples into the diffusion model, add noise to the image using the forward process of the diffusion model, that is, generate random Gaussian noise ε for each time t image t and get the final pure noise image X t , the calculation process is as follows:

[0079]

[0080] wherein, α i =1-β i , β i ∈(0,1) is the variance hyperparameter of Gaussian distribution, and X0 is the initial image.

[0081] Step 2.2: extract the features of the pure noise image using the feature extraction model U-Net, that is, get ε θ (X t ,t), define the loss function as follows to optimize the above feature extraction model U-Net:

[0082]

[0083] wherein, θ is the parameter of the feature extraction model U-Net.

[0084] Step 2.3: the backward denoising process removes noise by time iteration, and the noise image at time t-1 can be obtained according to the noise image at time t by the following formula, to complete the generation of the unlabelled image:

[0085]

[0086] wherein, σ t is the variance, α t =1-β t , β t ∈(0,1) is the variance hyperparameter of Gaussian distribution.

[0087] Step 3: jointly pre-train the teacher encoder and student encoder using public datasets. As shown in Figure 4 , the teacher encoder HRNet and the student encoder light-HRNet in this embodiment respectively extract features according to four scales of downsampling 1 / 4, 1 / 8, 1 / 16 and 1 / 32. Among them, the student encoder light-HRNet is a structured pruning of HRNet, which reduces the number of convolution layers to realize lightweight development.

[0088] Specifically, the joint pre-training in step 3 includes the following steps:

[0089] Step 3.1: Pre-train the teacher encoder using a public dataset (e.g. mini-imagenet), and calculate the softmax function δ(Z t ; T = τ) at temperature T = τ. Where Z t represents the prediction of soft labels made by the teacher model.

[0090] Pre-train the student encoder using a public dataset, and calculate the softmax function δ(Z s ; T = τ) at temperature T = τ. Where Z s represents the prediction of soft labels made by the student model.

[0091] Calculate their cross-entropy loss:

[0092]

[0093] Where b represents the current pre-training batch, and b max represents the maximum pre-training batch.

[0094] Pre-train the student encoder using a public dataset, and calculate the softmax function δ(Z s ; T = 1) at temperature T = 1.

[0095] Calculate the cross-entropy loss between the hard labels predicted by the student model and the real labels:

[0096]

[0097] Where y represents the real image label.

[0098] Step 3.2: Optimize the following loss function to obtain the pre-training weights, and freeze the weights of the teacher encoder.

[0099] L pre = ζLoss soft + ηLoss hard

[0100] Where ζ ∈ (0, 1), and η = 1 - ζ.

[0101] Step 4: Use the teacher encoder to perform multi-scale feature extraction on the support set, and concatenate to obtain the overall feature embedding. And calculate the sub-prototype with the mask, and then concatenate to obtain the proportionally balanced prototype, and form the clustering factor with the support set samples. Including the following steps:

[0102] Step 4.1: Perform multi-scale feature embedding on the support set original image through the teacher model multi-scale feature extractor f. For each image I j,c of class c, generate a set of feature embeddings at multiple scales where is the embedding feature of the support set sample image j at the i-th scale, and are the number of channels, height and width of the feature of the sample at the i-th scale, respectively.

[0103] Scale the multi-scale feature of the support set sample j to have dimension and concatenate it as the overall feature embedding FSev j,c of the class c. The computation process is as follows:

[0104]

[0105] where, α i is a series of learnable scale-specific scalar parameters, which can be used as adjustment factors to adjust the proportion of embedding features extracted at each scale.

[0106] Step 4.2: Mask average pooling the embedding feature of a certain scale with the scale-adjusted mask to obtain the sub-prototype of the corresponding scale. The calculation process of mask average pooling is as follows:

[0107]

[0108] where, denotes the scale adjustment of the mask image;

[0109] Scale the sub-prototypes of different scales to have dimension and concatenate them to obtain the final proportion-balanced prototype of the cluster class, the calculation process is as follows:

[0110]

[0111] Step 4.3: Aggregate the support set samples and class prototypes within the cluster to obtain the cluster factor RS c , which is used as a feature dictionary, the calculation process is as follows:

[0112] RS c = [s 1,c ,s 2,c ,...,s K,c ,psev c ]

[0113] Step 5: Multi-scale feature extraction on the unlabeled set using the teacher encoder and concatenation to get the overall feature embedding. Obtain the feature coefficient vector between the sample and the support set cluster, measure the distance between the unlabeled set sample and each cluster, predict the cluster category, get the pseudo label, and predict the mask. Including the following steps:

[0114] Step 5.1: Multi-scale feature embedding of the unlabeled set original image through the teacher model multi-scale feature extractor f. For each image I l , generate a set of feature embeddings at multiple scales where is the embedded feature of the image I l at the i-th scale, respectively, the channel number, height and width of the feature at the i-th scale.

[0115] Scale it and concatenate it as the final proportionally balanced total feature of the l-th unlabeled image belonging to class c

[0116] Step 5.2: Scale the overall feature embedding FUev l of the l-th image in the unlabeled set to Calculate the feature coefficient vector between it and a support set cluster The calculation process is as follows:

[0117]

[0118] where ε is a constant value; l = 1, 2,..., L; L is the total number of image samples in the unlabeled set, is the unlabeled image sample, C, H, and W are the channel number, height and width respectively; is the cluster factor of this class; is the overall feature of this unlabeled image sample; is the overall feature of the unlabeled image sample u l and the sub-coefficient vector related to the cluster of class c;

[0119] Use the reconstruction error to calculate the similarity distance du l between the unlabeled image sample u l,c and the support set cluster, the calculation process is as follows:

[0120]

[0121] where Similarity() represents the calculation of cosine similarity;

[0122] Calculate the pixel-level segmentation probability score map of the unlabeled sample belonging to a cluster The calculation process is as follows:

[0123]

[0124] wherein is each class in the label class set , and N represents the number of classes;

[0125] Upsampling the pixel-level segmentation probability score mapping to predict the final pseudo-segmentation mask of the pixel-level label annotation The calculation process is as follows:

[0126]

[0127] wherein the upsampling process is realized by means of bilinear interpolation.

[0128] Step 6: Calculate the distance between the support set samples belonging to a cluster and the class center thereof, so as to obtain the average distance as the cluster radius; compare the distance between the unlabeled sample and the cluster, and complete the screening of self-training. Including the following steps:

[0129] Step 6.1: For the total feature embedding FSev j,c of the image j in the support set, calculate the feature coefficient vector between it and a cluster category Use the reconstruction error to obtain the distance ds j,c between the support set sample and the cluster.

[0130] Step 6.2: According to the maximum similarity and the minimum similarity between the support set sample belonging to the category c and the prototype, the category radius is obtained, and the calculation process is as follows:

[0131]

[0132] Step 6.3: Use the similarity measure function allclose(du l,c , r c ) to compare the similarity du l,c of the unlabeled sample to the cluster and the closeness of the category radius r c , and only keep the unlabeled samples whose value is true.

[0133] Step 7: Calculate the prototype (inverse prototype) of the unlabeled set and the cluster factor using the predicted mask of the unlabeled set, measure the distance between the support set sample and the unlabeled cluster, and predict the mask of the support set sample. Calculate the loss with the real mask. Including the following steps:

[0134] Step 7.1: The embedding feature of the unlabeled set image at a certain scale Compared with the scaled mask Perform masked average pooling to obtain sub-prototypes of the corresponding scale.

[0135] Scale the sub-prototypes at different scales to make their dimensions change. Then, by splicing them together, a prototype puev representing the final balanced proportion of cluster categories in the unlabeled set is obtained. c .

[0136] Step 7.2: Obtain the clustering factor RU from the unlabeled set samples and class prototypes within the cluster. c And use it as a feature dictionary.

[0137] Step 7.3: For the multi-scale feature embedding FSev of the support set image j, j,c Calculate the feature coefficient vector between it and a certain cluster category. The similarity ds between the sample and the unlabeled cluster can be obtained using the reconstruction error. j,c Calculate the pixel-level segmentation probability score mapping of support set samples belonging to a certain cluster. Upsample the sample to predict the pseudo-segmentation mask for the support set.

[0138] Step 7.4: Calculate the cross-entropy loss using the support set prediction mask and the ground truth image mask. The calculation process is as follows:

[0139]

[0140]

[0141] Where n is the total number of pixels in the image.

[0142] Step 8: Update the sample set in the corresponding cluster using the pseudo-labels of the unlabeled samples filtered in Step 6, and recalculate the support set prototype. This includes the following steps:

[0143] Step 8.1: Calculate the similarity du between unlabeled image samples and a certain cluster. l,c The clustering category α of the unlabeled sample can be obtained. l The calculation process is as follows:

[0144]

[0145] Update samples in clusters The calculation process is as follows:

[0146]

[0147] in, For αl The cluster of the class only contains the samples of the support set.

[0148] Step 8.2: The mask of the unlabeled image belonging to the class l The mask of the unlabeled image belonging to the class Adjust the dimension, and obtain the updated sub-prototypes of different scales by using mask average pooling and splicing, to obtain the overall feature embedding of the cluster class, i.e. the updated prototype Calculate the cluster factor using the samples in the cluster and the class prototype and use it as a feature dictionary.

[0149] Step 9: Use the student encoder to perform multi-scale feature extraction on the query set samples, measure the distance between the query set samples and each support set cluster, obtain the pseudo label, and predict the mask, and calculate the loss with the real mask of the query set sample. Including the following steps:

[0150] Step 9.1: The original image of the query set is input into the student encoder for multi-scale feature embedding, and for each image I k , a set of feature embeddings of multiple scales is generated wherein is the embedding feature of the image I k at the i-th scale, respectively, the channel number, height and width of the feature at the i-th scale;

[0151] Scale the multi-scale feature embedding of the query set, and connect it as the overall feature FQev k of the single sample k belonging to the class c;

[0152] Step 9.2: For the overall feature embedding FQev k of the image k in the query set, calculate the feature coefficient vector between it and a cluster class, and calculate the similarity d k between the sample and the cluster using the reconstruction error. Calculate the pixel-level segmentation probability score map of the sample belonging to the cluster, and upsample it to predict the final pseudo segmentation mask

[0153] Step 9.3: Calculate the cross-entropy loss using the predicted mask of the query set and the real mask, and the calculation process is as follows:

[0154]

[0155]

[0156] Step 10: Calculate the loss of the algorithm, back-propagate to update the network parameters, continue the iteration of the training task until the training endpoint is reached, evaluate the model performance on the target task, and the calculation process is as follows:

[0157] Loss = Loss s + λLoss c

[0158] where λ is usually a time-varying Gaussian function Loss = Loss s and Loss c is balanced, where p is the current training batch, p max is the maximum number of training times.

[0159] Based on the same inventive concept, the embodiment of the present application discloses a remote sensing image semantic segmentation system based on diffusion model and knowledge distillation, comprising: a preprocessing module for dividing a remote sensing image dataset into a support set and a query set, each sample containing both an original image and a mask image; a knowledge distillation model pre-training module for jointly pre-training a multi-scale feature extraction teacher encoder and its lightweight student encoder; a diffusion model generated image module for inputting all image data samples into a diffusion model to generate unlabeled images using the forward noise adding process and the backward noise removing process of the diffusion model, obtaining an unlabeled set for semi-supervised learning; a semi-supervised learning training module for using a teacher encoder to perform multi-scale feature extraction on support set samples, scaling and splicing the multi-scale features to obtain overall features, performing mask average pooling on the features of each scale to obtain sub-prototypes, splicing the sub-prototypes of different scales to obtain a support set prototype, and aggregating the support set samples to obtain a clustering factor; using a teacher encoder to perform multi-scale feature extraction on the unlabeled set, measuring the distance between its features and each support set cluster to obtain pseudo labels and predict the mask; at the same time, calculating the average distance between the cluster center and the support set samples belonging to the class as the clustering radius, and using it to filter the unlabeled samples; calculating the unlabeled set prototype and the clustering factor using the predicted mask of the unlabeled set, measuring the distance between the support set samples and the unlabeled cluster, and predicting the mask of the support set samples, and calculating the loss with the true mask; updating the sample set in the corresponding cluster by the pseudo label of the filtered unlabeled sample, and recalculating the support set prototype; using a student encoder to perform multi-scale feature extraction on query set samples, measuring the distance between the query set samples and each support set cluster to obtain pseudo labels and predict the mask, and calculating the loss with the true mask of the query set sample; and combining the mask loss of the support set sample and the mask loss of the query set sample as the total loss, back-propagating to update the network parameters, continuing the iteration of the training task until the training endpoint is reached.

[0160] Based on the same inventive concept, the embodiment of the present application discloses a computer system, which comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor, and when the computer program is loaded into the processor, the steps of the semantic segmentation method for remote sensing images based on diffusion model and knowledge distillation are implemented.

Claims

1. A remote sensing image semantic segmentation method based on a diffusion model and knowledge distillation, characterized in that, The method comprises the following steps: Step 1: dividing a remote sensing image dataset into a support set and a query set, each sample containing an original image and a mask image; Step 2: inputting all image data samples into a diffusion model, using the forward noise adding process and the backward noise removing process of the diffusion model to generate images, and obtaining an unlabeled set; Step 3: jointly pre-training a multi-scale feature extraction teacher encoder and a lightweight student encoder; Step 4: using the teacher encoder to perform multi-scale feature extraction on the support set samples, scaling the multi-scale features and then splicing to obtain overall features, performing mask average pooling on the features of each scale to obtain sub-prototypes, splicing the sub-prototypes of different scales to obtain a support set prototype, and aggregating the support set prototype with the support set samples to obtain a clustering factor; Step 5: using the teacher encoder to perform multi-scale feature extraction on the unlabeled set, measuring the distance between the features and each support set cluster to obtain pseudo labels, and predicting a mask; Step 6: calculating the average distance between the cluster center and the support set samples belonging to the cluster as a cluster radius, and using the cluster radius to screen the unlabeled samples; Step 7: calculating the unlabeled set prototype and the clustering factor using the predicted mask of the unlabeled set, measuring the distance between the support set samples and the unlabeled cluster, and predicting the mask of the support set samples, and calculating the loss with the real mask; Step 8: updating the sample set in the corresponding cluster by the pseudo label of the unlabeled sample screened in step 6, and recalculating the support set prototype; Step 9: using the student encoder to perform multi-scale feature extraction on the query set samples, measuring the distance between the query set samples and each support set cluster to obtain pseudo labels, and predicting a mask, and calculating the loss with the real mask of the query set sample; Step 10: combining the mask loss of the support set sample in step 7 and the mask loss of the query set sample in step 9 as a total loss, updating the network parameters by back propagation, and continuing to iterate the training task until the training endpoint is reached.

2. The method of claim 1, wherein, The step 1 selects N classes from all classes of the image dataset, and selects K samples in each class as a support set S = {(s 1,1 , M 1,1 ), (s 2，1 , M 2，1 ),..., (s j，c , M j，c ),..., (s K,N , M K，N )} where c = {1, 2,..., N} represents the image class, and j = {1, 2,..., K} represents the order of the sample in the sample set of its corresponding class; and then selects one image from the samples not selected in these classes as a query set Q = {(q1, M1), (q2, M2),..., (q k , M k ),..., (q N , M N )}. and represent the original images of the support set and the query set samples respectively, M j，c and M k represent the mask images respectively, k = {1, 2,..., N}, C, H, and W are the channel number, height, and width of the original image respectively.

3. The method of claim 1, wherein, In step 3, the teacher encoder HRNet and the student encoder light-HRNet perform feature extraction according to four scales of 1 / 4, 1 / 8, 1 / 16 and 1 / 32 downsampling respectively; wherein the student encoder light-HRNet is a structured pruning of the HRNet, which reduces the number of convolutional layers to realize lightweight development.

4. The method of claim 1, wherein, Step 4 comprises the following steps: Step 4.1: The support set original image is embedded with multi-scale features by the teacher encoder, and the feature embedding set at multiple scales is generated for the jth image I of the c class j，c wherein is the image I j，c embedding feature at the ith scale, and are the channel number, height and width of the feature at the ith scale, respectively;​ Each scale feature is scaled in dimension to and concatenated into an overall feature embedding belonging to class c, computed as follows: where α i is a set of learnable scale-specific scalar parameters, FSev j,c is the overall feature of the jth image of the cth class, C, H, and W are the channel number, height, and width of the original image, respectively. Step 4.2: embedding features of certain scales masking with the scaled mask mask average pooling to get sub-prototypes of corresponding scales The sub-prototypes of different scales are scaled to have dimensions of and spliced to obtain the final proportionally balanced prototype of the clustering category, and the calculation process is as follows: Step 4.3: Aggregating the support set samples and class prototypes within a cluster to obtain the cluster factor RS c Using it as the feature dictionary, the calculation process is as follows: RS c = [s 1，c , s 2，c ,..., s K，c , psev c ] where s j，c represents the original image supporting the cth class jth sample, j = {1, 2,..., K}, K is the number of samples in each class.

5. The method of claim 1, wherein, Step 5 comprises the following steps: Step 5.1: The unlabeled set original images are passed through the teacher encoder for multi-scale feature embedding, for the l-th unlabeled image I l , a set of feature embeddings at multiple scales are generated where is the image I l embedding feature at the i-th scale, is the channel number, height and width of the feature at the i-th scale, respectively; scaling it and concatenating it as the final proportionally balanced total feature FUevbelonging to class c of the 1st unlabeled image l ; Step 5.2: Compute the overall feature embedding FUevfor the l-th image in the unlabeled set l Scale it to have unit norm Compute the feature coefficient vector The computation proceeds as follows: wherein, ε is a constant value; l = 1, 2,..., L; L is the total number of image samples in the unlabeled set, is an unlabeled image sample, C, H, W are the channel number, height and width respectively; is the cluster factor of the category; is the overall feature of the unlabeled image sample u is the overall feature of the unlabeled image sample u l is the sub-coefficient vector related to the cluster of the category c. The no-label image sample u is calculated using the reconstruction error l and the similarity distance between the support clusters du l，c The calculation process is as follows: Wherein, Similarity() represents the calculation of cosine similarity; Computing pixel-level segmentation probability score maps of whether an unlabeled sample belongs to a certain cluster The computation proceeds as follows: wherein is a set of label classes each class in the set of classes, N denotes the number of classes. Upsample the pixel-level segmentation probability score map to predict the final pseudo segmentation mask of pixel-level label annotation The calculation process is as follows: Wherein, the upsampling process is realized by bilinear interpolation.

6. The method of claim 1, wherein, Step 6 comprises the following steps: Step 6.1: For the support set of the jth image sample belonging to class c, compute its feature coefficient vector with respect to class c, the similarity distance ds between the support set sample and the cluster can be obtained using the reconstruction error j，c ; Step 6.2: The maximum similarity between the support set samples and the prototypes belonging to class c and the minimum similarity The class radius is obtained by the following calculation procedure: Step 6.3: The closeness of the unlabeled samples to the cluster similarity du l，c and the class radius r c is compared using the similarity measure function allclose(du l，c , r c ) and only the unlabeled samples whose value is true are kept.

7. The method of claim 1, wherein, Step 7 comprises the following steps: Step 7.1: embedding features of the i-th scale of the first image in the unlabeled set masking with the scale-adjusted mask mask average pooling to obtain the sub-prototype of the corresponding scale scaling the sub-prototypes of different scales and splicing them to obtain the final proportionally balanced prototype of the cluster class in the unlabeled set; Step 7.2: obtaining a clustering factor from the unlabeled set samples in the cluster and the class prototype, and using the clustering factor as a feature dictionary; Step 7.3: For the multi-scale feature embedding FSev of the jth image in the support set j，c , compute its feature coefficient vector with respect to a cluster class, and use the reconstruction error to obtain the similarity ds between the sample and the unlabeled cluster j，c ; mapping the support set sample to a pixel-level segmentation probability score based on the similarity computation and upsampled, predicting a pseudo segmentation mask for the support set sample Step 7.4: calculating the cross-entropy loss using the predicted mask of the support set and the real image mask.

8. The method of claim 1, wherein the method is characterized by, Step 9 comprises the following steps: Step 9.1: The original images of the query set are embedded with multi-scale features by the student encoder, and the kth image I k is embedded with multi-scale features to generate a set of feature embeddings at multiple scales , where is the image I k , and the embedded features at the ith scale are , respectively, the number of channels, height, and width of the features at the ith scale. The multi-scale feature embeddings of the query set are scaled by the scale and connected to obtain the overall feature FQev k ; Step 9.2: For the overall feature embedding FQev k , compute its feature coefficient vector with respect to a cluster class; compute the similarity between the sample and the cluster using the reconstruction error, and based on the similarity compute the pixel-level segmentation probability score map of the sample belonging to the cluster and upsample it to predict the final pseudo segmentation mask Step 9.3: calculating the cross-entropy loss using the predicted mask of the query set and the real mask.

9. A remote sensing image semantic segmentation system based on a diffusion model and knowledge distillation, characterized in that, It comprises: A preprocessing module for dividing a remote sensing image dataset into a support set and a query set, each sample containing an original image and a mask image; The knowledge distillation model pre-training module is configured to jointly pre-train the multi-scale feature extraction teacher encoder and the lightweight student encoder thereof. The diffusion model generated image module is configured to input all image data samples into the diffusion model, generate unlabelled images by using the forward noise adding process and the backward noise removing process of the diffusion model, and obtain an unlabelled set for semi-supervised learning. The semi-supervised learning training module is configured to use the teacher encoder to perform multi-scale feature extraction on the support set samples, splice the multi-scale features after scale scaling to obtain overall features, perform mask average pooling on the features of each scale to obtain sub-prototypes, splice the sub-prototypes of different scales to obtain support set prototypes, and aggregate the support set prototypes and the support set samples to obtain clustering factors; use the teacher encoder to perform multi-scale feature extraction on the unlabelled set, measure distances between the features and each support set cluster, obtain pseudo labels, and predict masks; meanwhile, calculate average distances between clustering centers and support set samples belonging to the cluster as clustering radii, and use the clustering radii to screen unlabelled samples; calculate unlabelled set prototypes and clustering factors by using the predicted masks of the unlabelled set, measure distances between the support set samples and unlabelled clusters, and predict masks of the support set samples; and calculate losses by using the masks and real masks of the support set samples. The sample set in the corresponding cluster is updated by the pseudo labels of the screened unlabelled samples, and the support set prototypes are recalculated. The student encoder is used to perform multi-scale feature extraction on the query set samples, measure distances between the query set samples and each support set cluster, obtain pseudo labels, and predict masks, and losses are calculated by using the masks and real masks of the query set samples. The mask loss of the support set samples and the mask loss of the query set samples are combined as a total loss, network parameters are updated by back propagation, and the training task is iterated until a training endpoint is reached.

10. A computer system comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The computer program is loaded into the processor to implement the steps of the diffusion model and knowledge distillation based remote sensing image semantic segmentation method according to any one of claims 1-8.

Images