Building element segmentation model training method, building element segmentation method and device

Abstract

The application provides a ground object element segmentation model training method, a ground object element segmentation method and device, and belongs to the technical field of computers.The model training method comprises the following steps: acquiring a labeled remote sensing image sample, a ground object element classification label of the labeled remote sensing image sample and an unlabeled remote sensing image sample; based on the labeled remote sensing image sample, the ground object element classification label and the unlabeled remote sensing image sample, performing semi-supervised training on a student network model until the comprehensive loss value is less than a loss threshold value or a preset training number of rounds is reached, and the student network model is used for segmenting ground object elements; and determining the trained student network model as a target ground object element segmentation model. By combining a supervised first loss value, a second loss value obtained by calculating a loss in a consistency regularization mode and a third loss value obtained by calculating a loss in a contrast learning mode, a comprehensive loss value is calculated, and then the model parameters are updated, so that the amount of data labeling data can be reduced, and the segmentation precision can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to a ground object element segmentation model training method and device and a ground object element segmentation method. BACKGROUND

[0002] Semantic segmentation, also known as semantic labeling, is one of the most basic and challenging tasks in remote sensing image understanding. The goal is to assign a pixel-level semantic class label to a given image. In particular, semantic segmentation in very high resolution (VHR) aerial images plays an increasingly important role in its wide range of applications, such as road extraction, urban planning, and land cover classification.

[0003] In the field of semantic segmentation, labeling is often difficult. Because the mask annotation requires close fitting with the target edge, otherwise it will bring additional loss on the boundary. On the contrary, the amount of unlabeled data is generally much larger than the amount of labeled data.

[0004] High-resolution remote sensing images have complex scenes, with complex ground features and rich background interference. The workload of pixel-level labeling is greater, and the general framework is not designed for remote sensing images. Therefore, it is a great challenge to accurately classify multiple temporal ground object elements in complex remote sensing image scenes with a small amount of fine-labeled data. SUMMARY

[0005] To solve the problems in the prior art, the embodiments of the present application provide a ground object element segmentation model training method, a ground object element segmentation method and device.

[0006] In a first aspect, the present application provides a ground object element segmentation model training method, comprising:

[0007] obtaining a labeled remote sensing image sample, a ground object element classification label of the labeled remote sensing image sample, and an unlabeled remote sensing image sample;

[0008] based on the labeled remote sensing image sample, the ground object element classification label and the unlabeled remote sensing image sample, performing semi-supervised training on a student network model until the comprehensive loss value is less than a loss threshold or a preset training number of rounds is reached, the student network model being used for segmenting ground object elements;

[0009] determining the trained student network model as a target ground object element segmentation model;

[0010] the semi-supervised training of the student network model comprises:

[0011] inputting the labeled remote sensing image sample into the student network model, and calculating a multi-class cross-entropy loss based on the ground feature element classification label and a first classification probability prediction map output by the student network model, to obtain a first loss value, the classification probability prediction map being used to represent a predicted classification probability of each pixel point in the remote sensing image;

[0012] inputting the unlabeled remote sensing image sample processed by the two strong data enhancement methods into two teacher network models, and averaging classification probability prediction maps output by the teacher network models to obtain a second classification probability prediction map, the teacher network model having the same structure as the student network model;

[0013] inputting the unlabeled remote sensing image sample processed by the weak data enhancement method into the student network model to obtain a third classification probability prediction map output by the student network model;

[0014] calculating a loss based on each first pixel point in the second classification probability prediction map and the third classification probability prediction map by a consistency regularization method to obtain a second loss value, the entropy value of the first pixel point being less than an entropy threshold value;

[0015] calculating a loss based on each second pixel point in the second classification probability prediction map and the third classification probability prediction map by a contrastive learning method to obtain a third loss value, the entropy value of the second pixel point being greater than or equal to the entropy threshold value;

[0016] determining the comprehensive loss value based on the first loss value, the second loss value and the third loss value;

[0017] updating the parameters of the student network model by minimizing the comprehensive loss value;

[0018] updating the parameters of each teacher network model by a sliding average method based on the parameters of the student network model.

[0019] Optionally, according to the ground feature element segmentation model training method provided in the present application, the calculating a loss based on each second pixel point in the second classification probability prediction map and the third classification probability prediction map by a contrastive learning method to obtain a third loss value comprises:

[0020] filtering each second pixel point in the second classification probability prediction map and each third pixel point in the third classification probability prediction map based on a classification probability threshold value to determine a plurality of fourth pixel points and a query sample of each classification, the entropy value of the third pixel point being greater than or equal to the entropy threshold value, the predicted classification probability of the fourth pixel point being less than the classification probability threshold value, and the predicted classification probability of the query sample being greater than or equal to the classification probability threshold value;

[0021] Uniform sampling is performed on the query samples of each category to obtain positive samples of each category;

[0022] For each target category in the plurality of categories, a plurality of to-be-sequenced pixel points are filtered out from the plurality of fourth pixel points, and based on similarities between the to-be-sequenced pixel points and the positive samples of the target category, the plurality of to-be-sequenced pixel points are sequenced in descending order of the similarities to obtain a pixel point sequence, and based on a preset sampling configuration, the negative samples of the target category are obtained by sampling features of the pixel point sequence, and a predicted category of the to-be-sequenced pixel points is not the target category;

[0023] Based on the query samples, the positive samples and the negative samples of each category, comparative learning is performed to obtain the third loss value;

[0024] The preset sampling configuration is used to indicate that a sampling ratio of a pixel point at a front of the pixel point sequence is higher than a sampling ratio of a pixel point at a rear of the pixel point sequence.

[0025] Optionally, according to the ground object element segmentation model training method provided by the present application, the comprehensive loss value is determined based on the first loss value, the second loss value and the third loss value, and the method comprises the following steps:

[0026] The comprehensive loss value is determined by the following loss value calculation formula:

[0027] L=L sup +λ u L unsup +L con ;

[0028] Wherein, L represents the comprehensive loss value, L sup represents the first loss value, L unsup represents the second loss value, L con represents the third loss value, and λ u represents an unsupervised loss weight.

[0029] Optionally, according to the ground object element segmentation model training method provided by the present application, the parameters of each teacher network model are updated by a sliding average method based on the parameters of the student network model, and the method comprises the following steps:

[0030] The parameters of each teacher network model are updated by the following sliding average calculation formula:

[0031] θ t =αθ t +(1-α)θ s ;

[0032] Wherein, θ sθ is a parameter in a student network model t α is a parameter in a teacher network model corresponding to θ s α is a preset sliding average parameter.

[0033] Optionally, the application provides a ground feature segmentation model training method, after the parameters of each teacher network model are updated by the sliding average method based on the parameters of the student network model, the method further comprises:

[0034] The entropy threshold is determined by the following adaptive threshold calculation formula:

[0035]

[0036] wherein, α i represents the entropy threshold used in the i-th training round, a0 represents the initial entropy threshold, and total epoch represents the preset number of training rounds.

[0037] In a second aspect, the application further provides a ground feature segmentation method, comprising:

[0038] obtaining a remote sensing image;

[0039] inputting the remote sensing image into a target ground feature segmentation model to obtain a ground feature segmentation prediction result output by the target ground feature segmentation model;

[0040] The target ground feature segmentation model is obtained by semi-supervised training according to the ground feature segmentation model training method of any one of the above.

[0041] In a third aspect, the application further provides a ground feature segmentation model training device, comprising:

[0042] a sample acquisition module configured to acquire a labeled remote sensing image sample, a ground feature classification label of the labeled remote sensing image sample, and an unlabeled remote sensing image sample;

[0043] a training module configured to perform semi-supervised training on a student network model based on the labeled remote sensing image sample, the ground feature classification label, and the unlabeled remote sensing image sample until a comprehensive loss value is less than a loss threshold or a preset number of training rounds is reached, wherein the student network model is used for segmenting ground features;

[0044] a model determination module configured to determine the trained student network model as a target ground feature segmentation model;

[0045] The semi-supervised training on the student network model comprises:

[0046] input the labeled remote sensing image sample to the student network model, and based on the ground feature element classification label and a first classification probability prediction map output by the student network model, calculate a multi-classification cross-entropy loss to obtain a first loss value, and the classification probability prediction map is used to represent the predicted classification probability of each pixel point in the remote sensing image;

[0047] input the unlabeled remote sensing image sample processed by the two strong data enhancement methods to two teacher network models, and obtain a second classification probability prediction map by averaging the classification probability prediction maps output by each teacher network model, and the structure of the teacher network model is the same as that of the student network model;

[0048] input the unlabeled remote sensing image sample processed by the weak data enhancement method to the student network model to obtain a third classification probability prediction map output by the student network model;

[0049] based on each first pixel point in the second classification probability prediction map and the third classification probability prediction map, calculate a loss by a consistency regularization method to obtain a second loss value, and the entropy value of the first pixel point is less than an entropy threshold value;

[0050] based on each second pixel point in the second classification probability prediction map and the third classification probability prediction map, calculate a loss by a contrastive learning method to obtain a third loss value, and the entropy value of the second pixel point is greater than or equal to the entropy threshold value;

[0051] based on the first loss value, the second loss value and the third loss value, determine the comprehensive loss value;

[0052] update the parameters of the student network model by minimizing the comprehensive loss value;

[0053] update the parameters of each teacher network model by a sliding average method based on the parameters of the student network model.

[0054] In a fourth aspect, the present application further provides a ground feature element segmentation device, comprising:

[0055] an image acquisition module for acquiring a remote sensing image;

[0056] a prediction module for inputting the remote sensing image to a target ground feature element segmentation model to obtain a ground feature element segmentation prediction result output by the target ground feature element segmentation model;

[0057] the target ground feature element segmentation model is obtained by semi-supervised training of the ground feature element segmentation model training device.

[0058] In a fifth aspect, the present application also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the feature segmentation model training method according to any one of the preceding aspects, or the processor implements the feature segmentation method according to any one of the preceding aspects.

[0059] In a sixth aspect, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the feature segmentation model training method according to any one of the preceding aspects, or the feature segmentation method according to any one of the preceding aspects.

[0060] The feature segmentation model training method, the feature segmentation method and the device provided by the present application can obtain a first classification probability prediction map and a first loss value of a student network model by inputting a labeled remote sensing image sample into the student network model for training, perform strong data enhancement on an unlabeled remote sensing image sample, input the unlabeled remote sensing image sample into two teacher network models for prediction, obtain two prediction maps and average them to obtain a second classification probability prediction map, perform pixel-level classification on the pixels of the second classification probability prediction map to divide them into first pixels with confidence and second pixels without confidence, perform weak data enhancement on the unlabeled remote sensing image sample, input the unlabeled remote sensing image sample into the student network model for prediction to obtain a third classification probability prediction map, then calculate a loss based on each first pixel in the second classification probability prediction map and the third classification probability prediction map by a consistency regularization method to obtain a second loss value, calculate a loss based on each second pixel in the second classification probability prediction map and the third classification probability prediction map by a contrastive learning method to obtain a third loss value, and then determine a comprehensive loss value based on the first loss value, the second loss value and the third loss value, update the parameters of the student network model using the comprehensive loss value, update the parameters of each teacher network model using a sliding average method, and obtain a target feature segmentation model through multiple rounds of iterative training, which can reduce the amount of data labeling, and the target feature segmentation model obtained through semi-supervised training under the condition of a small amount of fine labeling data can be used to accurately classify multi-temporal features in a complex remote sensing image scene. BRIEF DESCRIPTION OF DRAWINGS

[0061] In order to more clearly illustrate the technical solutions of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0062] Figure 1is one of the flowcharts of the ground feature element segmentation model training method provided by the present application;

[0063] Figure 2 is one of the flowcharts of the ground feature element segmentation model training method provided by the present application;

[0064] Figure 3 is the flowchart of the ground feature element segmentation method provided by the present application;

[0065] Figure 4 is the structural schematic diagram of the ground feature element segmentation model training device provided by the present application;

[0066] Figure 5 is the structural schematic diagram of the ground feature element segmentation device provided by the present application;

[0067] Figure 6 is the structural schematic diagram of the electronic device provided by the present application. DETAILED DESCRIPTION

[0068] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0069] Figure 1 is one of the flowcharts of the ground feature element segmentation model training method provided by the present application, as shown in Figure 1 The execution subject of the method can be an electronic device, such as a server, etc. The method comprises:

[0070] Step 101, obtaining a labeled remote sensing image sample, a ground feature element classification label of the labeled remote sensing image sample, and an unlabeled remote sensing image sample.

[0071] Specifically, the labeled remote sensing image sample refers to a sample in which the ground feature element classification information has been labeled in the remote sensing image, including buildings, roads, water bodies, etc. These samples can be used to train machine learning models.

[0072] Corresponding to the labeled remote sensing image sample is the unlabeled remote sensing image sample, which refers to a sample in which the ground feature element classification information has not been labeled in the remote sensing image. These samples can be processed by unsupervised learning methods, thereby realizing automatic segmentation and identification of ground feature elements in remote sensing images.

[0073] In step 102, the student network model is semi-supervised trained based on the labeled remote sensing image sample, the ground object element classification label and the unlabeled remote sensing image sample until the comprehensive loss value is less than the loss threshold value or the preset training number of rounds is reached, and the student network model is used for segmenting the ground object element.

[0074] Specifically, the semi-supervised training of the student network model comprises:

[0075] The labeled remote sensing image sample is input into the student network model, and a multi-class cross-entropy loss is calculated based on the ground object element classification label and the first classification probability prediction map output by the student network model to obtain a first loss value, and the classification probability prediction map is used to represent the predicted classification probability of each pixel point in the remote sensing image.

[0076] The unlabeled remote sensing image sample processed by the two kinds of strong data enhancement methods is input into the two teacher network models, and the classification probability prediction maps output by each teacher network model are averaged to obtain a second classification probability prediction map, and the structure of the teacher network model is the same as that of the student network model.

[0077] The unlabeled remote sensing image sample processed by the weak data enhancement method is input into the student network model to obtain a third classification probability prediction map output by the student network model.

[0078] Based on each first pixel point in the second classification probability prediction map and the third classification probability prediction map, a loss is calculated by a consistency regularization method to obtain a second loss value, and the entropy value of the first pixel point is less than an entropy threshold value.

[0079] Based on each second pixel point in the second classification probability prediction map and the third classification probability prediction map, a loss is calculated by a contrastive learning method to obtain a third loss value, and the entropy value of the second pixel point is greater than or equal to the entropy threshold value.

[0080] Based on the first loss value, the second loss value and the third loss value, a comprehensive loss value is determined.

[0081] The parameters of the student network model are updated by minimizing the comprehensive loss value.

[0082] Based on the parameters of the student network model, the parameters of each teacher network model are updated by a sliding average method.

[0083] It should be noted that image enhancement is an important part of image processing, and its main purpose is to improve the quality of the image by improving the visual effect of the image or making some details of the image more easily recognized. Image enhancement does not increase new information of the image, but through processing of the existing image, the information in the image becomes more obvious and clear.

[0084] The weak data augmentation manner refers to weak image augmentation, which can be image change operations such as flipping, cropping, copying or rotating. The strong data augmentation manner refers to strong image augmentation, which can be changes to the categories or features of the image, such as classmix, which mixes the categories in two images, and cutmix, which copies a part of an image onto another image.

[0085] Consistency regularization is a semi-supervised learning technique, and its basic idea is to encourage the model to remain unchanged for slight input changes. By using image augmentation with different intensities, samples of various difficulties can be generated to guide the model to better learn and adapt to complex and variable real environments. By using strong image augmentation, the difficulty of the teacher network to maintain consistency during training is increased, and the consistency of the model can be improved through training, thereby improving the generalization performance of the data.

[0086] It can be understood that the above semi-supervised learning process involves three models: a student network model and two teacher network models. Among them, the labeled remote sensing image samples are input into the student network model for training, and the multi-class cross-entropy loss is calculated to obtain a first loss value L sup The unlabeled remote sensing image samples are input into the two teacher network models, and the average of the output classification probability prediction maps is taken to obtain a second classification probability prediction map. In addition, the unlabeled remote sensing image samples are processed using a weak data augmentation manner to obtain a third classification probability prediction map output by the student network model. The loss is calculated by the second classification probability prediction map and the third classification probability prediction map to obtain a second loss value L unsup and a third loss value L con The first, second and third loss values are combined to determine a comprehensive loss value, and the parameters of the student network model are updated by minimizing the comprehensive loss value. Finally, the parameters of each teacher network model are updated using a sliding average method.

[0087] Optionally, the ground feature element classification labels of the labeled remote sensing image samples can be obtained by the following method: the ground feature elements are divided into 9 categories, namely grassland, forest land, cultivated land, bare land, water body, road, impervious surface, residential area and industrial area, and pixel-level labeling is performed using PhotoShop software.

[0088] Optionally, the student network model can be a DeepLab v3+ model with an encoding-decoding structure, and a resnet101 network is used as the backbone network.

[0089] Optionally, the labeled remote sensing image sample can be an original RGB image with a dimension of HxWx3 (H represents the height of the image, and W represents the width of the image), and the labeled remote sensing image sample is input into the student network model, so that a class probability map with a size of HxWxC output by the student network model can be obtained, where C represents the number of feature element categories, and the class probability map is a first classification probability prediction map.

[0090] Optionally, Figure 2 is a flowchart of a feature element segmentation model training method provided by the present application, as shown in Figure 2 The first classification probability prediction map output by the student network model is calculated with the feature element classification label (for example, GroundTruth) to supervise the multi-class cross-entropy loss, and a first loss value L sup .

[0091] It can be understood that the two teacher network models (for example, Teacher network 1 and Teacher network 2) have the same architecture as the student network, the parameters of the teacher network are the sliding average of the parameters of the student network, the parameters of the two teacher networks are the same, and the image enhancement methods used for inputting the unlabeled remote sensing image samples into the two teacher networks are different.

[0092] Optionally, the ClassMix image enhancement method can be used for strong data enhancement of the unlabeled remote sensing image sample, and then the data-enhanced unlabeled remote sensing image sample is input into the Teacher network 1; the CutMix image enhancement method can be used for strong data enhancement of the unlabeled remote sensing image sample, and then the data-enhanced unlabeled remote sensing image sample is input into the Teacher network 2.

[0093] Optionally, as shown in Figure 2 The unlabeled images subjected to different strong image enhancement are input into the Teacher 1 and the Teacher network 2, respectively output prediction maps with a size of HxWxC, and the two segmentation maps are summed and averaged to obtain a second classification probability prediction map.

[0094] Optionally, the unlabeled images subjected to weak image enhancement are input into the Student network, and a third classification probability prediction map with a size of HxWxC can be output.

[0095] Optionally, after obtaining the last output average logits of the teacher network model, the SoftMax function can be used to convert it into a probability distribution, and after obtaining the probability distribution, the entropy value of each pixel is calculated according to the following formula:

[0096]

[0097] H(pij ) represents the entropy value of the jth pixel of the ith image, p ij (c) represents the probability output by the SoftMax function, and C represents the number of feature element categories.

[0098] It can be understood that, as Figure 2 shown, a threshold of entropy is set, and the label pixel points below the entropy threshold are confident class pixel points, that is, first pixel points, and the label pixel points above the entropy threshold are unconfident class pixel points, that is, second pixel points.

[0099] Step 103, determining the trained student network model as a target feature element segmentation model.

[0100] Specifically, when performing feature element segmentation, an accurate target feature element segmentation model is needed, and the student network model that has been trained in the above semi-supervised manner is a relatively ideal choice. By using this model, various feature elements on a map can be effectively segmented, and the speed and accuracy of feature element segmentation can be improved.

[0101] The feature element segmentation model training method provided by the application trains the student network model by inputting the labeled remote sensing image sample into the student network model, obtains the first classification probability prediction map of the student network model and the first loss value, uses strong data enhancement on the unlabeled remote sensing image sample, and inputs the unlabeled remote sensing image sample into two teacher network models for prediction, obtains two prediction maps and averages them to obtain a second classification probability prediction map, classifies the pixel points of the second classification probability prediction map at the pixel level into confident first pixel points and unconfident second pixel points, uses weak data enhancement on the unlabeled remote sensing image sample, and inputs the unlabeled remote sensing image sample into the student network model for prediction to obtain a third classification probability prediction map, and then based on each first pixel point in the second classification probability prediction map and the third classification probability prediction map, the loss is calculated by consistency regularization to obtain a second loss value, and based on each second pixel point in the second classification probability prediction map and the third classification probability prediction map, the loss is calculated by contrastive learning to obtain a third loss value, and then based on the first loss value, the second loss value and the third loss value, a comprehensive loss value is determined, and then the parameters of the student network model can be updated using the comprehensive loss value, and the parameters of each teacher network model can be updated using the sliding average method, and through multiple rounds of iterative training, a target feature element segmentation model can be obtained, which can reduce the amount of data labeling, and in the case of a small amount of fine labeling data, the target feature element segmentation model obtained by the semi-supervised training method can be used to accurately classify multiple time-phase feature elements in a complex remote sensing image scene.

[0102] Optionally, the application provides a ground feature element segmentation model training method based on the second classification probability prediction map and the third classification probability prediction map, a loss is calculated by a contrast learning method to obtain a third loss value, including:

[0103] Based on the classification probability threshold, each second pixel point in the second classification probability prediction map and each third pixel point in the third classification probability prediction map are screened to determine a plurality of fourth pixel points and query samples of each classification. The entropy value of the third pixel point is greater than or equal to the entropy threshold, the predicted classification probability of the fourth pixel point is less than the classification probability threshold, and the predicted classification probability of the query sample is greater than or equal to the classification probability threshold.

[0104] For the query sample of each classification, uniform sampling is performed to obtain positive samples of each classification.

[0105] For each target classification in the plurality of classifications, a plurality of to-be-sequenced pixel points are screened out from the plurality of fourth pixel points, and based on the similarity between the to-be-sequenced pixel points and the positive samples of the target classification, the plurality of to-be-sequenced pixel points are sequenced in order from high to low similarity to obtain a pixel point sequence, and based on a preset sampling configuration, the pixel point sequence is feature-sampled to obtain negative samples of the target classification. The predicted classification of the to-be-sequenced pixel point is not the target classification.

[0106] Based on the query samples, the positive samples and the negative samples of each classification, contrast learning is performed to obtain the third loss value.

[0107] The preset sampling configuration is used to indicate that the sampling proportion of the pixel points at the front of the pixel point sequence is higher than the sampling proportion of the pixel points at the back of the pixel point sequence.

[0108] Specifically, based on the classification probability threshold, the pixel points in the second and third classification probability prediction maps are screened to determine the fourth pixel points and the query samples. The query samples are uniformly sampled to obtain the positive samples. For the target classification, the to-be-sequenced pixel points are screened out from the fourth pixel points, and the pixel point sequence is sequenced in order from high to low similarity to obtain the negative samples. Based on the query samples, the positive samples and the negative samples, contrast learning is performed to obtain the third loss value. The sampling configuration is used to indicate that the sampling proportion of the pixel points at the front of the pixel point sequence is higher than the sampling proportion of the pixel points at the back.

[0109] Optionally, the plurality of fourth pixel points can be placed in a data structure called Memory bank, and then based on the Memory bank, the plurality of fourth pixel points are screened, sorted and sampled according to two principles to obtain negative samples. The first principle is that the pixel point feature and the positive sample and the Query cannot belong to the same class, which is to ensure that the selected negative sample can fully represent other classes, and the first principle corresponds to the process of screening a plurality of pixel points to be sorted from the plurality of fourth pixel points. The second principle is that the similarity of the feature and the positive sample and the Query must be high enough, so that the selected negative sample is closer to the positive sample in the feature space, and such negative sample can provide stronger contrast, thereby making learning more effective, and the second principle corresponds to the process of sorting the plurality of pixel points to be sorted.

[0110] According to the two principles, the features in the Memory bank are sorted according to the similarity from high to low. For the classes ranked in the front (i.e. high similarity), more features will be sampled, while for the classes ranked in the back (i.e. low similarity), fewer features will be sampled. After this process, all the sampled features are used as negative samples for subsequent contrast learning.

[0111] Memory bank is a data structure used to store a large amount of data, commonly used in deep learning and machine learning for specific tasks such as large-scale unsupervised learning, self-supervised learning and semi-supervised learning. In these tasks, it may be necessary to process millions of samples and maintain the state of these samples throughout the training process. Memory bank stores data in an array or list in memory. Memory bank here acts as a buffer, storing all pixel features to be sorted.

[0112] It can be understood that, due to the intra-class dissimilarity and inter-class similarity of remote sensing images, the difficulty of feature classification is reflected in the feature space. By using contrast learning, more learning weight is given to difficult-to-classify samples, and the difficulty of contrast learning is increased, making the classification of the feature space more effective, which can improve the accuracy of remote sensing feature segmentation compared to the prior art, and has important significance.

[0113] Optionally, the loss function used in contrast learning can be an InfoNCE loss function.

[0114] Optionally, according to the ground feature element segmentation model training method provided by the application, the comprehensive loss value is determined based on the first loss value, the second loss value and the third loss value, which comprises:

[0115] The comprehensive loss value is determined by the following loss value calculation formula:

[0116] L=L sup +λ u L unsup +L con ;

[0117] Wherein, L represents the comprehensive loss value, L sup represents the first loss value, L unsup represents the second loss value, L con represents the third loss value, and λ u represents an unsupervised loss weight.

[0118] Specifically, as Figure 2 indicated, the unsupervised loss L sup obtained in the consistency regularization manner is combined with the supervised loss L unsup and the contrastive loss L con obtained in the contrastive learning, so as to determine the comprehensive loss value L , and the parameters of the student network model can be updated by minimizing the comprehensive loss value.

[0119] Optionally, the unsupervised loss weight λ u may be a dynamic weight, which is reduced as the proportion of the number of untrustworthy pixels in the total number of pixels decreases, so as to improve the training effect.

[0120] Optionally, according to the ground object element segmentation model training method provided in the present application, the parameters of each teacher network model are updated by a sliding average manner based on the parameters of the student network model, which comprises the following steps:

[0121] The parameters of each teacher network model are updated by the following sliding average calculation formula:

[0122] θ t =αθ t +(1-α)θ s ;

[0123] Wherein, θ s is a parameter in the student network model, θ t is a parameter corresponding to θ s in the teacher network model, and α is a preset sliding average parameter.

[0124] Specifically, since the structure of the teacher network model is the same as that of the student network model, the θ s corresponding to θ t can be found in the teacher network model, and after the parameters of the student network model are updated by minimizing the comprehensive loss value, the parameters of each teacher network model can be updated by the above sliding average calculation formula.

[0125] Optionally, the application provides a ground feature segmentation model training method, after the parameters of the student network model are used to update the parameters of each teacher network model by a sliding average method, the method further comprises:

[0126] The entropy threshold is determined by the following adaptive threshold calculation formula:

[0127]

[0128] Wherein, α i represents the entropy threshold used in the i-th training round, a0 represents the initial entropy threshold, and total epoch represents the preset number of training rounds.

[0129] Specifically, as the more the training rounds are, the higher the confidence of the image is, and the fewer the untrustworthy pixels are, the threshold should be adaptively reduced to continue filtering the untrustworthy pixels and improve the model training effect.

[0130] Figure 3 is a flowchart of a ground feature segmentation method provided by the application, as Figure 3 shown, the execution subject of the method can be an electronic device, such as a server, etc. The method comprises:

[0131] Step 301, acquiring a remote sensing image;

[0132] Step 302, inputting the remote sensing image into a target ground feature segmentation model to obtain a ground feature segmentation prediction result output by the target ground feature segmentation model;

[0133] The target ground feature segmentation model is obtained by semi-supervised training according to any one of the ground feature segmentation model training methods described above.

[0134] Specifically, before ground feature segmentation, a remote sensing image needs to be acquired as input. These remote sensing images can be obtained by a drone, a satellite or other remote sensing equipment. Next, these images can be input into a target ground feature segmentation model, which can be trained and optimized by a deep learning algorithm to identify and segment different ground features in the image, such as buildings, roads, water bodies, etc. Finally, ground feature segmentation prediction results can be obtained, which can be used in map making, urban planning or environmental monitoring, etc.

[0135] The ground feature segmentation model training device provided by the application is described below, and the ground feature segmentation model training device described below can be correspondingly referred to the ground feature segmentation model training method described above.

[0136] Figure 4 is a structural diagram of a ground feature segmentation model training device provided by the application, asFigure 4 As shown, the device comprises a sample acquisition module 401, a training module 402 and a model determination module 403, wherein:

[0137] The sample acquisition module 401 is configured to acquire a labeled remote sensing image sample, a ground feature element classification label of the labeled remote sensing image sample, and an unlabeled remote sensing image sample.

[0138] The training module 402 is configured to perform semi-supervised training on a student network model based on the labeled remote sensing image sample, the ground feature element classification label and the unlabeled remote sensing image sample until a comprehensive loss value is less than a loss threshold or a preset training number of rounds is reached, the student network model being used for segmenting a ground feature element.

[0139] The model determination module 403 is configured to determine the trained student network model as a target ground feature element segmentation model.

[0140] The semi-supervised training on the student network model comprises:

[0141] inputting the labeled remote sensing image sample into the student network model, and calculating a multi-class cross-entropy loss based on the ground feature element classification label and a first classification probability prediction map output by the student network model, to obtain a first loss value, the classification probability prediction map being used to represent a predicted classification probability of each pixel point in the remote sensing image.

[0142] inputting the unlabeled remote sensing image sample processed by the two kinds of strong data augmentation methods into two teacher network models, and averaging classification probability prediction maps output by each teacher network model to obtain a second classification probability prediction map, the teacher network model having the same structure as the student network model.

[0143] inputting the unlabeled remote sensing image sample processed by the weak data augmentation method into the student network model to obtain a third classification probability prediction map output by the student network model.

[0144] based on each first pixel point in the second classification probability prediction map and the third classification probability prediction map, calculating a loss by a consistency regularization method to obtain a second loss value, an entropy value of the first pixel point being less than an entropy threshold.

[0145] based on each second pixel point in the second classification probability prediction map and the third classification probability prediction map, calculating a loss by a contrastive learning method to obtain a third loss value, an entropy value of the second pixel point being greater than or equal to the entropy threshold.

[0146] based on the first loss value, the second loss value and the third loss value, determining the comprehensive loss value.

[0147] updating parameters of the student network model by minimizing the comprehensive loss value;

[0148] updating parameters of each teacher network model by a sliding average manner based on the parameters of the student network model.

[0149] The following describes the ground feature element segmentation device provided by the present application, which can be mutually corresponding with the ground feature element segmentation method described above.

[0150] Figure 5 is a structural schematic diagram of the ground feature element segmentation device provided by the present application, as Figure 5 shown, the device comprises an image acquisition module 501 and a prediction module 502, wherein:

[0151] The image acquisition module 501 is configured to acquire a remote sensing image.

[0152] The prediction module 502 is configured to input the remote sensing image into a target ground feature element segmentation model, and acquire a ground feature element segmentation prediction result output by the target ground feature element segmentation model.

[0153] The target ground feature element segmentation model is obtained by semi-supervised training of any one of the ground feature element segmentation model training devices described above.

[0154] Figure 6 is a structural schematic diagram of the electronic device provided by the present application, as Figure 6 shown, the electronic device can comprise a processor 610, a communications interface 620, a memory 630 and a communications bus 640, wherein the processor 610, the communications interface 620 and the memory 630 complete mutual communication through the communications bus 640. The processor 610 can invoke a logical instruction in the memory 630 to execute a ground feature element segmentation model training method, which comprises:

[0155] acquiring a labeled remote sensing image sample, a ground feature element classification label of the labeled remote sensing image sample and an unlabeled remote sensing image sample;

[0156] based on the labeled remote sensing image sample, the ground feature element classification label and the unlabeled remote sensing image sample, semi-supervised training a student network model until a comprehensive loss value is less than a loss threshold value or a preset training round number is reached, the student network model being configured to segment a ground feature element;

[0157] determining the trained student network model as a target ground feature element segmentation model;

[0158] The semi-supervised training of the student network model comprises:

[0159] inputting the labeled remote sensing image sample into the student network model, and calculating a multi-class cross-entropy loss based on the surface feature element classification label and a first classification probability prediction map output by the student network model, to obtain a first loss value, the classification probability prediction map being used to represent the predicted classification probability of each pixel point in the remote sensing image;

[0160] inputting the unlabeled remote sensing image sample processed by the two types of strong data enhancement into two teacher network models, and averaging the classification probability prediction maps output by each teacher network model to obtain a second classification probability prediction map, the structure of the teacher network model being the same as that of the student network model;

[0161] inputting the unlabeled remote sensing image sample processed by the weak data enhancement into the student network model to obtain a third classification probability prediction map output by the student network model;

[0162] based on each first pixel point in the second classification probability prediction map and the third classification probability prediction map, calculating a loss by a consistency regularization method to obtain a second loss value, the entropy value of the first pixel point being less than an entropy threshold value;

[0163] based on each second pixel point in the second classification probability prediction map and the third classification probability prediction map, calculating a loss by a contrastive learning method to obtain a third loss value, the entropy value of the second pixel point being greater than or equal to the entropy threshold value;

[0164] based on the first loss value, the second loss value and the third loss value, determining the comprehensive loss value;

[0165] updating the parameters of the student network model by minimizing the comprehensive loss value;

[0166] updating the parameters of each teacher network model by a sliding average method based on the parameters of the student network model;

[0167] alternatively, a surface feature element segmentation method is performed, the method comprising:

[0168] obtaining a remote sensing image;

[0169] inputting the remote sensing image into a target surface feature element segmentation model to obtain a surface feature element segmentation prediction result output by the target surface feature element segmentation model;

[0170] the target surface feature element segmentation model is obtained by semi-supervised training according to the surface feature element segmentation model training method according to any one of the above.

[0171] Further, the logic instructions in the memory 630 described above can be implemented in the form of software functional units and sold or used as standalone products, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that make contributions to the prior art, or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0172] In another aspect, the present application also provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the building feature segmentation model training method provided by the above method, the method comprises:

[0173] obtaining a labeled remote sensing image sample, a building feature classification label of the labeled remote sensing image sample, and an unlabeled remote sensing image sample;

[0174] Based on the labeled remote sensing image sample, the building feature classification label, and the unlabeled remote sensing image sample, the student network model is semi-supervisedly trained until the comprehensive loss value is less than the loss threshold or the preset training number of rounds is reached, and the student network model is used for segmenting building features;

[0175] determining the trained student network model as a target building feature segmentation model;

[0176] The semi-supervised training of the student network model comprises:

[0177] inputting the labeled remote sensing image sample into the student network model, and based on the building feature classification label and the first classification probability prediction map output by the student network model, calculating a multi-class cross-entropy loss to obtain a first loss value, and the classification probability prediction map is used to represent the predicted classification probability of each pixel point in the remote sensing image;

[0178] inputting the unlabeled remote sensing image sample processed by the two kinds of strong data enhancement methods into two teacher network models, and averaging the classification probability prediction maps output by each teacher network model to obtain a second classification probability prediction map, the structure of the teacher network model is the same as that of the student network model;

[0179] input the unlabeled remote sensing image sample processed by the weak data enhancement manner to the student network model, and obtain a third classification probability prediction map output by the student network model;

[0180] based on each first pixel point in the second classification probability prediction map and the third classification probability prediction map, a loss is calculated by a consistency regularization manner to obtain a second loss value, and an entropy value of the first pixel point is less than an entropy threshold value;

[0181] based on each second pixel point in the second classification probability prediction map and the third classification probability prediction map, a loss is calculated by a contrastive learning manner to obtain a third loss value, and an entropy value of the second pixel point is greater than or equal to an entropy threshold value;

[0182] based on the first loss value, the second loss value and the third loss value, the comprehensive loss value is determined;

[0183] the parameters of the student network model are updated by minimizing the comprehensive loss value;

[0184] based on the parameters of the student network model, the parameters of each teacher network model are updated by a sliding average manner;

[0185] alternatively, the above-mentioned each method provides a ground feature element segmentation method, which comprises:

[0186] obtaining a remote sensing image;

[0187] inputting the remote sensing image to a target ground feature element segmentation model to obtain a ground feature element segmentation prediction result output by the target ground feature element segmentation model;

[0188] the target ground feature element segmentation model is obtained by semi-supervised training according to the ground feature element segmentation model training method of any one of the above.

[0189] The device embodiments described above are only schematic, wherein the units shown as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e., they can be located in one place, or distributed on multiple network units. According to actual needs, part or all of the modules can be selected to achieve the purpose of the embodiment scheme. Those skilled in the art can understand and implement it without creative labor.

[0190] Those skilled in the art can clearly understand the technical solutions of the various embodiments from the above description of the embodiments, and the various embodiments can be implemented by means of software with the necessary general hardware platforms, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0191] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for training a feature element segmentation model, characterized in that, The method comprises the following steps: obtaining labeled remote sensing image samples, ground feature element classification labels of the labeled remote sensing image samples, and unlabeled remote sensing image samples; performing semi-supervised training on a student network model based on the labeled remote sensing image samples, the ground feature element classification labels, and the unlabeled remote sensing image samples until a comprehensive loss value is less than a loss threshold or a preset number of training rounds is reached, the student network model being used for segmenting ground feature elements; determining the trained student network model as a target ground feature element segmentation model; the semi-supervised training on the student network model comprises the following steps: inputting the labeled remote sensing image samples into the student network model, and calculating a multi-class cross-entropy loss based on the ground feature element classification labels and a first classification probability prediction map output by the student network model, the classification probability prediction map being used to represent the predicted classification probability of each pixel point in the remote sensing image, to obtain a first loss value; inputting the unlabeled remote sensing image samples processed by two types of strong data augmentation into two teacher network models, and averaging the classification probability prediction maps output by each teacher network model to obtain a second classification probability prediction map, the structure of the teacher network model being the same as that of the student network model; inputting the unlabeled remote sensing image samples processed by a weak data augmentation into the student network model to obtain a third classification probability prediction map output by the student network model; calculating a loss based on each first pixel point in the second classification probability prediction map and the third classification probability prediction map by a consistency regularization method to obtain a second loss value, the entropy value of the first pixel point being less than an entropy threshold; calculating a loss based on each second pixel point in the second classification probability prediction map and the third classification probability prediction map by a contrastive learning method to obtain a third loss value, the entropy value of the second pixel point being greater than or equal to the entropy threshold; determining the comprehensive loss value based on the first loss value, the second loss value, and the third loss value; updating the parameters of the student network model by minimizing the comprehensive loss value; updating the parameters of each teacher network model by a sliding average method based on the parameters of the student network model; the calculating of the loss based on each second pixel point in the second classification probability prediction map and the third classification probability prediction map by the contrastive learning method to obtain the third loss value comprises the following steps: filtering each second pixel point in the second classification probability prediction map and each third pixel point in the third classification probability prediction map based on a classification probability threshold to determine a plurality of fourth pixel points and query samples of each classification, the entropy value of the third pixel point being greater than or equal to the entropy threshold, the predicted classification probability of the fourth pixel point being less than the classification probability threshold, and the predicted classification probability of the query sample being greater than or equal to the classification probability threshold; performing uniform sampling on the query samples of each classification to obtain positive samples of each classification; For each target classification in the plurality of classifications, a plurality of to-be-sequenced pixel points are screened out from the plurality of fourth pixel points, and based on similarities between the to-be-sequenced pixel points and positive samples of the target classification, the to-be-sequenced pixel points are sequenced in descending order of the similarities, a pixel point sequence is obtained, and based on a preset sampling configuration, the pixel point sequence is sampled to obtain negative samples of the target classification, and a predicted classification of the to-be-sequenced pixel points is not the target classification; Based on the query samples, the positive samples and the negative samples of each classification, comparative learning is performed to obtain the third loss value; The preset sampling configuration is used to indicate that a sampling proportion of pixel points at the front of the pixel point sequence is higher than a sampling proportion of pixel points at the back of the pixel point sequence. 2.The method of claim 1, wherein, The comprehensive loss value is determined based on the first loss value, the second loss value and the third loss value, including: The comprehensive loss value is determined by the following loss value calculation formula: L = L sup + λ u L unsup + L con ; wherein L denotes the combined loss value, L sup denotes the first loss value, L unsup denotes the second loss value, L con denotes the third loss value, λ u denotes the unsupervised loss weight. 3.The method of claim 1, wherein, The parameters of the student network model are used to update the parameters of each teacher network model in a sliding average manner, including: The parameters of each teacher network model are updated by the following sliding average calculation formula: θ t = a θ t + (1 - a) θ s ; where θ s is a parameter in the student network model, θ t is a parameter in the teacher network model corresponding to θ s , and α is a preset sliding average parameter.

4. The method of claim 1-3, wherein, After the parameters of the student network model are used to update the parameters of each teacher network model in a sliding average manner, it further includes: The entropy threshold is determined by the following adaptive threshold calculation formula: wherein a i represents the entropy threshold adopted in the i-th round of training, a0represents the initial entropy threshold, and totalepochrepresents the preset number of training rounds.

5. A method of feature segmentation, characterized by, Including: Obtaining a remote sensing image; Inputting the remote sensing image into a target ground feature element segmentation model to obtain a ground feature element segmentation prediction result output by the target ground feature element segmentation model; The target ground feature element segmentation model is obtained by semi-supervised training according to the ground feature element segmentation model training method in any one of claims 1-4. 6.A device for training a feature element segmentation model, characterized in that, Including: A sample acquisition module is configured to acquire labeled remote sensing image samples, ground feature element classification labels of the labeled remote sensing image samples, and unlabeled remote sensing image samples; A training module is configured to perform semi-supervised training on a student network model based on the labeled remote sensing image samples, the ground feature element classification labels and the unlabeled remote sensing image samples until a comprehensive loss value is less than a loss threshold or a preset training number of rounds is reached, and the student network model is used to segment ground feature elements; A model determination module is configured to determine the trained student network model as a target ground feature element segmentation model; The semi-supervised training on the student network model includes: Inputting the labeled remote sensing image samples into the student network model, and calculating a multi-class cross-entropy loss based on the ground feature element classification labels and a first classification probability prediction map output by the student network model to obtain a first loss value, and the classification probability prediction map is used to represent a predicted classification probability of each pixel point in the remote sensing image; Inputting the unlabeled remote sensing image samples processed by two types of strong data enhancement into two teacher network models, and averaging classification probability prediction maps output by each teacher network model to obtain a second classification probability prediction map, and the structure of the teacher network model is the same as that of the student network model. inputting the unlabeled remote sensing image sample processed by the weak data enhancement manner into the student network model, and obtaining a third classification probability prediction map output by the student network model; calculating a loss based on each first pixel point in the second classification probability prediction map and the third classification probability prediction map by a consistency regularization manner, obtaining a second loss value, and the entropy value of the first pixel point being less than an entropy threshold; calculating a loss based on each second pixel point in the second classification probability prediction map and the third classification probability prediction map by a contrastive learning manner, obtaining a third loss value, and the entropy value of the second pixel point being greater than or equal to the entropy threshold; determining the comprehensive loss value based on the first loss value, the second loss value and the third loss value; updating the parameters of the student network model by minimizing the comprehensive loss value; updating the parameters of each teacher network model by a sliding average manner based on the parameters of the student network model; the calculating a loss based on each second pixel point in the second classification probability prediction map and the third classification probability prediction map by a contrastive learning manner, obtaining a third loss value, comprises: filtering each second pixel point in the second classification probability prediction map and each third pixel point in the third classification probability prediction map based on a classification probability threshold, determining a plurality of fourth pixel points and a query sample of each classification, the entropy value of the third pixel point being greater than or equal to the entropy threshold, the prediction classification probability of the fourth pixel point being less than a classification probability threshold, and the prediction classification probability of the query sample being greater than or equal to a classification probability threshold; performing uniform sampling on the query sample of each classification to obtain a positive sample of each classification; for each target classification in the plurality of classifications, filtering a plurality of to-be-sequenced pixel points from the plurality of fourth pixel points, and based on the similarity between the to-be-sequenced pixel points and the positive sample of the target classification, sorting the plurality of to-be-sequenced pixel points in order of similarity from high to low to obtain a pixel point sequence, and based on a preset sampling configuration, performing feature sampling on the pixel point sequence to obtain a negative sample of the target classification, and the prediction classification of the to-be-sequenced pixel point being not the target classification; performing contrastive learning based on the query sample, the positive sample and the negative sample of each classification to obtain the third loss value; the preset sampling configuration is used to indicate that the sampling proportion of the pixel points at the front of the pixel point sequence is higher than the sampling proportion of the pixel points at the back of the pixel point sequence.

7. A feature segmenting apparatus characterized by comprising: comprises: an image acquisition module configured to acquire a remote sensing image; a prediction module configured to input the remote sensing image into a target ground feature segmentation model, and obtain a ground feature segmentation prediction result output by the target ground feature segmentation model; the target ground feature segmentation model is obtained by semi-supervised training of the ground feature segmentation model training apparatus of claim 6.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, the processor implements the ground feature segmentation model training method of any one of claims 1 to 4 when executing the program, or the processor implements the ground feature segmentation method of claim 5 when executing the program. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the feature element segmentation model training method according to any one of claims 1 to 4, or the computer program is executed by the processor to implement the feature element segmentation method according to claim 5.

Images