Remote sensing image segmentation method, device, and electronic device based on semi-supervised learning

Abstract

Embodiments of the present invention disclose a remote sensing image segmentation method, device, and electronic device based on semi-supervised learning. The method includes: obtaining an annotation set based on a preset remote sensing image, the annotation set including a training set and a validation set; training a target segmentation model using the training set; inputting the validation set into the target segmentation model to obtain a predicted segmentation result for the validation set; if the predicted segmentation result of the validation set does not meet the preset validation requirements, updating the training set with the validation images belonging to validation samples whose segmentation results in the validation set meet the preset prediction requirements and the predicted category labels corresponding to the validation samples; returning to the step of training at least one target segmentation model using the training set; and inputting a remote sensing image to be tested into at least one target segmentation model to obtain a predicted segmentation result for the remote sensing image to be tested. This solution can quickly and efficiently segment the remote sensing image to be tested, saving time for manual segmentation and annotation.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a remote sensing image segmentation method, device, electronic device, storage medium and computer program product based on semi-supervised learning. Background Art

[0002] Land cover classification methods based on remote sensing images have important application prospects in land resource management, urban planning, agricultural yield estimation, and environmental protection. In recent years, the spatial resolution of remote sensing images has continued to increase. Meter-level high-resolution remote sensing images contain a wealth of detailed information about land features, such as texture, shape, and spatial distribution, which is particularly important for identifying fine-scale land classifications. However, the interpretation and analysis of meter-level high-resolution remote sensing images requires extensive manual annotation. Due to the large image size of remote sensing images, which can contain hundreds of millions of pixels, this manual annotation method is time-consuming and inefficient, which to some extent restricts its application in land cover classification. Summary of the Invention

[0003] The present invention was developed in response to the aforementioned challenges. It provides a remote sensing image segmentation method, apparatus, electronic device, storage medium, and computer program product based on semi-supervised learning. This approach trains one or more target segmentation models based on a small number of labeled samples. Using these trained target segmentation models, the remote sensing image under test can be segmented quickly and efficiently, significantly reducing manual segmentation and annotation time.

[0004] According to one aspect of the present invention, a remote sensing image segmentation method based on semi-supervised learning is provided. The method comprises: obtaining an annotation set based on a preset remote sensing image, the annotation set comprising a training set and a validation set, the training set comprising at least one training image and category labels for each image region on at least one training image, and the validation set comprising at least one validation image and category labels for each image region on at least one validation image; using the training set to train at least one target segmentation model, the at least one target segmentation model being used to predict the category of each image region of the input image; inputting the validation set into the at least one target segmentation model to obtain a predicted segmentation result of the validation set, and when the predicted segmentation result of the validation set does not meet the preset validation requirements, the validation image to which the validation sample whose sample segmentation result meets the preset prediction requirements belongs and the predicted category corresponding to the validation sample are inputted. The label is updated to the training set, and the step of training at least one target segmentation model using the training set is returned to execute until the predicted classification result of the verification set meets the preset verification requirements, wherein the predicted segmentation result of the verification set includes the sample segmentation result corresponding to each verification sample in the verification set, and the sample segmentation result is used to indicate the predicted category label corresponding to the verification sample. The verification sample is the image area in the verification image. The preset prediction requirement includes that the predicted probability of the predicted category label corresponding to the verification sample meets the first preset reliability, and the preset verification requirement includes that the predicted probability of the predicted category label corresponding to each verification sample in the verification set meets the second preset reliability; the remote sensing image to be measured is input into at least one target segmentation model to obtain the predicted segmentation result of the remote sensing image to be measured.

[0005] Optionally, the number of target segmentation models is greater than 1, and the network architecture of each target segmentation model is different. The verification set is input into at least one target segmentation model to obtain the predicted segmentation result of the verification set, including: inputting the verification set into each target segmentation model respectively to obtain the model segmentation result output by each target segmentation model, and the model segmentation result is used to indicate the predicted probability that each pixel in the verification set belongs to each preset category; determining the target weight of each target segmentation model according to the model segmentation result output by each target segmentation model; for each preset category in multiple preset categories, for each pixel in the verification set, the predicted probability of the pixel belonging to the preset category is weightedly summed according to the target weight of each target segmentation model, and the preset category with the largest weighted result is determined as the predicted category label of the pixel, the predicted category label corresponding to the pixel belongs to one of multiple preset categories, and the predicted category label corresponding to the verification sample is the predicted category label of each pixel contained in the verification sample.

[0006] Optionally, the target weight of each target segmentation model is determined according to the model segmentation results output by each target segmentation model, including: determining the overall accuracy of each target segmentation model in a one-to-one correspondence according to the model segmentation results output by each target segmentation model; determining the target weight of each target segmentation model in a one-to-one correspondence according to the overall accuracy of each target segmentation model.

[0007] Optionally, for any pixel, the predicted probability that the pixel belongs to the preset category is weighted using the following formula, and the preset category with the largest weighted result is determined as the predicted category label of the pixel:

[0008]

[0009] Where A represents the predicted category label of the pixel, n represents the number of target segmentation models, and W i represents the target weight of the i-th target segmentation model, P i (x) represents the probability that the pixel belongs to the x-th preset category in the model segmentation result output by the i-th target segmentation model.

[0010] Optionally, the training set includes a first training sample and a second training sample, the first training sample and the second training sample are image areas in the training image, and the area of ​​the first training sample is larger than the area of ​​the second training sample; using the training set to train at least one target segmentation model includes: performing a first training operation, including: for each target segmentation model, inputting the training set into the target segmentation model; using the focal loss function to adjust the parameters of the target segmentation model until the target segmentation model meets the first optimization requirement, wherein during the parameter adjustment process, the learning rate of the target segmentation model maintains a preset standard learning rate unchanged, and the first optimization requirement includes that the predicted probability corresponding to the predicted category label obtained by predicting the first training sample meets a third preset confidence level: after performing the first training operation, performing a second training operation, including: for each target segmentation model, inputting the training set into the target segmentation model; using the focal loss function to adjust the parameters of the target segmentation model until the target segmentation model meets the second optimization requirement, wherein the parameters adjusted during the parameter adjustment process include the learning rate, and the second optimization requirement includes that the predicted probability corresponding to the predicted category label obtained by predicting the second training sample meets a fourth preset confidence level.

[0011] Optionally, a balance factor used by the focus loss function in the first training operation is smaller than a balance factor used in the second training operation. The formula of the focus loss function is as follows:

[0012] FL(p t )=-α t (1-p t ) γlog(p t )

[0013] Where p t It represents the correct sample rate of prediction, which is equal to the ratio of the number of training samples whose predicted probability corresponding to the predicted category label meets the corresponding preset confidence to the total number of training samples. t represents the balance factor, and γ represents the preset adjustment factor.

[0014] Optionally, obtaining a labeling set based on a preset remote sensing image includes: determining reliable sample data and sample data to be labeled based on the preset remote sensing image, the reliable sample data including target reliable samples and category labels of the target reliable samples, the sample data to be labeled including samples to be labeled, and the target reliable samples and samples to be labeled are image areas in the preset remote sensing image; training the labeling model using the images to which the target reliable samples belong; inputting the images to which the samples to be labeled belong into the labeling model to obtain relabeled samples, and when the number of target reliable samples does not meet the preset number requirement, adding the reliable samples in the relabeled samples to the target reliable samples, and returning to execute the step of training the labeling model using the images to which the target reliable samples belong, until the number of target reliable samples meets the preset number requirement, determining the images to which the target reliable samples belong as the images included in the labeling set; wherein, the reliable samples in the relabeled samples include samples whose predicted category labels output by the labeling model meet the preset label requirements, and samples obtained after manual relabeling of samples whose predicted category labels output by the labeling model do not meet the preset label requirements.

[0015] Optionally, the labeling model is trained using images of target reliable samples, including: performing image enhancement on the images of target reliable samples; inputting the images of target reliable samples before image enhancement and the images of target reliable samples after image enhancement into the labeling model to calculate the consistency loss value of the target reliable samples before and after image enhancement; and adjusting the parameters of the labeling model using the consistency loss value.

[0016] Optionally, reliable sample data and sample data to be labeled are determined based on a preset remote sensing image, including: segmenting the preset remote sensing image into multiple image blocks; inputting the multiple image blocks into a pre-trained segmentation model to obtain image block segmentation results of the multiple image blocks output by the pre-trained segmentation model, the image block segmentation results being used to indicate predicted category labels of each image area contained in each of the multiple image blocks; in response to a user's re-labeling operation of at least part of the multiple image blocks with predicted category labels obtained based on the pre-trained segmentation model, determining new predicted category labels for at least part of the image areas in at least part of the image blocks; in response to the user's determination operation, determining at least part of the image area that currently has a predicted category label as a target reliable sample, and determining at least part of the image area that currently does not have a predicted category label as a sample to be labeled.

[0017] Optionally, the training set includes a first training sample and a second training sample, the first training sample and the second training sample are image areas in the training image, and the area of ​​the first training sample is larger than the area of ​​the second training sample; after determining the image to which the target reliable sample belongs as the image included in the annotation set, the method also includes: performing image enhancement on the second training sample in the training set; obtaining a new training image based on the initial training image to which the target background sample belongs by randomly embedding the image-enhanced second training sample into any target background sample in the training set, adding the new training image to the training set or replacing the initial training image with the new training image, wherein the area occupied by the target background sample in the initial training image is the largest or greater than a preset area threshold.

[0018] According to another aspect of the present invention, an image segmentation device is also provided, which includes: an acquisition module for acquiring an annotation set based on a preset remote sensing image, the annotation set including a training set and a verification set, the training set including at least one training image and category labels for each image area on at least one training image, and the verification set including at least one verification image and category labels for each image area on at least one verification image; a training module for training at least one target segmentation model using the training set, the at least one target segmentation model being used to predict the category of each image area of ​​the input image; a first input module for inputting the verification set into the at least one target segmentation model to obtain a predicted segmentation result of the verification set, and when the predicted segmentation result of the verification set does not meet the preset verification requirements, the verification sample whose sample segmentation result in the verification set meets the preset prediction requirements is input into the verification sample. The verification image and the predicted category label corresponding to the verification sample are updated to the training set, and the step of training at least one target segmentation model using the training set is returned to execute until the predicted classification result of the verification set meets the preset verification requirements, wherein the predicted segmentation result of the verification set includes the sample segmentation result corresponding to each verification sample in the verification set, and the sample segmentation result is used to indicate the predicted category label corresponding to the verification sample. The verification sample is the image area in the verification image. The preset prediction requirement includes that the predicted probability of the predicted category label corresponding to the verification sample meets the first preset reliability, and the preset verification requirement includes that the predicted probability of the predicted category label corresponding to each verification sample in the verification set meets the second preset reliability; the second input module is used to input the remote sensing image to be measured into at least one target segmentation model to obtain the predicted segmentation result of the remote sensing image to be measured.

[0019] According to another aspect of the present invention, an electronic device is provided, comprising: a processor and a memory, wherein the memory stores computer program instructions, and the computer program instructions are used by the processor to execute the above-mentioned remote sensing image segmentation method based on semi-supervised learning when the processor is running.

[0020] According to yet another aspect of the present invention, a storage medium is provided on which program instructions are stored. The program instructions are used to execute the above-mentioned remote sensing image segmentation method based on semi-supervised learning when running.

[0021] According to yet another aspect of the present invention, a computer program product is provided, comprising computer program instructions, which are used to execute the above-mentioned remote sensing image segmentation method based on semi-supervised learning when running.

[0022] The above technical solution uses a training set to train the target segmentation model and a validation set to verify the performance of the target segmentation model, which is conducive to ensuring that the accuracy of the categories of each image area of ​​the input remote sensing image predicted by the trained target segmentation model meets the actual prediction requirements. In particular, when the predicted segmentation result obtained after the validation set is input into the target segmentation model does not meet the preset verification requirements, that is, when the performance of the target segmentation model does not meet the actual prediction requirements, the validation image to which the validation sample whose segmentation result in the validation set meets the preset prediction requirements and the predicted category label corresponding to the validation sample are updated to the training set without having to additionally obtain the training image and the category label of each image area on the training image. This training method based on semi-supervised learning can effectively save the time of re-acquiring the training image and the category label of each image area to expand the training set, and this method of updating the training set helps to expand the number of training images in the training set and the number of category labels of each image area on the training image. By updating the training set multiple times to train the target segmentation model multiple times, the performance of the target segmentation model can be effectively improved. On the other hand, by inputting the remote sensing image to be tested into the trained target segmentation model, the category corresponding to each image area on the remote sensing image to be tested can be determined quickly and accurately, which greatly saves the time of manual segmentation and labeling and is more efficient.

[0023] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above and other objects, features, and advantages of the present invention will become more apparent through a more detailed description of the embodiments of the present invention with reference to the accompanying drawings. The accompanying drawings are provided to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and are not intended to limit the present invention. In the drawings, the same reference numerals generally represent the same components or steps.

[0025] Figure 1 A schematic flow chart of a remote sensing image segmentation method based on semi-supervised learning according to one embodiment of the present invention is shown;

[0026] Figure 2 A schematic block diagram of a remote sensing image segmentation device based on semi-supervised learning according to an embodiment of the present invention is shown;

[0027] Figure 3 A schematic block diagram of an electronic device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of the present invention more apparent, exemplary embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments of the present invention described in the present invention, all other embodiments obtained by those skilled in the art without creative work should fall within the scope of protection of the present invention.

[0029] To at least partially address the above technical issues, embodiments of the present invention provide a remote sensing image segmentation method, apparatus, electronic device, storage medium, and computer program product based on semi-supervised learning. This approach trains one or more object segmentation models based on a small number of labeled samples. Using these trained object segmentation models, the remote sensing image under test can be segmented quickly and efficiently, significantly reducing manual segmentation and annotation time.

[0030] See also Figure 1 , which is a schematic flow chart of a remote sensing image segmentation method based on semi-supervised learning according to an embodiment of the present invention. According to one aspect of the present invention, a remote sensing image segmentation method based on semi-supervised learning is provided, the method comprising steps S110, S120, S130, and S140.

[0031] In step S110, an annotation set is obtained based on a preset remote sensing image. The annotation set includes a training set and a verification set. The training set includes at least one training image and a category label for each image region on at least one training image. The verification set includes at least one verification image and a category label for each image region on at least one verification image.

[0032] For example, the preset remote sensing image can be acquired using an optical sensor or radar sensor on a satellite platform, or an optical camera, multispectral camera, thermal imager, or other equipment on a manned aircraft or unmanned aerial vehicle (UAV) on an aerial platform. There can be one or more preset remote sensing images, and a labeling set can be generated based on the preset remote sensing images. Specifically, the labeling set can include a training set, which can include one or more training images, and at least a portion of the image area of ​​each training image can have a corresponding category label. The training image can be the entire preset remote sensing image or an image block segmented from the preset remote sensing image. Similarly, the labeling set can include a validation set, which can include one or more validation images, and at least a portion of the image area of ​​each validation image can have a corresponding category label. The validation image can be the entire preset remote sensing image or an image block segmented from the preset remote sensing image. The training images included in the training set and the validation images included in the validation set can be the same or different. When the training and validation images are the same, the image areas with category labels in the training images and the image areas with category labels in the validation images are at least partially different. Category labels can include, for example, cultivated land, forest land, grassland, urban and rural residential and public facility land, and industrial and mining land. These can be defined by the user based on actual classification needs. It is preferable to define category labels based on the classification method of the "Land Use Classification." A single image region can have one or more category labels. For example, the category label for a single image region can be paddy field. Another example is that the category labels for a single image region can include cultivated land and paddy field, where cultivated land can be used as a primary category label and paddy field can be used as a secondary category label. Each image region can be a single pixel of the image or a connected domain consisting of multiple pixels in the image.

[0033] In step S120, at least one target segmentation model is trained using the training set, and the at least one target segmentation model is used to predict the category of each image region of the input image.

[0034] Exemplarily, the target segmentation model can be, for example, a spatial pyramid pooling network (Pyramid Scene Parsing Network, PSPNet), an encoder-decoder based void convolution semantic segmentation network (DeepLabv3+), a high-resolution network (High-Resolution Net, HRNet), a full convolutional network (Fully Convolutional Networks, FCN), a dual attention network (Dual Attention Network, DANet), or any other semantic segmentation model that can be used to predict the category of each image region of the input image. In some embodiments, the initial target segmentation model can be an open source semantic segmentation model pre-trained with a large data set. The target segmentation model can be trained using a training set in the annotation set to further optimize the target segmentation model.

[0035] In step S130, the verification set is input into at least one target segmentation model to obtain a predicted segmentation result of the verification set. When the predicted segmentation result of the verification set does not meet the preset verification requirements, the verification image to which the verification sample whose sample segmentation result in the verification set meets the preset prediction requirements and the predicted category label corresponding to the verification sample are updated to the training set, and the step of training at least one target segmentation model using the training set is returned to be executed until the predicted classification result of the verification set meets the preset verification requirements, wherein the predicted segmentation result of the verification set includes the sample segmentation results corresponding to each verification sample in the verification set, the sample segmentation results are used to indicate the predicted category label corresponding to the verification sample, the verification sample is the image area in the verification image, the preset prediction requirement includes that the predicted probability of the predicted category label corresponding to the verification sample meets the first preset reliability, and the preset verification requirement includes that the predicted probability of the predicted category label corresponding to each verification sample in the verification set meets the second preset reliability.

[0036] Exemplarily, for each validation image in the validation set, part or all of the image region of the validation image may have a corresponding category label, and the image region with the category label may serve as a validation sample. After each training of the target segmentation model using the training set, the validation set may be input into the target segmentation model, and the target segmentation model may output a predicted segmentation result for the validation set. Specifically, the predicted segmentation result for the validation set may include a sample segmentation result corresponding to each validation sample in the validation set, and the sample segmentation result may indicate the predicted category label corresponding to the validation sample. Specifically, for each validation sample, the target segmentation model may determine and output the predicted category label for the validation sample based on the predicted probabilities of each category label predicted for the validation sample. For example, the target segmentation model may determine the category label with the highest predicted probability as the predicted category label for the validation sample, and optionally output the predicted probability corresponding to the category label. For another example, the target segmentation model may output a specific number of category labels as the predicted category labels for the validation sample in descending order of priority based on predicted probability, and optionally output the predicted probabilities corresponding to the output category labels. For another example, the target segmentation model may determine a category label with a predicted probability greater than or equal to a specific value as the predicted category label of the verification sample, and optionally output the predicted probability corresponding to the output category label. In a specific embodiment, for a certain verification sample, the sample segmentation result of the verification sample may include: woodland (prediction probability = 96.7%), grassland (prediction probability = 2.2%), and farmland (prediction probability = 0.4%). In another specific embodiment, for a certain verification sample, the sample segmentation result of the verification sample may include: woodland (prediction probability = 96.7%). In yet another specific embodiment, for a certain verification sample, the sample segmentation result of the verification sample may include: cultivated land (prediction probability = 99.3%), paddy field (prediction probability = 97.4%), irrigated land (prediction probability = 1.4%), woodland (prediction probability = 0.4%), and swampland (prediction probability = 0.33%). In this embodiment, cultivated land and woodland are used as primary category labels, paddy field and irrigated land are secondary category labels belonging to cultivated land, and swampland is a secondary category label belonging to woodland. Each validation sample can correspond to a predicted class label output by the target segmentation model, and can also correspond to its own class label. It is understood that for each validation sample, the greater the predicted probability of the predicted class label output by the target segmentation model for that validation sample that is consistent with the class label of the validation sample itself, the better the performance of the target segmentation model.

[0037] Exemplarily, for each verification sample, when the predicted probability of the predicted category label corresponding to the verification sample meets the first preset reliability, the sample segmentation result corresponding to the verification sample can be considered to meet the preset prediction requirements. In some embodiments, when the predicted category label with the largest predicted probability corresponding to the verification sample is consistent with the category label of the verification sample itself, and the predicted probability of the corresponding predicted category label with the largest predicted probability is greater than or equal to the first preset probability threshold, the predicted probability of the predicted category label corresponding to the verification sample can be considered to meet the first preset reliability. The first preset probability thresholds corresponding to category labels belonging to different preset categories may be the same or different. In other embodiments, when the predicted category label with the largest predicted probability corresponding to the verification sample is consistent with the category label of the verification sample itself, and the predicted probability of any predicted category label other than the predicted category label with the largest predicted probability is less than the second preset probability threshold, the predicted probability of the predicted category label corresponding to the verification sample can be considered to meet the first preset reliability.

[0038] Exemplarily, when the predicted segmentation result of the validation set does not meet the preset validation requirements, the validation image to which the validation sample whose sample segmentation result in the validation set meets the preset prediction requirements belongs and the predicted category label corresponding to the validation sample can be updated to the training set. Specifically, when the validation image to which the validation sample belongs is the same image as a training image in the training set, the category label of the image area corresponding to the validation sample can be updated in the training image, and the category label of the image area in the training image after the update can be the predicted category label with the maximum predicted probability of the corresponding validation sample or the predicted category label with a predicted probability greater than or equal to a specific probability threshold. Alternatively, the validation image to which the validation sample belongs can be directly added to the training set as a training image, and the category label of the image area corresponding to the validation sample in the training image can be updated to the predicted category label with the maximum predicted probability of the validation sample or the predicted category label with a predicted probability greater than or equal to a specific probability threshold. When the verification image to which the verification sample belongs is different from any training image in the training set, the verification image to which the verification sample belongs can be directly added to the training set as a training image, and the category label of the image area corresponding to the verification sample in the training image is updated to the predicted category label with the maximum predicted probability of the verification sample or the predicted category label with a predicted probability greater than or equal to a specific probability threshold. For example, when the predicted segmentation result of the verification set meets the preset verification requirements, it can be considered that the target segmentation model meets the preset optimization requirements. The preset verification requirements may include: the predicted probability of the predicted category label corresponding to each verification sample in the verification set meets the second preset reliability. For each verification sample in the verification set, whether the predicted probability of the predicted category label corresponding to the verification sample meets the second preset reliability can be determined by referring to the aforementioned situation where the predicted probability of the predicted category label corresponding to the verification sample meets the first preset reliability, which will not be repeated here. It should be noted that the first preset probability threshold and / or the second preset probability threshold in the second preset reliability can be the same as or different from the first preset probability threshold and / or the second preset probability threshold in the first preset reliability. When the predicted probability of the predicted category label corresponding to each verification sample meets the second preset confidence level, it can be considered that the predicted segmentation result of the verification set meets the preset verification requirement.

[0039] In step S140 , the remote sensing image to be measured is input into at least one target segmentation model to obtain a predicted segmentation result of the remote sensing image to be measured.

[0040] For example, after completing the training of the target segmentation model, the remote sensing image to be tested can be input into the target segmentation model to obtain a predicted segmentation result for the remote sensing image to be tested. The predicted segmentation result for the remote sensing image to be tested can refer to the description of the predicted segmentation result of the validation set in the previous embodiment, and will not be repeated here. In some embodiments, the remote sensing image to be tested and the preset remote sensing image can be related images. For example, the remote sensing image to be tested and the preset remote sensing image can be the same image. In this case, the union of the image areas of the images included in the annotation set can be a partial image area of ​​the preset remote sensing image (i.e., the remote sensing image to be tested). For another example, the preset remote sensing image is an image block in the remote sensing image to be tested. For another example, the remote sensing image to be tested and the preset remote sensing image belong to a set of remote sensing images collected by the same device for the target area. The set of remote sensing images can include multiple images, some of which can serve as the remote sensing images to be tested, and other images can serve as the preset remote sensing images. In other embodiments, the remote sensing image to be tested and the preset remote sensing image can be remote sensing images collected independently. Exemplarily, when there are multiple target segmentation models, the predicted segmentation result of the remote sensing image under test can be comprehensively determined based on the segmentation results output by each target segmentation model. For example, for each pixel in the remote sensing image under test, each target segmentation model can output a predicted probability that the pixel belongs to each preset category. Each target segmentation model can have a target weight. The predicted probabilities of the pixel belonging to the preset category are weighted and summed according to the target weights of each target segmentation model, and the preset category with the largest weighted sum is determined as the predicted category label for the pixel. In this case, the predicted category label corresponding to each image region of the remote sensing image under test is the predicted category label for each pixel contained in that image region. For another example, for each pixel in the remote sensing image under test, each target segmentation model can output a predicted probability that the pixel belongs to each preset category. The average of the predicted probabilities output by each target segmentation model for that pixel belonging to the preset category can be calculated, and the preset category with the largest average value is determined as the predicted category label for that pixel. Similarly, the predicted category label corresponding to each image region of the remote sensing image under test is the predicted category label for each pixel contained in that image region.

[0041] The above technical solution uses a training set to train the target segmentation model and a validation set to verify the performance of the target segmentation model, which is conducive to ensuring that the accuracy of the categories of each image area of ​​the input remote sensing image predicted by the trained target segmentation model meets the actual prediction requirements. In particular, when the predicted segmentation result obtained after the validation set is input into the target segmentation model does not meet the preset verification requirements, that is, when the performance of the target segmentation model does not meet the actual prediction requirements, the validation image to which the validation sample whose segmentation result in the validation set meets the preset prediction requirements and the predicted category label corresponding to the validation sample are updated to the training set without having to additionally obtain the training image and the category label of each image area on the training image. This training method based on semi-supervised learning can effectively save the time of re-acquiring the training image and the category label of each image area to expand the training set, and this method of updating the training set helps to expand the number of training images in the training set and the number of category labels of each image area on the training image. By updating the training set multiple times to train the target segmentation model multiple times, the performance of the target segmentation model can be effectively improved. On the other hand, by inputting the remote sensing image to be tested into the trained target segmentation model, the category corresponding to each image area on the remote sensing image to be tested can be determined quickly and accurately, which greatly saves the time of manual segmentation and labeling and is more efficient.

[0042] Optionally, the number of target segmentation models is greater than 1, and the network architecture of each target segmentation model is different. The verification set is input into at least one target segmentation model to obtain the predicted segmentation result of the verification set, including: inputting the verification set into each target segmentation model respectively to obtain the model segmentation result output by each target segmentation model, and the model segmentation result is used to indicate the predicted probability that each pixel in the verification set belongs to each preset category; determining the target weight of each target segmentation model according to the model segmentation result output by each target segmentation model; for each preset category in multiple preset categories, for each pixel in the verification set, the predicted probability of the pixel belonging to the preset category is weightedly summed according to the target weight of each target segmentation model, and the preset category with the largest weighted result is determined as the predicted category label of the pixel, the predicted category label corresponding to the pixel belongs to one of multiple preset categories, and the predicted category label corresponding to the verification sample is the predicted category label of each pixel contained in the verification sample.

[0043] Exemplarily, there may be multiple target segmentation models. In this case, the network architectures of each target segmentation model may be different. In step S130, for each target segmentation model, a validation set may be input into the target segmentation model to obtain a model segmentation result output by the target segmentation model. The model segmentation result may indicate the predicted probability that each pixel in the validation set belongs to each preset category. Specifically, for the same pixel, each target segmentation model may output multiple predicted probabilities that the pixel belongs to multiple preset categories, with each preset category corresponding to a predicted probability. The preset categories may be the categories indicated by the category labels of the aforementioned embodiments, and may include, for example, cultivated land, forest land, grassland, urban and rural residential and public facility land, industrial and mining land, etc. A target weight for each target segmentation model may be determined based on the model segmentation results output by each target segmentation model. For example, for each target segmentation model, the preset category with the maximum predicted probability output by the target segmentation model for each pixel may be used as the predicted category for that pixel. The prediction accuracy of the target segmentation model for each preset category may be determined based on the predicted category of each pixel and the category label corresponding to each pixel. The target weight of each target segmentation model can be determined according to the minimum accuracy of each target segmentation model, and the target weight of each target segmentation model can also be determined according to the overall accuracy of each target segmentation model. The overall accuracy can be, for example, equal to the ratio of the number of pixels whose predicted category is consistent with the category label to the total number of predicted pixels. Exemplarily, after determining the target weight of each target segmentation model, for each preset category, the predicted probability that each pixel in the verification set belongs to the preset category can be weighted and summed according to the target weight of each target segmentation model, and the preset category with the largest weighted result is determined as the predicted category label of the pixel. For example, at least one target segmentation model may include a first target segmentation model, a second target segmentation model, and a third target segmentation model, and their respective target weights are 0.5, 0.3, and 0.2, respectively. For a certain pixel, the first target segmentation model outputs the model segmentation results of: preset category a = 70%, preset category b = 20%, preset category c = 10%; the second target segmentation model outputs the model segmentation results of: preset category a = 60%, preset category b = 10%, preset category d = 30%; the third target segmentation model outputs the model segmentation results of: preset category a = 80%, preset category b = 20%. Then, the weighted predicted probability of the pixel belonging to preset category a = 0.5*70% + 0.3*60% + 0.2*80% = 69%, the weighted predicted probability of belonging to preset category b = 0.5*20% + 0.3*10% + 0.2*20% = 17%, the weighted predicted probability of belonging to preset category c = 0.5*10% = 5%, and the predicted probability of belonging to preset category d = 0.3*30% = 9%.In this embodiment, if the weighted predicted probability of preset category a is the largest, then preset category a can be determined as the predicted category label for the pixel. It will be understood that the predicted category label corresponding to each pixel belongs to one of the multiple preset categories, and the predicted category label corresponding to the validation sample can be the predicted category label of each pixel included therein.

[0044] The above technical solution inputs the verification set into multiple target segmentation models respectively and determines the target weight of each target segmentation model based on the model segmentation results output by each of the multiple target segmentation models. The predicted probability of each pixel in the verification set belonging to each preset category is weighted and summed according to each target weight to determine the predicted category label of each pixel. This can comprehensively determine the predicted category label of each verification sample based on the model segmentation results of each of the multiple target segmentation models. The determined predicted category label can, to a certain extent, reflect the comprehensive performance of the multiple target segmentation models. When training each target segmentation model, the performance of the target segmentation model is verified based on the predicted category label of the comprehensively determined verification sample to see whether it meets the actual prediction requirements. Compared with verifying whether the performance of the target segmentation model meets the actual prediction requirements based on the predicted category label of the verification sample determined separately by the target segmentation model, the requirements for the performance of the target segmentation model itself are lower, the training difficulty is less, and training time can be saved. On the other hand, this method of determining the predicted category label of the verification sample is conducive to improving the fault tolerance rate of each target segmentation model that has completed training, and the accuracy of the predicted segmentation result of the remote sensing image to be tested is finally determined.

[0045] Optionally, the target weight of each target segmentation model is determined according to the model segmentation results output by each target segmentation model, including: determining the overall accuracy of each target segmentation model in a one-to-one correspondence according to the model segmentation results output by each target segmentation model; determining the target weight of each target segmentation model in a one-to-one correspondence according to the overall accuracy of each target segmentation model.

[0046] Exemplarily, for each target segmentation model, the overall accuracy of the target segmentation model can be determined based on the model segmentation result output by the target segmentation model. For example, the ratio of the number of pixels whose predicted category is consistent with the category label in the model segmentation result output by any target segmentation model to the total number of pixels contained in the verification sample can be used as the overall accuracy of the target segmentation model. The relevant description of the predicted category can refer to the aforementioned embodiment and will not be repeated here. The target weight of each target segmentation model can be determined one-to-one based on the overall accuracy of each target segmentation model. For example, at least one target segmentation model may include a first target segmentation model, a second target segmentation model, and a third target segmentation model. If the overall accuracy of the first target segmentation model is f1, the overall accuracy of the second target segmentation model is f2, and the overall accuracy of the third target segmentation model is f3, then the target weight of the first target segmentation model can be f1 / (f1+f2+f3), the target weight of the second target segmentation model can be f2 / (f1+f2+f3), and the target weight of the third target segmentation model can be f3 / (f1+f2+f3).

[0047] The above technical solution determines the target weight of each target segmentation model through the overall accuracy of each target segmentation model, which can assign higher weights to target segmentation models with better performance. This helps the obtained predicted category labels of each pixel to better reflect the comprehensive performance of each target segmentation model, thereby helping to accurately judge whether the performance of each target segmentation model meets the actual prediction requirements.

[0048] Optionally, for any pixel, the predicted probability that the pixel belongs to the preset category is weighted using the following formula, and the preset category with the largest weighted result is determined as the predicted category label of the pixel:

[0049]

[0050] Where A represents the predicted category label of the pixel, n represents the number of target segmentation models, and W i represents the target weight of the i-th target segmentation model, P i (x) represents the probability that the pixel belongs to the x-th preset category in the model segmentation result output by the i-th target segmentation model.

[0051] Exemplarily, taking n=3 as an example, the multiple target segmentation models include a first target segmentation model (i=1), a second target segmentation model (i=2), and a third target segmentation model (i=3), wherein W1=0.4, W2=0.3, and W3=0.3. The number of preset categories is, for example, 2, including a first preset category (x=1) and a second preset category (x=2). P1(1)=0.8, P1(2)=0.2, P2(1)=0.7, P2(2)=0.3, P3(1)=0.75, P3(2)=0.25, then A=argmax(x=1, 0.4*0.8+0.3*0.7+0.3*0.75=0.755, x=2, 0.4*0.2+0.3*0.3+0.3*0.25=0.245)=the second preset category.

[0052] The above technical solution can quickly and accurately determine the preset category with the largest weighted result and use the preset category as the predicted label category of the corresponding pixel. In particular, when there are many preset categories, the weighted summation formula has a small amount of calculation and can efficiently determine the predicted label category of each pixel.

[0053] Optionally, the training set includes a first training sample and a second training sample, the first training sample and the second training sample are image areas in the training image, and the area of ​​the first training sample is larger than the area of ​​the second training sample; using the training set to train at least one target segmentation model includes: performing a first training operation, including: for each target segmentation model, inputting the training set into the target segmentation model; using the focal loss function to adjust the parameters of the target segmentation model until the target segmentation model meets the first optimization requirement, wherein during the parameter adjustment process, the learning rate of the target segmentation model maintains a preset standard learning rate unchanged, and the first optimization requirement includes that the predicted probability corresponding to the predicted category label obtained by predicting the first training sample meets a third preset confidence level: after performing the first training operation, performing a second training operation, including: for each target segmentation model, inputting the training set into the target segmentation model; using the focal loss function to adjust the parameters of the target segmentation model until the target segmentation model meets the second optimization requirement, wherein the parameters adjusted during the parameter adjustment process include the learning rate, and the second optimization requirement includes that the predicted probability corresponding to the predicted category label obtained by predicting the second training sample meets a fourth preset confidence level.

[0054] Exemplarily, the training set may include a first training sample and a second training sample. The first training sample and the second training sample are image regions within a training image. In other words, each first training sample or each second training sample is an image region within one of the training images in the training set. The area of ​​any first training sample is larger than the area of ​​any second training sample. That is, each first training sample of each first training sample occupies a larger area within the corresponding training image and can be considered a large target region within the corresponding training image. Correspondingly, each second training sample of each second training sample occupies a smaller area within the corresponding training image and can be considered a small target region within the corresponding training image. The first training sample and the second training sample can be determined by pixel statistics and setting an area threshold. Specifically, the area of ​​each image region can be determined through pixel statistics. By setting a specific area threshold, image regions with an area greater than or equal to the threshold can be determined as first training samples, and image regions with an area less than the threshold can be determined as second training samples. In step S120, for each target segmentation model, training the target segmentation model may include a first training operation and a second training operation. Specifically, the first training operation may be performed first. When performing the first training operation, the training set can be input into the target segmentation model, and the target segmentation model can be adjusted using the focal loss function. During the parameter adjustment process, the learning rate of the target segmentation model can always maintain the preset standard learning rate unchanged, and one or more parameters other than the learning rate can be adjusted according to the loss value obtained based on the focal loss function until the target segmentation model meets the first optimization requirement. The first optimization requirement may include: the predicted probability corresponding to the predicted category label obtained for the first training sample prediction meets the third preset reliability. For each first training sample in the training set, it is judged whether the predicted probability of the predicted category label corresponding to the first training sample meets the third preset reliability. The situation that the predicted probability of the predicted category label corresponding to each verification sample meets the first preset reliability can be referred to. This will not be repeated here. It should be noted that the first preset probability threshold and / or the second preset probability threshold in the third preset reliability can be the same as or different from the numerical value of the first preset probability threshold and / or the second preset probability threshold in the first preset reliability. When the predicted probability of the predicted class label corresponding to each first training sample meets the third preset confidence level, the target segmentation model can be considered to meet the first optimization requirement. More specifically, since the first training sample belongs to a large target region, it can be understood that when the target segmentation model meets the first optimization requirement, it can be considered that the target segmentation model can accurately segment the large target region in the image. In other words, the first training operation is primarily used to enable the target segmentation model to accurately segment the large target region in the image.Since the segmentation task for large target areas is relatively simple, the target segmentation model can be easily optimized to meet the first optimization requirement even without adjusting the learning rate. Using a fixed learning rate can simplify the first training operation, reduce the time for finding the optimal learning rate, and save the computing resources required to find the optimal learning rate.

[0055] Exemplarily, after executing the first training operation, a second training operation can be executed. When executing the second training operation, for any target segmentation model, similarly, the training set can be input into the target segmentation model, and the target segmentation model can be tuned using the focal loss function. During the parameter tuning process, the adjustable parameters of the target segmentation model include the learning rate. One or more parameters of the target segmentation model, including the learning rate, can be adjusted based on the loss value obtained based on the focal loss function until the target segmentation model meets the second optimization requirement. The second optimization requirement may include: the predicted probability corresponding to the predicted category label obtained for the second training sample meets the fourth preset confidence level. For each second training sample in the training set, whether the predicted probability of the predicted category label corresponding to the second training sample meets the fourth preset confidence level can be determined by referring to the situation where the predicted probability of the predicted category label corresponding to each validation sample meets the first preset confidence level, which will not be described in detail here. It should be noted that the first preset probability threshold and / or the second preset probability threshold in the fourth preset confidence level may be the same as or different from the first preset probability threshold and / or the second preset probability threshold in the first preset confidence level. When the predicted probability of the predicted category label corresponding to each second training sample meets the fourth preset confidence, the target segmentation model can be considered to meet the second optimization requirement. More specifically, since the second training sample belongs to the small target area, it can be understood that when the target segmentation model meets the second optimization requirement, it can be considered that the target segmentation model can accurately segment out the small target area in the image. In other words, the second training operation is mainly used to enable the target segmentation model to accurately segment out the small target area in the image. Since the segmentation task for small target areas is more difficult, adjusting the learning rate can help the target segmentation model converge faster and can better improve the performance of the target segmentation model, which is conducive to ensuring that the target segmentation model can segment out small target areas more accurately, so as to optimize the target segmentation model to meet the second training requirement. In some embodiments, in order to enhance feature extraction for small target areas, a target segmentation model including an attention mechanism module or a pyramid pooling module can be used.

[0056] The above technical solution optimizes the target segmentation model by performing the first training operation and the second training operation, which can take into account the performance of the target segmentation model in segmenting large target areas and segmenting small target areas. The trained target segmentation model has better performance and is not prone to losing small target areas.

[0057] Optionally, a balance factor used by the focus loss function in the first training operation is smaller than a balance factor used in the second training operation. The formula of the focus loss function is as follows:

[0058] FL(p t )=-α t (1-p t ) γ log(p t )

[0059] Where p t It represents the correct sample rate of prediction, which is equal to the ratio of the number of training samples whose predicted probability corresponding to the predicted category label meets the corresponding preset confidence to the total number of training samples. t represents the balance factor, and γ represents the preset adjustment factor.

[0060] Exemplarily, the first training sample whose predicted probability corresponding to the predicted category label obtained by prediction meets the third preset reliability, and the second training sample whose predicted probability corresponding to the predicted category label obtained by prediction meets the fourth preset reliability can be used as the correctly predicted samples, and the ratio between the number of correctly predicted samples and the total number of the first training samples and the second training samples can be used as the correctly predicted sample rate. The preset adjustment factor γ can be defined by the user according to actual conditions, and the embodiment of the present invention does not limit the specific value of the preset adjustment factor γ. In both the first training operation and the second training operation, the above-mentioned focal loss function can be used to calculate the loss value of the first training sample and the second training sample. The balance factor α used by the focal loss function in the first training operation is t is less than the α used in the second training operation t .

[0061] The above-mentioned technical solution uses a focal loss function to optimize the target segmentation model, which can better adapt to the imbalance in the number of first and second training samples, thereby improving the robustness of the target segmentation model. Furthermore, by setting the preset adjustment factor of the focal loss function higher in the second training operation, this helps make the target segmentation model more sensitive to the smaller second training samples, thereby ensuring that the trained target segmentation model can focus on small areas in the image.

[0062] Optionally, obtaining a labeling set based on a preset remote sensing image includes: determining reliable sample data and sample data to be labeled based on the preset remote sensing image, the reliable sample data including target reliable samples and category labels of the target reliable samples, the sample data to be labeled including samples to be labeled, and the target reliable samples and samples to be labeled are image areas in the preset remote sensing image; training the labeling model using the images to which the target reliable samples belong; inputting the images to which the samples to be labeled belong into the labeling model to obtain relabeled samples, and when the number of target reliable samples does not meet the preset number requirement, adding the reliable samples in the relabeled samples to the target reliable samples, and returning to execute the step of training the labeling model using the images to which the target reliable samples belong, until the number of target reliable samples meets the preset number requirement, determining the images to which the target reliable samples belong as the images included in the labeling set; wherein, the reliable samples in the relabeled samples include samples whose predicted category labels output by the labeling model meet the preset label requirements, and samples obtained after manual relabeling of samples whose predicted category labels output by the labeling model do not meet the preset label requirements.

[0063] For example, in step S110, reliable sample data and unlabeled sample data can be determined based on a preset remote sensing image. The reliable sample data can include target reliable samples and category labels of the target reliable samples. The target reliable samples can be image regions within the preset remote sensing image, and the category labels of the target reliable samples can indicate the preset categories corresponding to the target reliable samples. The unlabeled sample data can include unlabeled samples, which can be image regions within the preset remote sensing image. It will be appreciated that the target reliable samples and the unlabeled samples are different image regions within the preset remote sensing image. The image to which the target reliable samples belong can be used to train the annotation model. The image to which the target reliable samples belong can be image blocks within the preset remote sensing image, and the target reliable samples can be image regions within the image blocks. The annotation model can be a deep learning network, such as an EfficientNet or Vision Transformers. After each training of the annotation model, the unlabeled samples can be input into the annotation model to obtain relabeled samples. The relabeled samples can include both reliable and unreliable samples. Specifically, reliable samples can include samples whose predicted category labels output by the annotation model meet preset label requirements. In some embodiments, for each re-labeled sample, the preset label requirements may include, for example, that each pixel in the re-labeled sample has a corresponding predicted category label, and / or that the predicted category labels for each pixel in the re-labeled sample are the same, and / or that the ratio of the number of pixels in the re-labeled sample corresponding to the predicted category label with the largest number of pixels to the total number of pixels in the re-labeled sample is greater than or equal to a preset ratio threshold. Whether the re-labeled sample is a reliable sample can be determined by testing the predicted category label of the re-labeled sample. In other embodiments, whether the predicted category label output by the labeling model for the re-labeled sample meets the preset label requirements can also be determined by the user. Specifically, the user can determine whether the predicted category label of the re-labeled sample contains errors or omissions. Reliable samples among the re-labeled samples may also include samples obtained through manual re-labeling of samples whose predicted category labels output by the labeling model do not meet the preset label requirements. Manually re-labeled samples may be a subset of all samples whose predicted category labels output by the labeling model do not meet the preset label requirements. For example, if the number of target reliable samples does not meet the preset number requirement, the reliable samples among the re-labeled samples may be added to the target reliable samples, and the process returns to the step of training the labeling model using the images to which the target reliable samples belong. It is understood that during the training of the annotation model, the number of target reliable samples gradually increases. When the number of target reliable samples meets a preset quantity requirement, the images to which the target reliable samples belong can be included in the annotation set. The preset quantity requirement may, for example, include the number of target reliable samples being greater than or equal to a preset sample quantity threshold.

[0064] The above technical solution trains the annotation model by using the images belonging to the target reliable samples. When the number of target reliable samples does not meet the preset number requirements, the reliable samples in the re-annotated samples are added to the target reliable samples. This can optimize the annotation model multiple times when the initial number of target reliable samples is small, and the number of target reliable samples used in each optimization of the annotation model gradually increases, which is conducive to improving the performance of the target model. In addition, after each optimization of the annotation model, more target reliable samples can be obtained, which can obtain target reliable samples that meet the preset number requirements more quickly and efficiently, so that the annotation set can be obtained quickly and efficiently.

[0065] Optionally, the labeling model is trained using images of target reliable samples, including: performing image enhancement on the images of target reliable samples; inputting the images of target reliable samples before image enhancement and the images of target reliable samples after image enhancement into the labeling model to calculate the consistency loss value of the target reliable samples before and after image enhancement; and adjusting the parameters of the labeling model using the consistency loss value.

[0066] Exemplarily, in the process of training the annotation model, the image to which the target reliable sample belongs can be enhanced. The image enhancement method can specifically include one or more methods such as rotation, cropping, translation, scaling, and color transformation. It can be understood that the predicted category labels of the corresponding image areas in the image to which the target reliable sample belongs before and after image enhancement are consistent. The image to which the target reliable sample belongs before image enhancement and the image to which the target reliable sample belongs after image enhancement are input into the annotation model, and the consistency loss value of the target reliable sample before and after image enhancement can be calculated. The consistency loss value can be used to adjust the parameters of the annotation model. Specifically, each time the image to which the target reliable sample belongs is used to train the annotation model, it can be determined that the current training is completed when the consistency loss value is less than or equal to the preset loss threshold.

[0067] The above technical solution performs image enhancement on the image to which the target reliable sample belongs, and optimizes the labeling model according to the consistency loss value of the target reliable sample before and after image enhancement. This is conducive to ensuring that the predicted category labels output by the trained labeling model for the image-enhanced image are highly consistent with the predicted category labels output for the original image, thereby making the trained labeling model more stable.

[0068] Optionally, reliable sample data and sample data to be labeled are determined based on a preset remote sensing image, including: segmenting the preset remote sensing image into multiple image blocks; inputting the multiple image blocks into a pre-trained segmentation model to obtain image block segmentation results of the multiple image blocks output by the pre-trained segmentation model, the image block segmentation results being used to indicate predicted category labels of each image area contained in each of the multiple image blocks; in response to a user's re-labeling operation of at least part of the multiple image blocks with predicted category labels obtained based on the pre-trained segmentation model, determining new predicted category labels for at least part of the image areas in at least part of the image blocks; in response to the user's determination operation, determining at least part of the image area that currently has a predicted category label as a target reliable sample, and determining at least part of the image area that currently does not have a predicted category label as a sample to be labeled.

[0069] Exemplarily, when determining reliable sample data and sample data to be labeled based on a preset remote sensing image, the preset remote sensing image can be segmented into multiple image blocks. For example, the preset remote sensing image can be segmented into 1024*1024 image blocks. After obtaining multiple image blocks based on the segmentation of the preset remote sensing image, the multiple image blocks can be input into a pre-trained segmentation model to obtain image block segmentation results of the multiple image blocks output by the pre-trained segmentation model. Specifically, the pre-trained segmentation model can be, for example, a convolutional neural network such as a residual network (Residual Network, ResNet) or a U-net network (U-Net) that can be used for image segmentation. The image block segmentation result can indicate the predicted category label of each image area contained in each image block. It should be noted that the predicted category label of each image area output by the pre-trained segmentation model may not be the true category of the corresponding image area. In other words, the predicted category label output by the pre-trained segmentation model can be regarded as a "pseudo-label". A user can perform a relabeling operation on at least some of the image blocks in the plurality of image blocks based on the predicted category labels predicted by the pre-trained segmentation model. For example, a predicted category label can be added to an image region that lacks a predicted category label. For another example, a predicted category label whose indicated category is inconsistent with the actual category corresponding to the image region can be corrected. For another example, the boundary of the image region corresponding to a predicted category label can be adjusted. In response to the user's relabeling operation, new predicted category labels can be determined for at least some of the image regions in at least some of the image blocks. The user can perform a first determination operation to determine the image regions corresponding to at least some of the predicted category labels output by the pre-trained segmentation model and / or the image regions corresponding to at least some of the predicted category labels re-determined through the relabeling operation as image regions that currently have predicted category labels, and other image regions can be determined as currently not having predicted category labels. In response to the user's second determination operation, the image regions indicated by the second determination operation in the image regions that currently have predicted category labels can be determined as target reliable samples, and the image regions indicated by the second determination operation in the image regions that currently do not have predicted category labels can be determined as samples to be labeled. In some embodiments, the first determination operation and the second determination operation may be the same operation, and the determination operation may include: determining the image area corresponding to at least part of the predicted category label output by the pre-trained segmentation model, and / or, the image area corresponding to at least part of the predicted category label re-determined by the re-labeling operation as the target reliable sample, and determining at least part of the image area in other image areas except the target reliable sample as the sample to be labeled.

[0070] The above technical solution can use a pre-trained segmentation model to output predicted category labels for reference, which can greatly reduce the workload of users to segment and label image regions one by one, thereby saving time in obtaining reliable sample data and sample data to be labeled.

[0071] Optionally, the training set includes a first training sample and a second training sample, the first training sample and the second training sample are image areas in the training image, and the area of ​​the first training sample is larger than the area of ​​the second training sample; after determining the image to which the target reliable sample belongs as the image included in the annotation set, the method also includes: performing image enhancement on the second training sample in the training set; obtaining a new training image based on the initial training image to which the target background sample belongs by randomly embedding the image-enhanced second training sample into any target background sample in the training set, adding the new training image to the training set or replacing the initial training image with the new training image, wherein the area occupied by the target background sample in the initial training image is the largest or greater than a preset area threshold.

[0072] Exemplarily, when the training set includes a first training sample and a second training sample, the second training sample in the training set can be image enhanced, and the image enhancement method can include one or more methods such as rotation, flipping, scaling, cropping, adjusting brightness, adjusting contrast, and adjusting saturation. After obtaining the image-enhanced second training sample, the image-enhanced second training sample can be randomly embedded in any target background sample in the training set, and a new training image can be obtained based on the initial training image to which the target background sample belongs. It can be understood that the corresponding target background sample in the new training image is embedded with the second training sample. Specifically, the target background sample can be the image region with the largest area occupied in the initial training image, or the target background sample can be the image region with an area occupied in the initial training image that is greater than a preset area threshold.

[0073] The above technical solution can effectively solve the problem of the small number of second training samples with smaller area in the training set, which is conducive to enhancing the diversity of training samples in the training set. By randomly selecting target background samples in the initial training image and embedding the image-enhanced second training samples into them, more real scenes can be simulated, and it helps the annotation model to focus on small target areas.

[0074] See also Figure 2 , which is a schematic block diagram of an image segmentation device according to one embodiment of the present invention. According to another aspect of the present invention, a remote sensing image segmentation device based on semi-supervised learning is further provided. The device 200 includes:

[0075] An acquisition module 210 is configured to acquire an annotation set based on a preset remote sensing image, the annotation set comprising a training set and a validation set, the training set comprising at least one training image and a category label for each image region on the at least one training image, and the validation set comprising at least one validation image and a category label for each image region on the at least one validation image;

[0076] A training module 220 is configured to train at least one target segmentation model using a training set, wherein the at least one target segmentation model is configured to predict a category of each image region of an input image;

[0077] A first input module 230 is configured to input a validation set into at least one target segmentation model to obtain a predicted segmentation result of the validation set. When the predicted segmentation result of the validation set does not meet a preset validation requirement, the validation image to which the validation sample whose sample segmentation result in the validation set meets the preset prediction requirement and the predicted category label corresponding to the validation sample are updated to the training set, and the step of training the at least one target segmentation model using the training set is returned to be executed until the predicted classification result of the validation set meets the preset validation requirement, wherein the predicted segmentation result of the validation set includes the sample segmentation result corresponding to each validation sample in the validation set, the sample segmentation result is used to indicate the predicted category label corresponding to the validation sample, the validation sample is an image region in the validation image, the preset prediction requirement includes that the predicted probability of the predicted category label corresponding to the validation sample meets a first preset reliability, and the preset verification requirement includes that the predicted probability of the predicted category label corresponding to each validation sample in the validation set meets a second preset reliability;

[0078] The second input module 240 is configured to input the remote sensing image to be measured into at least one target segmentation model to obtain a predicted segmentation result of the remote sensing image to be measured.

[0079] See also Figure 3 As shown, it is a schematic block diagram of an electronic device according to an embodiment of the present invention. On the other hand, an electronic device is also provided according to the present invention. The electronic device 300 includes: a processor 310 and a memory 320, wherein the memory 320 stores computer program instructions, and the computer program instructions are used by the processor 310 to execute the above-mentioned remote sensing image segmentation method based on semi-supervised learning when the processor 310 is running.

[0080] According to another aspect of the present invention, a storage medium is provided, on which program instructions are stored. When the program instructions are executed by a computer or processor, the computer or processor executes the corresponding steps of the above-mentioned remote sensing image segmentation method based on semi-supervised learning according to the embodiment of the present invention, and is used to implement the corresponding modules in the above-mentioned image segmentation device according to the embodiment of the present invention or the corresponding modules in the above-mentioned image segmentation device. The storage medium may include, for example, a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, or any combination of the above-mentioned storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media.

[0081] According to yet another aspect of the present invention, a computer program product is provided, comprising computer program instructions, which are used to execute the above-mentioned remote sensing image segmentation method based on semi-supervised learning when running.

[0082] A person skilled in the art can understand the specific implementation and beneficial effects of the remote sensing image segmentation device, electronic device, storage medium and computer program product based on semi-supervised learning by reading the above detailed description of the remote sensing image segmentation method based on semi-supervised learning. For the sake of brevity, they will not go into details here.

[0083] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely illustrative and are not intended to limit the scope of the present invention. Various changes and modifications may be made therein by those skilled in the art without departing from the scope and spirit of the present invention. All such changes and modifications are intended to be included within the scope of the present invention as claimed in the appended claims.

[0084] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0085] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units is merely a logical function division. In actual implementation, other division methods may be used. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not performed.

[0086] In the description provided herein, numerous specific details are described. However, it is understood that embodiments of the present invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.

[0087] Similarly, it should be understood that in order to streamline the present invention and aid in understanding one or more of the various inventive aspects, in the description of exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this approach to the present invention should not be interpreted as reflecting the intention that the claimed invention requires more features than those explicitly recited in each claim. More precisely, as reflected in the corresponding claims, the inventive point is that the corresponding technical problem can be solved with fewer features than all the features of a single disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into the detailed description, with each claim itself serving as a separate embodiment of the present invention.

[0088] It will be understood by those skilled in the art that, except where mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or apparatus disclosed herein may be combined in any combination. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature providing the same, equivalent, or similar purpose.

[0089] Furthermore, those skilled in the art will appreciate that although some embodiments herein include certain features included in other embodiments but not other features, combinations of features from different embodiments are intended to be within the scope of the present invention and to form different embodiments. For example, in the claims, any of the claimed embodiments may be used in any combination.

[0090] The various component embodiments of the present invention can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all of the functions of some modules in the image segmentation apparatus according to an embodiment of the present invention. The present invention can also be implemented as a device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program for implementing the present invention can be stored on a computer-readable medium, or can be in the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0091] It should be noted that the above embodiments illustrate rather than limit the invention, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between brackets should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising several different elements and by means of appropriately programmed computers. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names.

[0092] The above is merely a description of specific embodiments of the present invention, and the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention are intended to be covered by the scope of protection of the present invention. The scope of protection of the present invention shall be based on the scope of protection of the claims.

Claims

1. A remote sensing image segmentation method based on semi-supervised learning, characterized in that: The method comprises: Acquire an annotation set based on a preset remote sensing image, the annotation set comprising a training set and a validation set, the training set comprising at least one training image and a category label for each image region on the at least one training image, and the validation set comprising at least one validation image and a category label for each image region on the at least one validation image; Using the training set to train at least one target segmentation model, the at least one target segmentation model is used to predict the category of each image region of the input image; The verification set is input into the at least one target segmentation model to obtain the predicted segmentation result of the verification set. When the predicted segmentation result of the verification set does not meet the preset verification requirements, the verification image to which the verification sample whose sample segmentation result in the verification set meets the preset prediction requirements and the predicted category label corresponding to the verification sample are updated to the training set, and the step of training at least one target segmentation model using the training set is returned to be executed until the predicted classification result of the verification set meets the preset verification requirements, wherein the predicted segmentation result of the verification set includes the sample segmentation result corresponding to each verification sample in the verification set, and the sample segmentation result is used to indicate the predicted category label corresponding to the verification sample, and the verification sample is the verification The image area in the image, the preset prediction requirement includes that the predicted probability of the predicted category label corresponding to the verification sample meets the first preset reliability, and when the predicted probability of the predicted category label corresponding to the verification sample meets the first preset reliability, the predicted category label with the largest predicted probability corresponding to the verification sample is consistent with the category label of the verification sample, the preset verification requirement includes that the predicted probability of the predicted category label corresponding to each verification sample in the verification set meets the second preset reliability, and when the predicted probability of the predicted category label corresponding to each verification sample in the verification set meets the second preset reliability, the predicted category label with the largest predicted probability corresponding to each verification sample in the verification set is consistent with the category label of the verification sample; The remote sensing image to be measured is input into the at least one target segmentation model to obtain a predicted segmentation result of the remote sensing image to be measured.

2. The method according to claim 1, characterized in that The number of the target segmentation models is greater than one, the network architecture of each target segmentation model is different, and the inputting the validation set into the at least one target segmentation model to obtain a predicted segmentation result of the validation set includes: Inputting the validation set into each target segmentation model to obtain a model segmentation result output by each target segmentation model, wherein the model segmentation result is used to indicate the predicted probability that each pixel in the validation set belongs to each preset category; Determine the target weight of each target segmentation model according to the model segmentation results output by each target segmentation model; For each preset category in multiple preset categories, for each pixel in the verification set, the predicted probability that the pixel belongs to the preset category is weighted and summed according to the target weights of each target segmentation model, and the preset category with the largest weighted result is determined as the predicted category label of the pixel. The predicted category label corresponding to the pixel belongs to one of the multiple preset categories, and the predicted category label corresponding to the verification sample is the predicted category label of each pixel contained in the verification sample.

3. The method according to claim 2, characterized in that The determining of the target weights of the target segmentation models according to the model segmentation results output by the target segmentation models includes: Determine the overall accuracy of each target segmentation model according to the model segmentation results output by each target segmentation model; The target weight of each target segmentation model is determined one-to-one according to the overall accuracy of each target segmentation model.

4. The method according to claim 2, characterized in that For any pixel, the predicted probability that the pixel belongs to the preset category is weighted using the following formula, and the preset category with the largest weighted result is determined as the predicted category label of the pixel: Where A represents the predicted category label of the pixel, n represents the number of target segmentation models, and W i represents the target weight of the i-th target segmentation model, P i (x) represents the probability that the pixel belongs to the x-th preset category in the model segmentation result output by the i-th target segmentation model.

5. The method according to any one of claims 1 to 4, characterized in that The training set includes a first training sample and a second training sample, the first training sample and the second training sample are image regions in the training image, and the area of ​​the first training sample is larger than the area of ​​the second training sample; The using the training set to train at least one target segmentation model includes: Perform a first training operation, including: for each target segmentation model, Inputting the training set into the target segmentation model; The target segmentation model is parameterized using a focal loss function until the target segmentation model meets a first optimization requirement, wherein during the parameter adjustment process, a learning rate of the target segmentation model maintains a preset standard learning rate unchanged, and the first optimization requirement includes that a predicted probability corresponding to a predicted class label obtained for the first training sample meets a third preset confidence level: After performing the first training operation, performing a second training operation includes: for each target segmentation model, Inputting the training set into the target segmentation model; The target segmentation model is adjusted using the focal loss function until the target segmentation model meets the second optimization requirement, wherein the parameters adjusted in the parameter adjustment process include the learning rate, and the second optimization requirement includes that the predicted probability corresponding to the predicted category label obtained for the second training sample prediction meets the fourth preset confidence.

6. The method according to claim 5, characterized in that The balance factor used by the focus loss function in the first training operation is smaller than the balance factor used in the second training operation. The formula of the focus loss function is as follows: FL(p t )=-a t (1-p t ) γ log(p t ) Where p t It represents the correct prediction rate, which is equal to the ratio of the number of training samples whose predicted probability corresponding to the predicted category label meets the corresponding preset confidence to the total number of training samples. t represents the balance factor, and γ represents the preset adjustment factor.

7. The method according to any one of claims 1 to 4, characterized in that The step of obtaining a label set based on a preset remote sensing image includes: Determining reliable sample data and sample data to be labeled based on the preset remote sensing image, wherein the reliable sample data includes target reliable samples and category labels of the target reliable samples, and the sample data to be labeled includes samples to be labeled, and the target reliable samples and the samples to be labeled are image areas in the preset remote sensing image; Using the images of the target reliable samples to train the annotation model; Inputting the image to which the sample to be labeled belongs into the labeling model to obtain a re-labeled sample; when the number of the target reliable samples does not meet the preset number requirement, adding the reliable samples in the re-labeled sample to the target reliable sample; returning to the step of training the labeling model using the image to which the target reliable sample belongs, until the number of the target reliable samples meets the preset number requirement, determining the image to which the target reliable sample belongs as the image included in the labeling set; Among them, the reliable samples in the re-labeled samples include samples whose predicted category labels output by the labeling model meet the preset label requirements, and samples obtained after manual re-labeling of samples whose predicted category labels output by the labeling model do not meet the preset label requirements.

8. The method according to claim 7, characterized in that The training of the annotation model using the image of the target reliable sample includes: Performing image enhancement on the image to which the target reliable sample belongs; Inputting the image to which the target reliable sample belongs before image enhancement and the image to which the target reliable sample belongs after image enhancement into the annotation model to calculate the consistency loss value of the target reliable sample before and after image enhancement; The consistency loss value is used to adjust the parameters of the annotation model.

9. The method according to claim 7, characterized in that The determining of reliable sample data and sample data to be labeled based on the preset remote sensing image includes: Segmenting the preset remote sensing image into a plurality of image blocks; Inputting the plurality of image blocks into a pre-trained segmentation model to obtain image block segmentation results of the plurality of image blocks output by the pre-trained segmentation model, wherein the image block segmentation results are used to indicate predicted category labels of respective image regions contained in each of the plurality of image blocks; In response to a relabeling operation of a user on at least part of the plurality of image blocks based on the predicted category labels obtained by the user based on the pre-trained segmentation model, determining new predicted category labels for at least part of the image regions in the at least part of the image blocks; In response to the user's determination operation, at least part of the image area currently having the predicted category label is determined as the target reliable sample, and at least part of the image area currently not having the predicted category label is determined as the sample to be labeled.

10. The method according to claim 7, characterized in that The training set includes a first training sample and a second training sample, the first training sample and the second training sample are image regions in the training image, and the area of ​​the first training sample is larger than the area of ​​the second training sample; After determining that the image to which the target reliable sample belongs is an image included in the annotation set, the method further includes: Performing image enhancement on the second training sample in the training set; A new training image is obtained based on an initial training image to which the target background sample belongs by randomly embedding the image-enhanced second training sample into any target background sample in the training set, and the new training image is added to the training set or the initial training image is replaced by the new training image, wherein the area occupied by the target background sample in the initial training image is the largest or greater than a preset area threshold.

11. A remote sensing image segmentation device based on semi-supervised learning, characterized in that: The device comprises: an acquisition module, configured to acquire an annotation set based on a preset remote sensing image, the annotation set comprising a training set and a validation set, the training set comprising at least one training image and a category label for each image region on the at least one training image, and the validation set comprising at least one validation image and a category label for each image region on the at least one validation image; a training module, configured to train at least one target segmentation model using the training set, wherein the at least one target segmentation model is configured to predict a category of each image region of an input image; The first input module is used to input the verification set into the at least one target segmentation model to obtain the predicted segmentation result of the verification set. When the predicted segmentation result of the verification set does not meet the preset verification requirements, the verification image to which the verification sample whose sample segmentation result in the verification set meets the preset prediction requirements and the predicted category label corresponding to the verification sample are updated to the training set, and the step of training at least one target segmentation model using the training set is returned to be executed until the predicted classification result of the verification set meets the preset verification requirements, wherein the predicted segmentation result of the verification set includes the sample segmentation result corresponding to each verification sample in the verification set, and the sample segmentation result is used to indicate the predicted category label corresponding to the verification sample. For the image area in the verification image, the preset prediction requirement includes that the predicted probability of the predicted category label corresponding to the verification sample meets the first preset reliability, and when the predicted probability of the predicted category label corresponding to the verification sample meets the first preset reliability, the predicted category label with the largest predicted probability corresponding to the verification sample is consistent with the category label of the verification sample, and the preset verification requirement includes that the predicted probability of the predicted category label corresponding to each verification sample in the verification set meets the second preset reliability, and when the predicted probability of the predicted category label corresponding to each verification sample in the verification set meets the second preset reliability, the predicted category label with the largest predicted probability corresponding to each verification sample in the verification set is consistent with the category label of the verification sample; The second input module is used to input the remote sensing image to be measured into the at least one target segmentation model to obtain a predicted segmentation result of the remote sensing image to be measured.

12. An electronic device comprising a processor and a memory, characterized in that: The memory stores computer program instructions, which are used by the processor to execute the remote sensing image segmentation method based on semi-supervised learning according to any one of claims 1 to 10 when the processor is running the computer program instructions.

13. A storage medium having program instructions stored thereon, characterized in that: The program instructions are used to execute the remote sensing image segmentation method based on semi-supervised learning as described in any one of claims 1 to 10 when running.

14. A computer program product comprising computer program instructions, characterized in that The computer program instructions are used to execute the remote sensing image segmentation method based on semi-supervised learning as described in any one of claims 1 to 10 when running.

Images