An unsupervised semantic segmentation method for remote sensing images driven by spectral index of ground objects

Abstract

The invention discloses an unsupervised remote sensing image semantic segmentation method driven by a spectral index of an object, comprising: calculating an object spectral index I of an object based on a remote sensing image to obtain an object spectral index map; marking the object spectral index map to obtain an initial pixel-by-pixel label; thresholding the object spectral index map using an extended threshold e to obtain a complete object; training a semantic segmentation network model using the initial pixel-by-pixel label to obtain a prediction result of a target object; integrating the prediction result of the target object with the complete object to obtain an extended area, and using the extended area to update the initial pixel-by-pixel label to obtain an extended label; continuing to train the semantic segmentation network model using the extended label to obtain a prediction result of the target object and enter the next iteration; until a maximum number of iterations is reached, and usable pixel-by-pixel labels are obtained; and training the semantic segmentation network model using the usable pixel-by-pixel labels for semantic segmentation.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing image semantic segmentation, and specifically relates to an unsupervised semantic segmentation method for remote sensing images driven by a ground object spectral index. Background Art

[0002] The semantic segmentation of remote sensing images aims to assign corresponding ground object categories to all pixels in the image, and has important application value in land cover mapping, urban management, environmental monitoring and other fields. The booming deep learning technology in recent years has greatly promoted the accuracy and efficiency of remote sensing image semantic segmentation. The semantic segmentation technology based on deep learning has become the mainstream means of automatically extracting ground object information from remote sensing images.

[0003] The fully convolutional neural network is the basic architecture of the deep semantic segmentation network. It uses convolution operations instead of fully connected layers to achieve intensive operations in the image space range. It is mainly composed of a downsampling part that extracts high-dimensional features and an upsampling part that restores detailed features. In addition, some optimization strategies are also used to improve the semantic segmentation effect, such as skip connections, dilated convolutions, multi-scale feature fusion, and fully connected conditional random fields.

[0004] The current implementation of semantic segmentation technology based on deep fully convolutional networks mainly includes the following steps:

[0005] Step 1: Select remote sensing image training data, manually draw pixel-by-pixel labels on remote sensing images, and refer to high-resolution images or conduct field surveys for unclear areas on the images. Then divide these data into training sets and validation sets.

[0006] Step 2: Select a suitable semantic segmentation network model based on the characteristics of the dataset, such as FCN, U-Net, DeepLab, HRNet, etc. You can also optimize the selected network model, such as multi-scale feature fusion, adding attention modules, applying regularized loss functions, etc.

[0007] Step 3: Input the training set into the semantic segmentation network for model training. Use the validation set to validate the current model every few training periods, and select the model with the highest validation set accuracy as the final model.

[0008] When using a semantic segmentation network for data prediction, you can choose to post-process the prediction results, such as using methods such as fully connected conditional random fields, to further optimize the boundaries and prediction accuracy.

[0009] Although fully convolutional neural networks can achieve good semantic segmentation results, the training of semantic segmentation networks requires a large amount of pixel-by-pixel labeled data, and the labeling cost is very expensive, which greatly restricts the large-scale promotion and application of this technology. Existing methods manually make pixel-by-pixel labels by referring to high-resolution images. This method is not only time-consuming and labor-intensive, but also requires good professional knowledge and even field investigations. Summary of the invention

[0010] Purpose of the invention: To solve the problem of expensive pixel-by-pixel labeling, the present invention proposes an unsupervised semantic segmentation method for remote sensing images driven by object spectral index, which avoids the high cost of manual visual interpretation to produce labeled data, automatically generates training samples using the object recognition ability of object spectral index, and designs targeted training methods to deal with the incompleteness and incorrectness of automatically generated samples, thereby obtaining a semantic segmentation model with both high accuracy and high integrity.

[0011] Technical solution: A method for unsupervised semantic segmentation of remote sensing images driven by spectral index of ground objects, comprising the following steps:

[0012] Step 1: Based on the remote sensing image, calculate the ground object spectral index I of the ground object and obtain the ground object spectral index map;

[0013] Step 2: Mark the object spectral index map based on formula (1) to obtain an initial pixel-by-pixel label, which includes a foreground label, a background label, and an ignored label; and threshold the object spectral index map using an extended threshold e, and regard the pixels whose object spectral index I is greater than the extended threshold e as complete objects;

[0014]

[0015] In the formula, t h is the foreground threshold used to extract the target object, t l is the background threshold used to extract non-target objects;

[0016] Among them, t l <e<t h ;

[0017] Step 3: Use the initial pixel-by-pixel labeling to train the semantic segmentation network model to obtain the prediction results of the target object;

[0018] Step 4: Integrate the prediction result of the target object with the complete object to obtain an extended area, and use the extended area to update the pixels with the ignored mark in the initial pixel-by-pixel mark to obtain an extended mark;

[0019] Step 5: Use the extended mark to continue training the semantic segmentation network model to obtain the prediction result of the target object, execute step 4, and enter the next iteration; until the maximum number of iterations is reached, and the usable pixel-by-pixel mark is obtained;

[0020] Step 6: Use the available pixel-by-pixel labels to train the semantic segmentation network model to obtain the final semantic segmentation network model;

[0021] Step 7: Input the remote sensing image to be semantically segmented into the final semantic segmentation network model to obtain the semantic segmentation result.

[0022] Furthermore, before executing step 1, the following steps are also included:

[0023] The remote sensing image is cropped and divided into blocks to obtain a plurality of remote sensing image sub-blocks;

[0024] Execute steps 1 to 5 for each remote sensing image sub-block to obtain a usable pixel-by-pixel label corresponding to each remote sensing image sub-block;

[0025] In step 6, the available pixel-by-pixel labels corresponding to all remote sensing image sub-blocks are used to train the semantic segmentation network model to obtain the final semantic segmentation network model.

[0026] Furthermore, the ground object spectral index includes one or more of a water body index, a vegetation index or a building index.

[0027] Furthermore, in step 4, the prediction result of the target object is integrated with the complete object to obtain the expansion area, and the specific operation is as follows:

[0028] Each connected domain in the complete object is judged. When any pixel in the connected domain is predicted as the target object in the prediction result of the target object, the connected domain is regarded as the target object and the expansion area is obtained.

[0029] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0030] (1) The pixel-by-pixel labeling method of the present invention avoids the key problem that a large number of training samples need to be manually produced in semantic segmentation network training, and can fully utilize the advantages of massive remote sensing image data;

[0031] (2) By adopting the unsupervised training method for semantic segmentation of remote sensing images of the present invention, the obtained semantic segmentation network can not only accurately distinguish the target object from other objects, but also extract the complete target object information, thereby realizing high-precision object information extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 Flowchart of unsupervised training of semantic segmentation network driven by spectral index of ground objects;

[0033] Figure 2 To extend the tag generation process. DETAILED DESCRIPTION

[0034] The technical solution of the present invention is now further described in conjunction with the accompanying drawings and embodiments.

[0035] The present invention proposes an unsupervised semantic segmentation method for remote sensing images driven by the spectral index of ground objects. The initial pixel-by-pixel labeling is generated based on the spectral index of ground objects. The complete pixel-by-pixel labeling obtained after optimization is used to train a deep fully convolutional neural network. Figure 1 After the remote sensing image data and the target object category are given, the method of the present invention has the following specific steps:

[0036] Step 1: Since remote sensing images are often large in size, the remote sensing images are cropped and divided into blocks according to the size of 512×512 pixels or 256×256 to obtain multiple remote sensing image sub-blocks; for each remote sensing image sub-block, S100 to S700 are executed to obtain a pixel-by-pixel label with both completeness and accuracy corresponding to each remote sensing image sub-block.

[0037] S100: Calculate the spectral index I of the ground object based on the remote sensing image, such as water body index, vegetation index, building index, etc.

[0038] S200: Based on the data characteristics, dual thresholds are set with high accuracy as the principle, including a foreground threshold t for extracting target objects. h and the background threshold t for extracting non-target objects l , collectively referred to as the safety threshold t. By assigning the safety threshold t to the ground object spectral index map, the initial pixel-by-pixel labeling can be obtained, including foreground pixels, background pixels, and ignored pixels. The calculation method is as shown in Formula 1.

[0039]

[0040] S300: Set the expansion threshold e based on the principle of high integrity, satisfying t l <e<t h , which is used to maximize the accuracy while ensuring the integrity of the target object. The extension threshold e is assigned to the object spectral index map, and the pixels with object spectral index greater than the extension threshold e are regarded as complete objects, that is, they have relatively complete target object information (but not accurate enough).

[0041] S400: According to the task requirements and data characteristics, select or design a suitable semantic segmentation network model, such as FCN, U-Net, DeepLab, HRNet, etc., use the initial pixel-by-pixel labeling obtained in step 3 to train the semantic segmentation network model, and output the prediction results of the target object after appropriate training rounds.

[0042] S500: Integrate the prediction result of the target object with the complete object in S300 to obtain a more accurate complete object (i.e. Figure 2 The relatively accurate complete feature is used to update the ignored pixels in the initial pixel-by-pixel labeling to obtain the extended labeling. The generation process of the extended labeling is as follows: Figure 2 shown.

[0043] The target object information obtained by thresholding the object spectral index using the extended threshold e is complete, but often interfered by other objects. The initial mark generated by the safety threshold t is used to train the segmentation network to predict a more accurate target object, and the prediction result is used to screen the complete object map obtained by the extended threshold. Specifically, each connected domain of the complete object result map is judged. If any pixel in the connected domain is predicted to be a target object at the corresponding position in the network output, the entire complete object connected domain is regarded as the target object, thereby obtaining an extended area, which is used to fill the ignored pixels in the initial mark.

[0044] S600: Continue to train the semantic segmentation network model using the extended markers, and output the prediction results of the target object after a suitable training round.

[0045] S700: Execute S500 to perform the next iteration. After multiple iterations, a pixel-by-pixel labeling with both completeness and accuracy is finally obtained.

[0046] Step 2: Use the complete and accurate pixel-by-pixel labels corresponding to all remote sensing image sub-blocks to train the semantic segmentation network model with the labeled data to achieve remote sensing image semantic segmentation for target object information extraction. The specific operation sub-steps are:

[0047] The pixel-by-pixel labeled data is divided into a training set and a validation set;

[0048] The training set is used to train the semantic segmentation network model, and the validation set is used to validate the trained semantic segmentation network model;

[0049] Select the model with the highest verification accuracy as the final semantic segmentation network model;

[0050] The remote sensing image to be semantically segmented is input into the final semantic segmentation network model to obtain the semantic segmentation result.

Claims

1. A method for unsupervised semantic segmentation of remote sensing images driven by spectral index of ground objects, characterized by: The following steps are involved: Step 1: Based on the remote sensing image, calculate the ground object spectral index I of the ground object and obtain the ground object spectral index map; Step 2: Mark the ground object spectral index map based on formula (1) to obtain an initial pixel-by-pixel label, which includes a foreground label, a background label, and an ignored label; and thresholding the ground object spectral index map using an extended threshold value e, and considering pixels whose ground object spectral index I is greater than the extended threshold value e as complete ground objects; Where, t h is the foreground threshold used to extract the target object, t l is the background threshold used to extract non-target objects; Among them, t l <e<t h ; Step 3: Use the initial pixel-by-pixel labeling to train the semantic segmentation network model to obtain the prediction results of the target object; Step 4: Integrate the prediction result of the target object with the complete object to obtain an extended area, and use the extended area to update the pixels with the ignored mark in the initial pixel-by-pixel mark to obtain an extended mark; Step 5: Use the extended mark to continue training the semantic segmentation network model to obtain the prediction result of the target object, execute step 4, and enter the next iteration; until the maximum number of iterations is reached, and the usable pixel-by-pixel mark is obtained; Step 6: Use the available pixel-by-pixel labels to train the semantic segmentation network model to obtain the final semantic segmentation network model; Step 7: Input the remote sensing image to be semantically segmented into the final semantic segmentation network model to obtain the semantic segmentation result.

2. The unsupervised semantic segmentation method of remote sensing images driven by spectral index of ground objects according to claim 1, characterized in that: Before executing step 1, the following steps are also included: The remote sensing image is cropped and divided into blocks to obtain a plurality of remote sensing image sub-blocks; Execute steps 1 to 5 for each remote sensing image sub-block to obtain a usable pixel-by-pixel label corresponding to each remote sensing image sub-block; In step 6, the available pixel-by-pixel labels corresponding to all remote sensing image sub-blocks are used to train the semantic segmentation network model to obtain the final semantic segmentation network model.

3. The unsupervised semantic segmentation method of remote sensing images driven by spectral index of ground objects according to claim 1, characterized in that: The ground object spectral index includes one or more of a water body index, a vegetation index or a building index.

4. The unsupervised semantic segmentation method of remote sensing images driven by spectral index of ground objects according to claim 1, characterized in that: In step 4, the prediction result of the target object is integrated with the complete object to obtain the expanded area. The specific operations are: Each connected domain in the complete object is judged. When any pixel in the connected domain is predicted as the target object in the prediction result of the target object, the connected domain is regarded as the target object and the expansion area is obtained.

Images