Land use classification method based on remote sensing image

Abstract

A land use classification method based on remote sensing images, comprising the following steps: step 1: obtaining the original image of remote sensing images of urban and township land, marking the pictures after screening, and a total of four categories: cultivated land, orchard, forest and building; step 2: cutting and data enhancement are carried out on the marked pictures to make a data set; step 3: the training set in the prepared data set is put into a feature extraction network and a multi-scale fusion up-sampling structure for training to obtain a segmentation model; step 4: the remote sensing image to be classified is put into the trained segmentation model to obtain the result; the purpose of the present application is to solve the problems of low small target precision and poor edge segmentation effect when the land categories are many and the features are complex in the land classification method based on remote sensing images, and a method capable of realizing more accurate identification result of various land types in remote sensing images is proposed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote sensing images, and particularly relates to a semantic segmentation method for land use classification. BACKGROUND

[0002] Remote sensing feature classification is a process of identifying and locating feature types in a scene using remote sensing images. Land classification can provide basic data support for various production tasks by classifying features in original remote sensing data images. Feature classification has very good application prospects and can play an important role in urban and rural planning, land use monitoring, disaster assessment and other tasks, and can be applied to agriculture, forestry, municipal and other fields. Today, remote sensing technology is developing rapidly, and a large amount of remote sensing data is generated every day. Remote sensing images have the advantages of strong timeliness, rich surface information coverage, wide observation range, and excellent use value. Using remote sensing images to classify land types has gradually become one of the research focuses. The first step of rational land use is to accurately understand the current land type. Current deep learning-based methods are widely used in land classification tasks. Since remote sensing images have the advantages of wide range, multiple features, and large quantity, they can meet the requirement of deep learning technology for large data sets, so current deep learning-based methods have achieved good results.

[0003] Although the application of deep learning technology to land classification tasks of remote sensing images has achieved good results, there are still many deficiencies. For example, in the article "Land Use Information Extraction from High-Resolution Remote Sensing Images Based on U-Net", Unet is used to segment 2m remote sensing images to obtain high-precision land classification results. However, most remote sensing images do not reach the 2m height range, and when higher resolution remote sensing images are used, there are problems such as small target classification and poor edge segmentation effect. SUMMARY

[0004] The purpose of the present application is to solve the technical problems of low accuracy of small targets and poor edge segmentation effect in land classification methods based on remote sensing images when there are many land categories and complex features.

[0005] A land use classification method based on remote sensing images, comprising the following steps:

[0006] Step 1: Obtain the original image of remote sensing images of urban and rural land, screen the image and mark it, which contains four categories: farmland, orchard, forest and building;

[0007] Step 2: Crop and data enhance the marked image to make a data set;

[0008] Step 3: Feed the training set from the prepared dataset into the feature extraction network and multi-scale fusion upsampling structure for training to obtain the segmentation model;

[0009] Step 4: Input the remote sensing image to be classified into the trained segmentation model to obtain the result;

[0010] In step 1, the original remote sensing images of urban and rural land are acquired, and the images are filtered and labeled. This process also includes the following sub-steps:

[0011] 1-1: Use drones to photograph urban and rural areas to obtain remote sensing images;

[0012] 1-2: Perform image processing such as stitching, shadow removal, and noise removal;

[0013] 1-3: Crop the images, remove images with a single category or those that do not contain the desired category, and label images containing forests, orchards, buildings, and farmland.

[0014] Step 3 specifically includes the following steps:

[0015] The first layer of the backbone feature extraction network → the second layer of the backbone feature extraction network → the third layer of the backbone feature extraction network → the fourth layer of the backbone feature extraction network → the fifth layer of the backbone feature extraction network → spatial pooling module;

[0016] The second layer of the backbone feature extraction network → the first composite pooling module; the third layer of the backbone feature extraction network → the second composite pooling module; the fourth layer of the backbone feature extraction network → the third composite pooling module.

[0017] Spatial pooling module, third composite pooling module → pooling module fusion layer, fourth layer of backbone feature extraction network → first multi-scale fusion upsampling module, third layer of backbone feature extraction network → second multi-scale fusion upsampling module, second layer of backbone feature extraction network → third multi-scale fusion upsampling module.

[0018] The spatial pooling module is as follows: the fifth layer of the backbone feature extraction network → POOL pooling layer → 1×1 convolutional layer (conv) → upsampling operation (upsample) → fusion (Concat);

[0019] The first multi-scale fusion upsampling module is: the fourth layer of the backbone feature extraction network → spatial attention SA → convolutional layer conv normalized BN activation ReLU, pooling module fusion layer → fusion Add;

[0020] The second multi-scale fusion up-sampling module is: the third layer of the backbone feature extraction network, spatial attention SA, a convolution layer conv, normalization BN, and activation Relu, and the first multi-scale fusion up-sampling module, fusion Add;

[0021] The third multi-scale fusion up-sampling module is: the second layer of the backbone feature extraction network, spatial attention SA, a convolution layer conv, normalization BN, and activation Relu, and the second multi-scale fusion up-sampling module, fusion Add.

[0022] In step 3, the loss function for network training is as follows:

[0023] The loss function is composed of a cross entropy loss Cross Entropy Loss with a weight, and the calculation formula is as follows:

[0024]

[0025] Wherein, M represents the number of categories, and in the present application, M is 5, that is, five categories of cultivated land, orchard, forest, building and background, c is a variable, representing a certain category, y c is a one-hot vector, and the elements only have two values of 0 and 1, if the category is the same as the category of the sample, it is 1, otherwise it is 0, p c represents the probability that the predicted sample belongs to c, and w c represents a weight coefficient, and the calculation formula is as follows:

[0026]

[0027] Wherein, N represents the total number of pixels, and Nc represents the number of pixels of the ground truth category c.

[0028] In step 4, the remote sensing image of the region needing land classification identification is cut into a picture of the required input size of the model, and is input into the trained segmentation model to obtain the segmentation result.

[0029] Compared with the prior art, the present application has the following technical effects:

[0030] (1) The present application provides a method for identifying land category regions through remote sensing images; compared with general manual statistics, it is more cost-effective and more efficient. Compared with the traditional method for identifying land classification regions on remote sensing images, the identification accuracy is higher, and the classified land can be identified on a larger scale;

[0031] (2) In constructing the network model, this invention uses an attention module to improve the segmentation accuracy of the model. Simultaneously, it improves the enhanced feature extraction module, enriches the size of the square pooling, enhances the segmentation effect for small targets, and adds a composite pooling structure. Utilizing feature maps at different levels, the model can better capture rich contextual information. Finally, a multi-layer upsampling and deep-shallow feature fusion structure is employed to compensate for the loss of edge information caused by increasing the number of network layers, optimizing the segmentation effect and obtaining better results. Attached Figure Description

[0032] The present invention will be further described below with reference to the accompanying drawings and embodiments:

[0033] Figure 1 This is a flowchart of the present invention;

[0034] Figure 2 This is a schematic diagram of the overall network structure of the present invention;

[0035] Figure 3 This is a schematic diagram of the composite pooling structure of the present invention. Detailed Implementation

[0036] The purpose of this invention is to address the technical problems in land classification methods based on remote sensing images, such as low accuracy for small targets and poor edge segmentation when there are many land categories and complex features. This invention proposes a method that can achieve more accurate identification results for multiple types of land in remote sensing images.

[0037] like Figure 1 As shown, a land classification and identification method based on remote sensing imagery includes the following steps:

[0038] Step 1: Obtain the original remote sensing images of urban and rural land, filter the images and label them, including four categories: farmland, orchard, forest and buildings;

[0039] Step 2: Crop and augment the labeled images to create a dataset;

[0040] Step 3: Feed the training set from the prepared dataset into the feature extraction network and multi-scale fusion upsampling structure for training to obtain the segmentation model;

[0041] Step 4: Input the remote sensing image to be classified into the trained segmentation model to obtain the result;

[0042] In step 1, the original remote sensing images of urban and rural land are acquired, and the images are filtered and labeled. This process also includes the following sub-steps:

[0043] 1-1: Use drones to photograph urban and rural areas to obtain remote sensing images;

[0044] 1-2: image stitching, shadow removal, noise removal, etc.

[0045] 1-3: crop the picture, remove images with single category and without the required category, and label pictures containing forests, orchards, buildings, and farmland.

[0046] In step 3, the feature extraction network and the multi-scale fusion up-sampling structure are composed of a backbone feature extraction network (such as ResNet-50 shown in FIG. 1, FIG. 2, FIG. 3, FIG. 4, and FIG. 5), a spatial pooling module 6, a composite pooling module (such as shown in FIG. 1, FIG. 2, FIG. 3, FIG. 4, and FIG. 5), a pooling module fusion layer 10, and a multi-scale fusion up-sampling module (such as shown in FIG. 1, FIG. 2, FIG. 3, FIG. 4, and FIG. 5). Figure 1 Figure 1 Figure 1

[0047] The first layer 1 of the backbone feature extraction network → the second layer 2 of the backbone feature extraction network → the third layer 3 of the backbone feature extraction network → the fourth layer 4 of the backbone feature extraction network → the fifth layer 5 of the backbone feature extraction network → the spatial pooling module 6.

[0048] The second layer 2 of the backbone feature extraction network → the first composite pooling module 7, the third layer 3 of the backbone feature extraction network → the second composite pooling module 8, and the fourth layer 4 of the backbone feature extraction network → the third composite pooling module 9.

[0049] The spatial pooling module 6 and the third composite pooling module 9 → the pooling module fusion layer 10, the fourth layer 4 of the backbone feature extraction network → the first multi-scale fusion up-sampling module 11, the third layer 3 of the backbone feature extraction network → the second multi-scale fusion up-sampling module 12, and the second layer 2 of the backbone feature extraction network → the third multi-scale fusion up-sampling module 13.

[0050] The spatial pooling module 6 is: the fifth layer 5 of the backbone feature extraction network → the pooling layer POOL (bin size is 1x1; 2x2; 3x3; 6x6; 12x12; 24x24) → the 1x1 convolution layer conv → the up-sampling operation upsample → the fusion Concat.

[0051] The first multi-scale fusion up-sampling module 11 is: the fourth layer 4 of the backbone feature extraction network → the spatial attention SA → the convolution layer conv standardization BN activation Relu, the pooling module fusion layer 10 → the fusion Add.

[0052] ​​​The second multi-scale fusion upsampling module 12 is: the third layer 3 of the backbone feature extraction network → spatial attention SA → convolutional layer conv normalized BN activation ReLU, and the first multi-scale fusion upsampling module 11 → fusion Add;

[0053] The third multi-scale fusion upsampling module 13 is: the second layer 2 of the backbone feature extraction network → spatial attention SA → convolutional layer conv normalized BN activation ReLU, and the second multi-scale fusion upsampling module 12 → fusion Add.

[0054] In step 3, the loss function used for network training is as follows:

[0055] The loss function consists of the weighted cross-entropy loss, calculated as follows:

[0056]

[0057] Where M represents the number of categories, y c It is a one-hot vector where each element takes only two values: 0 and 1. The element is set to 1 if the class matches the sample's class, and 0 otherwise. c w represents the probability that the predicted sample belongs to c. c The weighting coefficient is represented by the following formula:

[0058]

[0059] Where N represents the total number of pixels, and Nc represents the number of pixels of category c in the ground truth category.

[0060] In step 4, the remote sensing image of the area requiring land classification and identification is segmented into images of the size required by the model, and then input into the pre-trained segmentation model to obtain the segmentation results.

[0061] In step 3, specifically:

[0062] 1) Construct a feature extraction network to extract target features;

[0063] 2) Construct a multi-scale feature fusion upsampling structure and use the features to obtain prediction results;

[0064] In step 1), a feature extraction network is constructed to obtain target features. The feature extraction network consists of two parts: a backbone feature extraction network and an enhanced feature extraction module. Specifically, it also includes the following steps:

[0065] 1-1) The ResNet-50 is divided into four parts, and channel and spatial attention are added to the first and last layers of the network: In the channel attention block, global max pooling and average pooling are first performed on the feature maps, then they are fed into a two-layer neural network, and the resulting features are then processed through a sigmoid activation function to obtain weight coefficients; finally, the weight coefficients are multiplied by the input features to obtain the scaled new features. In the spatial attention block, max pooling and average pooling are first performed on each channel to obtain two channel descriptions, and then these two descriptions are concatenated by channel. Then, a 7×7 convolutional layer and a sigmoid activation function are used to obtain weight coefficients, and finally, the weight coefficients are multiplied by the input features to obtain the scaled new features.

[0066] 1-2) The feature map obtained from ResNet-50 is input into the enhanced feature extraction module, which consists of two parts: spatial pooling and compound pooling. The first part divides the input feature map F4 into six regions of different sizes: 1×1, 2×2, 3×3, 6×6, 12×12, and 24×24. Then, average pooling is performed on each region to obtain feature layers of different sizes. After upsampling, these are concatenated with the input F4 to obtain feature map F6. The second part first inputs F1 into the compound pooling module mppm, then downsamples the output and concatenates it with F2 before inputting it back into the mppm module. The output is then downsampled and concatenated with F3 before being input back into the mppm module to obtain feature map F5. The composite pooling module mppm divides the feature map into two elongated regions of size H×1 and 1×W and a region of size H×W of the original image for strip pooling. Then, it expands the two output feature maps along the left and right and up and down respectively through convolution. The corresponding positions of the expanded feature maps are summed to obtain the H×W feature map.

[0067] Step 2) further includes the following steps:

[0068] 2-1): Obtain the F1, F2, and F3 layers from the backbone feature extraction network and input them into a spatial attention layer. The spatial attention layer performs channel-dimensional average pooling and max pooling operations on the input feature maps F1, F2, and F3 respectively, and concatenates the two results to obtain a feature description. Then, after passing through a 7×7 convolutional layer and a sigmoid function, a spatial weight coefficient is generated. The spatial weight coefficient is multiplied by the original input feature map to obtain the output results. Then, the output results are input into a convolutional, normalization, and activation function layer to adjust the number of channels to obtain feature maps F7, F8, and F9.

[0069] 2-2): Obtain feature maps F5 and F6 output by the enhanced feature extraction module. F5 and F6 are fused by adding pixels together and then upsampled to obtain feature map F10. Then, F10 is fused with F7 and upsampled to obtain feature map F11. F11 is then fused with F8 and upsampled to obtain feature map F12. Finally, F12 is fused with F9 and upsampled to output the final pixel-level classification result.

[0070] In step 3, the loss function used for network training is defined as follows:

[0071] The loss function consists of the weighted cross-entropy loss, calculated as follows:

[0072]

[0073] Where M represents the number of categories, y c It is a one-hot vector where each element takes only two values: 0 and 1. The element is set to 1 if the class matches the sample's class, and 0 otherwise. c w represents the probability that the predicted sample belongs to c. c The weighting coefficient is represented by the following formula:

[0074]

[0075] Where N represents the total number of pixels, and Nc represents the number of pixels of category c in the ground truth category.

[0076] In step 4, the remote sensing image to be identified needs to be cut into images of the size required by the model, put into the model for recognition, and then stitch the result images together.

[0077] Example:

[0078] The land classification and extraction method of this invention on remote sensing images is carried out in the following manner:

[0079] Step 1: The drone acquires remote sensing images, filters the images, selects those with diverse categories and rich features, and uses ENVI software to label farmland, orchards, forests, and building areas in the remote sensing images, while using areas other than these as the background.

[0080] Step 2: After labeling, the images are cropped and augmented. The remote sensing images and label maps are cut into 512×512 three-channel images to create a dataset. The main data augmentation operations for images include the following:

[0081] (1) Rotation: Random rotation with the center of the image as the rotation center;

[0082] (2) Translation: The image is translated randomly along the X-axis or Y-axis;

[0083] (3) Scaling: The image is randomly enlarged or reduced according to the proportion to achieve multi-scale training;

[0084] (4) Color jitter: Randomly changing the exposure, saturation and hue of an image to create images under different lighting and colors;

[0085] (5) Gaussian noise: Adding noise makes the image blurry.

[0086] Step 3: Input the image into the feature extraction network to obtain the feature map. Input F4 from the backbone feature extraction network into the spatial pooling module, and input F1, F2, and F3 into the composite pooling structure to enhance feature extraction. Then, fuse the results from the two parts.

[0087] Step 4: The result of fusing the spatial pooling module and the composite pooling module is processed through a multi-scale fusion upsampling structure to obtain the final prediction result.

[0088] Table 1. Results of different semantic segmentation methods on remote sensing datasets.

[0089]

[0090] The ablation experiment compared five current mainstream algorithms and the algorithm of this invention. The IOU of this invention is not much different from other algorithms in orchards and forests, but it outperforms other algorithms in the IOU of farmland and buildings, which are two target categories with particularly complex scales and contain many small scales.

[0091] Step 5: When training the network, the loss function is defined as follows:

[0092] The loss function consists of the weighted cross-entropy loss, calculated as follows:

[0093]

[0094] Where M represents the number of categories, y c It is a one-hot vector where each element takes only two values: 0 and 1. The element is set to 1 if the class matches the sample's class, and 0 otherwise. c w represents the probability that the predicted sample belongs to c. c The weighting coefficient is represented by the following formula:

[0095]

[0096] Where N represents the total number of pixels, and Nc represents the number of pixels of category c in the ground truth category.

[0097] Step 6: Cut the remote sensing image of the area to be classified and identified into images of the required size for the model. Then, cut the remote sensing image to be identified into images of the required size for the model, put them into the model for identification, and stitch the results together after the identification is completed.

Claims

1. A land use classification method based on remote sensing imagery, characterized in that, Includes the following steps: Step 1: Obtain the original remote sensing images of urban and rural land, filter the images and label them, including four categories: farmland, orchard, forest and buildings; Step 2: Crop and augment the labeled images to create a dataset; Step 3: Feed the training set from the prepared dataset into the feature extraction network and multi-scale fusion upsampling structure for training to obtain the segmentation model; Step 4: Input the remote sensing image to be classified into the trained segmentation model to obtain the result; Step 3 specifically includes the following steps: The first layer (1) of the backbone feature extraction network → the second layer (2) of the backbone feature extraction network → the third layer (3) of the backbone feature extraction network → the fourth layer (4) of the backbone feature extraction network → the fifth layer (5) of the backbone feature extraction network → spatial pooling module (6); The second layer (2) of the backbone feature extraction network → the first composite pooling module (7), the third layer (3) of the backbone feature extraction network → the second composite pooling module (8), the fourth layer (4) of the backbone feature extraction network → the third composite pooling module (9); Spatial pooling module (6), third composite pooling module (9) → pooling module fusion layer (10), fourth layer of backbone feature extraction network (4) → first multi-scale fusion upsampling module (11), third layer of backbone feature extraction network (3) → second multi-scale fusion upsampling module (12), second layer of backbone feature extraction network (2) → third multi-scale fusion upsampling module (13).

2. The method according to claim 1, characterized in that, In step 1, the original remote sensing images of urban and rural land are acquired, and the images are filtered and labeled. This process also includes the following sub-steps: 1-1: Use drones to photograph urban and rural areas to obtain remote sensing images; 1-2: Perform image stitching, shadow removal, and noise removal; 1-3: Crop the images, remove images with a single category or those that do not contain the desired category, and label images containing forests, orchards, buildings, and farmland.

3. The method according to claim 1, characterized in that, The spatial pooling module (6) is as follows: the fifth layer (5) of the backbone feature extraction network → pooling layer POOL → passing through 1×1 convolutional layer, 2×2 convolutional layer, 3×3 convolutional layer, 6×6 convolutional layer, 12×12 convolutional layer, and 24×24 convolutional layer respectively → performing upsampling operation on the features output by each convolutional layer → fusion Concat; The first multi-scale fusion upsampling module (11) is specifically: the fourth layer (4) of the backbone feature extraction network → spatial attention SA → convolutional layer conv normalized BN activation ReLU, pooling module fusion layer (10) → fusion Add; The second multi-scale fusion upsampling module (12) is specifically: the third layer (3) of the backbone feature extraction network → spatial attention SA → convolutional layer conv normalized BN activation ReLU, the first multi-scale fusion upsampling module (11) → fusion Add; The third multi-scale fusion upsampling module (13) is as follows: the second layer (2) of the backbone feature extraction network → spatial attention SA → convolutional layer conv normalized BN activation ReLU, the second multi-scale fusion upsampling module (12) → fusion Add.

4. According to claim 1, in step 3, the loss function used for network training is as follows: The loss function consists of the weighted cross-entropy loss, calculated as follows: in, M represents the number of categories, y c It is a one-hot vector where each element takes only two values: 0 and 1. The element is set to 1 if the class matches the sample's class, and 0 otherwise. c w represents the probability that the predicted sample belongs to c. c The weighting coefficient is represented by the following formula: Where N represents the total number of pixels, and Nc represents the number of pixels of category c in the ground truth category.

5. The method according to claim 1, characterized in that, In step 4, the remote sensing image of the area requiring land classification and identification is segmented into images of the size required by the model, and then input into the pre-trained segmentation model to obtain the segmentation results.

Images