Method and system for local climate zone classification based on multispectral data

Abstract

The application provides a local climate zone classification method based on multispectral data, comprising: preprocessing an original Sentinel-2 image to obtain a to-be-classified image; building an initial multi-layer convolutional neural network in a keras framework of Tensorflow based on deep learning; adding a residual connection and an attention mechanism module in the initial multi-layer convolutional neural network to obtain a first optimization stage convolutional neural network; replacing an ordinary convolutional layer in the first optimization stage convolutional neural network with a deep separable convolutional layer to obtain a final convolutional neural network; training the final convolutional neural network with a training data set based on LCZ classification to obtain an optimal model; and inputting the to-be-classified image into the optimal model for classification to obtain an LCZ classification map of a target region. The application constructs a brand-new optimal model, the optimal model can be used for abstracting and learning meaningful complex context features, the overall classification accuracy is higher than that of a peer model, and the optimal model has stronger robustness.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method and system for classifying local climate zones based on multispectral data. Background Technology

[0002] In 2012, climatologists formally proposed the concept of Local Climate Zones (LCZs), which are areas with similar land cover, urban structures and building materials, and human activities on a horizontal scale ranging from hundreds to thousands of meters. Among the 17 standard local climate zones, there are 10 urban building categories and seven natural categories. Each zone can also be quantitatively divided by 10 specific parameters, including sky visibility, street width-to-height ratio, building footprint ratio, impermeable layer footprint ratio, permeable layer footprint ratio, average height, topographic roughness grade, surface conductivity, surface reflectivity, and anthropogenic heat flux. The classification of local climate zones provides a fundamental framework for research on the urban heat island effect and normative global temperature observations.

[0003] However, the classification criteria for land cover within different categories of local climate zones are complex and diverse. Traditional image classification methods, such as supervised and unsupervised classification, struggle to effectively extract features from local climate zones, thus affecting the classification performance of LCZ (Local Climate Zone Classification). In recent years, the rapid development of deep learning technology in computer vision has propelled remote sensing image classification towards higher accuracy and automation. Currently, deep learning-based remote sensing image classification technology for local climate zones has made some progress and has been applied to research on urban heat islands, human activities, and climate assessment. The potential of deep learning-based local climate zone classification technology has not been fully explored, largely because most studies are based on different training datasets, making it difficult to fairly and consistently compare the performance of different deep learning models in local climate zone classification.

[0004] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a method for classifying local climate zones based on multispectral data, comprising:

[0006] S1: Obtain the original Sentinel-2 image of the target region, preprocess the original Sentinel-2 image to obtain the image to be classified;

[0007] S2: Build an initial multi-layer convolutional neural network in the Keras framework of Tensorflow, which is based on deep learning;

[0008] S3: Add residual connections and attention mechanism modules to the initial multi-layer convolutional neural network to obtain the first-stage optimized convolutional neural network;

[0009] S4: Replace the ordinary convolutional layers in the first optimization stage convolutional neural network with depthwise separable convolutional layers to obtain the final convolutional neural network;

[0010] S5: Train the final convolutional neural network using a training dataset based on LCZ classification to obtain the optimal model;

[0011] S6: Input the image to be classified into the optimal model for classification to obtain the LCZ classification map of the target region.

[0012] Preferably, step S1 specifically includes:

[0013] S11: Crop the Sentinel-2 image to the same extent as the target region to obtain the original Sentinel-2 image of the target region;

[0014] S12: Perform atmospheric correction, registration, cropping, and band synthesis on the original Sentinel-2 image to obtain the processed Sentinel-2 image;

[0015] S13: Select 10 bands with spatial resolutions of 10 meters and 20 meters from the processed Sentinel-2 image, perform band synthesis, and uniformly resample to 10-meter spatial resolution to obtain the image to be classified.

[0016] Preferably, the initial multilayer convolutional neural network includes: a variable-scale convolutional module, multiple feature extraction modules, a global average pooling layer, and a fully connected layer;

[0017] The variable-scale convolutional module includes: a 5*5*16 convolutional layer, a 3*3*32 convolutional layer, a 1*1*16 convolutional layer, and a fusion layer; the outputs of the 5*5*16 convolutional layer, the 3*3*32 convolutional layer, and the 1*1*16 convolutional layer are connected to the input of the fusion layer.

[0018] The feature extraction module includes: a first ordinary convolutional layer, a second ordinary convolutional layer, and a fusion layer; the output of the first ordinary convolutional layer is connected to the input of the second ordinary convolutional layer, and the output of the second ordinary convolutional layer is connected to the input of the fusion layer.

[0019] The variable-scale convolutional modules, feature extraction modules, global average pooling layers, and fully connected layers are sequentially connected to form the initial multi-layer convolutional neural network.

[0020] Preferably, step S3 specifically includes:

[0021] S31: In the initial multi-layer convolutional neural network, an attention mechanism module i is added between feature extraction module i and feature extraction module i+1. The attention mechanism module includes: channel attention mechanism module and spatial attention mechanism module; where i is the number of the feature extraction module.

[0022] S32: Establish a residual connection between the fusion layer of the variable-scale convolutional module and the fusion layer of the first feature extraction module, and establish a residual connection between the fusion layer of the attention mechanism module i and the feature extraction module i+1.

[0023] Preferably, step S4 specifically includes:

[0024] Replace the first and second ordinary convolutional layers in the feature extraction modules with odd numbers with depthwise separable convolutional layers;

[0025] Replace the second ordinary convolutional layer in the even-numbered feature extraction module with a depthwise separable convolutional layer.

[0026] Preferably, step S6 specifically includes:

[0027] S61: Divide the image to be classified into 32*32 image blocks. If the edge blocks are smaller than 32*32, pad them with null values ​​to obtain the padded segmented image.

[0028] S62: Input the padded segmented image into the optimal model for prediction and classification. Sort the images according to the predicted probabilities of each LCZ category, and assign the category with the highest probability value to the corresponding LCZ classification image.

[0029] A local climate zone classification system based on multispectral data, comprising:

[0030] The image acquisition unit is used to acquire the original Sentinel-2 image of the target region, preprocess the original Sentinel-2 image, and obtain the image to be classified.

[0031] The network building unit is used to build an initial multi-layer convolutional neural network in the Keras framework of Tensorflow, which is based on deep learning.

[0032] The first optimization unit is used to add residual connections and attention mechanism modules to the initial multi-layer convolutional neural network to obtain the first-stage convolutional neural network.

[0033] The second optimization unit is used to replace the ordinary convolutional layer in the convolutional neural network of the first optimization stage with a depthwise separable convolutional layer to obtain the final convolutional neural network.

[0034] The training unit is used to train the final convolutional neural network using a training dataset based on LCZ classification to obtain the optimal model.

[0035] The classification unit is used to input the image to be classified into the optimal model for classification, and obtain the LCZ classification map of the target region.

[0036] The present invention has the following beneficial effects:

[0037] This invention adds residual connections and attention mechanism modules to the initial multi-layer convolutional neural network, and replaces ordinary convolutional layers with depthwise separable convolutional layers to construct a novel optimal model. This optimal model can be used to abstract and learn meaningful and complex contextual features, and has a higher overall classification accuracy than peer models, as well as stronger robustness, thereby improving the accuracy and efficiency of remote sensing mapping in the field of urban climate. Attached Figure Description

[0038] Figure 1 This is a flowchart of a method according to an embodiment of the present invention;

[0039] Figure 2 This is a structural diagram of the attention mechanism module;

[0040] Figure 3 This is a structural diagram of a depth-separable convolutional layer;

[0041] Figure 4 This is the structure diagram of the final convolutional neural network;

[0042] Figure 5 LCZ classification map for the target region;

[0043] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0044] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0045] Reference Figure 1 This invention provides a method for classifying local climate zones based on multispectral data, comprising:

[0046] S1: Obtain the original Sentinel-2 image of the target region, preprocess the original Sentinel-2 image to obtain the image to be classified;

[0047] S2: Build an initial multi-layer convolutional neural network in the Keras framework of Tensorflow, which is based on deep learning;

[0048] S3: Add residual connections and attention mechanism modules to the initial multi-layer convolutional neural network to obtain the first-stage optimized convolutional neural network;

[0049] S4: Replace the ordinary convolutional layers in the first optimization stage convolutional neural network with depthwise separable convolutional layers to obtain the final convolutional neural network;

[0050] S5: Train the final convolutional neural network using a training dataset based on LCZ classification to obtain the optimal model;

[0051] S6: Input the image to be classified into the optimal model for classification to obtain the LCZ classification map of the target region.

[0052] In this embodiment, step S1 specifically includes:

[0053] S11: Crop the Sentinel-2 image to the same extent as the target region to obtain the original Sentinel-2 image of the target region;

[0054] Specifically, multispectral Sentinel-2 images of the study area can be downloaded from the websites of the U.S. Geological Survey or the European Space Agency. Taking the Sentinel-2 images of Wuhan and Hefei as examples, the Sentinel-2 image of Wuhan is dated April 9, 2022; the Sentinel-2 image of Hefei is dated November 9 and November 12, 2020.

[0055] S12: Perform atmospheric correction, registration, cropping, and band synthesis on the original Sentinel-2 image to obtain the processed Sentinel-2 image;

[0056] Specifically, atmospheric correction, registration, cropping, and band synthesis are performed on the original Sentinel-2 images in SNAP software or scripting languages. For scripting languages, the sen2cor plugin needs to be downloaded.

[0057] S13: Select 10 bands with spatial resolutions of 10 meters and 20 meters from the processed Sentinel-2 image, perform band synthesis, and uniformly resample to 10-meter spatial resolution to obtain the image to be classified.

[0058] Specifically, based on the characteristics of the training dataset So2SatLCZ42, 10 bands (B2, B3, B4, B5, B6, B7, B8, B8a, B11, B12) with spatial resolutions of 10 meters and 20 meters were selected for band synthesis and uniformly resampled to a spatial resolution of 10 meters.

[0059] In this embodiment, a deep learning environment is configured, and an initial multi-layer convolutional neural network is built in the Keras framework;

[0060] The initial multi-layer convolutional neural network includes: a variable-scale convolutional module, multiple feature extraction modules, a global average pooling layer, and a fully connected layer;

[0061] The variable-scale convolutional module includes: a 5*5*16 convolutional layer, a 3*3*32 convolutional layer, a 1*1*16 convolutional layer, and a fusion layer; the outputs of the 5*5*16 convolutional layer, the 3*3*32 convolutional layer, and the 1*1*16 convolutional layer are connected to the input of the fusion layer.

[0062] The feature extraction module includes: a first ordinary convolutional layer, a second ordinary convolutional layer, and a fusion layer; the output of the first ordinary convolutional layer is connected to the input of the second ordinary convolutional layer, and the output of the second ordinary convolutional layer is connected to the input of the fusion layer.

[0063] The variable-scale convolutional modules, feature extraction modules, global average pooling layers, and fully connected layers are sequentially connected to form the initial multi-layer convolutional neural network.

[0064] In this embodiment, step S3 specifically includes:

[0065] S31: In the initial multi-layer convolutional neural network, an attention mechanism module i is added between feature extraction module i and feature extraction module i+1. The attention mechanism module includes: channel attention mechanism module and spatial attention mechanism module; where i is the number of the feature extraction module.

[0066] S32: Establish a residual connection between the fusion layer of the variable-scale convolutional module and the fusion layer of the first feature extraction module, and establish a residual connection between the fusion layer of the attention mechanism module i and the feature extraction module i+1.

[0067] Specifically, the output calculation formula for residual connections is as follows:

[0068]

[0069] In the above formula, x l and x l+1 These are the input and output of layer L, respectively, W i Here are the parameters related to layer L, and Γ represents the residual learning. The activation function that affects the Γ path (the non-skipped portion);

[0070] The structure diagram of the attention mechanism module (CBAM) is as follows: Figure 2 As shown;

[0071]

[0072] In the above formula, M c(F) is the feature map output by the channel attention mechanism module in CBAM, σ is the sigmoid function, and W0 and W1 are the shared weights in the MLP, which share weights with both the input feature map and the ReLU activation function. For M c (F) The new feature map generated after average pooling. For M c (F) The new feature map generated after max pooling;

[0073]

[0074] In the above formula, M s (F) is the feature map output by the spatial attention mechanism module in CBAM, where σ is the sigmoid function, and f 3×3 For convolution operations, the kernel size is 3×3. For M s (F) The new feature map generated after average pooling. For M s (F) The new feature map generated after max pooling.

[0075] In this embodiment, step S4 specifically includes:

[0076] Replace the first and second ordinary convolutional layers in the feature extraction modules with odd numbers with depthwise separable convolutional layers;

[0077] Replace the second ordinary convolutional layer in the even-numbered feature extraction module with a depthwise separable convolutional layer;

[0078] Specifically, the structure diagram of a depth-separable convolutional layer is as follows: Figure 3 As shown, the second ordinary convolutional layer in each feature extraction module is replaced with a depthwise separable convolutional layer (DSE) first. The second depthwise separable convolutional layer needs to process more feature maps, thereby maximizing the parameter advantage of DSE.

[0079] The final structure diagram of the convolutional neural network is as follows: Figure 4 As shown;

[0080] When the number of input and output feature maps is the same, the formulas for calculating the number of parameters and flop in DSE are as follows:

[0081] B K × B K × M +M× N (4)

[0082] B K ×B K × M × B W × B H + M × N × B W × B H (5)

[0083] In the above formula, B K The value represents the size of the convolution kernel, M and N are the number of input feature maps and the number of output feature maps, respectively, and B... w and B H This indicates the width and height of the output feature map;

[0084] The ratio of the number of parameters and FLOPs of DSE to that of standard convolution is as follows:

[0085]

[0086]

[0087] Equations (6) and (7) show that DSE reduces the number of parameters and computation by splitting a normal convolution (by depth and points) into two convolutions.

[0088] The specific training process of the final convolutional neural network in step S5 of this embodiment is as follows:

[0089] Based on So2SatLCZ42, the largest dataset in the current LCZ classification field, and employing relevant training strategies, the final convolutional neural network was trained. Guided by the results of a series of ablation experiments, the optimal model LLNet was obtained. The training strategy employed Keras 2.3.1 framework based on TensorFlow backend and Python 3.6, using an Nvidia GTX 1060 6G GPU to accelerate computation and training. All convolutional layers were initialized using the he_Normal method, and the NesterovAdam optimizer was used to optimize the training process. The training batch size was 64, and the maximum iteration period was 100. The initial learning value was set to 0.1, decaying to half every 5 periods. Furthermore, to accelerate training and prevent overfitting, an "early stopping" mechanism was introduced. The detection metric for "early stopping" was the validation loss, with a detection interval of 30. If the validation loss did not decrease within 30 periods, training was stopped, and the optimal model with the highest validation accuracy was saved.

[0090] In this embodiment, step S6 specifically includes:

[0091] S61: Divide the image to be classified into 32*32 image blocks. If the edge blocks are smaller than 32*32, pad them with null values ​​to obtain the padded segmented image.

[0092] S62: Input the padded segmented image into the optimal model for prediction and classification. Sort the images according to the predicted probabilities of each LCZ category, and assign the category with the highest probability value to the corresponding LCZ classification image.

[0093] For details, please refer to Figure 5 , Figure 5 The images show the LCZ classification maps of the target areas obtained by this invention. (a) is the LCZ classification map of Wuhan City; (b) is the LCZ classification map of Hefei City. The overall classification accuracy of the two cities is 74.6% and 77.5%, respectively. The Kappa coefficients of the two cities are 0.72 and 0.75, respectively. It can be seen that the model of this invention not only has high accuracy, but also good robustness / transferability.

[0094] This invention provides a local climate zone classification system based on multispectral data, comprising:

[0095] The image acquisition unit is used to acquire the original Sentinel-2 image of the target region, preprocess the original Sentinel-2 image, and obtain the image to be classified.

[0096] The network building unit is used to build an initial multi-layer convolutional neural network in the Keras framework of Tensorflow, which is based on deep learning.

[0097] The first optimization unit is used to add residual connections and attention mechanism modules to the initial multi-layer convolutional neural network to obtain the first-stage convolutional neural network.

[0098] The second optimization unit is used to replace the ordinary convolutional layer in the convolutional neural network of the first optimization stage with a depthwise separable convolutional layer to obtain the final convolutional neural network.

[0099] The training unit is used to train the final convolutional neural network using a training dataset based on LCZ classification to obtain the optimal model.

[0100] The classification unit is used to input the image to be classified into the optimal model for classification, and obtain the LCZ classification map of the target region.

[0101] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0102] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. In the unit claims listing several devices, several of these devices may be embodied by the same hardware item. The use of the terms first, second, and third, etc., does not indicate any order and can be interpreted as identifiers.

[0103] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method for classifying local climate zones based on multispectral data, characterized in that, include: S1: Obtain the original Sentinel-2 image of the target region, preprocess the original Sentinel-2 image to obtain the image to be classified; S2: Build an initial multi-layer convolutional neural network in the Keras framework of Tensorflow, which is based on deep learning; S3: Add residual connections and attention mechanism modules to the initial multi-layer convolutional neural network to obtain the first-stage optimized convolutional neural network; S4: Replace the ordinary convolutional layers in the first optimization stage convolutional neural network with depthwise separable convolutional layers to obtain the final convolutional neural network; S5: Train the final convolutional neural network using a training dataset based on LCZ classification to obtain the optimal model; S6: Input the image to be classified into the optimal model for classification to obtain the LCZ classification map of the target region; Step S4 is as follows: Replace the first and second ordinary convolutional layers in the feature extraction modules with odd numbers with depthwise separable convolutional layers; Replace the second ordinary convolutional layer in the even-numbered feature extraction module with a depthwise separable convolutional layer.

2. The local climate zone classification method based on multispectral data according to claim 1, characterized in that, Step S1 is as follows: S11: Crop the Sentinel-2 image to the same extent as the target region to obtain the original Sentinel-2 image of the target region; S12: Perform atmospheric correction, registration, cropping, and band synthesis on the original Sentinel-2 image to obtain the processed Sentinel-2 image; S13: Select 10 bands with spatial resolutions of 10 meters and 20 meters from the processed Sentinel-2 image, perform band synthesis, and uniformly resample to 10-meter spatial resolution to obtain the image to be classified.

3. The local climate zone classification method based on multispectral data according to claim 1, characterized in that, The initial multilayer convolutional neural network includes: a variable-scale convolutional module, multiple feature extraction modules, a global average pooling layer, and a fully connected layer; The variable-scale convolutional module includes: a 5*5*16 convolutional layer, a 3*3*32 convolutional layer, a 1*1*16 convolutional layer, and a fusion layer; the outputs of the 5*5*16 convolutional layer, the 3*3*32 convolutional layer, and the 1*1*16 convolutional layer are connected to the input of the fusion layer. The feature extraction module includes: a first ordinary convolutional layer, a second ordinary convolutional layer, and a fusion layer; the output of the first ordinary convolutional layer is connected to the input of the second ordinary convolutional layer, and the output of the second ordinary convolutional layer is connected to the input of the fusion layer. The variable-scale convolutional modules, feature extraction modules, global average pooling layers, and fully connected layers are sequentially connected to form the initial multi-layer convolutional neural network.

4. The local climate zone classification method based on multispectral data according to claim 1, characterized in that, Step S3 is as follows: S31: In the initial multi-layer convolutional neural network, an attention mechanism module i is added between feature extraction module i and feature extraction module i+1. The attention mechanism module includes: channel attention mechanism module and spatial attention mechanism module; where i is the number of the feature extraction module. S32: Establish a residual connection between the fusion layer of the variable-scale convolutional module and the fusion layer of the first feature extraction module, and establish a residual connection between the fusion layer of the attention mechanism module i and the feature extraction module i+1.

5. The local climate zone classification method based on multispectral data according to claim 1, characterized in that, Step S6 is as follows: S61: Divide the image to be classified into 32*32 image blocks. If the edge blocks are smaller than 32*32, pad them with null values ​​to obtain the padded segmented image. S62: Input the padded segmented image into the optimal model for prediction and classification. Sort the images according to the predicted probabilities of each LCZ category, and assign the category with the highest probability value to the corresponding LCZ classification image.

6. A local climate zone classification system based on multispectral data, used to implement the method according to any one of claims 1 to 5, characterized in that, include: The image acquisition unit is used to acquire the original Sentinel-2 image of the target region, preprocess the original Sentinel-2 image, and obtain the image to be classified. The network building unit is used to build an initial multi-layer convolutional neural network in the Keras framework of Tensorflow, which is based on deep learning. The first optimization unit is used to add residual connections and attention mechanism modules to the initial multi-layer convolutional neural network to obtain the first-stage convolutional neural network. The second optimization unit is used to replace the ordinary convolutional layer in the convolutional neural network of the first optimization stage with a depthwise separable convolutional layer to obtain the final convolutional neural network. The training unit is used to train the final convolutional neural network using a training dataset based on LCZ classification to obtain the optimal model. The classification unit is used to input the image to be classified into the optimal model for classification, and obtain the LCZ classification map of the target region.

Images