Classification method for generating sub-mainland scale land coverage image by using coarse resolution time sequence image and CNN and semantic segmentation model

Abstract

The invention provides a classification method for generating a secondary mainland scale land coverage image by using a coarse resolution time sequence image, a CNN (Convolutional Neural Network) and a semantic segmentation model, and belongs to the technical field of satellite image classification. Constructing a reference image data set based on a 1D CNN model; establishing a land coverage category; converting image data in the batch processing model reference image data set into a raster format to generate a plurality of masking reference images; sorting marked areas and unmarked areas in the masking reference image; dividing the masked reference image and the main input image into preset sizes; repairing the masked input image to be the same as the corresponding mark mask in size, and sorting the masked input image; and training and optimizing the classifier model, and adjusting the learning rate of the classifier model. According to the invention, the land coverage type of the secondary mainland scale can be identified more accurately. Diversified data sets and training strategies are used, and the adaptability and generalization ability of the model in different scenes are improved.

Description

Technical Field

[0001] The invention belongs to the technical field of satellite image classification, and in particular relates to a classification method for generating sub-continental scale land cover images by using coarse-resolution time series images and CNN and semantic segmentation models. Background Art

[0002] The classification of land cover images at the sub-continental scale mainly involves the comprehensive application of remote sensing technology, image processing technology and machine learning / deep learning algorithms.

[0003] Remote sensing technology obtains images and data of the earth's surface through satellites, aircraft and other carriers, providing a rich data source for land cover classification. After obtaining remote sensing images, image processing technology is needed to preprocess and extract features. Machine learning and deep learning algorithms can automatically learn features from image data and build classification models.

[0004] Currently, DL techniques are generally more effective and robust than classical machine learning (ML) algorithms such as support vector machines (SVM) and random forests (RF) for research using high-resolution images. However, most studies applying deep learning techniques aim to classify high and medium resolution (<=30m) remote sensing images, including satellite, unmanned aerial vehicle (UAV) and / or aerial imagery, and hyperspectral data, to generate local-scale / small-area land cover maps. Nevertheless, the properties / applicability of the classifier are highly related to the properties of the input image, including its spatial resolution and the extent of the study area, which play an important role in controlling the inter- and intra-class heterogeneity of land cover types.

[0005] However, no study has applied deep learning techniques to the classification of coarse-resolution imagery (>100 m) for large-area / large-scale maps, especially by leveraging labeled (training and evaluation) datasets collected as shapefiles.

[0006] Therefore, in this study, we aim to evaluate the performance of the most commonly used DL techniques such as 1D CNN, 2-CNN, 3-CNN, and U-net semantic segmentation models in coarse-resolution (1 km), time-series image classification for sub-continental scale land cover mapping. In addition, we intend to compare the results with RF, which is one of the best classical ML techniques. To achieve our goal, we processed 9 months, coarse-resolution (1 km), 10-day synthetic images collected by the Fengyun-3C (FY-3C) Visible and Infrared Radiometer (VIRR). The labeled dataset was collected as shapefiles and divided into three datasets, namely training, validation, and testing datasets. Subsequently, the shapefiles were rasterized and preprocessed to meet the input shape format requirements of different models. After generating the reference dataset, we conducted several experiments using the training and validation samples to find the best model. Then, the generalization ability of the optimized model was tested on the test dataset and the performance of the model was evaluated. Summary of the invention

[0007] The present invention provides a classification method for generating sub-continental scale land cover images by using coarse-resolution time series images, CNN and semantic segmentation models. The method can more comprehensively capture feature information in the images, thereby improving the accuracy of land cover classification.

[0008] Methods include:

[0009] S101: configuring a classifier model based on a reference dataset of a pixel model, a reference dataset of a batch model, and a reference image dataset of a U-net model;

[0010] S102: construct a reference image dataset (X, y) based on a 1D CNN model;

[0011] Where: X represents the input image in the form of a 2D tensor, and y represents the corresponding label in the form of a 1D tensor / array; the reference dataset (X, y) is dimensionally transformed so that the dimension of X is converted to a 3D tensor and the y category is converted to a 2D tensor;

[0012] S103: in the batch model reference image dataset, establishing a land cover category corresponding to each reference image; and then converting the image data in the batch model reference image dataset into a raster format to generate a plurality of mask reference images;

[0013] S104: Slicing the masked reference image, sorting the marked areas and unmarked areas in the masked reference image; when performing a multi-classification task on a single input image, dividing the masked reference image and the main input image into preset sizes;

[0014] S105: Based on the reference image dataset of the U-net model, the masked input image is patched to the same size as its corresponding labeled mask and sorted to separate patches with image information from the masked areas;

[0015] S106: training and optimizing the classifier model using the training data set and the validation data set, and adjusting the learning rate of the classifier model using parameter adjustment and regularization methods;

[0016] S107: Quantitatively evaluate the classifier model based on confusion / error matrix, recall, and precision.

[0017] It should be further explained that step S102 further includes: generating three mask layers consisting of marked pixels based on rasterizing the three shapefiles;

[0018] If the three shapefiles are marked with zero, it means the area is unmarked or masked;

[0019] The unmarked regions are eliminated by y=y[y>0], removing all zero-valued regions and retaining the remaining pixels with values ​​between 1 and 8.

[0020] It should be further explained that there are 8 labeled categories in the reference image dataset;

[0021] The eight labeled categories include: bare sparse vegetation, built vegetation, cropland, forest, herbaceous wetland, herbaceous vegetation, shrubs, and water bodies.

[0022] It should be further explained that step S103 also includes: constructing a training reference image dataset, a verification reference image dataset and a test reference image dataset;

[0023] Split the 8 marker categories into 24 different shapefiles and generate 24 mask layers;

[0024] The mask layer consists of pixels with values ​​of 1 and 0, representing the marked and unmarked areas, respectively;

[0025] Using the mask layer to mask the main input image;

[0026] The SIIMG pixel spectral values ​​and spatial information corresponding to the mask layer pixels with a value of 1 are retained, and the remaining pixels are cleared and set to be masked.

[0027] It should be further explained that step S104 also includes:

[0028] Configure 2x 2 tile patches, 3x 3 tile patches, and 5x 5 tile patches;

[0029] Patches with information and patches without useful information were classified;

[0030] Label each area that contains classified information to match the original land cover class.

[0031] It should be further explained that a 6-bit floating point number is used to configure a unique label for each region.

[0032] It should be further explained that in S106, the depth of the classifier model is adjusted, the kernel size and number are selected, padding and batch normalization are applied, and the size and location of pooling, dropout, and l1 and l2 regularization are determined.

[0033] The present application also provides a system, which includes: a model configuration module, a data set construction module, an image data conversion module, an image sorting and classification module, a patch masking region division module, a training optimization module, and a quantitative evaluation module;

[0034] Model configuration module, which configures the classifier model based on the reference dataset of the pixel model, the reference dataset of the batch model, and the reference image dataset of the U-net model;

[0035] The dataset construction module is used to construct a reference image dataset (X, y) based on the 1D CNN model;

[0036] Where: X represents the input image in the form of a 2D tensor, and y represents the corresponding label in the form of a 1D tensor / array; the reference dataset (X, y) is dimensionally transformed so that the dimension of X is converted to a 3D tensor and the y category is converted to a 2D tensor;

[0037] An image data conversion module is used to establish a land cover category corresponding to each reference image in a batch model reference image data set; and then convert the image data in the batch model reference image data set into a raster format to generate a plurality of mask reference images;

[0038] An image sorting and classification module is used to slice the masked reference image and sort the marked areas and unmarked areas in the masked reference image; and when performing a multi-classification task on a single input image, the masked reference image and the main input image are divided into preset sizes;

[0039] The patch mask region segmentation module patches the masked input image to the same size as its corresponding labeled mask based on the reference image dataset of the U-net model and sorts it to separate patches with image information from the masked area;

[0040] A training optimization module is used to train and optimize the classifier model using the training data set and the validation data set, and to adjust the learning rate of the classifier model using parameter adjustment and regularization methods;

[0041] The quantitative evaluation module quantitatively evaluates the classifier model based on confusion / error matrix, recall rate, and precision.

[0042] According to another embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the method for generating a sub-continental scale land cover image classification method using coarse-resolution time series images and a CNN and semantic segmentation model are implemented.

[0043] According to another embodiment of the present application, a storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for generating a sub-continental scale land cover image classification method using coarse-resolution time series images and CNN and semantic segmentation models are implemented.

[0044] It can be seen from the above technical solutions that the present invention has the following advantages:

[0045] The application provides a method for classifying sub-continental scale land cover images using coarse-resolution time series images and CNN and semantic segmentation models. By combining multiple models and refined data processing steps, it is possible to more accurately identify sub-continental scale land cover types. The use of diverse data sets and training strategies improves the adaptability and generalization ability of the model in different scenarios.

[0046] The combined use of batch processing models and U-net models, as well as the preprocessing and slicing of image data, significantly improves the processing efficiency of image classification. In addition, by generating mask images and patching missing areas, it supports refined analysis of land cover types. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0048] Figure 1 A flowchart of the method for generating sub-continental scale land cover image classification using coarse resolution time series images and CNN and semantic segmentation models;

[0049] Figure 2 This is a 1D CNN synthetic image map;

[0050] Figure 3 Figure 2: 2D CNN synthesized image using 2x2 input patches.

[0051] Figure 4 Synthesize an image using 2D CNN with 5x5 patches.

[0052] Figure 5 Synthesize an image using a 3D CNN with a 2x2 input size.

[0053] Figure 6 Synthesize image map using 3x 3 input chip for U-net;

[0054] Figure 7 Synthesized image for U-net using 5x 5 input chip;

[0055] Figure 8 Schematic diagram of the hardware structure of the electronic device. DETAILED DESCRIPTION

[0056] The following will describe in detail the CNN and semantic segmentation model involved in the present application in the sub-continental scale land cover mapping coarse resolution satellite image classification method, for the purpose of illustration rather than limitation, specific details such as specific system structures, technologies, etc. are proposed to thoroughly understand the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details.

[0057] It should be understood that when used in the present specification, the term "includes" indicates the presence of the described features, integral bodies, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integral bodies, steps, operations, elements, components and / or their collections. The terms "include", "comprise", "have" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0058] It should be understood that the "one or more" mentioned in this application refers to one, two or more than two, and the "multiple" mentioned in this application refers to two or more. In the description of this application, unless otherwise specified, " / " means or, for example, A / B can mean A or B. The "and / or" in this article is just a way to describe the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0059] The phrases such as "one embodiment" or "some embodiments" described in the present application mean that the specific features, structures or characteristics described in the embodiment are included in one or more embodiments of the present application. Therefore, the phrases such as "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments" etc. that appear in different places in the present application do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways.

[0060] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] See also Figure 1 The flowchart of a method for generating a sub-continental scale land cover image classification method using a coarse-resolution time series image and a CNN and a semantic segmentation model in a specific embodiment is shown, and the method includes:

[0062] S101: configuring a classifier model based on a reference dataset of a pixel model, a reference dataset of a batch model, and a reference image dataset of a U-net model.

[0063] In some embodiments, pixel-based classification methods are used as a benchmark, and models that can process batch data, such as convolutional neural networks (CNNs) in deep learning, are used to capture the spatial features of images. The U-net network is used in combination with a deep learning model for image segmentation to perform more sophisticated image analysis and processing.

[0064] S102: Construct a reference image dataset (X, y) based on a 1D CNN model.

[0065] It should be noted that the input image (X) and the corresponding label (y) are converted into a format suitable for 1D CNN processing. This mainly involves the dimension transformation of the data to ensure the consistency of the model input.

[0066] Where: X represents the input image in the form of a 2D tensor, y represents the corresponding label in the 1D tensor / array format; the reference dataset (X, y) is dimensionally transformed so that the dimension of X is converted to a 3D tensor and y is classified and converted to a 2D tensor.

[0067] S103: in the batch model reference image dataset, establishing a land cover category corresponding to each reference image; and then converting the image data in the batch model reference image dataset into a raster format to generate a plurality of mask reference images.

[0068] In some embodiments, a land cover class is established for each reference image and converted into a raster format to generate a mask image, which helps to distinguish different types of land cover and facilitates subsequent processing.

[0069] S104: Slicing the masked reference image, sorting the marked areas and unmarked areas in the masked reference image; when performing a multi-classification task on a single input image, dividing the masked reference image and the main input image into preset sizes.

[0070] This embodiment performs slicing processing on the masked image so as to analyze each region more finely. At the same time, the marked and unmarked regions are sorted to optimize the subsequent classification process.

[0071] S105: Based on the reference image dataset of the U-net model, the masked input image is patched to the same size as its corresponding labeled mask and sorted to separate patches with image information from the masked areas.

[0072] This embodiment uses a U-net model to inpaint the masked area to restore the missing information in the image. The inpainted image is aligned with the original marked mask and sorted to distinguish the inpainted area from the original image area.

[0073] S106: Use the training data set and the validation data set to train and optimize the classifier model, and use parameter adjustment and regularization methods to adjust the learning rate of the classifier model.

[0074] This embodiment uses a training data set and a validation data set to train the classifier model, and optimizes the model performance through parameter adjustment and regularization methods. This may include adjusting hyperparameters such as learning rate and batch size, and using regularization techniques to prevent overfitting.

[0075] S107: Quantitatively evaluate the classifier model based on confusion / error matrix, recall, and precision.

[0076] This embodiment performs quantitative evaluation on the classifier model based on evaluation indicators such as confusion matrix, recall rate and precision, which can fully reflect the classification performance of the model.

[0077] Based on the above method of using coarse-resolution time series images and CNN and semantic segmentation models to generate sub-continental scale land cover image classification, by combining multiple models and refined data processing steps, it is possible to more accurately identify sub-continental scale land cover types. The use of diverse data sets and training strategies improves the adaptability and generalization ability of the model in different scenarios.

[0078] The combined use of batch processing models and U-net models, as well as the preprocessing and slicing of image data, significantly improves the processing efficiency of image classification. In addition, by generating mask images and patching missing areas, it supports refined analysis of land cover types.

[0079] Furthermore, as a refinement and extension of the specific implementation methods of the above-mentioned embodiment, in order to fully illustrate the specific implementation process in this embodiment, a method for generating a sub-continental scale land cover image classification method using a coarse-resolution time series image and a CNN and a semantic segmentation model is provided, which specifically includes the following specific methods.

[0080] In some embodiments, since different models require different input and reference data shapes, this embodiment uses three methods to generate reference data sets for pixel-based models (1D CNN), batch-based models (2D CNN and 3D CNN), and U-net models.

[0081] In order to prepare the reference dataset for the pixel-based model (1D CNN), the reference dataset can be rasterized to obtain three shapefiles to generate three mask layers consisting of labeled pixels. If the three mask layers are marked with zero, it means an unlabeled or masked area. Otherwise, it means a labeled or masked area. Shapefile is a non-topological simple format for storing the geometric location and attribute information of geographic features. Geographic features in shapefiles can be represented as points, lines, or surfaces (areas).

[0082] Subsequently, unmarked pixels are eliminated by y=y[y>0], removing all zero-valued pixels and retaining the remaining pixels with values ​​between 1 and 8.

[0083] This embodiment represents 8 labeled categories, including: bare sparse vegetation, built vegetation, farmland, forest, herbaceous wetland, herbaceous vegetation, shrubs and water bodies.

[0084] This embodiment uses the filtered mask layer (1D tensor of labels) to slice the corresponding input image (X). This process produces an output dataset (X, y).

[0085] Where: X represents the input image in the form of a 2D tensor and y represents the corresponding label in the format of a 1D tensor / array. Then, the array is reshaped to meet the 1D CNN model input shape requirement, i.e. X is converted to a 3D tensor by expanding its dimensions and y is converted to a 2D tensor by converting it to a categorical one.

[0086] In some embodiments, in the CNN model (2D CNN and 3D CNN), the association between the reference image and its label is one-to-one, that is, each reference image corresponds to a land cover type / category. A training reference image dataset, a verification reference image dataset, and a test reference image dataset can be configured here.

[0087] For the training reference image dataset, the verification reference image dataset, and the test reference image dataset, each dataset contains 8 classes. In this embodiment, one reference image is created for each class, for a total of 24 reference images.

[0088] This example splits the three created shapefile datasets (each with 8 classes) into 24 different shapefiles, each with only one class. Subsequently, all shapefiles are converted to raster format to generate 24 mask layers consisting of pixels with values ​​of 1 and 0, representing marked and unmarked areas, respectively.

[0089] This embodiment uses the mask layer to mask the main input image (SIIMG). Here, the spectral value and spatial information of the SIIMG pixel corresponding to the mask layer pixel with a value of 1 are retained, and the remaining pixels are cleared and masked.

[0090] In this embodiment, a total of 24 masked reference images can be generated through the training reference image dataset, the verification reference image dataset, and the test reference image dataset, each dataset containing 8 different masked reference images representing 8 categories. Each masked reference image is cut into blocks / patches, classified with the image information in the fully masked area / pixel, and re-labeled back to the class it represents.

[0091] In some embodiments, a preprocessing stage of the sliced ​​mask reference image (MRImg) is also involved, and the main steps include: slicing the masked reference image (MRImg) and being able to sort the marked areas in MRImg, that is, the unmasked and unlabeled areas / pixels.

[0092] When using a patch-based model for multi-class classification tasks with a single input image, the masked reference image and the main input image (SIIMG) can be divided into appropriate patch / chip sizes.

[0093] This embodiment, when processing a coarse-resolution input image, can include various land cover types in one chip because dividing the reference dataset and the input image into large patches increases the pixel heterogeneity within the patch. This may cause minor classes represented by only a few pixels and covering a small area to be ignored or not identified as separate independent classes. Instead, they will be homogenized into the majority class and classified as part of the dominant class within the patch because the patch-based model tends to assign the chip to a single class.

[0094] It should be noted that segmenting the coarse-resolution input image into large-sized patches may lead to misclassification / errors in the prediction process, especially for small classes.

[0095] For example, in a heterogeneous area, a 5 x 5 chip (covering 25 square kilometers) is likely to contain two or more different classes at different proportions. In this case, the model will homogenize the pixels and predict / classify them as the dominant class in the chip, resulting in misclassification.

[0096] In view of this, this embodiment configures three groups of patches of size 2x 2, 3x 3 and 5x 5 to select the best patch size and analyze the impact of patch size on performance and output map or classification image. After generating patches, patches with information are classified from patches without useful information (shielded chips).

[0097] Finally, each chip containing valuable information is relabeled to match the original land cover type (class), and this is the name of the ROI / shapefile.

[0098] During the re-labeling process, in order to give each chip a unique record / name, this embodiment uses a 6-digit floating point number (e.g., 1.090000), where the number before the decimal (1) indicates the category it represents; and the number after the decimal (.090000) indicates the uniqueness of the chip.

[0099] The decimal number of this embodiment is set to be updated by -1 (-1) so that the chips are arranged continuously. For example, for farmland (category 3), the classified chips are re-labeled as farmland 3.090000, farmland 3.089999.npy, farmland 3.089998.npy, etc.

[0100] Similarly, for the combination (class 2), the labels are generated as built-up.2.090000, built-up-2.089999.npi, built-up.2.089998.npy, etc. Finally, when the chip / reference image is fed to the model, the number before the decimal is extracted and used as the label (y) and the chip is read as the input image (x).

[0101] In some specific embodiments, the segmentation model expects the labeled dataset as separate rasters, unlike patch-based CNN models where the labels are attached to the chip.

[0102] It should be noted that the segmentation model requires two separate reference rasters to train the model: the image / chip and its corresponding labeled mask. The three sets of shapefiles are rasterized to create a labeled mask containing class label (1 to 8) pixels, and if the class label pixel is within the ROI / shapefile, the unlabeled / masked area is configured to zero. Subsequently, the labeled mask is divided into blocks and classified to distinguish between patches with labels and patches without labels.

[0103] In order to prepare the input reference image / patch, this embodiment uses three mask layers to mask the main input image (SIIMG), which are composed of pixels with values ​​of 1 and 0 representing the label and unlabeled areas respectively. Here, three masked input images are generated. In the masked input image, the pixels aligned with the mask layer pixels with a value of 1 retain the spectral value and spatial information, while the remaining pixels are set to zero (0), effectively masking. Subsequently, the masked input image is patched to the same size as the corresponding labeled mask and sorted to separate patches with image information from completely masked areas / pixels. It can be seen that this embodiment generates two sets of training data sets of different sizes, 3x 3 and 5x 5, to analyze the impact of patch size and compare the results with the CNN model.

[0104] In the deep learning model training of this embodiment, parameter adjustment and regularization can be performed to prevent / minimize overfitting and enhance generalization ability. In order to optimize the three CNN and U-net models and obtain the best architecture for each model, this embodiment can use training and validation data sets to conduct multiple experiments and adopt common parameter adjustment and regularization methods.

[0105] Parameter adjustment and regularization methods can include adjusting the depth of the model, selecting the kernel size and number, applying padding and batch normalization, and determining the size and location of pooling, dropout, and l1 and l2 regularization. This embodiment can also use early stopping based on different batch sizes and adjust the learning rate of the model optimizer. Optionally, an Adam optimizer deep learning model is used. Finally, the best architecture is selected for each model based on the overall accuracy of the validation data set and the training validation accuracy map.

[0106] Once the model is optimized, it is evaluated on the test dataset to assess its ability to generalize to unseen data and achieve classification / prediction purposes. The model is then quantitatively evaluated using accuracy measures such as confusion / error matrix, recall, precision, and OA.

[0107] It should be noted that OA is a technology for quantifying the accuracy of various classifiers (including deep learning models).

[0108] In some embodiments, it also involves reconstructing the predicted chips / pixels to generate a complete land cover map of the study area, followed by modal filtering to remove noise.

[0109] In pixel-based classification (1D CNN), the output / classified pixels are reconstructed into the original input image; for clarification, here the prediction vector is reshaped into a 2D shape. Patch-based CNNs (2D CNN and 3D CNN) and segmentation (U-net) models operate differently and produce different types of outputs, namely, the CNN assigns a class to the chip. Therefore, two different methods are used to reconstruct the predicted chips from the two models. To reconstruct the predicted code chips from the CNN model, you can initialize an output image of the same size as the original image (zeros), then iterate over the predicted code chips to determine the row and column indices of each block, and fill the output image with the block and the predicted class.

[0110] For the U model, to reconstruct the prediction chip, we can reshape the prediction chip in (row, column) format, create an empty output array with the same shape as the original image, then loop over the chip and fill the output array with the original labels obtained by mapping the predicted labels to the original labels.

[0111] This embodiment is based on three CNN models (1D CNN, 2D CNN and 3D CNN) and U-net segmentation for time series coarse-resolution image classification of large-area land cover mapping, and generates a reference dataset involved in each model.

[0112] There was significant variation in performance due to the type of model and / or size of chip input, but the best models for each DL classifier (CNN and U-net) achieved high accuracy (>=85%).

[0113] Among the DL models, 1D CNN achieves the highest accuracy on both validation (OA=0.89, k=0.86) and test datasets (OA=0.87, k=0.84), outperforming the best results of 2D CNN and U-net models on the same reference datasets by 5-4% and 3-2%, respectively.

[0114] The best 2D CNN model produced OA=0.84 and k=0.81 with OA=0.

[0115] When manipulating the smallest chip size (2 x 2), k = 0.82 on the test dataset.

[0116] In this example, the accuracy (OA and k) of 3D CNN is 1% higher than that of 2D CNN on the same validation dataset, but the OA and k of the test samples are the same. On the other hand, when processing 3x 3 chips, the best U-net model achieves similar accuracy to 2D CNN, which means that if the correct size (i.e., smaller chip dataset) is processed during coarse resolution image classification, the patch-based CNN model and the segmentation model can produce comparable overall accuracy.

[0117] This embodiment can handle larger patches that negatively impact the accuracy of the patch-based CNN and U-net models. In 2D CNN, when the chip size increases from 2 to 5, the accuracy drops by 9%, and the model produces the smallest OA (0.76) and k (0.71). In U-net, increasing the patch size from 3 to 5 reduces the accuracy of the validation dataset by 3% and the accuracy of the test data by 1%, which shows that increasing the patch size has less impact on the U-net model than the patch-based CNN model.

[0118] The DL techniques involved in this implementation can be effectively used for coarse-resolution image classification. Choosing the right model and / or chip size is the main way to achieve the best results, as the performance of the model is highly affected by the type of model adopted and / or the patch size processed, especially for patch-based models. Among the DL techniques, the pixel-based 1D CNN method outperforms all patch-based models (2D, 3D) and U-net segmentation techniques; therefore, 1D CNN is a more viable DL technique compared to patch-based and U-net DL models. Regarding the performance of patch-based (2D CNN and 3D CNN) and U-net segmentation models, these models show fairly high accuracy when using reference datasets with small patch sizes; while the performance of patch-based models is adversely affected as the tile / chip size increases. Using larger patches in coarse-resolution image classification can lead to partial or complete loss of some smaller classes, resulting in classification errors.

[0119] In one embodiment of the present invention, a possible embodiment is given below to illustrate its specific implementation scheme in a non-limiting manner.

[0120] Coarse resolution (1 km) FY-3C VIRR 10-day composite images received from the National Satellite Meteorological Center (NSMC) of China. These images include 297 band indicators consisting of 11 different bands, including 4 visible light, 3 infrared, 3 thermal infrared and Normalized Difference Vegetation Index (NDVI). In order to operate all 297 channels as one image, these images were stacked in ENVI 5.3 software.

[0121] Specifically, if Figures 2 to 7 Shown are the confusion matrices generated using the test datasets of various CNN, U-net, and ML models and the corresponding final land cover products. Figure 2 This is a 1D CNN synthesized image. Figure 3 Figure 2: 2D CNN synthesized image using 2x2 input patches. Figure 4 Synthesize an image using 2D CNN with 5x5 patches. Figure 5 Synthesize an image using 3DCNN with 2x2 input size;

[0122] Figure 6 Synthesize image map using 3x 3 input chip for U-net; Figure 7 Synthesized image for U-net using a 5x 5 input chip.

[0123] Labeled datasets were collected as regions of interest (ROIs) / polygons representing eight land cover classes (bare sparse vegetation, built-up area, cropland, forest, herbaceous wetland, herbaceous vegetation, shrubs, and water bodies) across the study area.

[0124] In ENVI software, ROIs (polygons representing specific classes) were annotated by using three techniques in parallel, namely Landsat 8 image interpretation, cross-checking and consulting existing maps, and Google Earth. After annotation, the ROIs were saved as shapefiles. The shapefiles were then systematically divided into three datasets for training, validation, and testing, with a ratio of 68%, 20%, and 12%, respectively, where each dataset contained all eight land cover classes. However, since shapefiles cannot be used directly as reference data, they were further manipulated to meet the reference dataset format / shape requirements of each model.

[0125] The following is an embodiment of a sub-continental scale land cover image classification system provided by an embodiment of the present disclosure. The system and the method for generating a sub-continental scale land cover image classification method using coarse-resolution time series images, CNN, and a semantic segmentation model in the above-mentioned embodiments belong to the same inventive concept. For details not described in detail in the embodiment of the sub-continental scale land cover image classification system, reference may be made to the above-mentioned embodiment of the method for generating a sub-continental scale land cover image classification method using coarse-resolution time series images, CNN, and a semantic segmentation model.

[0126] A mobile terminal implementing various embodiments of the present invention will now be described with reference to the accompanying drawings. In the subsequent description, suffixes such as "module", "component" or "unit" used to represent elements are used only to facilitate the description of the embodiments of the present invention and have no specific meaning in themselves. Therefore, "module" and "component" can be used interchangeably.

[0127] The system includes: a model configuration module, a data set construction module, an image data conversion module, an image sorting and classification module, a patch masking region division module, a training optimization module and a quantitative evaluation module.

[0128] The model configuration module configures the classifier model based on the reference dataset of the pixel model, the reference dataset of the batch model, and the reference image dataset of the U-net model.

[0129] The dataset construction module is used to construct a reference image dataset (X, y) based on the 1D CNN model.

[0130] Where: X represents the input image in the form of a 2D tensor, y represents the corresponding label in the 1D tensor / array format; the reference dataset (X, y) is dimensionally transformed so that the dimension of X is converted to a 3D tensor and y is classified and converted to a 2D tensor.

[0131] The image data conversion module is used to establish the land cover category corresponding to each reference image in the batch model reference image data set; and then convert the image data in the batch model reference image data set into a raster format to generate multiple mask reference images.

[0132] The image sorting and classification module is used to slice the masked reference image and sort the marked areas and unmarked areas in the masked reference image; when performing multi-classification tasks on a single input image, the masked reference image and the main input image are divided into preset sizes.

[0133] The patch mask region segmentation module, based on the reference image dataset of the U-net model, patches the masked input image to the same size as its corresponding labeled mask and sorts it to separate patches with image information from the masked regions.

[0134] The training optimization module is used to train and optimize the classifier model using the training data set and the validation data set, and to adjust the learning rate of the classifier model using parameter adjustment and regularization methods.

[0135] The quantitative evaluation module quantitatively evaluates the classifier model based on confusion / error matrix, recall rate, and precision.

[0136] As for the classification method of sub-continental scale land cover image generated by using coarse resolution time series image and CNN and semantic segmentation model provided in this embodiment, it can be applied to specific electronic devices for implementation. Figure 8 A schematic diagram of the hardware structure of an electronic device implementing various embodiments of the present invention.

[0137] The electronic device 500 includes but is not limited to components such as a processor 501 , a network module 502 , an audio output unit 503 , an input unit 504 , a display unit 506 , a user input unit 507 , an interface unit 508 , a memory 509 and a power module 511 .

[0138] Those skilled in the art will appreciate that the electronic device structure involved in the embodiments of the present invention does not constitute a limitation on the electronic device, and the electronic device may include more or fewer components than shown, or combine certain components, or arrange different components. In the embodiments of the present invention, the electronic device includes but is not limited to a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of the present application described and / or required herein.

[0139] In the embodiment of the present application, the processor 501 can be implemented by using at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a processor, a controller, a microcontroller, a microprocessor, and an electronic unit designed to perform the functions described herein. In some cases, such an implementation can be implemented in a controller. For software implementation, implementations such as processes or functions can be implemented with separate software modules that allow execution of at least one function or operation. The software code can be implemented by a software application (or program) written in any appropriate programming language, and the software code can be stored in a memory and executed by a controller.

[0140] The processor 501 provides the user with wireless Internet access through the network module 502, thereby realizing the transmission and communication of data information in this application.

[0141] The audio output unit 503 may provide audio output related to a specific function executed by the processor 501. The audio output unit 503 includes a speaker, a buzzer, a receiver, and the like.

[0142] The input unit 504 is used to receive audio or video signals. The input unit 504 may include a graphics processing unit (GPU) and a microphone, and the graphics processor processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The processed image frame can be displayed on the display unit 506. The image frame processed by the graphics processor can be stored in the memory 509 (or other storage medium) or sent through the network module 502. The microphone can receive sound and can process such sound into audio data.

[0143] The display unit 506 is used to display information input by the user or information provided to the user. The display unit 506 may include a display panel, which may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc.

[0144] The user input unit 507 may be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the electronic device. The interface unit 508 is an interface for connecting an external device to the electronic device 500. The memory 509 may be used to store software programs and various data. The memory 509 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0145] For the purpose of the example learning, the electronic device can be a laptop equipped with a 2.7GHz processor, 32GB RAM, and an NVIDIA GeForceRTX 3050Ti laptop GPU. The GPU has 4GB dedicated memory and 16GB shared memory. Python is used for data preprocessing, model training and evaluation, classification, and post-processing. Python libraries / modules such as osgeo, gdal, ogr, numpy, sklearn, TensorFlow, keras, pickle, matplotlib, geolands, etc. are used to perform various tasks. In addition, ENVI 5.3 and Arcmap 10.3 are used for pre-processing and post-processing operations, including training data collection and ROI / shapefile operations, image stacking, and output visualization.

[0146] The electronic device of this embodiment is a unit and algorithm step of each example described in the embodiment of the method for classifying coarse-resolution satellite images for subcontinental scale land cover mapping in combination with CNN and semantic segmentation model, which can be implemented by electronic hardware, computer software or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0147] In the storage medium provided in the present application, a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary implementations of the present disclosure described in the above "Exemplary Method" section of the present specification.

[0148] The storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0149] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating a sub-continental scale land cover image classification using coarse resolution time series images and CNN and semantic segmentation models, characterized in that: Methods include: S101: configuring a classifier model based on a reference dataset of a pixel model, a reference dataset of a batch model, and a reference image dataset of a U-net model; S102: construct a reference image dataset (X, y) based on a 1D CNN model; Where: X represents the input image in the form of a 2D tensor, and y represents the corresponding label in the form of a 1D tensor / array; the reference dataset (X, y) is dimensionally transformed so that the dimension of X is converted to a 3D tensor and the y category is converted to a 2D tensor; S103: in the batch model reference image dataset, establishing a land cover category corresponding to each reference image; and then converting the image data in the batch model reference image dataset into a raster format to generate a plurality of mask reference images; S104: Slicing the masked reference image, sorting the marked areas and unmarked areas in the masked reference image; when performing a multi-classification task on a single input image, dividing the masked reference image and the main input image into preset sizes; S105: Based on the reference image dataset of the U-net model, the masked input image is patched to the same size as its corresponding labeled mask and sorted to separate patches with image information from the masked areas; S106: training and optimizing the classifier model using the training data set and the validation data set, and adjusting the learning rate of the classifier model using parameter adjustment and regularization methods; S107: Quantitatively evaluate the classifier model based on confusion / error matrix, recall, and precision.

2. The method for generating sub-continental scale land cover image classification using coarse resolution time series images and CNN and semantic segmentation models according to claim 1, characterized in that: Step S102 further includes: generating three mask layers consisting of marked pixels based on rasterizing the three shapefiles; If the three shapefiles are marked with zero, it means the area is unmarked or masked; The unmarked regions are eliminated by y=y[y>0], removing all zero-valued regions and retaining the remaining pixels with values ​​between 1 and 8.

3. The method for generating sub-continental scale land cover image classification using coarse resolution time series images and CNN and semantic segmentation models according to claim 2, characterized in that: The reference image dataset has 8 labeled categories; The eight labeled categories include: bare sparse vegetation, built vegetation, cropland, forest, herbaceous wetland, herbaceous vegetation, shrubs, and water bodies.

4. The method for generating sub-continental scale land cover image classification using coarse resolution time series images and CNN and semantic segmentation models according to claim 3, characterized in that: Step S103 also includes: constructing a training reference image dataset, a verification reference image dataset, and a test reference image dataset; Split the 8 marker categories into 24 different shapefiles and generate 24 mask layers; The mask layer consists of pixels with values ​​of 1 and 0, representing the marked and unmarked areas, respectively; Using the mask layer to mask the main input image; The SIIMG pixel spectral values ​​and spatial information corresponding to the mask layer pixels with a value of 1 are retained, and the remaining pixels are cleared and set to be masked.

5. The method for generating sub-continental scale land cover image classification using coarse resolution time series images and CNN and semantic segmentation models according to claim 3, characterized in that: Step S104 also includes: Configure 2x 2 tile patches, 3x 3 tile patches, and 5x 5 tile patches; Patches with information and patches without useful information were classified; Label each area that contains classified information to match the original land cover class.

6. The method for generating sub-continental scale land cover image classification using coarse resolution time series images and CNN and semantic segmentation models according to claim 5, characterized in that: Use a 6-bit floating point number to assign a unique label to each region.

7. The method for generating sub-continental scale land cover image classification using coarse resolution time series images and CNN and semantic segmentation models according to claim 1 or 2, characterized in that: In S106, the depth of the classifier model is adjusted, the kernel size and number are selected, padding and batch normalization are applied, and the size and location of pooling, dropout, and l1 and l2 regularization are determined.

8. A system, characterized in that: The system is used to implement the method for generating a sub-continental scale land cover image classification method using coarse resolution time series images and CNN and semantic segmentation models as described in any one of claims 1 to 7; The system includes: model configuration module, data set construction module, image data conversion module, image sorting and classification module, patch masking region division module, training optimization module and quantitative evaluation module; Model configuration module, which configures the classifier model based on the reference dataset of the pixel model, the reference dataset of the batch model, and the reference image dataset of the U-net model; The dataset construction module is used to construct a reference image dataset (X, y) based on the 1D CNN model; Where: X represents the input image in the form of a 2D tensor, and y represents the corresponding label in the form of a 1D tensor / array; the reference dataset (X, y) is dimensionally transformed so that the dimension of X is converted to a 3D tensor and the y category is converted to a 2D tensor; An image data conversion module is used to establish a land cover category corresponding to each reference image in a batch model reference image data set; and then convert the image data in the batch model reference image data set into a raster format to generate a plurality of mask reference images; An image sorting and classification module is used to slice the masked reference image and sort the marked areas and unmarked areas in the masked reference image; and when performing a multi-classification task on a single input image, the masked reference image and the main input image are divided into preset sizes; The patch mask region segmentation module patches the masked input image to the same size as its corresponding labeled mask based on the reference image dataset of the U-net model and sorts it to separate patches with image information from the masked area; A training optimization module is used to train and optimize the classifier model using the training data set and the validation data set, and to adjust the learning rate of the classifier model using parameter adjustment and regularization methods; The quantitative evaluation module quantitatively evaluates the classifier model based on confusion / error matrix, recall rate, and precision.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of generating a sub-continental scale land cover image classification method using coarse-resolution time series images and CNN and semantic segmentation models are implemented as described in any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for generating a sub-continental scale land cover image classification using coarse-resolution time series images and CNN and semantic segmentation models are implemented as claimed in any one of claims 1 to 7.

Images