Sentinel-2 dataset generation method and device based on small samples and deep learning model

Abstract

The present invention discloses a method and device for generating a Sentinel‑2 dataset based on small samples and a deep learning model. The method comprises the following steps: first, a support vector machine classifier generates a high-precision classification result by manually selecting a small sample. Then, the Sentinel‑2 image is divided into multiple image blocks according to the category of the classification result. Then, a DenseNet model that can extract the depth features of the sample is used to obtain the depth features of each image block. Finally, the confidence of each image block is determined based on these features, and the image blocks with high confidence are retained to generate a dataset. Experimental results show that after multiple iterations, when the error rate is reasonable, the incorrectly labeled image blocks in the dataset can be eliminated, and the number of image blocks in each type of land object can reach a stable state. Comparative experiments also found that this method does not delete the incorrectly labeled image blocks, and the deeper the fully connected layers and the better the performance of the deep learning model.

Description

Technical Field

[0001] The present invention relates to the technical field of dataset generation, and in particular to a method and device for generating a Sentinel-2 dataset based on small samples and a deep learning model. Background Art

[0002] In recent decades, deep learning methods have been widely used in remote sensing image classification due to their high classification accuracy. Convolutional neural networks (CNNs) are one of the most popular deep learning models. CNNs can identify intrinsic image features through a complex combination of numerous convolutions, nonlinear transformations, and other techniques, constructing powerful semantic features and accurate classifiers from the images. Unlike shallow classifiers, CNN models have a very large number of parameters; for example, VGG16 has 133 million parameters. This feature allows CNN models to be trained with more samples than shallow models. Current literature indicates that training a CNN model requires a training set containing at least tens of thousands of labeled images, especially when training from scratch. Too few training samples can lead to overfitting and poor model generalization. However, labeling large numbers of training samples is a very expensive process, requiring significant human, material, and financial resources.

[0003] In the prior art, two approaches are commonly used to obtain datasets for deep learning models. The first is data augmentation, which is well-suited for small-sample image classification problems. This method works by applying various transformations to the data (for example, rotating each image 30° to the left to double the number of training samples) to enhance the original training set. While these transformations can increase the number of training samples, most image features, such as spectral characteristics and the Normalized Vegetation Difference Index, remain unchanged. Since the dataset contains a large amount of similar data, this method cannot truly produce a large dataset. The second approach is manual labeling. For training deep learning models, many labeled datasets are available. The PatternNet, NWPU-resisc45, and WHU-RS datasets are primarily manually labeled. Manually labeled datasets have high accuracy, but the workload required to create them is significant. Although the Aerial Image Dataset (AID) and ORNL datasets are automatically annotated, they directly use the classification results without automatically checking the accuracy of the annotations.

[0004] It can be seen from this that the methods in the prior art have technical problems of low dataset generation efficiency or low dataset quality. Summary of the Invention

[0005] The present invention provides a method and device for generating a Sentinel-2 dataset based on a small sample and a deep learning model, which is used to solve or at least partially solve the technical problems of low dataset generation efficiency or low dataset quality in the prior art.

[0006] In order to solve the above technical problems, the first aspect of the present invention provides a method for generating a Sentinel-2 dataset based on a small sample and a deep learning model, which is characterized by comprising:

[0007] S1: Extract the features required for classification from the original Sentinel-2 image;

[0008] S2: Manually select a small sample from the original Sentinel-2 image;

[0009] S3: Input the features extracted in step S1 and the small sample selected in step S2 into a pre-built support vector machine classifier, and use the support vector machine classifier to classify the small sample;

[0010] S4: Use a sliding window to scan the classification results, divide the Sentinel-2 image into multiple image blocks, and calculate the proportion of each ground object category in each image block. Then, mark the ground object category of the image block based on the relationship between the proportion of each ground object category and the threshold. All image blocks constitute the initial dataset;

[0011] S5: Divide the initial dataset into training sets according to a preset ratio, train the DenseNet model, use the fully connected output of the trained DenseNet model as the deep feature of each image block, calculate the confidence of the deep feature of each image block, and generate the final Sentinel-2 dataset based on the confidence.

[0012] In one embodiment, step S1 includes:

[0013] Bands 1, 5, 6, 7, 8A, 9, 10, 11, and 12 of Sentinel-2 images were oversampled using the nearest neighbor algorithm to achieve a spatial resolution of 10 meters.

[0014] Spectral features, texture features, and shape features are extracted from the oversampled Sentinel-2 image as the features required for classification. The spectral features include the normalized vegetation difference index, the normalized moisture difference index, and the red edge index.

[0015] In one embodiment, the small samples manually selected from the original Sentinel-2 image are Sentinel-2 images, including the sample size of each block, the number of sample blocks, and the number of pixels.

[0016] In one embodiment, when the support vector machine classifier is used to classify the small sample in step S3, the spectral characteristics and shape characteristics are comprehensively used for classification of rivers and lakes in the water body, wherein the shape characteristics are characterized by the shape index, which is calculated as follows:

[0017]

[0018] Where s represents the area, p represents the perimeter, and the shape feature K is used to describe the regularity of the object's shape. The more irregular the shape, the smaller the shape index.

[0019] In one embodiment, the size of the sliding window in step S4 is 64 pixels × 64 pixels, the scanning step is 10 pixels, and the method for calculating the proportion of each ground object category in each image block is: for each 64 pixel × 64 pixel image block, the number of pixels belonging to each ground object category is counted and then divided by the total number of pixels.

[0020] In one embodiment, the depth feature of each image block is calculated in step S5 as follows:

[0021] conf m =exp(-αv(x m )) (2)

[0022] Among them, α is the factor that controls the decay rate, ranging from 2 to 5, conf m is the confidence of the depth feature of the mth image block, f m is the output of the fully connected layer of the DenseNet model, v(x m 0 is the violation factor, which is calculated as:

[0023]

[0024] For each class c∈L, there is a set of cluster centers The number is K c , n∈K c , represents the nth cluster center in category c, L is the set of categories, d(·,·) is the Euclidean distance, λ is a non-negative boundary parameter, is a counter function, l m For categories, Represents category l m The nth (x m ) cluster centers, x m represents the mth image block, Represents category l m The nth cluster center in .

[0025] In one embodiment, generating a final Sentinel-2 dataset based on the confidence level includes:

[0026] If the confidence is greater than the confidence threshold, the corresponding image block is retained, and all retained image blocks are used as the final Sentinel-2 dataset.

[0027] Based on the same inventive concept, the second aspect of the present invention provides a Sentinel-2 dataset generation device based on a small sample and a deep learning model, comprising:

[0028] Feature extraction module, used to extract features required for classification from the original Sentinel-2 image;

[0029] Small sample selection module, used to manually select small samples from raw Sentinel-2 images;

[0030] A preliminary classification module is used to input the features extracted by the feature extraction module and the small samples selected by the small sample selection module into a pre-built support vector machine classifier, and use the support vector machine classifier to classify the small samples;

[0031] The image block partitioning module is used to scan the classification results using a sliding window, divide the Sentinel-2 image into multiple image blocks, calculate the proportion of each ground object category in each image block, and then label the ground object category of the image block based on the relationship between the proportion of each ground object category and the threshold. All image blocks constitute the initial dataset;

[0032] The dataset generation module is used to divide the initial dataset into training sets according to a preset ratio, train the DenseNet model, use the fully connected output of the trained DenseNet model as the deep feature of each image block, calculate the confidence of the deep feature of each image block, and generate the final Sentinel-2 dataset based on the confidence.

[0033] Based on the same inventive concept, the third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the method described in the first aspect when the program is executed.

[0034] Based on the same inventive concept, the fourth aspect of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the program.

[0035] Compared with the prior art, the advantages and beneficial technical effects of the present invention are as follows:

[0036] The present invention proposes a method for generating a Sentinel-2 dataset based on a small sample and a deep learning model. First, the features required for classification are extracted from the original Sentinel-2 image, and a small sample is manually selected from the original Sentinel-2 image. Then, a support vector machine classifier is used to obtain the classification result. Then, the Sentinel-2 image is divided into multiple image blocks according to the category of the classification result. Then, a DenseNet model that can extract sample depth features is used to obtain the depth features of each image block. Finally, the confidence of each image block is determined based on these features, and image blocks with high confidence are retained to generate the final dataset. Since the method of the present invention uses manual selection of small samples, the workload can be reduced and the generation efficiency can be improved. In addition, the support vector machine classifier is used to generate high-precision classification results using the manually selected small samples. The Sentinel-2 image is further divided into multiple image blocks according to the category of the classification result. The DenseNet model can accurately obtain the depth features of each image block, and a high-quality dataset can be generated based on the confidence level. Experimental results show that after multiple iterations, with a reasonable error rate, incorrectly labeled image blocks can be removed from the dataset, and the number of image blocks in each type of terrain can reach a stable state. Comparative experiments also found that this method does not remove incorrectly labeled image blocks, and that deep learning models with larger and deeper fully connected layers perform better. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 This is a flowchart of the Sentinel-2 dataset generation method based on small samples and deep learning models provided in an embodiment of the present invention.

[0039] Figure 2 Schematic diagram of the classification results of the SVM classifier in an embodiment of the present invention.

[0040] Figure 3 Schematic diagram of a water body in an embodiment of the present invention.

[0041] Figure 4 Schematic diagram of rivers, seas and lakes in an embodiment of the present invention.

[0042] Figure 5 This is a schematic diagram for further classification in an embodiment of the present invention.

[0043] Figure 6 1 are the projection results corresponding to different iteration numbers in the embodiment of the present invention, where (a) is the first iteration, (b) is the second iteration, (c) is the third iteration, (d) is the fourth iteration, (e) is the fifth iteration, (f) is the sixth iteration, (g) is the seventh iteration, (h) is the eighth iteration, (i) is the ninth iteration, and (j) is the tenth iteration.

[0044] Figure 7 The number of incorrectly marked image blocks under different error rates in the embodiments of the present invention is (a) 1%, (b) 5%, (c) 8%, and (d) 10%. The horizontal axis is the number of iterations, and the vertical axis is the number of incorrectly marked image blocks.

[0045] Figure 8 For the incorrectly labeled image blocks that are not deleted in the embodiment of the present invention, the incorrect labels and correct labels of each image block are: (a) crops / industrial area; (b) road / river; (c) river / sea and lake; (d) and (e) pasture / crops; (f) river / sea and lake; (g) crops / industrial area; (h) river / sea and lake; (i) sea and lake / river.

[0046] Figure 9 The number of image blocks after each iteration for different error rates in the embodiments of the present invention is: (a) 1%; (b) 5%; (c) 8%; (d) 10%. The horizontal axis is the number of iterations, and the vertical axis is the number of image blocks.

[0047] Figure 10 The number of image blocks in different models according to the present invention is shown in Figure 2: (a) LeNet, (b) VGG, and (c) DenseNet. The horizontal axis represents the number of iterations, and the vertical axis represents the number of incorrectly labeled image blocks.

[0048] Figure 11 The change in the total number of image blocks with a 5% error rate in the following examples: (a) LeNet; (b) VGG; (c) DenseNet. The horizontal axis is the number of iterations, and the vertical axis is the total number of image blocks.

[0049] Figure 12 This is a structural block diagram of a Sentinel-2 dataset generation device based on small samples and a deep learning model in an embodiment of the present invention;

[0050] Figure 13 A schematic diagram of the structure of a computer-readable storage medium provided in an embodiment of the present invention;

[0051] Figure 14 A schematic diagram of the structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0052] DL (deep learning) is currently the most popular method in the field of pattern recognition due to its high classification accuracy. However, DL requires very large samples, and the noise in these samples must be as small as possible, close to pure values. Manually collecting a large number of pure samples is an expensive and time-consuming task. The present invention establishes a semi-automatic training set labeling framework based on full consideration of small samples, noisy samples, automation and accuracy, which reduces the manual burden. The framework selects small samples, uses an SVM classifier for classification, and then divides the Sentinel-2 image into multiple labeled image blocks according to the classification results through a sliding window. The deep features of the image blocks are extracted using the fully connected layer of the DenseNet model. The confidence of each image block is obtained in the feature space, and the image blocks with low confidence are removed. Finally, a large number of pure image blocks for DL ​​are collected.

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0054] Example 1

[0055] The embodiment of the present invention provides a method for generating a Sentinel-2 dataset based on a small sample and a deep learning model, including:

[0056] S1: Extract the features required for classification from the original Sentinel-2 image;

[0057] S2: Manually select a small sample from the original Sentinel-2 image;

[0058] S3: Input the features extracted in step S1 and the small sample selected in step S2 into a pre-built support vector machine classifier, and use the support vector machine classifier to classify the small sample;

[0059] S4: Use a sliding window to scan the classification results, divide the Sentinel-2 image into multiple image blocks, and calculate the proportion of each ground object category in each image block. Then, mark the ground object category of the image block based on the relationship between the proportion of each ground object category and the threshold. All image blocks constitute the initial dataset;

[0060] S5: Divide the initial dataset into training sets according to a preset ratio, train the DenseNet model, use the fully connected output of the trained DenseNet model as the deep feature of each image block, calculate the confidence of the deep feature of each image block, and generate the final Sentinel-2 dataset based on the confidence.

[0061] See Figure 1 , which is a flowchart of the Sentinel-2 dataset generation method based on small samples and deep learning models provided in an embodiment of the present invention.

[0062] In general, the method of the present invention includes five steps: (1) extracting features required for classification from Sentinel-2 images; (2) manually selecting a small sample; (3) inputting the features obtained in step (1) and the sample selected in step (2) into a support vector machine classifier to classify the Sentinel-2 images; (4) using a sliding window, dividing the Sentinel-2 images into multiple 64 pixel × 64 pixel image blocks according to the classification results, for constructing an initial data set; (5) judging whether the image blocks are correctly labeled based on their confidence, and retaining image blocks with high confidence to generate the final data set.

[0063] In specific implementation, Sentinel-2 is part of the Sentinel series of satellites, a dedicated series developed by the European Space Agency (ESA) and funded by the European Commission (EC) under the Space Segment (GSC) of the Copernicus Programme (formerly known as the Global Monitoring for Environment and Security Programme). August is summer in the Northern Hemisphere, and compared to other months, August has clear skies and less cloud cover, which helps prevent cloud cover from affecting data. The experimental data is Sentinel-2-S2A data from Karlsruhe, Germany, on a specific day of a specific year. Sentinel-2 carries a multispectral imager (MSI), which samples 13 spectral bands at spatial resolutions of 10, 20, and 60 meters.

[0064] In order to obtain enough samples to train the convolutional neural network, the selected images are relatively large, with a size of 10980 pixels × 10980 pixels. The size of Sentinel-2 images is 1000 pixels × 1000 pixels.

[0065] In one embodiment, step S1 includes:

[0066] Bands 1, 5, 6, 7, 8A, 9, 10, 11, and 12 of Sentinel-2 images were oversampled using the nearest neighbor algorithm to achieve a spatial resolution of 10 meters.

[0067] Spectral features, texture features, and shape features are extracted from the oversampled Sentinel-2 image as the features required for classification. The spectral features include the normalized vegetation difference index, the normalized moisture difference index, and the red edge index.

[0068] Specifically, sufficient features are conducive to obtaining good classification results. The method of this example uses the features shown in Table 1 for classification.

[0069] Table 1 Description of the feature set of SENTINEL-2

[0070]

[0071]

[0072] Since vegetation and water bodies belong to the terrain category, the Normalized Difference Vegetation Index (NDVI), Normalized Water Difference Index (NDWI) and Red Edge Index (only Sentinel-2 has a red edge band for free and open remote sensing data) are used as classification features.

[0073] To extract built-up areas, this implementation calculates the image contrast in 10 directions and uses the minimum value as the texture feature. In addition, to improve data processing efficiency, the data is first normalized using the zero-mean normalization method.

[0074] In one embodiment, the small samples manually selected from the original Sentinel-2 image are Sentinel-2 images, including the sample size of each block, the number of sample blocks, and the number of pixels.

[0075] Specifically, SVM is suitable for small samples and has good generalization ability. Manual selection of small samples requires less labor.

[0076] Because highways are elongated linear objects, each highway sample is 2 pixels by 2 pixels. Other land cover areas are relatively large, with each sample measuring 15 pixels by 15 pixels or 20 pixels by 20 pixels. The small sample statistics used by the SVM are shown in Table 2. This allows for sample selection in a very short time.

[0077] Table 2 Statistics of small samples used by support vector machines

[0078]

[0079] In one embodiment, when the support vector machine classifier is used to classify the small sample in step S3, the spectral characteristics and shape characteristics are comprehensively used for classification of rivers and lakes in the water body, wherein the shape characteristics are characterized by the shape index, which is calculated as follows:

[0080]

[0081] Where s represents the area, p represents the perimeter, and the shape feature K is used to describe the regularity of the object's shape. The more irregular the shape, the smaller the shape index.

[0082] Among shallow classifiers, support vector machines (SVMs) have very high classification accuracy, a fact confirmed by numerous studies. The accuracy of classification results significantly influences the selection of training samples for subsequent CNNs. In optical imagery, due to the common phenomenon of different spectra of the same object and the appearance of foreign objects within the same spectrum, unsupervised classification methods can easily lead to confusion when classifying objects. For example, when using ISODATA for unsupervised classification, the applicant found that roads and residential areas were easily confused, and forests and crops were easily confused. Therefore, the initial dataset was obtained by partitioning the Sentinel-2 imagery according to the classification results of the supervised SVM classifier.

[0083] Based on statistical learning theory based on the Vapnik-Chervonenkis dimension theory and the principle of minimum structural risk, SVM seeks an optimal compromise between model complexity and learning ability to achieve optimal generalization when sample information is limited. The accuracy of classification results is related to the purity of the final dataset. This step can avoid most possible labeling errors.

[0084] When using SVM for classification, rivers and lakes are considered as one category, namely water. ENVI software is used to calculate the classification results of the support vector machine. Then extract Figure 2 The water bodies marked in Figure 3 The object shown.

[0085] Water bodies fall into two categories: rivers and lakes. In optical images, these bodies are identical except for their boundary shapes. A shape index is used to distinguish these bodies during classification. For objects with identical spectral characteristics but different shapes, combining spectral and shape features can improve object recognition accuracy.

[0086] Calculate the shape index of each object and divide the water bodies into rivers and seas and lakes, such as Figure 4 As shown. Finally, the classification results are as follows Figure 5 shown.

[0087] In one embodiment, the size of the sliding window in step S4 is 64 pixels × 64 pixels, the scanning step is 10 pixels, and the method for calculating the proportion of each ground object category in each image block is: for each 64 pixel × 64 pixel image block, the number of pixels belonging to each ground object category is counted and then divided by the total number of pixels.

[0088] Specifically, the classification results were scanned using a fixed 64×64 pixel scanning window and a scanning step size of 10 pixels. The Sentinel-2 image was then divided into image patches based on the scan results. The proportion of each object class within each image patch was calculated. If the maximum proportion exceeded a threshold, the image patch was labeled with the corresponding object class. Roads and rivers, unlike the other six object classes, are long and narrow. Their thresholds differed from those of the other object classes. Finally, the image patches from the Sentinel-2 image formed the initial dataset.

[0089] In one embodiment, the depth feature of each image block is calculated in step S5 as follows:

[0090] conf m =exp(-αv(x m )) (2)

[0091] Among them, α is the factor that controls the decay rate, ranging from 2 to 5, conf m is the confidence of the depth feature of the mth image block, f m is the output of the fully connected layer of the DenseNet model, v(x m ) is the violation factor, which is calculated as:

[0092]

[0093] For each class c∈L, there is a set of cluster centers The number is K c , n∈K c , represents the nth cluster center in category c, L is the set of categories, d(·,·) is the Euclidean distance, λ is a non-negative boundary parameter, is a counter function, l m For categories, n(x m ) represents category l m The nth (x m ) cluster centers, x m represents the mth image block, Represents category l m The nth cluster center in .

[0094] Specifically, although SVM classification accuracy is high, it cannot guarantee that all image blocks are labeled correctly. The present invention retains image blocks with high confidence. This step is completed automatically by the program and is very efficient.

[0095] This implementation uses the fully connected layers of the DenseNet model as features for each image block to calculate confidence. The DenseNet model used in this method is derived from Keras's DenseNet201. Each land cover image block in the initial dataset is split into training and validation samples in a 3:7 ratio, and the DenseNet model is trained. The confidence of each image block is calculated using the output of the fully connected layers of the DenseNet model.

[0096] There is a set of image blocks m Represents the original information of an image block, is the true label x m (unknown during training), and l m ∈L is the observed label x m , there is a probability difference Our goal is to train a model M using the noisy training set X to predict the true label for each image patch. Given a model and an input image patch, we denote the intermediate output (e.g., the output of the fully connected layer) as f m .

[0097] The confidence evaluation is based on the clustering assumption. The clustering assumption is that data from the same class should form a cluster and be similar to each other, which is the basic assumption of many machine learning techniques. Specifically, for each class c∈L, there is a set of K c Cluster Center Since the number of image blocks for each land cover is different, this implementation sets a cluster center for every 200 image blocks. Then a violation factor is introduced. We are concerned about abnormal samples that violate the clustering assumption, that is, the distance from the sample to its own class center is greater than the distance to the centers of other classes. f(x) is a counter function that counts the number of violations observed. It can be a hard counter, if x is true, f(x) = 1, otherwise it is 0, or it can be a soft counter f(x) =, taking into account the degree of each violation. Obviously, v(x) m ) is larger, x m The more likely it is to be mislabeled, the closer it is to the center of many other classes than to the center of its own observed label.

[0098] Based on the violation factor, the confidence level can be directly defined, which is a monotonically decreasing function of the violation factor. In this embodiment, a simple function is used and the value is normalized to [0, 1].

[0099] In one embodiment, generating a final Sentinel-2 dataset based on the confidence level includes:

[0100] If the confidence is greater than the confidence threshold, the corresponding image block is retained, and all retained image blocks are used as the final Sentinel-2 dataset.

[0101] The method of the present invention is described below through specific examples.

[0102] The results of the method in this embodiment are presented in two forms: the confidence scores of some image blocks and the clustering of each image block, which is represented by dimensionality reduction from the high-dimensional feature space of the fully connected layer to a two-dimensional space using t-SNE technology. The following comparative experiments with different focuses are conducted using these two methods.

[0103] 1. Experimental Results

[0104] 1.1 Confidence Results

[0105] Image blocks with a confidence value less than 0.1 are considered to be mislabeled and will be deleted. This method was used to perform 10 iterative calculations on the experimental data. However, due to space limitations, Table 3 only lists the confidence values ​​of the 8 image blocks of each land cover after the first iteration. The confidence values ​​of the first 7 image blocks are the last 7 confidence rankings of this iteration. The confidence of the last image block is 1, which serves as a reference data point. From a visual effect perspective, there is a certain difference between image blocks with confidence values ​​less than 0.1 and 1. This also shows that the method proposed in the present invention can indeed delete image blocks that do not seem to be correctly labeled.

[0106] Table 3 Confidence after the first iteration

[0107]

[0108] 1.2 Visualization of data image patches in the initial dataset

[0109] Each image block in the dataset is input into the trained DenseNet model to obtain the output of the corresponding fully connected layer. The 1902-dimensional feature space of the fully connected layer is projected into two dimensions using t-SNE technology. The projection result is as follows Figure 6 shown.

[0110] like Figure 6 As shown in Figure 3, during the iterative feature learning process, due to the elimination of feature outliers, the image patches of the wrong categories in various ground objects gradually disappear.

[0111] 2. Results under different ratios of mislabeled image blocks

[0112] To verify the effectiveness of the proposed method, a certain number of mislabeled image patches are added to each category. The percentages are 1%, 5%, 8%, and 10%, respectively. The cardinality of each category is shown in Table 4.

[0113] After each iteration, the number of mislabeled image blocks changes as follows: Figure 7 Generally, after 10 iterations, mislabeled image blocks of different proportions can be eliminated, or their number can be close to 0.

[0114] When the proportion of mislabeled image blocks is 1%, all mislabeled image blocks are eliminated by the end of the third iteration.

[0115] Table 4 Cardinality of each type of landform

[0116]

[0117]

[0118] When the percentage of incorrectly labeled image blocks is 5%, starting from iteration 4, the number of remaining incorrectly labeled image blocks is stable. The number of incorrectly labeled image blocks in crops, roads, and rivers is 1. There is an interesting phenomenon. Figure 8 As shown in (a), based on the classification results, the image patch highlighted as an industrial area is labeled as an industrial area. However, the visual interpretation indicates that its category should be crops. Coincidentally, this image patch happened to be randomly assigned to crops as a mislabeled image patch. Such an image patch is called a "mislabeled image patch." After multiple iterations of the method, the image patch is still retained. This proves that the method does not delete a mislabeled image patch. Figure 8 The image patch in (b) is actually a mixture of road and water. Visually, it is best labeled as road. This is also a mislabeled image patch, which is saved. Figure 8 (c) is a water body. Since the proportion of water bodies in the entire image block is very large, it is difficult to distinguish whether the image block is a river or a sea or lake without surrounding information. Therefore, when an image block is placed in a river as a mislabeled image block, the method of this embodiment does not consider it to be not a river. This also means that the proposed method is not perfect. However, the method of this embodiment can accurately distinguish between rivers and seas and lakes through the shape index when constructing the initial dataset, so there is no confusion between river image blocks and sea and lake image blocks.

[0119] When the percentage of incorrectly labeled image blocks is 8%, two image blocks incorrectly labeled as grass and one image block incorrectly labeled as river are not deleted. Crops and pastures are two easily confused ground features because their base colors are the same (actually green). Figure 8 (d) This embodiment mistakenly identifies it as a pasture. However, Figure 8 (e) is not deleted, which exposes a small flaw in the method of this embodiment. Figure 8 The reason (f) was not deleted is Figure 8 (c) was not deleted for the same reason.

[0120] When the percentage of incorrectly labeled image patches is 10%, there are 3 incorrectly labeled image patches in crops, rivers, and seas and lakes that are not deleted. Figure 8 (g) is also a mislabeled image patch. Figure 8 (h) and (i) are cases where the river and the sea are mislabeled. Figure 8 In (i), there are rivers and lakes on the image patch, but no surrounding information.

[0121] In summary, this method can preserve mislabeled image patches. However, for easily confused image patches, such as crops and pastures, and rivers and lakes, the proposed method's performance is unsatisfactory. However, crops and pastures can be accurately distinguished by the classifier, so they do not appear in each other's categories as mislabeled image patches. The same is true for rivers and lakes.

[0122] The number of image blocks for each type of ground object changes as follows: Figure 9 As shown in the figure, the lower the error rate, the fewer iterations required to reach a stable state for the number of image patches. As the number of erroneous image patches increases, the intra-class differences become greater. This method sacrifices more correctly labeled image patches in order to eliminate as many erroneously labeled image patches as possible. When the error rate is 8% or less, the number of image patches per class of ground objects stabilizes at a relatively reasonable level.

[0123] The number of image blocks for each type of ground object changes as follows: Figure 9 As shown in the figure, the lower the error rate, the fewer iterations required to reach a stable state for the number of image patches. As the number of erroneous image patches increases, the intra-class differences become greater. This method sacrifices more correctly labeled image patches in order to eliminate as many erroneously labeled image patches as possible. When the error rate is 8% or less, the number of image patches per class of ground objects stabilizes at a relatively reasonable level.

[0124] 3. Impact of different CNN models on image patch selection

[0125] To verify whether the deep learning model has an impact on the final results, this embodiment conducted the following comparative experiment. This embodiment uses the LeNet model and the VGG model instead of the DenseNet model. The experimental data consists of 5% of the base number of mislabeled image blocks for each type of ground object.

[0126] The LeNet model, a very simple CNN model released 20 years ago, is one of the earliest CNN models. It is primarily used for simple datasets such as MNIST. The comparative experiments use the sixth fully connected layer of the LeNet model, which has 84 parameters.

[0127] The VGG model is the preferred algorithm for extracting CNN features from images. The comparative experiment uses the 21st fully connected layer of the VGG model, which has a total of 2048 parameters.

[0128] Figure 10 and Figure 11 The changes of incorrectly labeled image blocks.

[0129] like Figure 10 As shown in the figure, the LeNet model requires only three iterations to reach a stable number of mislabeled image blocks. The VGG model requires nine iterations, and the DenseNet model requires four. Regarding the number of residual mislabeled image blocks, the DenseNet model has the fewest residual mislabeled image blocks, followed by the VGG model, and then the LeNet model. Considering the above, the DenseNet model achieves the best results, the VGG model performs second, and the LeNet model achieves the worst results.

[0130] like Figure 11 As shown in (a), the number of image blocks in the LeNet model decreases sharply, and the number of image blocks for the five types of objects approaches zero. Figure 11 As shown in (b), the number of image patches retained by the VGG model is slightly improved compared to the LeNet model, but the number of sea-lakes is still small. Figure 11 As shown in (c), among the three models, the DenseNet model performs the best. The number of image blocks reaches a stable state the fastest and the effect is the best.

[0131] In summary, from Figure 10 and Figure 11 As can be seen, the DenseNet model performed best in eliminating mislabeled image patches and retaining correctly labeled ones. Since the only difference between the three experimental methods is the structure of the deep learning models, the structural differences between the LeNet, VGG, and DenseNet models must have led to the differences in the final results. In the LeNet model, the 84 parameters of the sixth fully connected layer are used as features for each image patch to generate its confidence score. In the VGG model, the 21st fully connected layer (2048 features) are used. In the DenseNet model, the 707th fully connected layer has 1920 parameters. The dimension of the DenseNet model's fully connected layers is similar to that of the VGG model, but much larger than that of the LeNet model. The depth of the DenseNet model's fully connected layers is much greater than that of the LeNet and VGG models. This demonstrates that deeper deep learning models yield more refined features.

[0132] Through comparative experiments, the applicant found that under the same mislabeling ratio, deeper deep learning models performed better.

[0133] Example 2

[0134] Based on the same inventive concept, this embodiment provides a Sentinel-2 dataset generation device based on small samples and deep learning models, see Figure 12 , the device comprises:

[0135] Feature extraction module, used to extract features required for classification from the original Sentinel-2 image;

[0136] Small sample selection module, used to manually select small samples from raw Sentinel-2 images;

[0137] A preliminary classification module is used to input the features extracted by the feature extraction module and the small samples selected by the small sample selection module into a pre-built support vector machine classifier, and use the support vector machine classifier to classify the small samples;

[0138] The image block partitioning module is used to scan the classification results using a sliding window, divide the Sentinel-2 image into multiple image blocks, calculate the proportion of each ground object category in each image block, and then label the ground object category of the image block based on the relationship between the proportion of each ground object category and the threshold. All image blocks constitute the initial dataset;

[0139] The dataset generation module is used to divide the initial dataset into training sets according to a preset ratio, train the DenseNet model, use the fully connected output of the trained DenseNet model as the deep feature of each image block, calculate the confidence of the deep feature of each image block, and generate the final Sentinel-2 dataset based on the confidence.

[0140] Since the device described in Example 2 of the present invention is used to implement the method for generating a Sentinel-2 dataset based on a small sample size and a deep learning model in Example 1 of the present invention, the specific structure and variations of the device are readily apparent to those skilled in the art based on the method described in Example 1 of the present invention, and thus will not be further described here. All devices used in the method described in Example 1 of the present invention fall within the scope of protection of the present invention.

[0141] Example 3

[0142] Based on the same inventive concept, see Figure 13 The present invention further provides a computer-readable storage medium 300 on which a computer program 311 is stored. When the program is executed, the method described in the first embodiment is implemented.

[0143] Since the computer-readable storage medium described in Example 3 of the present invention is the computer-readable storage medium used to implement the method for generating a Sentinel-2 dataset based on a small sample and deep learning model in Example 1 of the present invention, those skilled in the art will be able to understand the specific structure and variations of the computer-readable storage medium based on the method described in Example 1 of the present invention, and therefore will not be described in detail here. All computer-readable storage media used in the method of Example 1 of the present invention fall within the scope of protection of the present invention.

[0144] Example 4

[0145] Based on the same inventive concept, the present application also provides a computer device, such as Figure 14 As shown, it includes a memory 401, a processor 402 and a computer program 403 stored in the memory and executable on the processor. When the processor executes the above program, the method in the first embodiment is implemented.

[0146] Since the computer device described in Example 4 of the present invention is used to implement the method for generating a Sentinel-2 dataset based on a small sample size and a deep learning model in Example 1 of the present invention, the specific structure and variations of the computer device are readily understood by those skilled in the art based on the method described in Example 1 of the present invention, and thus will not be further described here. All computer devices used in the method described in Example 1 of the present invention fall within the scope of protection of the present invention.

[0147] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0148] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0149] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0150] Obviously, those skilled in the art may make various changes and modifications to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, if such changes and modifications of the embodiments of the present invention fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A method for generating the Sentinel-2 dataset based on small samples and deep learning models, characterized by: include: S1: Extract features required for classification from the original Sentinel-2 image. Specifically, the method uses the nearest neighbor algorithm to oversample bands 1, 5, 6, 7, 8A, 9, 10, 11, and 12 of the Sentinel-2 image to achieve a spatial resolution of 10 meters. Spectral, texture, and shape features are extracted from the oversampled Sentinel-2 image as features required for classification. Spectral features include the Normalized Difference Index of Vegetation, the Normalized Difference Index of Water, and the Red Edge Index. S2: Manually select a small sample from the original Sentinel-2 image; S3: Input the features extracted in step S1 and the small sample selected in step S2 into a pre-built support vector machine classifier, and use the support vector machine classifier to classify the small sample. When using the support vector machine classifier to classify the small sample, for rivers and lakes in water bodies, spectral features and shape features are comprehensively used for classification. The shape feature is represented by the shape index, which is calculated as follows: （1） in Indicates area, Indicates perimeter and shape characteristics Used to describe the regularity of an object's shape. The more irregular the shape, the smaller the shape index. S4: Use a sliding window to scan the classification results, divide the Sentinel-2 image into multiple image blocks, and calculate the proportion of each ground object category in each image block. Then, mark the ground object category of the image block based on the relationship between the proportion of each ground object category and the threshold. All image blocks constitute the initial dataset; S5: Divide the initial dataset into training sets according to a preset ratio, train the DenseNet model, use the fully connected output of the trained DenseNet model as the depth feature of each image block, calculate the confidence of the depth feature of each image block, and generate the final Sentinel-2 dataset based on the confidence. The depth feature of each image block is calculated as follows: （2） in, is a factor that controls the decay rate, ranging from 2 to 5. is the confidence of the depth feature of the m-th image block, is the output of the fully connected layer of the DenseNet model, is the violation factor, which is calculated as: （3） For each category , there is a set of cluster centers , the number is , , represents the nth cluster center in category c, is a collection of categories, is the Euclidean distance, is a non-negative boundary parameter, is a counter function, For categories, , Representation category Middle Cluster centers, represents the mth image block, Representation category The nth cluster center in .

2. The method for generating a Sentinel-2 dataset based on a small sample and a deep learning model according to claim 1, wherein: The small samples manually selected from the original Sentinel-2 image are Sentinel-2 images, including the sample size of each block, the number of sample blocks, and the number of pixels.

3. The method for generating a Sentinel-2 dataset based on a small sample and a deep learning model according to claim 1, wherein: The size of the sliding window in step S4 is 64 pixels × 64 pixels, and the scanning step is 10 pixels. The ratio of each ground object category in each image block is calculated as follows: for each 64 pixel × 64 pixel image block, the number of pixels belonging to each ground object category is counted and then divided by the total number of pixels.

4. The method for generating a Sentinel-2 dataset based on a small sample and a deep learning model according to claim 1, wherein: The final Sentinel-2 dataset is generated based on the confidence level, including: If the confidence is greater than the confidence threshold, the corresponding image block is retained, and all retained image blocks are used as the final Sentinel-2 dataset.

5. A Sentinel-2 dataset generation device based on small samples and deep learning models, characterized by: include: The feature extraction module is used to extract the features required for classification from the original Sentinel-2 image. Specifically, it uses the nearest neighbor algorithm to oversample bands 1, 5, 6, 7, 8A, 9, 10, 11, and 12 of the Sentinel-2 image to achieve a spatial resolution of 10 meters. The spectral features, texture features, and shape features are extracted from the oversampled Sentinel-2 image as the features required for classification. The spectral features include the Normalized Difference Index of Vegetation, the Normalized Difference Index of Water, and the Red Edge Index. Small sample selection module, used to manually select small samples from raw Sentinel-2 images; The preliminary classification module is used to input the features extracted by the feature extraction module and the small samples selected by the small sample selection module into the pre-built support vector machine classifier, and use the support vector machine classifier to classify the small samples. When using the support vector machine classifier to classify small samples, for rivers and lakes in water bodies, spectral features and shape features are comprehensively used for classification. The shape feature is represented by the shape index, which is calculated as follows: （1） in Indicates area, Indicates perimeter and shape characteristics Used to describe the regularity of an object's shape. The more irregular the shape, the smaller the shape index. The image block partitioning module is used to scan the classification results using a sliding window, divide the Sentinel-2 image into multiple image blocks, calculate the proportion of each ground object category in each image block, and then label the ground object category of the image block based on the relationship between the proportion of each ground object category and the threshold. All image blocks constitute the initial dataset; The dataset generation module is used to divide the initial dataset into training sets according to a preset ratio, train the DenseNet model, use the fully connected output of the trained DenseNet model as the depth feature of each image block, calculate the confidence of the depth feature of each image block, and generate the final Sentinel-2 dataset based on the confidence. The depth feature of each image block is calculated as follows: （2） in, is a factor that controls the decay rate, ranging from 2 to 5. is the confidence of the depth feature of the m-th image block, is the output of the fully connected layer of the DenseNet model, is the violation factor, which is calculated as: （3） For each category , there is a set of cluster centers , the number is , , represents the nth cluster center in category c, is a collection of categories, is the Euclidean distance, is a non-negative boundary parameter, is a counter function, For categories, , Representation category Middle Cluster centers, represents the mth image block, Representation category The nth cluster center in .

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed, the Sentinel-2 dataset generation method based on small samples and deep learning models is implemented as described in any one of claims 1 to 4.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method for generating a Sentinel-2 dataset based on a small sample and a deep learning model according to any one of claims 1 to 4 is implemented.

Images