A crop classification method based on time-series remote sensing images

Abstract

The application discloses a crop classification method based on time sequence remote sensing images, and crop classification is performed through a crop classification model, wherein the crop classification model comprises a learnable position encoding module, a spatial domain encoding module, a time domain encoding module and a classification decoder; the learnable position encoding module is used for encoding calendar time to obtain a time dimension related to crop growth; the spatial domain encoding module is used for extracting features of a single multispectral remote sensing image of Sentinel 2 to obtain spatial domain features; the time domain encoding module is used for mixing the spatial domain features and the time dimension to extract time domain features; and the classification decoder is used for obtaining a crop classification result according to the time domain features. The application encodes calendar time, mixes the calendar time with spatial domain features, and uses cross-domain time encoding to eliminate the problem of inconsistent time distribution of a source data set and a target data set; and then a semi-supervised method with time shift enhancement and attenuation is used to fully mine spatial geometric information of the target data set, so that the crop classification precision is improved.

Description

Technical Field

[0001] This invention belongs to the field of crop classification technology and relates to a crop classification method based on time-series remote sensing images. Background Technology

[0002] Agricultural crop type mapping provides a valuable tool for agricultural management, environmental monitoring, and decision-making. The multispectral image time series provided by Sentinel-2 has enabled deep learning-based agricultural monitoring applications. The subsequent development of datasets such as the Sentinel-2 million-labeled sample dataset TimeSen2Crop (see reference 1) and the multi-year, multi-country benchmark dataset Sen4AgriNet (see reference 2) has facilitated large-scale crop classification.

[0003] However, the training results on existing datasets are not ideal when directly applied to regions of interest, mainly in two aspects: First, due to different climatic conditions and planting patterns, the phenological characteristics of the same crop type vary significantly in different regions; second, even in the same region, the phenological characteristics are different in different years. For example, the extreme weather in the Midwestern United States in 2019 caused floods that delayed planting by 1-2 months.

[0004] To address the cross-domain time series crop classification problem, Wang et al. proposed the Phenological Alignment Network (PAN) (see reference 3). Specifically, it employs the MMD algorithm within domain invariance to align source and target data in the feature space. Although PAN learns domain-invariant features, it does not explicitly handle the time dimension, thus failing to directly resolve the time shift problem. Joachim et al. pointed out that the main difference between the source and target domains is the time dimension shift, and proposed the Time Matching algorithm (see reference 4). The first step involves training the model in the source domain using shiftAug to reduce the model's sensitivity to time shift, thereby minimizing application error in the target domain. The second step involves using the trained model... The first step directly estimates the time offset of the target domain. The second step uses the offset estimate to train pseudo-labels on the unlabeled target region. The third step repeats steps 2 and 3 until the model's estimate of the time offset of the target domain approaches 0, thus achieving an adaptive transformation from the source domain to the target domain. Soon after, Joachim et al. proposed an improved algorithm based on thermal location encoding (TPE) based on the time matching algorithm (see reference 5). TPE points out that since crop growth is related to thermal time but not calendar time, if the unit of time domain is changed from calendar time to thermal time, the alignment of the time dimension can be directly achieved. The TPE algorithm greatly enhances the robustness of the model to time changes and shows extremely high versatility on the European dataset. In the TPE algorithm, thermal time needs to be calculated for both the source and target data. The calculation of thermal time requires accumulating the average temperature of each day, but most publicly available multi-time series satellite datasets do not include temperature data.

[0005] List of cited references:

[0006] Reference 1, G. Weikmann, C. Paris and L. Bruzzone, "TimeSen2Crop: AMillion Labeled Samples Dataset of Sentinel 2 Image Time Series for Crop-TypeClassification," in IEEE Journal of Selected Topics in Applied EarthObservations and Remote Sensing, vol. 14, pp. 4699-4708, 2021, doi: 10.1109 / JSTARS.2021.3073965.

[0007] Citation 2, D. Sykas, I. Papoutsis and D. Zografakis, "Sen4AgriNet: AHarmonized Multi-Country, Multi-Temporal Benchmark Dataset for AgriculturalEarth Observation Machine Learning Applications," 2021 IEEE InternationalGeoscience and Remote Sensing Symposium IGARSS, 2021, pp. 5830-5833, doi:10.1109 / IGARSS47720.2021.9553603.

[0008] Citation 3, Ziqiao Wang, Hongyan Zhang, Wei He, and Liangpei Zhang.Phenology alignment network: A novel framework for cross regional time seriescrop classifification. In Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition, pages 2940–2949, 2021. 2.

[0009] Citation 4, Joachim Nyborg, Charlotte Pelletier, Sebastien Lefevre, andIra Assent. TimeMatch: Unsupervised cross-region adaptation by temporal shiftestimation. arXiv preprint, abs / 2111.02682, 2021. 1, 2, 4, 6, 7, 8

[0010] Cited document 5, Nyborg, Joachim, Charlotte Pelletier, and Ira Assent. "Generalized classification of satellite image time series with thermalpositional encoding." Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition. 2022. Summary of the Invention

[0011] To address the technical problem in the field of crop classification based on multi-temporal remote sensing imagery that training results on existing datasets cannot be directly applied to cross-domain regions of interest, this invention provides a semi-supervised crop classification algorithm based on learnable location encoding to improve crop classification accuracy.

[0012] This invention provides a crop classification model, including a learnable location encoding module, a spatial encoding module, a temporal encoding module, and a classification decoder. The learnable location encoding module is used to encode calendar time to obtain a time dimension related to crop growth. The spatial encoding module is used to extract features from a single multispectral remote sensing image of Sentinel-2 to obtain spatial features. The temporal encoding module is used to combine the spatial features and the time dimension to extract temporal features. The classification decoder is used to derive crop classification results based on the temporal features.

[0013] Furthermore, the learnable location encoding module consists of a GRU module and a fully connected layer; the spatial encoding module uses a CNN module or a pixel encoder (PSE); the temporal encoding module uses one of a GRU module, an LSTM module, or a temporal attention encoder (TAE); and the classification decoder uses a CNN module or a support vector machine (SVM).

[0014] The present invention also provides a training method for the above-mentioned crop classification model. The crop classification model M is trained on the source dataset by means of time-shift enhancement to obtain the initially trained crop classification model Ms. The crop classification model Ms is further trained in the source domain and the target domain in a semi-supervised manner. During training, the time-shift enhancement decays with the increase of the number of training times until the sum of the cross-entropy loss and the squared difference loss reaches the minimum, and the final crop classification model Msd is obtained.

[0015] The crop classification method based on time-series remote sensing imagery disclosed in this invention inputs time-series multispectral remote sensing imagery into the aforementioned crop classification model Msd to obtain crop classification results.

[0016] This invention proposes to encode calendar time and integrate it with spatial features, using cross-domain time encoding to eliminate the problem of inconsistent time distribution between the source and target datasets; then, a semi-supervised method with time-shifted attenuation is used to fully mine the spatial geometric information of the target dataset to improve crop classification accuracy. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the crop classification model in Example 1;

[0018] Figure 2 This is a structural block diagram of the learnable position encoding module in Example 1;

[0019] Figure 3 This is a schematic diagram of semi-supervised learning in Example 2;

[0020] Figure 4 This is a comparison chart of the accuracy rates of the four crop classification methods in Example 3. Detailed Implementation

[0021] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The embodiments of the present invention are given for illustrative and descriptive purposes only, and are not intended to be exhaustive or to limit the invention to the forms disclosed. Many modifications and variations will be apparent to those skilled in the art. The embodiments were chosen and described to better illustrate the principles and practical application of the invention, and to enable those skilled in the art to understand the invention and design various embodiments with various modifications suitable for a particular purpose. Example

[0022] A crop classification model, such as Figure 1 As shown, it includes a learnable location encoding module, a spatial encoding module, a temporal encoding module, and a classification decoder.

[0023] The learnable location encoding module is used to encode calendar time to obtain a time dimension related to crop growth. In this embodiment, the learnable location encoding module consists of a GRU module and a fully connected layer, such as... Figure 2 As shown, the calendar time T[nt] in days is input into the learnable location encoding module to obtain the encoded time E[ntc1].

[0024] The spatial coding module is used to extract features from a single multispectral remote sensing image X[ntc0s] of Sentinel-2 to obtain spatial features F1[ntc1]. Sentinel-2 is an Earth observation mission under the European Space Agency's Copernicus program, primarily used to observe the Earth's surface and provide related telemetry services, such as forest monitoring, land cover change detection, and natural disaster management. This part is a routine operation for crop classification, which can be performed using a CNN module to extract image features, or a pixel encoder (PSE) can be used to extract some statistical features, such as the mean, standard deviation, maximum value, and minimum value of the target pixels.

[0025] The temporal coding module is used to combine the spatial feature F1[ntc1] and the temporal dimension E[ntc1] to extract the temporal feature F2[ntc2]. Commonly used temporal coding modules in this field include GRU, LSTM, and Temporal Attention Encoder (TAE).

[0026] The classification decoder is used to derive the crop classification result Yp[nc] based on the temporal feature F2[ntc2]. It can be a CNN module or a support vector machine (SVM).

[0027] It is worth noting that this embodiment does not aim to propose how to perform spatial coding, temporal coding, and classification decoding, as these can all be achieved based on existing technologies. Rather, it aims to integrate learnable positional coding into the crop classification model, model the time dimension related to crop growth, and then combine it with spatial features to solve the time offset problem that was not addressed in reference 3, as well as the problem that references 4 and 5 are too complex to be widely applied. Example

[0028] The training method for the crop classification model described in Example 1 involves training the crop classification model M on the source dataset using time-shift augmentation to obtain the initially trained crop classification model Ms.

[0029] The crop classification model Ms is further trained in a semi-supervised manner on both the source and destination domains, such as... Figure 3 As shown, the time-shift enhancement during training decays with the increase of training times until the sum of the interaction entropy loss and the squared difference loss reaches its minimum, resulting in the final crop classification model Msd.

[0030] Figure 3 In this context, Yg[nc] represents the ground truth of the labeled data, which is the labeling result; Yz[nc] represents the average of the results obtained from multiple inferences during the training process for the unlabeled data, which is the average of the prediction results. Since semi-supervised learning algorithms are existing technology, they will only be briefly introduced here and will not be elaborated upon.

[0031] Time offset enhancement and statistical enhancement are data enhancement methods for calendar time input and multispectral remote sensing image input, respectively. How to implement them is a matter of existing technology and will not be elaborated here.

[0032] Since the reliability of the learnable temporal encoding module gradually increases with model performance, and significant time-shift augmentation can inhibit the modeling of crop growth time dimension by the learnable temporal encoding, the impact of time-shift augmentation should be gradually reduced as the number of training iterations increases, similar to how the learning rate decays with the number of training iterations. In short, during semi-supervised learning, time-shift augmentation initially takes effect, but as the time-shift augmentation diminishes, the learnable positional encoding gradually begins to take effect until the time-shift augmentation decays to zero.

[0033] Therefore, this embodiment defines a cosine descent formula for time-shift enhancement attenuation. in, Characterizing the initial time shift enhancement magnitude, This indicates which number is currently being represented. How many represent Then time shift enhancement decays to For example, take This indicates that the maximum time shift enhancement is 60 days. Example

[0034] A crop classification method based on temporal remote sensing imagery, wherein temporal multispectral remote sensing imagery is input into the crop classification model in Example 2. The crop classification results were obtained.

[0035] To verify the improvement in crop classification accuracy based on the crop classification model and training method proposed in this invention, crop classification accuracy tests were conducted in four groups.

[0036] 1. Group 1 (LPE+SA):

[0037] First, the crop classification model described in Example 1 is adopted, wherein learnable positional encoding (using...) The (representation) module consists of The module consists of fully connected layers. The spatial coding module uses a pixel encoder (PSE), the temporal coding module uses a temporal attention encoder (TAE), and the classification decoder uses a 3×3 CNN module with a ReLU activation function. Secondly, the training method described in Example 2 is used. Take 60, Set to 120, SA represents time-shift enhancement attenuation;

[0038] Finally, the pre-prepared test set (TimeMatch dataset, source domain AT1+DK1+FR2, destination domain FR1) is input into the crop classification model Msd to obtain the classification results and verify the accuracy.

[0039] 2. Second group (FPE+SA):

[0040] Unlike the first group, a fixed sinusoidal position code (represented by FPE) is used instead of a learnable position code.

[0041] 3. The third group (LPE+a=0):

[0042] Unlike the first group, no time-shift enhancement processing was performed during the model training process (denoted by a=0).

[0043] 4. The fourth group (LPE+t=0):

[0044] Unlike the first group, time-shift enhancement is performed during model training, but no time-shift enhancement is decayed during semi-supervised learning (denoted as t=0).

[0045] The accuracy verification results of the four crop classification methods are as follows: Figure 4 As shown.

[0046] Comparing the first and second groups, it can be seen that learnable location coding, when used in crop classification models, achieves better classification accuracy than fixed location coding.

[0047] Comparing the first group with the third and fourth groups, it can be seen that time-shift augmentation without decay will inhibit the ability of learnable positional encoding to model the crop growth time dimension, affecting classification accuracy; while without time-shift augmentation, the model may have insufficient generalization ability in the target domain, low confidence of generated pseudo-labels, and ultimately cause the entire model to fail to converge in semi-supervised learning, resulting in very low classification accuracy.

[0048] Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art and related fields based on the embodiments of the present invention without inventive effort should fall within the scope of protection of the present invention.

Claims

1. A crop classification method based on time-series remote sensing imagery, characterized in that, Crop classification is performed using a crop classification model that integrates spatial features and the temporal dimension; The crop classification model includes a learnable location encoding module, a spatial encoding module, a temporal encoding module, and a classification decoder; The learnable location encoding module is used to encode calendar time to obtain the time dimension related to crop growth; The spatial coding module is used to extract features from a single multispectral remote sensing image of Sentinel 2 to obtain spatial features; The temporal coding module is used to combine spatial features and the temporal dimension to extract temporal features; The classification decoder is used to derive crop classification results based on temporal features; The training method of the crop classification model is as follows: First, the crop classification model M is trained on the source dataset using time-off augmentation to obtain the pre-trained crop classification model Ms. Then, the crop classification model Ms is trained in a semi-supervised manner on the source and destination domains. During training, the time-shift enhancement decays as the number of training iterations increases until the sum of the interaction entropy loss and the squared difference loss reaches its minimum, thus obtaining the final crop classification model Msd. The temporal multispectral remote sensing images are input into the final crop classification model Msd to obtain the crop classification results.

2. The crop classification method based on time-series remote sensing imagery according to claim 1, characterized in that, The learnable location encoding module consists of a GRU module and a fully connected layer.

3. The crop classification method based on time-series remote sensing imagery according to claim 1 or 2, characterized in that, The spatial coding module uses a CNN module or a pixel encoder (PSE); the temporal coding module uses one of a GRU module, an LSTM module, or a temporal attention encoder (TAE); and the classification decoder uses a CNN module or a support vector machine (SVM).

Images