Method for identifying crops from time series remote sensing data based on crop phenology knowledge

Abstract

The present invention relates to a method for identifying crops from time-series remote sensing data based on crop phenological knowledge, comprising the following steps: Step S1: Construct phenological indices for key growth periods according to the crop growth curve, and extract crop phenological characteristics; Step S2: Construct a feature extraction model for time-series remote sensing data based on the LSTM network, and integrate a fully convolutional neural network into the LSTM network; Step S3: Based on a multi-modal learning framework, construct a neural network integrating phenological knowledge and time-series remote sensing data, namely the PST-LSTM model; Step S4: Obtain training sample data, and train and optimize the parameters of the PST-LSTM model; Step S5: Identify the remote sensing images to be identified based on the trained PST-LSTM model. The present invention overcomes the problems of large regional and data dependence and weak migration in the traditional method for crop identification, and improves the identification accuracy and applicability of crops.

Description

Technical Field

[0001] The present invention relates to the field of automatic classification of crops in remote sensing images, and particularly to a method for identifying crops from time-series remote sensing data based on crop phenological knowledge. Background Art

[0002] Food security is an important foundation of national security. Scientifically and effectively carrying out crop cultivation is the key means to ensure food security. The spatial distribution information of crop cultivation is of great significance for aspects such as modern agricultural production and optimization of the spatial configuration of the planting structure, and provides basic data support for individual farmers and relevant departments to carry out agricultural digital management. The existing methods for collecting crop cultivation spatial distribution data rely heavily on manual work, are time-consuming and laborious, and have poor timeliness, making it difficult to meet the needs of smart agriculture. Conducting automatic classification of crop cultivation based on satellite remote sensing data is an effective solution to this problem. Currently, the existing methods for automatically classifying crops from remote sensing data mostly use single phenological feature rules or conventional time-series classification methods. There is little research on comprehensively using the phenology, time, and space characteristics of crops, especially integrating multi-modal information under a unified framework. As a result, the existing identification methods have problems of high regional and data dependence and weak migration ability, which limit the large-scale and cross-regional application of automatic classification of crops from remote sensing data. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide a method for identifying crops from time-series remote sensing data based on crop phenological knowledge, which overcomes the problems of high regional and data dependence and weak migration ability in crop identification by traditional methods, and improves the identification accuracy and applicability of crops.

[0004] To achieve the above purpose, the present invention adopts the following technical solutions:

[0005] A method for identifying crops from time-series remote sensing data based on crop phenological knowledge, comprising the following steps:

[0006] Step S1: Construct a phenological index for the critical growth period according to the crop growth curve, and extract crop phenological characteristics;

[0007] Step S2: Construct a feature extraction model for time-series remote sensing data based on the LSTM network, and integrate a fully convolutional neural network in the LSTM network;

[0008] Step S3: Based on the multi-modal learning framework, construct a neural network integrating phenological knowledge and time-series remote sensing data, namely the PST-LSTM model;

[0009] Step S4: Obtain training sample data, and train and optimize the parameters of the PST-LSTM model;

[0010] Step S5: Identify the remotely sensed image to be recognized based on the trained PST-LSTM model.

[0011] Further, the specific steps of step S1 are as follows:

[0012] Step S11: Obtain the temporal growth curve of the target crop index and perform fitting processing on the growth curve using the GAM method;

[0013] Step S12: Obtain the phenological information of the target crop in key phenological periods such as the sowing season, maturity season, and harvesting season, and perform feature analysis on the crop index time series curve based on this;

[0014] Step S13: Construct a phenological feature index according to the characteristics of the crop index time series curve and extract crop phenological variables.

[0015] Further, the specific steps of step S2 are as follows: Based on the LSTM network, construct a temporal feature extractor for remote sensing data to extract the temporal features of crop growth; Based on the LSTM network, construct a spatial feature extractor for remote sensing data based on the fully convolutional neural network module to extract the spatial features of crop types.

[0016] Further, for the LSTM network, constructing a temporal feature extractor for remote sensing data to extract the temporal features of crop growth is specifically as follows:

[0017] (1) Construct a bidirectional feature fusion module, which consists of 2 encoders and 1 concatenation layer. Each encoder includes a spatio-temporal convolutional layer, a normalization layer, an activation function, and an attention mechanism layer;

[0018] (2) Based on the bidirectional feature fusion module, extract the temporal features and phenological features of the input data set and perform bidirectional feature fusion processing;

[0019] (3) Select LSTM as the basic network, combine the attention mechanism to obtain the deep temporal domain information of the fusion features, and extract the temporal features of crop growth.

[0020] Further, based on the LSTM network, constructing a spatial feature extractor for remote sensing data based on the fully convolutional neural network module to extract the spatial features of crop types is specifically as follows:

[0021] (1) Perform dimension permutation on the remote sensing time series data to convert it from (N, Q, M) to (N, M, Q); where N is the total number of samples, Q is the maximum time step, and M is the number of variables processed at each time step;

[0022] (2) Construct a fully convolutional neural network module to extract the spatial features of crop types. The fully convolutional neural network module consists of 3 encoders and 1 global average pooling layer. Each encoder includes a spatio-temporal convolutional layer, a normalization layer, and an activation function. The first two encoders end with an attention mechanism.

[0023] Furthermore, the specific steps of step S3 are as follows:

[0024] Step S31: Construct a multi-modal learning framework to integrate phenological knowledge and temporal remote sensing data;

[0025] Step S32: Based on the multi-modal learning framework, fuse the phenological, temporal, and spatial features output by the LSTM module and the fully convolutional neural network module, and use a fully connected layer to complete the classification output.

[0026] Furthermore, the specific steps of step S4 are as follows:

[0027] Step S41: Based on high-resolution optical images, use the method of visual interpretation to collect training sample data;

[0028] Step S42: Construct training sample data, assign the target crop label as 1 and the non-target crop label as 0. The dimension of the temporal dataset is (N, Q t ×M), and the dimension of the phenological variable dataset is (N, Q p ), where N is the total number of samples, Q t is the maximum time step of the temporal dataset, M is the number of variables processed at each time step, and Q p is the total number of phenological variables;

[0029] Step S43: Train the model based on the training sample set. The training uses the Adam optimizer and sets the initial learning rate as Lr, the training batch is set as B, and the number of iterations is set as R;

[0030] Step S44: Analyze the influence of different combinations of phenological variables on the recognition results, determine the optimal combination, obtain the corresponding model weight file, and get the trained PST-LSTM model;

[0031] The present invention has the following beneficial effects compared with the prior art:

[0032] 1. Based on multi-modal learning, the present invention constructs a unified learning framework, integrates phenological knowledge and temporal remote sensing data, automatically fuses and extracts the phenological-time-space features of crops, and greatly improves the crop recognition ability of the model. The present invention can overcome the problems of a large number of false extractions, missed extractions, and serious fragmentation existing in the traditional method for crop recognition, and improve the transferability of the model.

[0033] 2. The present invention further improves the accuracy of crop identification by automatically combining phenological knowledge and time-series remote sensing data, providing technical support for the production management of the agricultural department. Description of the Drawings

[0034] Figure 1 It is a schematic flowchart of the method according to the embodiment of the present invention.

[0035] Figure 2 It is a structural diagram of the phenology spatio-temporal long short-term memory model (PST-LSTM) according to the embodiment of the present invention.

[0036] Figure 3 It is a structural diagram of the LSTM unit in the model according to the embodiment of the present invention.

[0037] Figure 4 It is a planting cycle diagram of the target crop extracted according to the embodiment of the present invention.

[0038] Figure 5 It is a schematic diagram of the construction of the time-series curve and index of the VH and VV indices of the SAR image of the target crop extracted according to the embodiment of the present invention.

[0039] Figure 6 It is a detailed diagram of part of the recognition results according to the embodiment of the present invention.

[0040] Figure 7 It is a detailed diagram of the migration result according to the embodiment of the present invention. Detailed Embodiment

[0041] The present invention will be further described below with reference to the drawings and embodiments.

[0042] In this embodiment, the Sentinel-1A time-series SAR remote sensing images of the study area are obtained, and preprocessing operations are performed on the images. The preprocessing steps include orbital file correction, radiometric correction, terrain correction, speckle filtering and other operations.

[0043] Please refer to Figure 1 , the present invention provides a method for identifying crops from time-series remote sensing data based on crop phenological knowledge, including the following steps:

[0044] Step S1: Construct a phenological index for the key growth period according to the growth curve of the tobacco crop, and extract the crop phenological characteristics;

[0045] Step S2: Based on the long short-term memory neural network model (LSTM), construct a time-series feature extractor for remote sensing data to extract the time features of crop growth;

[0046] Step S3: On this basis, based on the fully convolutional neural network module (FCN), construct a spatial feature extractor for remote sensing data to extract the spatial features of crop types;

[0047] Step S4: Based on the multi-modal learning framework, integrate phenological knowledge and temporal remote sensing data to create a neural network that fuses phenology, time, and space, namely the Phenological Spatio-Temporal - Long Short-Term Memory Model (PST-LSTM);

[0048] Step S5: Obtain training sample data, and train and optimize the parameters of the PST-LSTM model.

[0049] Step S6: Based on the trained PST-LSTM model, perform automatic crop recognition in different regions to verify the effectiveness and transferability of the model;

[0050] In this embodiment, step S1 specifically includes the following steps:

[0051] Step S11: Based on the SAR temporal data, obtain the temporal growth curves of indices such as VH and VV of tobacco crops, and use the GAM method to fit the growth curves;

[0052] Step S12: Obtain the phenological information of tobacco at key phenological periods such as the film mulching period, transplanting period, topping period, growth period, and mature harvesting period, and conduct feature analysis on the temporal curves of VH and VV accordingly:

[0053] Since tobacco is affected by the greenhouse film during the film mulching period, the VH value of tobacco decreases. After the transplanting period, as tobacco grows and the leaves expand, the VH value increases. During the topping period, the growth of tobacco is restricted by apical dominance, and a local minimum appears in the VH curve. After the topping work, tobacco continues to grow and the VH value continues to increase. At the mature harvesting period, tobacco begins to be harvested and the VH value drops suddenly. The change trend of the VV backscattering coefficient of tobacco during the phenological period is the same as that of the VH coefficient;

[0054] Step S13: According to the characteristics of the tobacco VH and VV temporal curves, construct phenological feature indices V1, V2, and V3, and extract crop phenological variables:

[0055] First, construct feature indices V1 and V3 according to the phenological characteristics of tobacco during the mature harvesting period. The V1 index is the sum of the slopes of the sequential VH images during the mature harvesting period of tobacco.

[0056]

[0057] The V3 index is the difference between the VV values at the end of the tobacco harvesting period and the beginning of the harvesting period.

[0058] V3 = VV 收割结束期 - VV 收割始期

[0059] Secondly, according to the phenological characteristics that the VH value of tobacco reaches local minimum values simultaneously before the transplanting period and at the topping stage, a characteristic index V2 is constructed. V2 is the difference between the two minimum values.

[0060] V2 = VH 打顶期极小值 - VH 移栽期前极小值

[0061] In this embodiment, in step S11, the GAM is defined as follows:

[0062] GAM is a non-parametric form of multiple regression model with great flexibility, and its expression is as follows:

[0063] g(u) = β 0 + s 1 (X 1 ) + s 2 (X 2 ) + … + s n (X n )

[0064] In the formula, s i (·) is a non-parametric smoothing function, such as kernel function, smoothing spline function, etc.

[0065] In this embodiment, step S2 specifically includes the following steps:

[0066] Step S21: Construct a bidirectional feature fusion module (BIFFM). The module consists of 2 encoders and 1 concatenation layer. Each encoder includes a spatio-temporal convolutional layer, a normalization layer, an activation function, and an attention mechanism layer;

[0067] Step S22: Based on the bidirectional feature fusion module, extract the time features and phenological features of the input data set, and perform bidirectional feature fusion processing;

[0068] Step S23: Select LSTM as the basic network, combine the attention mechanism to obtain the deep time-domain information of the fusion features, and extract the time features of crop growth;

[0069] In this embodiment, in step S23, the LSTM is defined as follows:

[0070] The LSTM cell has a gating mechanism inside to determine the retention and update of information. The input data of the LSTM cell at the t-th time step consists of the current input signal x t and the previous time output signal h t-1 . Inside the LSTM, the input data is output through functions such as the sigmod function and the tanh function. The specific calculation formula is as follows:

[0071] f t = σ(W f·[h t-1 ,x t +b f )

[0072] i t =σ(W i ·[h t-1 ,x t +b i )

[0073] o t =σ(W o ·[h t-1 ,x t +b o )

[0074]

[0075]

[0076]

[0077] In the formula, f t 、i t and o t are the "forget gate", "input gate" and "output gate" respectively; W f 、W i 、W o and W c are weight factors; b f 、b i 、b o and b c are bias vectors.

[0078] In this embodiment, step S3 specifically includes the following steps:

[0079] Step S31: Perform dimensional permutation on the remote sensing time series data to convert it from (N, Q, M) to (N, M, Q), where N is the total number of samples, Q is the maximum time step, and M is the number of variables processed at each time step. When M < Q, the dimensional permutation operation can improve the model performance;

[0080] Step S32: On the basis of step S31, construct a fully convolutional neural network module to extract the spatial features of the crop type. The fully convolutional neural network module consists of 3 encoders and 1 global average pooling layer. Each encoder includes a spatio-temporal convolutional layer, a normalization layer, and an activation function. The first two encoders end with an attention mechanism;

[0081] In this embodiment, step S4 specifically includes the following steps:

[0082] Step S41: On the basis of steps S2 and S3, construct a multi-modal learning framework to integrate phenological knowledge and temporal remote sensing data;

[0083] Step S42: Based on the multi-modal learning framework, fuse the phenological, temporal, and spatial features output by the LSTM module and the fully convolutional neural network module, and use a fully connected layer to complete the classification output;

[0084] In this embodiment, step S5 specifically includes the following steps:

[0085] Step S51: Based on Sentinel-2 images, use visual interpretation methods to collect training sample data;

[0086] Step S52: Based on step S51, make a training sample set, assign the tobacco label as 1, and the non-tobacco label as 0. The dimension of the time series data set is (N, Q t ×M), the dimension of the phenological variable data set is (N, Q p ), N = 71227 is the total number of samples, Q t = 15 is the maximum time step of the time series data set, M = 2 is the number of variables processed at each time step, Q p is the total number of phenological variables;

[0087] Step S53: Train the model based on the training sample set. The training uses the Adam optimizer and sets the initial learning rate to 10 -3 , the training batch is set to 128, and the number of iterations is set to 250;

[0088] Step S54: Analyze the influence of different combinations of phenological variables on the recognition results, determine the optimal combination, obtain the corresponding model weight file, and obtain the trained PST-LSTM model;

[0089] Specifically, this embodiment uses Ninghua County, Sanming City, Fujian Province as the study area, and uses 15 scenes of Sentinel-1A radar time series remote sensing images from February to July 2020. After preprocessing, the VH and VV backscattering coefficient values are obtained. This embodiment uses 29234 tobacco samples and 41993 non-tobacco samples for model training.

[0090] Such as Figure 2As shown in the figure, it is the structural diagram of the phenological spatio-temporal long short-term memory model (PST-LSTM) constructed in this embodiment. The model extracts the time features of crop growth based on the long short-term memory neural network module (LSTM), integrates the fully convolutional neural network module (FCN) on the LSTM to extract the spatial features of crop types, and finally, under the multi-modal learning framework, fuses the phenological, time, and spatial features to obtain the spatial distribution of crops. The input of PST-LSTM is the time series remote sensing image data and the phenological feature dataset. The phenological features and time features are fused by the bidirectional feature fusion module (BiFFM). BiFFM consists of 2 encoders and 1 concatenation layer. Each encoder includes a spatio-temporal convolutional layer, a normalization layer, an activation function, and an attention mechanism layer. The fused features obtain the time features of crop growth through the LSTM module. The FCN module consists of 3 encoders and 1 global average pooling layer. Each encoder includes a spatio-temporal convolutional layer, a normalization layer, and an activation function. The first two encoders end with the CBAM attention mechanism. The time series remote sensing image data obtains the spatial features of crop types through the FCN module. Finally, the concatenation layer is used to fuse the phenological, time, and spatial features of the crops to complete the recognition of the target crops.

[0091] As Figure 5 shown, it is the phenological feature indices V1, V2, and V3 constructed in this embodiment according to the VH and VV time series curve features of tobacco. The V1 index is the sum of the successive slopes of VH during the mature harvesting period of tobacco. The V2 index is the difference between the minimum values of VH of tobacco in February (before the transplanting period) and April (topping period). The V3 index is the difference between the VV values at the end of the tobacco harvesting period and the beginning of the harvesting period.

[0092] As Figure 6 shown, it is the local detail map of tobacco identified in Ninghua County, Sanming City, Fujian Province in this embodiment. In this embodiment, 3 groups of different phenological variable combinations are set for model training, specifically V1+V2, V1+V3, V1+V2+V3. It can be seen from the figure that there are significant differences in the tobacco recognition results with the addition of different phenological variables, but most of the tobacco planting areas can still be recognized, with relatively high recognition accuracy. After accuracy verification, the model weight of the V1+V3 combination is the best, and the overall accuracy reaches 94.95%, so this is selected as the transfer prediction model.

[0093] As Figure 7 shown, it is the detail map of the transfer results in Pucheng County, Nanping City, Fujian Province, Shanghang County, Longyan City, Fujian Province, and Anfu County, Ji'an City, Jiangxi Province in the embodiment of the present invention. It can be seen from the figure that most of the tobacco planting areas in the three regions can be recognized using the proposed method, with relatively high transferability.

[0094] The above are only the preferred embodiments of the present invention. All equivalent changes and modifications made according to the scope of the patent application of the present invention shall fall within the scope covered by the present invention.

Claims

1. A method for identifying crops from time - series remote sensing data based on crop phenological knowledge, characterized in that, it includes the following steps: Step S1: Construct a phenological index for the critical growth period according to the crop growth curve, and extract the crop phenological characteristics; Step S2: Construct a feature extraction model for time - series remote sensing data based on the LSTM network, and integrate a fully convolutional neural network into the LSTM network; Step S3: Based on the multi - modal learning framework, construct a neural network that integrates phenological knowledge and time - series remote sensing data, namely the PST - LSTM model; Step S4: Obtain training sample data, and train and optimize the parameters of the PST - LSTM model; Step S5: Identify the remote sensing image to be recognized based on the trained PST - LSTM model; The specific content of step S2 is as follows: Based on the LSTM network, construct a time - series feature extractor for remote sensing data to extract the time features of crop growth; Based on the LSTM network, construct a spatial feature extractor for remote sensing data based on the fully convolutional neural network module to extract the spatial features of crop types; Based on the LSTM network, constructing a time - series feature extractor for remote sensing data to extract the time features of crop growth is specifically: (1) Construct a bidirectional feature fusion module, which consists of 2 encoders and 1 concatenation layer. Each encoder includes a spatio - temporal convolutional layer, a normalization layer, an activation function, and an attention mechanism layer; (2) Based on the bidirectional feature fusion module, extract the time features and phenological features of the input data set, and perform bidirectional feature fusion processing; (3) Select LSTM as the basic network, and combine the attention mechanism to obtain the deep - layer time - domain information of the fusion features, and extract the time features of crop growth; Based on the LSTM network, constructing a spatial feature extractor for remote sensing data based on the fully convolutional neural network module to extract the spatial features of crop types is specifically: (1) Perform dimension permutation on the remote sensing time - series data to convert it from (N, Q, M) to (N, M, Q); where N is the total number of samples, Q is the maximum time step, and M is the number of variables processed at each time step; (2) Construct a fully convolutional neural network module to extract the spatial features of crop types. The fully convolutional neural network module consists of 3 encoders and 1 global average pooling layer. Each encoder includes a spatio - temporal convolutional layer, a normalization layer, and an activation function, and the first two encoders end with an attention mechanism.

2. The method for identifying crops from time - series remote sensing data based on crop phenological knowledge according to claim 1, characterized in that, the specific content of step S1 is as follows: Step S11: Obtain the time - series growth curve of the target crop index, and use the GAM method to fit the growth curve; Step S12: Obtain the phenological information of the target crop at the critical phenological periods of the sowing season, maturity season, and harvesting season, and perform feature analysis on the crop index time - series curve accordingly; Step S13: According to the characteristics of the crop index time - series curve, construct a phenological feature index and extract crop phenological variables.

3. The method for identifying crops from time - series remote sensing data based on crop phenological knowledge according to claim 1, characterized in that, the specific content of step S3 is as follows: Step S31: Construct a multi-modal learning framework to integrate phenological knowledge and temporal remote sensing data; Step S32: Based on the multi-modal learning framework, fuse the phenological, temporal, and spatial features output by the LSTM module and the fully convolutional neural network module, and use a fully connected layer to complete the classification output.

4. The method for identifying crops from time series remote sensing data based on crop phenological knowledge according to claim 1, characterized in that the specific steps of step S4 are as follows: Step S41: Based on high-resolution optical images, use visual interpretation methods to collect training sample data; Step S42: Construct training sample data, assign the target crop label as 1, and the non-target crop label as 0. The dimension of the time series dataset is (N, Q t ×M), and the dimension of the phenological variable dataset is (N, Q p ). N is the total number of samples, Q t is the maximum time step of the time series dataset, M is the number of variables processed at each time step, and Q p is the total number of phenological variables; Step S43: Train the model based on the training sample set. The training uses the Adam optimizer and sets the initial learning rate to Lr, the training batch is set to B, and the number of iterations is set to R; Step S44: Analyze the influence of different combinations of phenological variables on the recognition results, determine the optimal combination, obtain the corresponding model weight file, and obtain the trained PST-LSTM model.

Images