Early remote sensing identification method of crop based on accumulated effective temperature

Abstract

The present application provides an early crop remote sensing recognition method fusing effective accumulated temperature, aiming at the early crop growth environment and its own characteristics, proposes a crop extraction method based on a space-time multi-level attention model and a long short-time memory neural network, integrates effective accumulated temperature and multi-source remote sensing data, fully utilizes the complementary information between SAR and optical data, fuses again in the feature layer, and further fuses with effective accumulated temperature data to realize the enhancement of crop environment information, so that the model better understands the relevance between different crop early characteristics and environment, and improves the early crop recognition performance.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of crop mapping, in particular to an early crop remote sensing identification method fusing effective accumulated temperature. BACKGROUND

[0002] Near real-time crop mapping can provide basic data support for agricultural resource allocation and help relevant departments and farmers optimize the use of limited resources in agricultural production. The commonly used machine learning method combined with satellite remote sensing data usually relies on complete time series remote sensing data covering the entire growing season, which has high calculation cost and poor timeliness. In addition, the single data source used leads to limited information mining, and the environmental factors of crops are not fully considered. Heat condition is an important factor restricting agricultural production, especially in the early growth stage of crops. Effective accumulated temperature is crucial for crop growth. Embedding effective accumulated temperature information can enhance the ability of the deep learning segmentation network to distinguish the similarities of different crops in the early stage, and is expected to further optimize the performance and efficiency of early crop identification. SUMMARY

[0003] The application provides an early crop remote sensing identification method fusing effective accumulated temperature. According to the early crop growth environment and its own characteristics, a spatio-temporal multi-level attention model and a long short-term memory neural network are proposed. The method integrates effective accumulated temperature and multi-source remote sensing data for crop extraction. The complementary information between SAR and optical data is fully utilized, and the features are fused at the feature layer. Then, the effective accumulated temperature data is further fused to enhance the crop environmental information, so that the model can better understand the relevance of the early characteristics of different crops and the environment, and improve the early crop identification performance.

[0004] The application adopts the following technical solutions.

[0005] The early crop remote sensing identification method fusing effective accumulated temperature comprises the following steps:

[0006] Step S1, acquiring medium-resolution time-series Sentinel-1 radar SAR images in a study area, and processing to obtain farmland plot-level backscattering coefficients VV and VH characteristics;

[0007] Step S2, acquiring medium-resolution time-series Sentinel-2 optical images in the study area, calculating kNDVI, ReG_RVI, ReG_NDVI, NDWI, REP and mNDVI705 characteristics, and screening out the best optical feature combination: kNDVI, NDWI, mNDVI705 and Red through a random forest feature optimization algorithm;

[0008] Step S3, acquiring daily ERA-5 air temperature data in the study area, and calculating effective accumulated temperature T cal ;

[0009] Step S4, making a crop sample dataset based on the farmland plot;

[0010] Step S5, constructing a double-branch network model framework based on deep learning semantic segmentation, the first branch being a multi-level network model integrated with a spatio-temporal attention mechanism, used for extracting SAR and optical image features, and the second branch being a long short-term memory neural network model with an attention mechanism, used for extracting effective accumulated temperature features;

[0011] Step S6, taking the multi-level network model with a spatio-temporal attention mechanism as the first branch, used for mining multi-scale spatio-temporal information of SAR and optical medium-resolution images; the branch is composed of two parallel multi-level feature extractors based on ResNet and a spatio-temporal attention mechanism module, SAR and optical data are input in parallel, spatial features are extracted through different receptive fields, and the spatio-temporal attention mechanism module is used to further extract multi-level spatio-temporal features, and the extracted high-level features of SAR and optical images are spliced and fused according to different levels;

[0012] Step S7, taking the long short-term memory neural network model (LSTM) with an attention mechanism for crop effective accumulated temperature information as the second branch, using long short-term memory neural network to introduce an attention mechanism, screening the extracted high temporal resolution information, extracting deep-level information of crop accumulated temperature, constructing the relationship between the labeled crop and its environment, and establishing effective crop accumulated temperature features;

[0013] Step S8, a double-branch feature fusion module, fusing medium and low-level satellite image features and effective accumulated temperature information, and then inputting into a multi-level cross-attention module to integrate different scale contributions and capture differences, and finally obtaining aggregated effective accumulated temperature-satellite remote sensing spatio-temporal features with rich multi-scale information;

[0014] Step S9, according to the early identification strategy of incremental classification, using the training set data to generate data sets of different time lengths to input the model for training; the training process is based on Adam optimizer, cross-entropy loss function and auxiliary loss function, and finally the classifiers trained for different time sequences are obtained;

[0015] Step S10, using the test set to evaluate the accuracy of the results of different classifiers, identifying the shortest time sequence and reaching a specific accuracy of the time sequence, and regarding the acquisition time of the last image as the earliest identifiable time of the crop, so as to realize the extraction of early crop planting area at the plot level; Step S11, using the above method and the classifier used to determine the earliest identifiable time to carry out regional early crop identification.

[0016] The step S1 is specifically implemented as follows:

[0017] Step S11, obtain ground range detection (GRD) data of the target year's Sentinel-1 radar SAR image in the early to middle growth period of the crop in an interferometric wide (IW) mode, and pre-process the data, including thermal noise removal, orbit file correction, radiometric calibration, filter correction, and Doppler terrain correction, to finally obtain backscatter information;

[0018] Step S12, calculate the backscatter features of the time-series Sentinel-1 radar SAR image in VV and VH polarizations, perform time-series smoothing through SG filtering, and clip the features to the study area through plot constraint;

[0019] The step S2 is specifically implemented as follows:

[0020] Step S21, obtain Level-1C products of the target year's Sentinel-2 optical image data in the early to middle growth period of the crop, which have been processed through radiometric calibration and atmospheric correction; in view of the characteristics that optical images are easily disturbed by clouds, first, select medium-resolution time-series Sentinel-2 optical images with a cloud cover percentage and a snow cover percentage both less than 20%, and then remove the clouds by using the QA60 band marker, mosaic the obtained cloud-free images, and clip them to the target area;

[0021] Step S22, calculate the vegetation index kNDVI, ReG_RVI, ReG_NDVI, NDWI, REP, and mNDVI705 images of the study area based on the medium-resolution time-series Sentinel-2 optical images, and the calculation formulas are as follows:

[0022]

[0023]

[0024] wherein NIR represents the Sentinel-2 near-infrared band, Red represents the red band, Green represents the green band, RE2 and RE1 represent red edge 1 and red edge 2 bands respectively, and SWIR1 represents the short-wave infrared 1 band; finally, clip the features to the study area through plot constraint;

[0025] Step S23, perform importance sorting on the extracted features by using random forest (RF); calculate the feature importance based on the Gini index by using random forest (RF), select only part of the classification feature variables with high scores to avoid the interference of redundant and irrelevant features, and effectively avoid overfitting and reduce the generalization error of classification;

[0026] Suppose that for a given data set D, which contains n categories, the probability that a sample belongs to the i-th category is P i The calculation formula of the Gini index is as follows:

[0027]

[0028] Gini index, namely Gini impurity, is used to characterize the purity of the sample set, the smaller the value, the smaller the sample misclassification probability, and the value is 0 when the data set is consistent in class; if the set D is divided into k parts according to a certain feature, the Gini index after division is:

[0029]

[0030] Where S j is the number of samples at the child node j, and S is the total number of samples; when the data set traverses all the partitions, the Gini index Gini(D) before partitioning is subtracted from the Gini split (D) after partitioning to obtain the Gini gain value, the larger the value, the higher the importance, and the better the classification effect when the node is divided according to the feature;

[0031] Step S24, based on the optical index and optical band beneficial to early identification of crops, different feature combinations are formed by changing the number of feature variables, the relationship between classification accuracy and feature number is obtained, and the best combination optical feature kNDVI, NDWI, mNDVI705, Red beneficial to early identification is screened out through the contribution rate of each time phase feature.

[0032] The step S3 is specifically implemented as follows:

[0033] Step S31, obtain the daily ERA-5Daily maximum and minimum temperature data of the target year covering the early to middle growth period of crops, and mosaic and crop the obtained temperature data to the target area;

[0034] Step S32, calculate the daily effective accumulated temperature of a specific crop using the maximum and minimum values of the daily ERA-5Daily data, and the calculation formula is as follows:

[0035]

[0036] Where t represents the day in the target year, and respectively represent the minimum and maximum temperature of the i-th day (i=1, 2, 3…t) from the sowing day to the target year, and T base is the effective accumulated temperature range of the crop, which has a fixed value according to different crops, after the calculation, the image is resampled to 10m resolution to match the satellite image in space, and is cropped to the study area through the block constraint.

[0037] Step S4, when making the crop sample dataset, the method is to process each SAR, optical, effective accumulated temperature feature time series data into a mat file with a size of HxWxT, and then crop the time series image into a dataset with a size of Nx128x128xT, and the label data is Nx128x128, where N represents the number of 128x128 small blocks cropped from the study area, and T is the time series length. Finally, all features are combined into a npz (NumPy Zip) file, and the proportion of the training set and the test set is 6:4. After processing, it is used as the model input data Step S4 also specifically includes the following steps:

[0038] Step S41, based on steps S1, S2 and S3, each feature is processed into a mat file with a size of HxWxT, and the high-time-resolution effective accumulated temperature feature is aggregated to the satellite image acquisition day in the year;

[0039] Step S42, set the overlay repetition rate of each time series feature data to 0.5, crop it to a specific length and width, i.e. 128*128, and record it as a list. Normalize according to the respective features and read it as a npz file, and organize it into a four-dimensional tensor TxCxHxW, where T and C represent the time series length and the corresponding channel number respectively, and HxW is the image length and width, which is used as input data;

[0040] Step S43, based on the historical crop distribution dataset, make crop samples based on the plot, and assign the crop sample data grid value to 1. Convert the plot vector to a grid data value to 2, and the background value to 0. First, calculate the grid to assign the cotton on the plot to 1, second, adopt a voting mechanism for each plot, i.e. the largest proportion of the land class is determined as the land class, and finally, assign 2 to other crops. Organize it into a three-dimensional tensor ClassxHxW, where Class is the class, which is used as the input label data.

[0041] The step S5 is specifically implemented as follows:

[0042] Step S51, a parallel branch network model coupled with multi-source remote sensing data is constructed;

[0043] Step S52, input the time series image through the encoder, and design two parallel branches, the first branch is used to extract the medium-resolution multi-source satellite image features, and the second branch is used to extract the effective accumulated temperature data features;

[0044] Step S53, fuse the different modal feature information of the two branches in the front end of the decoder.

[0045] The step S6 is specifically implemented as follows:

[0046] Step S61, based on the STMA model framework, a satellite remote sensing data feature extraction network of double-branch ResNet-based multi-level feature extractor and space-time attention mechanism module is constructed;

[0047] Step S62, the encoder is a ResNet18-based multi-level feature front-end extractor, including a standard 7x7 convolution block, a maximum pooling layer and four residual blocks, wherein each residual block contains two 3x3 convolution operations, batch normalization BN, rectified linear unit ReLU and a skip connection layer composed of a standard 1x1 convolution operation and BN; After convolution and pooling, the spatial size of the input patch cube is reduced by 1 / 2, 1 / 4 and 1 / 8 respectively, and the channel dimension gradually increases; ResNet extracts multi-scale features in multiple receptive fields, which correspond to shallow features, middle features and high features extracted based on different spatial resolutions, respectively, and extracts local spatial features of each time in turn, and then concatenates the output features along the time dimension, thereby retaining the time dimension information and promoting the time feature extraction of the subsequent time attention module;

[0048] Step S63, the spatial attention position embedding is used before entering the space-time attention, and an embedded learnable function is used to flexibly extract important position information related to the task, and the feature cube extracted from the ResNet module is used to learn the complex position relationship;

[0049] x′ n =x n ×σ(DConv(x n ))

[0050] wherein, represents the input feature cube and the output label embedding extracted based on the CNN ResNet module; DConv(.) represents a depth convolution operation, the convolution kernel is 3x3, and the padding is 1, which extracts spatial weights from the input sequence; The depth convolution applies a convolution filter to each input channel, collects spatial information conditioned on the local neighborhood of the input label, and maintains translational equivalence while maintaining a small number of trainable parameters. By assigning different weight coefficients, each pixel in the extracted feature cube can adaptively obtain position embedding, and the spatial information is retained in the output label embedding;

[0051] Step S64, the STSA is composed of spatial self-attention, temporal self-attention and feature fusion modules; The spatial self-attention and the temporal self-attention part independently extract spatial and temporal information according to different flattened forms of the encoded feature cube. The feature fusion module contains convolution operation, LayerNorm layer LN and multilayer perceptron MLP layer, so that the concatenated features are integrated to obtain robust space-time features, and the influence of artificial intervention on the concatenation order of space-time features is avoided; First, the input label embedding x′n Flattening into two embedded sequences along spatial and temporal dimensions and Then, the flattened embedded sequences are fed into multi-head self-attention MSA along spatial and temporal axes to capture spatial and temporal information. and z

[0052] z s ′ = MSA(LN(z s ))+z s

[0053] z T ′ = MSA(LN(z T ))+z T

[0054] MSA(z) = Concat(head1,...,head h )W O

[0055]

[0056] where h is the number of heads in MSA, W O is a linear transformation matrix for feature space conversion; represent the key, query, value of each head, respectively; is a linear transformation matrix where parameters are learnable; denotes a scaling factor;

[0057] The Softmax activation function provides attention values from 0 to 1, and the self-attention matrices of spatial and temporal dimensions are obtained with the size of [H n W n , H n W n ] and [T, T], respectively, to reflect the weight distribution of different pixels and time; the feature fusion module realizes the aggregation of spatial and temporal features, concatenates the extracted features, and performs a feedforward network similar to the original transformer; the output of STSA is expressed in the following form:

[0058]

[0059] is the output spatiotemporal feature. The MLP layer consists of two fully connected layers with Gaussian Error Linear Unit (GELU) activation. is a learnable network parameter for adaptive fusion. denotes z S ′ and zT The concatenated results in the third dimension, so that the local structure information is preserved in the convolution operation, avoiding insufficient fusion. After STSA, the output spatiotemporal feature d is reshaped to a two-dimensional image space to generate a discriminative spatiotemporal feature For different multi-scale features obtained by ResNet, the STSA module can also obtain middle-level features and high-level features

[0060] The step S7 is specifically implemented as follows:

[0061] Step S71, an encoder for fully extracting deep-level information of crop effective accumulated temperature is constructed, which is composed of a 2D convolution block, an attention mechanism and an LSTM, and the LSTM introduces the attention mechanism as a filtering mechanism for extracting high temporal resolution information;

[0062] Step S72, two 2D convolution blocks of the effective accumulated temperature data encoder are responsible for dimension compression of data, and the reduced calculation pressure is input into the LSTM; Step S73, the signal output by the LSTM passes through the attention mechanism, highlights the key information of the image by calculating the image feature weight, so as to help the model make more accurate decisions, and finally outputs the extracted features through the Linear layer and the FC layer;

[0063] In step S72, the LSTM is defined as follows: each LSTM unit includes a memory unit (C t ) and three gating mechanisms, namely an input gate Input t , a forget gate Forget t and an output gate Output t . The input data of the LSTM unit at the tth time step is composed of the current unit input signal x t and the previous unit output signal h t . When the input data enters the LSTM, the input gate determines which information can enter the memory unit through the Sigmoid function; the forget gate determines which information needs to be discarded previously through the Sigmoid function. The information output of the input gate and the forget gate determines the information update of the memory unit C t , and the input data after the hyperbolic tangent function tanh is multiplied by the input gate output information, while the forget gate output information is multiplied by the previous memory unit C t-1 , and finally the sum of the two generates a new memory value C t . The output gate determines the final information output through the Sigmoid function, and the generated new memory value C t is multiplied by the output gate output information through the hyperbolic tangent function, to obtain the final output signal h twhere σ is a sigmoid function, W is a weight, and tanh represents a hyperbolic tangent function. The specific calculation process is as follows:

[0064] Input t = σ(W i [h t-1 , x t ])

[0065] Forget t = σ(W f [h t-1 , x t ])

[0066] Output t = σ(W o [h t-1 , x t ])

[0067] C' t = σ(W c [h t-1 , x t ])

[0068] C t = Multiply[Forget t , C t-1 ] + Multiply[Input t , C' t ]

[0069] h t = Multiply[Output t , tanh(C t )];

[0070] In step S72, the attention mechanism is defined as follows: the LSTM cell unit output signal h t is processed through an attention mechanism to capture important time features related to the target object. As an information screening mechanism, the attention mechanism is located between the encoder and the decoder. By calculating the image feature weight, the attention mechanism highlights the key information of the image, thereby helping the model make more accurate decisions. The attention mechanism assigns a context vector v to the input sequence data, which depends on the weighted sum of the input sequence h = {h1, h2,... h t}, and can reflect the most important information at the current time step. a ijThe attention weight is a softmax function of the model output, indicating the importance of each time step to the model output. This weight calculation method enables the model to focus on the most important part of the input sequence, thereby improving the performance and accuracy of the model. ij The vector related to the input sequence measures the importance of information, and a is a feedforward neural network.

[0071]

[0072] a ij = softmax(e ij )

[0073] e ij = a(v i-1 , h j ).

[0074] The step S8 is implemented as follows:

[0075] Step S81, a cascading feature fusion method combined with attention mechanism is used to construct a dual-branch feature fusion module to realize the fusion of features extracted by the first branch and the second branch.

[0076] Step S82, in Dual-STMA, the optical branch and the SAR branch are composed of different branches, and different branches extract different levels of spatiotemporal features through ResNet and STSA modules. The features output by the optical branch are represented as z1 Opt , z2 Opt , z3 Opt , and the features output by the SAR branch are represented as z1 SAR , z2 SAR , z3 SAR . After output by STSA, the corresponding scales are fused. The different levels of features extracted after the fusion of optical and SAR data are used to balance the number of channels by using 1×1 convolution and average pooling operation to unify the size of different multi-scale features, and z1, z2, z3 are obtained. The fusion process is as follows:

[0077] z n = AvgPool(Conv(Concat(z n Opt +z n SAR ))

[0078] Where n represents different levels.

[0079] Step S83, the accumulated temperature data passes through the 2D convolution module and the corresponding dimension transformation, and then enters the LSTM and attention mechanism, and then passes through the Linear and FC layers to extract high-level features and perform dimension matching, as the input for subsequent fusion with the satellite data.

[0080] The features z4 extracted from the ALSTM model accumulated temperature data are fused with the z3 and z2 features to form z3' and z2', which are then input into the multi-level attention module (MCA) with z1 respectively, to integrate the contributions of different scales and capture the differences between multi-scale features.

[0081]

[0082] wherein, Q, K and V represent the query, key and value of the MCA mechanism respectively. represents the outer product operation. Q and M K are the size values of Q c , K c respectively; finally, the valuable semantic information of the high-level feature z3 is added to the generated feature to obtain the aggregated temperature-satellite remote sensing spatio-temporal feature z m .

[0083] The step S9 is specifically implemented as follows:

[0084] Step S91, the entire network is trained based on the minimum principle cross-entropy (CE) loss function and the auxiliary loss function used, the minimum principle cross-entropy loss is used for training the entire network to learn the mapping from the input time series to the ground reference data, and the auxiliary loss is used to supervise the optimization process of the multi-level attention to prevent the loss of detailed information in the intermediate network layer, so that the network architecture can learn better multi-scale spatio-temporal feature representation, thereby improving the mapping performance of crops in the semantic view;

[0085]

[0086] Step S92, the model total loss is expressed as follows: α is used to balance the trade-off between the minimum principle cross-entropy loss and the auxiliary loss; the auxiliary loss function is calculated by minimizing the minimum principle cross-entropy loss between the auxiliary map y aux generated by the CNN-based decoder and the ground reference map ; only the training stage is adopted to not affect the test stage; the minimum principle cross-entropy loss function is based on considering the output crop mapping result y and the ground reference data reconstruction loss between the input data and the output data of the model; the multi-class minimum principle cross-entropy loss calculation is as the formula of step S91 above, and N represents the number of pixels; and y i indicates the one-hot encoding of the real label on the i-th image pixel and the corresponding softmax output of the model;

[0087] L All = alpha L CE + (1-alpha) L Aux .

[0088] Step S93, divide the input data into a training set and a validation set according to a ratio of 6:4; train using the training set data, use an Adam optimizer with an initial learning rate Lr as the optimizer for model training, set the training batch of the model to N, set the iteration number of the model to R, measure the goodness of the model prediction based on the above loss function, and propagate the output error back to the network parameters in each batch to fit the output of the sample; use accuracy to measure the performance of the model in each batch, and save the optimal weight file of the model when the accuracy of the validation set is the highest.

[0089] The step S10 is specifically implemented as follows:

[0090] Step S101, train the model weight file based on different time sequence lengths, crop the image according to the corresponding time sequence when splicing to generate crop identification results of different time sequence lengths;

[0091] Step S102, evaluate the crop identification results of each time sequence using ground truth samples, and use an incremental classification strategy for early identification, that is, the acquisition time of the latest image captured by the time sequence with the shortest time sequence and reaching a certain precision is regarded as the earliest identifiable time of the crop;

[0092] The step S11 is specifically implemented as follows:

[0093] Step S111, obtain Sentinel-1A, Sentinel-2, and ERA5 Daily temperature data in the application range area and the time span from the crop sowing period to the earliest identifiable time, and process them according to steps S1, S2, and S3 to form model input data;

[0094] Step S112, perform early identification on the data to be applied based on the model weight file trained based on the corresponding time sequence length of the earliest identifiable time, and obtain the extraction result of early crop identification.

[0095] The application proposes a crop early remote sensing identification method fusing effective accumulated temperature, which is a crop early identification method fusing deep learning and effective accumulated temperature, based on a spatio-temporal multi-level attention model and a long short-term memory neural network, integrates an effective accumulated temperature and a crop extraction method of multi-source remote sensing data, fuses the complementary information between SAR and optical data at a feature layer, and further fuses the effective accumulated temperature data to realize enhancement of crop environmental information, so that the model can better understand the relevance between different crop early characteristics and environment, and improve the crop early identification performance.

[0096] Compared with the traditional method, the dual-branch network fusing effective accumulated temperature and satellite remote sensing data has the advantages of fully using limited early crop growth images, comprehensively analyzing the environment and characteristics of crops, realizing fine crop early spatial distribution automatic extraction under small sample conditions, and considering the influence of multi-features and model structure on early crop identification results. BRIEF DESCRIPTION OF DRAWINGS

[0097] The application will be further described in detail below in combination with the drawings and specific embodiments:

[0098] The accompanying drawings are schematic diagrams of the method flow of the preferred embodiment of the application; Figure 1 The accompanying drawings are schematic diagrams of the data processing of the preferred embodiment of the application;

[0099] Figure 2 The accompanying drawings are schematic diagrams of the early identification method of the preferred embodiment of the application;

[0100] The accompanying drawings are schematic diagrams of the TDSTMA network model structure of the preferred embodiment of the application; Figure 3 The accompanying drawings are schematic diagrams of the feature fusion module structure of the preferred embodiment of the application;

[0101] Figure 4 The accompanying drawings are schematic diagrams of the feature fusion module structure of the preferred embodiment of the application;

[0102] The accompanying drawings are schematic diagrams of the feature fusion module structure of the preferred embodiment of the application; Figure 5 The accompanying drawings are schematic diagrams of the feature fusion module structure of the preferred embodiment of the application;

[0103] Figure 6 The accompanying drawings are schematic diagrams of the feature fusion module structure of the preferred embodiment of the application; DETAILED DESCRIPTION

[0104] The technical solutions of the application will be described in detail below in combination with the drawings.

[0105] It should be pointed out that the following detailed description is exemplary and is intended to provide further description of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as generally understood by those skilled in the art to which the present application belongs.

[0106] ​​​It is to be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments according to the present application; as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, steps, operations, elements, components, and / or groups thereof, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0107] As shown in the figure, the early remote sensing identification method of crop fused effective accumulated temperature includes the following steps:

[0108] Step S1, acquire medium-resolution time-series Sentinel-1 radar SAR images in the study area, and process to obtain farmland plot-level backscattering coefficients VV and VH features;

[0109] Step S2, acquire medium-resolution time-series Sentinel-2 optical images in the study area, calculate kNDVI, ReG_RVI, ReG_NDVI, NDWI, REP, and mNDVI705 features, and screen out the best optical feature combination: kNDVI, NDWI, mNDVI705, and Red through a random forest feature optimization algorithm.

[0110] Step S3, acquire daily ERA-5 temperature data in the study area, and calculate effective accumulated temperature T cal ;

[0111] Step S4, make a crop sample data set based on farmland plots;

[0112] Step S5, construct a dual-branch network model framework based on deep learning semantic segmentation, the first branch being a multi-level network model integrated with a spatio-temporal attention mechanism, used for extracting SAR and optical image features, and the second branch being a long short-term memory neural network model with an attention mechanism, used for extracting effective accumulated temperature features;

[0113] Step S6, use the multi-level network model with a spatio-temporal attention mechanism as the first branch, used for mining multi-scale spatio-temporal information of SAR and optical medium-resolution images; this branch is composed of two parallel multi-level feature extractors based on ResNet and a spatio-temporal attention mechanism module, SAR and optical data are input in parallel, spatial features are extracted through different receptive fields, and the spatio-temporal attention mechanism module is used to further extract multi-level spatio-temporal features, and the extracted high-level features of SAR and optical images are spliced and fused according to different levels;

[0114] Step S7, a long short-term memory neural network model (LSTM) with an attention mechanism for crop effective accumulated temperature information is used as a second branch, the long short-term memory neural network is used to introduce an attention mechanism, the extracted high temporal resolution information is screened, deep information of crop accumulated temperature is extracted, the relationship between the labeled crop and its environment is constructed, and effective crop accumulated temperature features are established;

[0115] Step S8, a double-branch feature fusion module is used to fuse the satellite image features and effective accumulated temperature information at the middle and low levels, and then input the multi-level cross-attention module to integrate different scale contributions and capture differences, so that the aggregated effective accumulated temperature-satellite remote sensing spatio-temporal features with rich multi-scale information are obtained;

[0116] Step S9, according to the early identification strategy of incremental classification, different time length datasets are generated by using the training set data to input the model for training; the training process is based on the Adam optimizer, the cross-entropy loss function and the auxiliary loss function, and finally the classifiers trained for different time sequences are obtained;

[0117] Step S10, the results of different classifiers are evaluated by using the test set, the shortest time sequence is identified and reaches a specific accuracy, and the acquisition time of the last image is regarded as the earliest identifiable time of the crop, so that the early crop planting area extraction at the plot level is realized; step S11, the above method and the classifier used to determine the earliest identifiable time are used to carry out regional early crop identification.

[0118] The step S1 is specifically implemented as follows:

[0119] Step S11, the ground range detection (GRD) data of the Sentinel-1 radar SAR image in the interference wide (IW) mode during the early to middle growth period of the target year covering crop is obtained, and the data is preprocessed, including thermal noise removal, orbit file correction, radiation scaling, filter correction and Doppler terrain correction, and finally the backscatter information is obtained;

[0120] Step S12, the backscatter features of the time sequence Sentinel-1 radar SAR image VV and VH polarization are calculated, time sequence smoothing is performed through SG filtering, and the features are cropped to the research area through plot constraint;

[0121] The step S2 is specifically implemented as follows:

[0122] Step S21, obtain the Level-1C product of the Sentinel-2 optical image data of the target year covering the early to middle growth period of the crop, which is processed by radiation calibration and atmospheric correction; in view of the characteristics that the optical image is easily disturbed by clouds, first, the medium resolution time series Sentinel-2 optical image with the cloud cover and snow cover percentage both less than 20% is screened, then the cloud is removed by using the QA60 band mark, and the cloud-free image is mosaicked and cropped to the target area;

[0123] Step S22, calculate the vegetation index kNDVI, ReG_RVI, ReG_NDVI, NDWI, REP, mNDVI705 images of the study area based on the medium resolution time series Sentinel-2 optical image, and the calculation formula is as follows:

[0124]

[0125] Wherein, NIR represents the Sentinel-2 near-infrared band, Red represents the red band, Green represents the green band, RE2 and RE1 represent the red edge 1 and red edge 2 bands respectively, and SWIR1 represents the short wave infrared 1 band; finally, it is cropped to the study area by block constraint;

[0126] Step S23, the importance of the extracted features is sorted by using random forest RF; the feature importance is calculated based on the Gini index by using random forest RF, and only some classification feature variables with high scores are selected to avoid the interference of redundant and irrelevant features, and to effectively avoid overfitting and reduce the generalization error of classification;

[0127] Suppose for a given data set D, which contains n categories, the probability that a sample belongs to the i-th category is P i , then the calculation formula of the Gini index is:

[0128]

[0129] The Gini index, i.e. the Gini impurity, is used to represent the purity of the sample set, the smaller the value, the smaller the probability of misclassification, and when the value is 0, the data set is consistent in category; if the set D is divided into k parts according to a certain feature, then the Gini index after division is:

[0130]

[0131] Where S j is the number of samples at the j-th node, and S is the total number of samples; when the data set traverses all the divisions, the Gini index before division Gini(D) is subtracted from Gini split (D) to obtain the Gini gain value, the greater the value, the higher the importance, and the better the classification effect when the node is divided according to the feature;

[0132] Step S24, based on the optical index and optical band conducive to early identification of crops, different feature combinations are formed by changing the number of feature variables, the relationship between classification accuracy and feature number is obtained, and the contribution rate of each time phase feature is accumulated to screen out the best combination of optical features kNDVI, NDWI, mNDVI705, Red conducive to early identification.

[0133] The step S3 is specifically implemented as follows:

[0134] Step S31, obtain the daily ERA-5Daily maximum and minimum temperature data of the target year covering the early to middle growth period of the crop, and splice and crop the obtained temperature data to the target area;

[0135] Step S32, using the maximum and minimum values of the daily ERA-5Daily data, calculate the daily effective accumulated temperature of the specific crop, and the calculation formula is as follows:

[0136]

[0137] Where, t represents the day in the target year, And respectively represent the minimum and maximum temperature of the i-th day (i=1, 2, 3…t) from the sowing day to the target year, and T base is the effective accumulated temperature range of the crop, which has a fixed value according to different crops, and after the calculation, the image is resampled to 10m resolution to match the satellite image in space, and is cropped to the study area by block constraint.

[0138] Step S4, when making the crop sample data set, the method is: process each SAR, optical, and effective accumulated temperature feature time series data into a mat file of size HxWxT, then crop the time series image into a data set of size Nx128x128xT, and the label data is Nx128x128. Wherein N represents the number of 128x128 small blocks cropped from the study area, and T is the time series length. Finally, all features are merged into a npz (NumPy Zip) file, wherein the proportion of training set and test set is 6:4, and after processing, it is used as model input data Step S4 further comprises the following steps:

[0139] Step S41, based on steps S1, S2, and S3, process each feature into a mat file of size HxWxT, and aggregate the high time resolution effective accumulated temperature feature to the mid-year day obtained from the satellite image.

[0140] Step S42, set the overlay of each time series feature data repetition rate to 0.5, crop to a specific length and width, i.e. 128*128, and record as a list, normalize according to the respective features and read as an npz file, and organize into a four-dimensional tensor TxCxHxW, where T and C represent the time series length and the corresponding channel number respectively, and HxW is the image length and width, as input data;

[0141] Step S43, crop sample production based on historical crop distribution data set, crop sample data grid value assignment is 1, and the block vector is converted to grid data assignment 2, and the background value is 0. First, the grid calculation is performed to assign the cotton on the block as 1, secondly, the voting mechanism is adopted for each block, i.e. the largest proportion of land class is determined as the land class, and finally, the value of 2 is assigned to other crops, and organized into a three-dimensional tensor ClassxHxW, where Class is the class, as the input label data.

[0142] The step S5 is specifically implemented as follows:

[0143] Step S51, a parallel branch network model coupled with multi-source remote sensing data is constructed;

[0144] Step S52, input time series images are transmitted through an encoder, and two parallel branches are designed, the first branch is used to extract medium resolution multi-source satellite image features, and the second branch is used to extract effective accumulated temperature data features;

[0145] Step S53, different modal feature information of the two branches is fused in the front end of the decoder.

[0146] The step S6 is specifically implemented as follows:

[0147] Step S61, based on the STMA model framework, a satellite remote sensing data feature extraction network of a double-branch multi-level feature extractor based on ResNet and a spatiotemporal attention mechanism module is constructed;

[0148] Step S62, the encoder is a multi-level feature front-end extractor based on ResNet18, including a standard 7x7 convolution block, a max pooling layer and four residual blocks, wherein each residual block contains two 3x3 convolution operations, batch normalization BN, rectified linear unit ReLU and a skip connection layer composed of a standard 1x1 convolution operation and BN; after convolution and pooling, the spatial size of the input patch cube is reduced by 1 / 2, 1 / 4 and 1 / 8 respectively, and the channel dimension gradually increases; ResNet extracts multi-scale features in multiple receptive fields, which correspond to shallow features, middle features and high features extracted based on different spatial resolutions, respectively, and then the output features are concatenated along the time dimension, so as to retain the time dimension information and promote the time feature extraction of the subsequent time attention module;

[0149] Step S63, the spatial attention position embedding is used before entering the spatio-temporal attention, and an embedded learnable function is used to flexibly extract important position information related to the task, and the complex position relationship is learned from the feature cube extracted from the ResNet module;

[0150] x′ n =x n ×σ(DConv(x n ))

[0151] wherein, indicates the input feature cube and output label embedding extracted based on the ResNet module of the CNN; DConv(.) represents a deep convolution operation, the convolution kernel is 3x3, the padding is 1, and the spatial weight is extracted from the input sequence; the deep convolution applies a convolution filter to each input channel, collects the spatial information conditioned on the local neighborhood of the input label, and maintains the translational equivalence while keeping a small number of trainable parameters. By assigning different weight coefficients, each pixel in the extracted feature cube can adaptively obtain position embedding, and the spatial information is retained in the output label embedding;

[0152] Step S64, the STSA is composed of spatial self-attention, temporal self-attention and feature fusion modules; the spatial self-attention and the temporal self-attention part independently extract spatial and temporal information according to different flat forms of the encoded feature cube. The feature fusion module contains convolution operation, LayerNorm layer LN and multilayer perceptron MLP layer, so that the integrated concatenated features can obtain robust spatio-temporal features, and the influence of artificial intervention on the concatenation order of spatio-temporal features is avoided; first, the input label embedding x′ n is flattened along the spatial and temporal dimensions into two embedding sequences and Then, the flattened embedded sequence is fed into multi-head self-attention MSA along the spatial and temporal axes to capture spatial and temporal information. and denote the spatial and temporal features of the spatial and temporal self-attention outputs, respectively:

[0153] z s ′ = MSA(LN(z S ))+z S

[0154] z T ′ = MSA(LN(z T ))+z T

[0155] MSA(z) = Concat(head1,..., head h W O

[0156]

[0157] where h is the number of heads in MSA, W O is a linear transformation matrix for feature space conversion; denote the key, query, and value of each head, respectively; is a linear transformation matrix where parameters are learnable; denotes a scaling factor;

[0158] The Softmax activation function provides attention values between 0 and 1, and the self-attention matrices of the spatial and temporal dimensions are obtained with sizes of [H n W n , H n W n ] and [T, T], respectively, to reflect the weight distribution of different pixels and times; the feature fusion module realizes the aggregation of spatial and temporal features, concatenates the extracted features, and performs a feedforward network similar to the original transformer; the output of STSA is expressed in the following form:

[0159]

[0160] is the output spatiotemporal feature. The MLP layer consists of two fully connected layers with Gaussian Error Linear Unit (GELU) activation. is a learnable network parameter for adaptive fusion. denote z S ′ and z TThe concatenated results in the third dimension, so that the local structure information is preserved in the convolution operation, and insufficient fusion is avoided. After STSA, the output spatiotemporal feature d is reshaped to a two-dimensional image space to generate a discriminative spatiotemporal feature For different multi-scale features obtained by ResNet, the STSA module can also obtain middle-level features and high-level features

[0161] The step S7 is specifically implemented as follows:

[0162] Step S71, an encoder for fully extracting deep-level information of crop effective accumulated temperature is constructed, which is composed of a 2D convolution block, an attention mechanism and an LSTM, and the LSTM introduces the attention mechanism as a filtering mechanism for extracting high temporal resolution information;

[0163] Step S72, two 2D convolution blocks of the effective accumulated temperature data encoder are responsible for dimension compression of data, and the reduced calculation pressure is input into the LSTM; Step S73, the signal output by the LSTM passes through the attention mechanism, highlights the key information of the image by calculating the image feature weight, so as to help the model make more accurate decisions, and finally outputs the extracted features through the Linear layer and the FC layer;

[0164] In step S72, the LSTM is defined as follows: each LSTM unit includes a memory unit (C t ) and three gating mechanisms, namely an input gate Input t , a forget gate Forget t and an output gate Output t . The input data of the LSTM unit at the tth time step is composed of the current unit input signal x t and the previous unit output signal h t . When the input data enters the LSTM, the input gate determines which information can enter the memory unit through the Sigmoid function; the forget gate determines which information needs to be discarded in the previous step through the Sigmoid function. The information output of the input gate and the forget gate determines the information update of the memory unit C t , and the input data after the hyperbolic tangent function tanh is multiplied by the input gate output information, while the forget gate output information is multiplied by the previous memory unit C t-1 , and finally the sum of the two generates a new memory value C t . The output gate determines the final information output through the Sigmoid function, and the generated new memory value C t is multiplied by the output gate output information through the hyperbolic tangent function, to obtain the final output signal h twhere σ is a sigmoid function, W is a weight, and tanh represents a hyperbolic tangent function. The specific calculation process is as follows:

[0165] Input t = σ(W i [h t-1 , x t ])

[0166] Forget t = σ(W f [h t-1 , x t ])

[0167] Output t = σ(W o [h t-1 , x t ])

[0168] C′ t = σ(W c [h t-1 , x t )

[0169] C t = Multiply[Forget t C t-1 ]+ Multiply[Input t , C′ t ]

[0170] h t = Multiply[Output t , tanh(C t )];

[0171] In step S72, the attention mechanism is defined as follows: the LSTM cell unit output signal h t is processed through an attention mechanism to capture important time features related to the target object. As an information screening mechanism, the attention mechanism is located between the encoder and the decoder. By calculating the image feature weight, the attention mechanism highlights the key information of the image, thereby helping the model make more accurate decisions. The attention mechanism assigns a context vector v to the input sequence data, which depends on the weighted sum of the input sequence h = {h1, h2,... h t}, and can reflect the most important information in the current time step. a ijThe attention weight is a softmax function of the model output, indicating the importance of each time step to the model output. This weight calculation method enables the model to focus on the most important part of the input sequence, thereby improving the performance and accuracy of the model. ij The vector related to the input sequence measures the importance of information, and a is a feedforward neural network.

[0172]

[0173] a ij = softmax(e ij )

[0174] e ij = a(v i-1 , h j ).

[0175] The step S8 is implemented as follows:

[0176] Step S81, a cascading feature fusion method combined with attention mechanism is used to construct a dual-branch feature fusion module to realize the fusion of features extracted by the first branch and the second branch.

[0177] Step S82, in Dual-STMA, the optical branch and the SAR branch are composed, different branches extract different levels of spatio-temporal features through ResNet and STSA modules respectively, the features output by the optical branch are represented as z1 Opt , z2 Opt , z3 Opt , and the features output by the SAR branch are represented as z1 SAR , z2 SAR , z3 SAR After output by STSA, the corresponding scales are fused, and the different levels of features extracted after the fusion of optical and SAR data are used to balance the number of channels by using 1×1 convolution and average pooling operation to unify the size of different multi-scale features, and z1, z2, z3 are obtained, the fusion process is represented as follows:

[0178] z n = AvgPool(Conv(Concat(z n Opt +z n SAR ))

[0179] Where n represents different levels.

[0180] Step S83, the accumulated temperature data passes through the 2D convolution module and the corresponding dimension transformation, and then enters the LSTM and attention mechanism, and then passes through the Linear and FC layers to extract high-level features and perform dimension matching, as the input for subsequent fusion with the satellite data.

[0181] The features z4 extracted from the ALSTM model accumulated temperature data are fused with the z3 and z2 features to form z3' and z2', which are then input into the multi-level attention module (MCA) with z1 respectively, to integrate the contributions of different scales and capture the differences between multi-scale features.

[0182]

[0183] wherein, Q, K and V represent the query, key and value of the MCA mechanism respectively. represents the outer product operation. Q and M K are the size values of Q c , K c ; finally, the valuable semantic information of the high-level feature z3 is added to the generated feature to obtain the aggregated temperature-satellite remote sensing spatio-temporal feature z m with rich multi-scale information.

[0184] The step S9 is specifically implemented as follows:

[0185] Step S91, the entire network is trained based on the minimum principle cross-entropy (CE) loss function and the auxiliary loss function used, the minimum principle cross-entropy loss is used for training the entire network to learn the mapping from the input time series to the ground reference data, and the auxiliary loss is used to supervise the optimization process of the multi-level attention to prevent the loss of detailed information in the intermediate network layer, so that the network architecture can learn better multi-scale spatio-temporal feature representation, thereby improving the mapping performance of crops in the semantic view;

[0186]

[0187] Step S92, the model total loss is expressed as follows: α is used to balance the trade-off between the minimum principle cross-entropy loss and the auxiliary loss; the auxiliary loss function is calculated by minimizing the minimum principle cross-entropy loss between the auxiliary map y aux generated by the CNN-based decoder and the ground reference map ; only the training stage is adopted to not affect the test stage; the minimum principle cross-entropy loss function is based on considering the output crop mapping result y and the ground reference data reconstruction loss between the two; the multi-class minimum principle cross-entropy loss calculation minimizes the principle cross-entropy and auxiliary loss, and N represents the number of pixels; and y i indicates the one-hot encoding of the real label on the i-th image pixel and the corresponding softmax output of the model;

[0188] L All = alpha L CE + (1-alpha) L Aux .

[0189] Step S93, divide the input data into a training set and a validation set according to a ratio of 6:4; train using the training set data, use an Adam optimizer with an initial learning rate Lr as the optimizer for model training, set the training batch of the model to N, set the iteration number of the model to R, measure the goodness of the model prediction based on the above loss function, and propagate the output error back to the network parameters in each batch to fit the output of the sample; use accuracy to measure the performance of the model in each batch, and save the optimal weight file of the model when the accuracy of the validation set is the highest.

[0190] The step S10 is specifically implemented as follows:

[0191] Step S101, train the model weight file based on different time sequence lengths, crop the image according to the corresponding time sequence when splicing to generate crop identification results of different time sequence lengths;

[0192] Step S102, evaluate the crop identification results of each time sequence using ground truth samples, and use an incremental classification strategy for early identification, that is, the acquisition time of the latest image captured by the time sequence with the shortest time sequence and reaching a certain precision is regarded as the earliest identifiable time of the crop;

[0193] The step S11 is specifically implemented as follows:

[0194] Step S111, obtain Sentinel-1A, Sentinel-2, and ERA5 Daily temperature data in the application range area and the time span from the crop sowing period to the earliest identifiable time, and process them according to steps S1, S2, and S3 to form model input data;

[0195] Step S112, perform early identification on the data to be applied based on the model weight file trained based on the corresponding time sequence length of the earliest identifiable time, and obtain the extraction result of early crop identification.

[0196] As Figure 1 shown, it is a method flowchart of the preferred embodiment of the application.

[0197] As Figure 2As shown in the figure, it is a data processing schematic diagram of the embodiment.

[0198] As shown in the figure, it is a data processing schematic diagram of the embodiment. Figure 3 As shown in the figure, it is an early identification method schematic diagram of the embodiment.

[0199] As shown in the figure, it is an early identification method schematic diagram of the embodiment. Figure 4 As shown in the figure, it is a TDSTMA network model structure diagram of the embodiment.

[0200] As shown in the figure, it is a feature fusion module structure diagram of the embodiment. Figure 5 As shown in the figure, it is a feature fusion module structure diagram of the embodiment.

[0201] As shown in the figure, it is a feature fusion module structure diagram of the embodiment. Figure 6 As shown in the figure, it is a result diagram obtained by using the proposed multi-branch network model and using different branches and corresponding data to extract cotton distribution information in the embodiment. As can be seen from the figure, when only using the SAR data set, cotton can be effectively extracted, but there are still some false extraction and missing extraction phenomena; when only using the optical data set, the missing extraction phenomenon is reduced; when using the optical + SAR data set, the false extraction phenomenon is obviously reduced; and when the accumulated temperature data is added, the effect is the best. Therefore, by gradually introducing multi-source satellite remote sensing images and effective accumulated temperature data, the model can make full use of the complementarity of various information, so that the false extraction is reduced.

[0202] The crop extraction method integrating effective accumulated temperature and multi-source remote sensing data of the application is early crop identification, a spatiotemporal multi-level attention model and a long short-term memory neural network are proposed, the complementary information between SAR and optical data is fully utilized, fusion is performed at the feature layer, and further fusion is performed with effective accumulated temperature data to realize enhancement of crop environmental information, so that the model better understands the relevance of different crop early characteristics and environment, and the crop early identification performance is improved. Compared with the traditional method, the double-branch network integrating effective accumulated temperature and satellite remote sensing data has the advantages that limited early crop growth images can be fully used, the crop environment and its own characteristics are more comprehensively analyzed, fine crop early spatial distribution automatic extraction is realized under the condition of small samples, and the method of the application considers the influence of multiple features and model structures on early crop identification results.

[0203] The above is the preferred embodiment of the application, and any changes made according to the technical solutions of the application, as long as the generated function does not exceed the scope of the technical solutions of the application, belong to the protection scope of the application.

Claims

1. A method for early remote sensing identification of crops that integrates effective accumulated temperature, characterized in that: Includes the following steps: Step S1: Acquire medium-resolution time-series radar SAR images of the study area and process them to obtain the backscattering coefficients VV and VH characteristics of cultivated land plots. Step S2: Obtain medium-resolution temporal optical images of the study area, calculate the features kNDVI, ReG_RVI, ReG_NDVI, NDWI, REG, and mNDVI705, and select the best combination of optical features: kNDVI, NDWI, mNDVI705, and Red using a random forest feature optimization algorithm. Step S3: Obtain daily ERA-5 temperature data for the study area and calculate the effective accumulated temperature T. cal ; Step S4: Create a crop sample dataset based on cultivated land plots; Step S5: Construct a dual-branch network model framework based on deep learning semantic segmentation. The first branch is a multi-level network model that integrates spatiotemporal attention mechanisms to extract SAR and optical image features. The second branch is a long short-term memory neural network model with attention mechanisms to extract effective accumulated temperature features. Step S6: The first branch is a multi-level network model with a spatiotemporal attention mechanism, which is used to mine multi-scale spatiotemporal information of SAR and optical medium-resolution images. This branch consists of two parallel ResNet-based multi-level feature extractors and a spatiotemporal attention mechanism module. SAR and optical data are input in parallel and spatial features are extracted through different receptive fields. The spatiotemporal attention mechanism module is used to further extract multi-level spatiotemporal features and stitch and fuse the extracted SAR and optical image high-level features according to different levels. Step S7: Using the Long Short-Term Memory Neural Network (LSTM) model with attention mechanism for crop effective accumulated temperature information as the second branch, the LSTM neural network is used to introduce the attention mechanism, filter the extracted high temporal resolution information, extract deep-level information of crop accumulated temperature, construct the relationship between the labeled crop and its environment, and establish effective crop accumulated temperature features. Step S8: The dual-branch feature fusion module fuses the mid-to-low-level satellite image features with the effective accumulated temperature information, and then inputs them into the multi-level cross-attention module to integrate contributions from different scales and capture differences, ultimately obtaining aggregated effective accumulated temperature-satellite remote sensing spatiotemporal features with rich multi-scale information. Step S9: Based on the early identification strategy of incremental classification, use the training set data to generate datasets of different time lengths and input them into the model for training. The training process is based on the Adam optimizer, cross-entropy loss function, and auxiliary loss function, and finally obtains classifiers trained for different time series lengths; Step S10: Use the test set to evaluate the accuracy of the results of different classifiers, identify the shortest time series that reaches a specific accuracy, and the acquisition time of the last image of the shortest time series is regarded as the earliest identifiable time of the crop, thereby realizing the extraction of early crop planting areas at the plot level. Step S11: Using the above methods and the classifier determined as the earliest identifiable time, carry out early identification of regional crops.

2. The method for early remote sensing identification of crops based on fused effective accumulated temperature according to claim 1, characterized in that: The image acquired in step S1 is a medium-resolution time-series Sentinel-1 radar SAR image of the study area. Step S1 is specifically implemented as follows: Step S11: Obtain ground range detection GRD data in interferometric wide-swath IW mode of Sentinel-1 radar SAR imagery covering early to mid-crop growth of the target year. Perform preprocessing on the data, including thermal noise removal, orbit file correction, radiometric calibration, filter correction, and Doppler terrain correction, and finally obtain backscatter information. Step S12: Calculate the VV and VH polarization backscattering characteristics of the time-series Sentinel-1 radar SAR image, perform time-series smoothing through SG filtering, and crop it to the study area through plot constraints. The image acquired in step S2 is a medium-resolution temporal Sentinel-2 optical image of the study area. Step S2 is implemented as follows: Step S21: Obtain Level-1C products of Sentinel-2 optical image data covering early to mid-stage crop growth in the target year. These products have undergone radiometric calibration and atmospheric correction. Given that optical images are susceptible to cloud interference, medium-resolution time-series Sentinel-2 optical images with cloud cover and snow cover percentages below 20% are first selected. Then, cloud removal is achieved using QA60 band labeling. The obtained cloudless images are then mosaicked and cropped to the target area. Step S22: Calculate the vegetation indices kNDVI, ReG_RVI, ReG_NDVI, NDWI, REG, and mNDVI705 images of the study area based on medium-resolution temporal Sentinel-2 optical images. The calculation formula is as follows: Wherein, NIR represents the Sentinel-2 near-infrared band, Red represents the red band, Green represents the green band, RE2 and RE1 represent the red edge 1 and red edge 2 bands respectively, and SWIR1 represents the shortwave infrared 1 band; finally, it is clipped to the study area by plot constraints; Step S23: Random Forest (RF) is used to rank the importance of the extracted features. Using RF, the importance of features is calculated based on the Gini index. By selecting only some of the top-scoring categorical feature variables, the interference of redundant and irrelevant features is avoided, and overfitting is effectively avoided and the generalization error of the classification is reduced. Suppose that for a given dataset D containing n classes, the probability of a sample belonging to class i is P. i The formula for calculating the Gini index is: The Gini index, also known as Gini impurity, is used to characterize the purity of a sample set. The smaller the value, the lower the probability of misclassification. A value of 0 indicates that the data set is of the same category. If a set D is divided into k parts based on a certain feature, the Gini index after segmentation is: Where S j S is the number of samples at child node j, and S is the total number of samples. When traversing all partitions of the dataset, the Gini index after the partition is subtracted from the Gini value after the partition. split (D) Obtain the Gini gain value. The larger the value, the higher the importance, and the better the classification effect when segmenting nodes based on features. Step S24: Based on optical indices and optical bands that are conducive to early crop identification, different feature combinations are formed by changing the number of feature variables to obtain the relationship between classification accuracy and the number of features. The contribution rate of each phase feature is accumulated to select the best combination of optical features kNDVI, NDWI, mNDVI705, and Red that are conducive to early identification.

3. The method for early remote sensing identification of crops based on effective accumulated temperature according to claim 1, characterized in that: Step S3 is implemented as follows: Step S31: Obtain the maximum and minimum daily ERA-5 Daily temperature data for the early to mid-stage growth of the target year's covered crops, mosaic the obtained temperature data, and crop it to the target area; Step S32: Using the maximum and minimum values ​​of the daily ERA-5Daily data, calculate the effective accumulated temperature for a specific crop each day. The calculation formula is as follows: Where t represents the mid-year date of the target year. and T represents the minimum and maximum temperatures from the sowing date to the i-th day of the target year, i = 1, 2, 3…t. base The effective accumulated temperature range for crops has a fixed value depending on the crop. After calculation, the image is resampled to a 10m resolution to match the satellite image spatially, and then cropped to the study area by plot constraints.

4. The method for early remote sensing identification of crops based on fused effective accumulated temperature according to claim 1, characterized in that: Step S4 involves creating the crop sample dataset as follows: Each SAR, optical, and effective accumulated temperature feature time-series data is processed into a .mat file of size H×W×T. The time-series images are then cropped into a dataset of size N×128×128×T, with label data of size N×128×128, where N represents the number of 128×128 patches cropped from the study area, and T is the time series length. Finally, all features are merged into an .npz file, with a training set to test set ratio of 6:4, and this processed data is used as the model input data. Step S4 also specifies Includes the following steps; Step S41: Based on steps S1, S2, and S3, each feature is processed into a mat file of the form H×W×T, and the effective accumulated temperature features with high temporal resolution are aggregated to the mid-year day acquired from satellite imagery. Step S42: Set the overlay of each time series feature data repetition rate to 0.5, crop it to 128*128, and record it as a list. Normalize it according to its own features and read it as an npz file. Organize it into a four-dimensional tensor T×C×H×W, where T and C represent the time series length and the corresponding number of channels, respectively, and H×W are the image length and width, which are used as input data. Step S43: Based on the historical crop distribution dataset, crop samples are created based on plots. The raster value of the crop sample data is assigned as 1, the plot vector is converted to raster data and assigned as 2, and the background value is 0. First, raster calculation is performed to assign a value of 1 to cotton on the plot. Then, a voting mechanism is adopted for each plot, that is, the plot with the largest proportion of land type is determined to be that land type. Finally, the plots assigned as 2 are other crops. The data is organized into a three-dimensional tensor Class×H×W, where Class is the category and serves as the input label data.

5. The method for early remote sensing identification of crops based on fused effective accumulated temperature according to claim 1, characterized in that: Step S5 is implemented as follows: Step S51: Construct a parallel branch network model that couples multi-source remote sensing data; Step S52: Input time-series images are transmitted through an encoder. Two parallel branches are designed: the first branch is used to extract features of medium-resolution multi-source satellite images, and the second branch is used to extract features of effective accumulated temperature data. Step S53: Fuse the different modal feature information of the two branches at the front end of the decoder.

6. The method for early remote sensing identification of crops based on fused effective accumulated temperature according to claim 1, characterized in that: Step S6 is implemented as follows: Step S61: Based on the STMA model framework, construct a satellite remote sensing data feature extraction network with a dual-branch ResNet-based multi-level feature extractor and a spatiotemporal attention mechanism module. Step S62: The encoder is a multi-level feature front-end extractor based on ResNet18, including a standard 7×7 convolutional block, a max pooling layer, and four residual blocks. Each residual block contains two 3×3 convolution operations, batch normalization (BN), rectified linear units (ReLU), and a skip connection layer consisting of a standard 1×1 convolution operation and BN. After convolution and pooling, the spatial size of the input patch cube is reduced by 1 / 2, 1 / 4, and 1 / 8, respectively, while the channel dimension gradually increases. ResNet extracts multi-scale features in multiple receptive fields, corresponding to shallow, mid-level, and high-level features extracted based on different spatial resolutions. Local spatial features at each time are extracted sequentially, and then the output features are concatenated along the time dimension to preserve time dimension information and facilitate the subsequent time attention module to extract time features. Step S63: Before entering the spatiotemporal attention phase, spatial attention position embedding is used. Embedded learnable functions are used to flexibly extract important positional information related to the task and learn complex positional relationships from the feature cube extracted by the ResNet module. x′ n =x n ×σ(DConv(x n )) in, This represents the input feature cube and output label embedding extracted by the ResNet module based on CNN; DConv(.) represents the depthwise convolution operation with a 3×3 kernel and padding of 1, which extracts spatial weights from the input sequence; depthwise convolution applies a convolutional filter to each input channel, collects spatial information conditioned on the local neighborhood of the input label, and maintains translation equivalence while keeping small trainable parameters; by assigning different weight coefficients, each pixel in the extracted feature cube can adaptively obtain a positional embedding, preserving spatial information in the output label embedding; Step S64, STSA consists of spatial self-attention, temporal self-attention, and feature fusion modules; the spatial and temporal self-attention parts independently extract spatial and temporal information according to different flattening forms of the encoded feature cube; the feature fusion module includes convolution operations, LayerNorm (LN) layers, and Multilayer Perceptron (MLP) layers, enabling the integration of concatenated features to obtain robust spatiotemporal features and avoiding the influence of manual intervention on the concatenation order of spatiotemporal features; firstly, the input label embedding x′ n Flattened into two embedding sequences along the spatial and temporal dimensions and Then, the flattened embedding sequence is fed into multi-head self-attention MSA along the spatial and temporal axes to capture spatial and temporal information; and Represent the spatial and temporal characteristics of the outputs of spatial self-attention and temporal self-attention, respectively: z S ′=MSA(LN(z S ))+z S z T ′=MSA(LN(z T ))+z T MSA(z)=Concat(head1,…,head h )W O Where h is the number of headers in the MSA, W O It is a linear transformation matrix used for feature space transformation; These represent the key, query, and value of each header, respectively. It is a linear transformation matrix in which the parameters are learnable; Indicates the scaling factor; The Softmax activation function provides attention values ​​from 0 to 1, and the self-attention matrices in the spatial and temporal dimensions are respectively set to [H]. n W n H n W n The magnitudes of [] and [T, T] are obtained to reflect the weight distribution of different pixels and times; the feature fusion module aggregates spatial and temporal features, concatenates the extracted features, and executes a feedforward network; the output of STSA is expressed in the following form: It represents the spatiotemporal characteristics of the output; the MLP layer consists of two fully connected layers, activated by Gaussian error linear units (GELUs); These are learnable network parameters used for adaptive fusion; Represents z S ′ and z T The concatenation result in the third dimension preserves local structural information during convolution operations, avoiding insufficient fusion; after STSA, the output spatiotemporal feature d is reshaped into a two-dimensional image space to generate discriminative spatiotemporal features. For the different multi-scale features obtained by ResNet, the STSA module also obtains mid-level features. and high-level characteristics 7. The method for early remote sensing identification of crops based on fused effective accumulated temperature according to claim 1, characterized in that: Step S7 is implemented as follows: Step S71: Construct an encoder that fully extracts deep information on the effective accumulated temperature of crops. It consists of 2D convolutional blocks, an attention mechanism, and an LSTM. The LSTM introduces an attention mechanism as a filtering mechanism for extracting high temporal resolution information. Step S72: The effective accumulated temperature data encoder uses two 2D convolutional blocks to compress the data in dimensions, reducing computational pressure before inputting it into the LSTM. Step S73: The signal output by the LSTM is passed through the attention mechanism to highlight key information of the image by calculating image feature weights, thereby helping the model to make more accurate decisions. Finally, the extracted features are output through the Linear layer and the FC layer. In step S72, the LSTM is defined as follows: each LSTM unit includes a memory unit C. t And three gating mechanisms, namely the input gate. t Forget Gate t and Output gate t The input data of the LSTM cell at time step t is determined by the current cell input signal x. t and the previous unit output signal h t Structure: When input data enters the LSTM, the input gate uses a sigmoid function to determine which information can enter the memory cell; the forget gate uses a sigmoid function to determine which previous information needs to be discarded; the outputs of the input and forget gates determine the memory cell c. t Information updates are performed by multiplying the input data by the hyperbolic tangent function tanh and the output information of the input gate, while the output information of the forget gate is multiplied by the previous memory unit C. t-1 Multiply them, and then sum them to generate a new memory value C. t The output gate determines the final information output through the Sigmoid function and generates a new memory value C. t The final output signal h is obtained by multiplying the hyperbolic tangent function with the output information of the output gate. t In the formula, σ is the Sigmoid function, w is the weight, and tanh represents the hyperbolic tangent function; the specific calculation process is as follows: Input t =σ(W i [h t-1 ,x t ]) Forget t =σ(W f [h t-1 ,x t ]) Output t =σ(W o [h t-1 ,x t ]) C′ t =σ(W c [h t-1 ,x t ]) C t =Multiply[Forget t ,C t-1 ]+Multiply[Input t ,C′ t ]h t =Multiply[Output t ,tanh(C t )]; In step S72, the attention mechanism is defined as follows: the LSTM cell unit output signal h t Processing through attention mechanisms to capture important temporal features relevant to the target object; Attention, as an information filtering mechanism, sits between the encoder and decoder. It highlights key information in an image by calculating image feature weights, thereby helping the model make more accurate decisions. The attention mechanism assigns a context vector v to the input sequence data, which depends on the input sequence h = {h1, h2, ... h...}. t The weighted sum of} reflects the most important information in the current time step; a ij The attention weights are the model output of a softmax function, representing the importance of each time step to the model output. This weighting method allows the model to focus on the most important parts of the input sequence, thereby improving model performance and accuracy; ij 'a' is a vector associated with the input sequence, which measures the importance of the information; 'a' is a feedforward neural network. a ij =softmax(e ij ) e ij =a(v i-1 ,h j )。 8. The method for early remote sensing identification of crops based on effective accumulated temperature according to claim 1, characterized in that: Step S8 is implemented as follows: Step S81: Construct a dual-branch feature fusion module using a cascaded feature fusion method that incorporates an attention mechanism to achieve the fusion of features extracted by the first branch and the second branch; Step S82: Dual-STMA consists of an optical branch and a SAR branch. Different branches extract different levels of spatiotemporal features via ResNet and STSA modules, respectively. The features output by the optical branch are represented as z1. Opt z2 Opt z3 Opt The SAR branch output feature is represented as z1 SAR z2 SAR z3 SAR After being output by STSA, the data is fused according to the corresponding scale. The features extracted from the optical and SAR data at different levels are fused. To unify the size of the features at different multi-scales, 1×1 convolution and average pooling operations are used to balance the number of channels, resulting in z1, z2, and z3. The fusion process is represented as follows: z n =AvgPool(Conv(Concat(z n Opt +z n SAR ))) Where n represents different levels; Step S83: After the accumulated temperature data is processed by the 2D convolution module and corresponding dimensional transformation, it enters the LSTM and attention mechanism, and then high-level features are extracted and dimensional matching is performed through the Linear and FC layers, which serve as the input for subsequent fusion with satellite data. The feature z4 extracted from the accumulated temperature data of the ALSTM model is fused with the features z3 and z2 to form z3′ and z2′, and then fed into the multi-level attention module MCA along with z1 to integrate the contributions of different scales and capture the differences between multi-scale features. in, These represent the query, key, and value in the MCA mechanism, respectively. Indicates the outer product operation; M Q and M K Q c K c The magnitude of the value is determined; finally, valuable semantic information from the high-level feature z3 is added to the generated features to obtain the aggregated temperature-satellite remote sensing spatiotemporal feature z with rich multi-scale information. m .

9. The method for early remote sensing identification of crops based on fused effective accumulated temperature according to claim 1, characterized in that: Step S9 is implemented as follows: Step S91: Train the entire network based on the Minimum Principle Cross-Entropy (CE) loss function and the auxiliary loss function. The Minimum Principle Cross-Entropy loss is used to train the entire network to learn mapping from the input time series to the ground reference data, while the auxiliary loss is used to supervise the optimization process of multi-level attention and prevent the loss of detailed information in the intermediate network layers. This enables the network to learn better multi-scale spatiotemporal feature representations, thereby improving the mapping performance of crops in the semantic view. Step S92, the total model loss is expressed as follows: α is used to balance the cross-entropy loss and auxiliary loss according to the minimization principle; the auxiliary loss function minimizes the auxiliary map y generated by the CNN-based decoder. aux ground reference map The cross-entropy loss is calculated using the principle of minimizing the cross-entropy, and is only used during the training phase so as not to affect the testing phase. The principle of minimizing the cross-entropy loss function is based on considering the output crop mapping result y and the ground reference data. The reconstruction loss between; the calculation of the multi-class minimization principle cross-entropy loss and auxiliary loss is as described in step S91 above, where N represents the number of pixels; and y i This represents the one-hot encoding of the true label on the i-th image pixel and the corresponding softmax output of the model; L All =αL CE +(1-α)L Aux ; Step S93: Divide the input data into a training set and a validation set in a 6:4 ratio; train the model using the training set data, using the Adam optimizer with an initial learning rate Lr as the optimizer for model training, setting the training batch size to N and the number of model iterations to R, and measuring the model's prediction performance based on the above loss function. For each batch, backpropagate the output error to the network parameters to fit the sample output; use accuracy to measure the model's performance in each batch, and save the optimal weight file of the model when the validation set accuracy is the highest.

10. The method for early remote sensing identification of crops based on fused effective accumulated temperature according to claim 1, characterized in that: Step S10 is implemented as follows: Step S101: Train model weight files based on different time series lengths, perform crop prediction during image stitching based on the corresponding time series cropped images, and generate crop recognition results for different time series lengths; Step S102: Use real ground samples to evaluate the crop identification results for each time series, and use an incremental classification strategy for early identification. The acquisition time of the latest image captured in the time series with the shortest time series and reaching a certain accuracy is regarded as the earliest identifiable time of the crop. The specific implementation of step S11 is as follows: Step S111: Obtain Sentinel-1A, Sentinel-2, and ERA5 Daily temperature data for the application area and time span from crop sowing to the earliest identifiable time, and process them according to steps S1, S2, and S3 to form model input data; Step S112: The model weight file, trained based on the corresponding time series length of the earliest identifiable time, is used to perform early identification on the data to be applied, and the extraction results of early crop identification are obtained.

Images