Crop early-stage remote sensing identification method fusing effective accumulated temperature

Abstract

The invention provides a crop early-stage remote sensing identification method fusing effective accumulated temperature, and provides a crop extraction method based on a space-time multi-level attention model and a long-short time memory neural network and integrating effective accumulated temperature and multi-source remote sensing data aiming at an early-stage crop growth environment and own characteristics. Complementary information between SAR and optical data is fully utilized, fusion is carried out in a feature layer, and further fusion with effective accumulated temperature data is carried out to realize enhancement of crop environment information, so that the model can better understand relevance between early-stage characteristics of different crops and the environment, and the early-stage recognition performance of the crops is improved.

Description

Technical Field

[0001] The invention relates to the technical field of crop mapping, in particular to an early remote sensing identification method for crops integrating effective accumulated temperature. Background Art

[0002] Near-real-time crop mapping can provide basic data support for agricultural resource allocation and help relevant departments and farmers optimize the use of limited resources in agricultural production. Common methods of combining machine learning with satellite remote sensing data usually rely on complete time series remote sensing data covering the entire growing season, which has high computational costs and poor timeliness. In addition, the relatively single data source used leads to limited information mining, and most of them fail to fully consider the environmental factors of the crops. Thermal conditions are an important factor restricting agricultural production, especially in the early stages of crop growth. Effective accumulated temperature is crucial for crop growth. Embedding effective accumulated temperature information can enhance the ability of deep learning segmentation networks to distinguish between similarities between different crops in the early stages, which is expected to further optimize the performance and efficiency of early crop identification. Summary of the invention

[0003] The present invention proposes an early remote sensing identification method for crops that integrates effective accumulated temperature. According to the early growth environment and characteristics of crops, a crop extraction method based on a spatiotemporal multi-level attention model and a long-short time memory neural network is proposed, which integrates effective accumulated temperature and multi-source remote sensing data. The complementary information between SAR and optical data is fully utilized, and the data is integrated at the feature layer. The data is then further integrated with the effective accumulated temperature data to enhance the crop environmental information, so that the model can better understand the correlation between the early characteristics of different crops and the environment, thereby improving the early crop identification performance.

[0004] The present invention adopts the following technical solutions.

[0005] The remote sensing identification method of crops in the early stage integrated with effective accumulated temperature includes the following steps:

[0006] Step S1, obtaining a medium-resolution time-series Sentinel-1 radar SAR image of the study area, and processing it to obtain the backscatter coefficient VV and VH characteristics of the cultivated land plot level;

[0007] Step S2, obtain medium-resolution time-series Sentinel-2 optical images of the study area, calculate kNDVI, ReG_RVI, ReG_NDVI, NDWI, REP, and mNDVI705 features, and select the best optical feature combination: kNDVI, NDWI, mNDVI705, and Red through the random forest feature optimization algorithm;

[0008] Step S3: Obtain daily ERA-5 temperature data for the study area and calculate the effective accumulated temperature T cal ;

[0009] Step S4, preparing a crop sample data set based on the cultivated land plots;

[0010] Step S5, constructing a dual-branch network model framework based on deep learning semantic segmentation, wherein the first branch is a multi-level network model integrating a spatiotemporal attention mechanism, which is used to extract SAR and optical image features, and the second branch is a long short-term memory neural network model of an attention mechanism, which is used to extract effective accumulated temperature features;

[0011] Step S6, using a multi-level network model of the spatiotemporal attention mechanism as the first branch, for mining the multi-scale spatiotemporal information of SAR and optical medium-resolution images; this branch consists of two parallel multi-level feature extractors based on ResNet and a spatiotemporal attention mechanism module, SAR and optical data are input in parallel, spatial features are extracted through different receptive fields, and the spatiotemporal attention mechanism module is used to further extract multi-level spatiotemporal features, and the extracted high-level features of SAR and optical images are spliced ​​and fused according to different levels;

[0012] Step S7, using a long short-term memory neural network model (LSTM) with an attention mechanism for crop effective accumulated temperature information as the second branch, using a long short-term memory neural network to introduce an attention mechanism, screen the extracted high-temporal resolution information, extract deep-level crop accumulated temperature information, construct a relationship between the labeled crop and its environment, and establish an effective crop accumulated temperature feature;

[0013] Step S8: A dual-branch feature fusion module fuses the low- and medium-level satellite image features with the effective accumulated temperature information, and then inputs the multi-level cross attention module to integrate contributions from different scales and capture differences, ultimately obtaining aggregated effective accumulated temperature-satellite remote sensing spatiotemporal features with rich multi-scale information;

[0014] Step S9: According to the early recognition strategy of incremental classification, the training set data is used to generate data sets of different time lengths and input into the model for training; the training process is based on the Adam optimizer, the cross entropy loss function and the auxiliary loss function, and finally a classifier trained for different time series lengths is obtained;

[0015] Step S10: Use the test set to evaluate the accuracy of the results of different classifiers, identify the shortest time series and the time series that reaches a specific accuracy, and the acquisition time of the last scene image is regarded as the earliest identifiable time of the crop, so as to realize the extraction of early crop planting areas at the plot level;

[0016] Step S11: using the above method and the classifier determined as the earliest identifiable time, carry out early identification of regional crops.

[0017] The step S1 is specifically implemented as follows:

[0018] Step S11, obtaining ground range detection GRD data in the interferometric wide-band IW mode of Sentinel-1 radar SAR images in the early to mid-growth period of the target year's cover crops, preprocessing the data, including thermal noise removal, orbit file correction, radiation calibration, filter correction, and Doppler terrain correction, and finally obtaining backscattering information;

[0019] Step S12, calculating the VV and VH polarization backscattering characteristics of the time series Sentinel-1 radar SAR images, performing time series smoothing through SG filtering, and clipping them to the study area through plot constraints;

[0020] The step S2 is specifically implemented as follows:

[0021] Step S21, obtaining the Level-1C product of Sentinel-2 optical image data of the early to mid-term growth of cover crops in the target year, which is processed by radiometric calibration and atmospheric correction; in view of the fact that optical images are easily disturbed by clouds, firstly select the medium-resolution time-series Sentinel-2 optical images with cloud cover and snow cover percentage less than 20%, then use the QA60 band marker to remove clouds, mosaic the obtained cloud-free images and crop them to the target area;

[0022] Step S22: Calculate the vegetation index kNDVI, ReG_RVI, ReG_NDVI, NDWI, REP, and mNDVI705 images of the study area based on the medium-resolution time-series Sentinel-2 optical images. The calculation formula is as follows:

[0023]

[0024]

[0025]

[0026] Among them, NIR represents the Sentinel-2 near-infrared band, Red represents the red band, Green represents the green band, RE2 and RE1 represent the red edge 1 and red edge 2 bands respectively, and SWIR1 represents the short-wave infrared 1 band. Finally, it is clipped to the study area through the plot constraint;

[0027] Step S23, using random forest RF to rank the extracted features by importance; using random forest RF, the feature importance is calculated based on the Gini index, and only some classification feature variables with high scores are selected to avoid interference from redundant and irrelevant features, and effectively avoid overfitting and reduce the generalization error of classification;

[0028] Assume that for a given data set D, which contains n categories, the probability that a sample belongs to the i-th category is P i, then the calculation formula of the Gini index is:

[0029]

[0030] The Gini index, also known as Gini impurity, is used to characterize the purity of a sample set. The smaller its value, the smaller the probability of sample misclassification. When the value is 0, the data set categories are consistent. If the set D is divided into k parts according to a certain feature, the Gini index after segmentation is:

[0031]

[0032] Where S j is the number of samples at child node j, S is the total number of samples; when the data set traverses all the splits, the Gini index before the split is subtracted from the Gini after the split split (D) The Gini gain value is obtained. The larger the value, the higher the importance, and the better the classification effect when splitting nodes according to the features;

[0033] Step S24: Based on the optical index and optical band that are beneficial to the early identification of crops, different feature combinations are formed by changing the number of feature variables to obtain the relationship between classification accuracy and feature quantity. The contribution rate of each phase feature is accumulated to screen out the best combination of optical features kNDVI, NDWI, mNDVI705, and Red that are beneficial to early identification.

[0034] The step S3 is specifically implemented as follows:

[0035] Step S31, obtaining the maximum and minimum ERA-5Daily temperature data from the early to mid-term growth period of the cover crop in the target year, mosaicking and synthesizing the obtained temperature data, and cropping them to the target area;

[0036] Step S32: Calculate the effective accumulated temperature of specific crops every day by using the maximum and minimum values ​​of the daily ERA-5Daily data. The calculation formula is as follows:

[0037]

[0038] Where t represents the day of the target year, and They represent the minimum and maximum temperatures from the i-th day (i=1, 2, 3…t) from the sowing day to the target year mid-day, T base It is the effective accumulated temperature range of crops, which has a fixed value according to different crops. After calculation, the image is resampled to 10m resolution to match the satellite image in space, and it is cropped to the study area through plot constraints.

[0039] When preparing the crop sample data set in step S4, the method is as follows: each SAR, optical, and effective accumulated temperature feature time series data is processed into a mat file of size H×W×T, and then the time series image is cropped into a data set of size N×128×128×T, and the label data is of size N×128×128, where N represents the number of 128×128 small blocks that the study area is cropped into, and T is the time series length. Finally, all features are merged into an npz (NumPyZip) file, in which the ratio of the training set to the test set is 6:4, and after processing, it is used as the model input data. Step S4 also specifically includes the following steps;

[0040] Step S41: Based on steps S1, S2, and S3, each feature is processed into a mat file in the form of H×W×T, and the effective accumulated temperature features with high temporal resolution are aggregated to the mid-year day obtained by satellite images;

[0041] Step S42, set the repetition rate overlay of each time series feature data to 0.5, crop it to a specific length and width, i.e., 128*128, and record it as a list, normalize it according to its own characteristics and read it as an npz file, organize it into a four-dimensional tensor T×C×H×W, where T and C represent the time series length and the corresponding number of channels, respectively, and H×W represents the length and width of the image, as input data;

[0042] Step S43, based on the historical crop distribution data set, crop samples are produced based on plots, the raster value of the crop sample data is assigned to 1, the plot vector is converted to raster data and assigned to 2, the background value is 0, firstly, raster calculation is performed to assign 1 to the cotton on the plot, and then a voting mechanism is adopted for each plot, that is, the plot with the largest proportion is judged as that type of land, and finally other crops are assigned 2, which are organized into a three-dimensional tensor Class×H×W, where Class is the category, which is used as the input label data.

[0043] The step S5 is specifically implemented as follows:

[0044] Step S51, constructing a parallel branch network model coupling multi-source remote sensing data;

[0045] Step S52: The input time series image is transmitted through the encoder, and two parallel branches are designed, the first branch is used to extract the medium-resolution multi-source satellite image features, and the second branch is used to extract the effective accumulated temperature data features;

[0046] Step S53: Fuse the different modal feature information of the two branches at the decoder front end.

[0047] The step S6 is specifically implemented as follows:

[0048] Step S61: Based on the STMA model framework, a satellite remote sensing data feature extraction network with a dual-branch ResNet-based multi-level feature extractor and a spatiotemporal attention mechanism module is constructed;

[0049] Step S62, the encoder is a multi-level feature front-end extractor based on ResNet18, including a standard 7×7 convolution block, a maximum pooling layer and four residual blocks, wherein each residual block contains two 3×3 convolution operations, batch normalization BN, a rectified linear unit ReLU and a jump connection layer consisting of a standard 1×1 convolution operation and BN; after convolution and pooling, the spatial size of the input patch cube is reduced by 1 / 2, 1 / 4 and 1 / 8 respectively, and the channel dimension gradually increases; ResNet extracts multi-scale features in multiple receptive fields, corresponding to shallow features, middle features and high-level features extracted based on different spatial resolutions, respectively, extracts local spatial features at each time in turn, and then concatenates the output features along the time dimension, thereby retaining the time dimension information and promoting the subsequent time attention module to realize time feature extraction;

[0050] Step S63: Use spatial attention position embedding before entering spatiotemporal attention, use embedded learnable functions to flexibly extract important position information related to the task, and learn complex position relationships from the feature cube extracted by the ResNet module;

[0051] x′ n =x n ×σ(DConu(x n ))

[0052] in, Represents the input feature cube and output tag embedding extracted by the CNN-based ResNet module; DConv(.) represents a deep convolution operation with a kernel of 3×3 and padding of 1, extracting spatial weights from the input sequence; deep convolution applies a convolution filter to each input channel, collects spatial information conditioned on the local neighborhood of the input tag, and maintains translation equivalence while keeping small trainable parameters. By assigning different weight coefficients, each pixel in the extracted feature cube can adaptively obtain a position embedding, preserving spatial information to the output tag embedding;

[0053] Step S64, STSA consists of spatial self-attention, temporal self-attention and feature fusion modules; spatial self-attention and temporal self-attention parts independently extract spatial and temporal information according to different flattened forms of the encoded feature cube. The feature fusion module contains convolution operations, LayerNorm layers LN and multi-layer perceptron MLP layers, so that the concatenated features are integrated to obtain robust spatiotemporal features and avoid the influence of manual intervention on the concatenation order of spatiotemporal features; first, the input tag embedding x′n Flattened into two embedding sequences along the spatial and temporal dimensions and Then, the flattened embedding sequence is fed into multi-head self-attention MSA along the spatial and temporal axes to capture spatial and temporal information; and Represent the spatial and temporal features of the output of spatial self-attention and temporal self-attention respectively:

[0054] zs′=MSA(LN(z s ))+z s

[0055] z T ′=MSA(LN(z T ))+z T

[0056] MSA(z)=Concat(head1...,head h )W o

[0057]

[0058] Where h is the number of heads in MSA, W O is the linear transformation matrix used for feature space transformation; Represents the key, query, and value of each header respectively; is a linear transformation matrix whose parameters can be learned; represents the scaling factor;

[0059] The Softmax activation function provides an attention value from 0 to 1, and the self-attention matrices in the spatial and temporal dimensions are [H n W n , H n W n ] and [T, T] to reflect the weight distribution of different pixels and time; the feature fusion module realizes the aggregation of spatial and temporal features, connects the extracted features in series, and performs a feedforward network similar to the original transformer; the output of STSA is expressed in the following form:

[0060]

[0061] is the spatiotemporal feature of the output. The MLP layer consists of two fully connected layers, using Gaussian Error Linear Unit (GELU) activation. are learnable network parameters used for adaptive fusion. Represents z S ′ and z T′ is the concatenated result in the third dimension, so that the local structure information is retained in the convolution operation to avoid insufficient fusion. After STSA, the output spatiotemporal feature d is reshaped into the two-dimensional image space to generate discriminative spatiotemporal features For different multi-scale features obtained by ResNet, the STSA module can also obtain mid-level features And high-level features

[0062] The step S7 is specifically implemented as follows:

[0063] Step S71, constructing an encoder for fully extracting deep-level information of crop effective accumulated temperature, which is composed of a 2D convolution block, an attention mechanism and an LSTM. The LSTM introduces the attention mechanism as a screening mechanism for extracting high-temporal-resolution information;

[0064] Step S72, the two 2D convolution blocks of the effective accumulated temperature data encoder are responsible for dimensional compression of the data, reducing the calculation pressure and then inputting into the LSTM; Step S73, the signal output by the LSTM is passed through the attention mechanism to highlight the key information of the image by calculating the image feature weights, thereby helping the model to make more accurate decisions, and finally output the extracted features through the Linear layer and FClayer;

[0065] In step S72, the LSTM is defined as follows: Each LSTM unit includes a memory unit (C t ) and three gating mechanisms, namely the input gate t , Forget t And output gate Output t The input data of the LSTM unit at the tth time step is composed of the current unit input signal x t and the previous unit output signal h t When the input data enters the LSTM, the input gate uses the Sigmoid function to determine which information can enter the memory unit; the forget gate uses the Sigmoid function to determine which previous information needs to be discarded. The information output of the input gate and the forget gate determines the memory unit C. t The input data is multiplied by the input gate output information after passing through the hyperbolic tangent function tanh, and the forget gate output information is multiplied by the previous memory unit C t-1 Multiply them, and finally sum them to generate a new memory value C t The output gate determines the final information output through the Sigmoid function and generates the new memory value C t Through the hyperbolic tangent function, multiply the output information of the output gate to get the final output signal h t, where σ is the Sigmoid function, W is the weight, and tanh represents the hyperbolic tangent function. The specific calculation process is as follows:

[0066] Input t =σ(W i [h t-1， x t ])

[0067] Forget t =σ(W f [h t-1 , x t ])

[0068] Output t =σ(W o [h t-1， x t ])

[0069] C′ t =σ(W c [h t-1 , x t ])

[0070] C t =Multiply[Forget t , C t-1 ]+Multiply[Input t , C′ t ]

[0071] h t =Multiply[Output t , tanh(C t )];

[0072] In step S72, the attention mechanism is defined as follows: LSTM cell unit output signal h t The attention mechanism is used to capture important temporal features related to the target object. As an information screening mechanism, the attention mechanism is located between the encoder and the decoder. It highlights the key information of the image by calculating the weight of the image features, thereby helping the model make more accurate decisions. The attention mechanism assigns a context vector v to the input sequence data, which depends on the input sequence h = {h1, h2, ...h t ) can reflect the most important information in the current time step. ijis the attention weight, which is the model output of a softmax function, indicating the importance of each time step to the model output. This weight calculation method enables the model to focus on the most important part of the input sequence, thereby improving the performance and accuracy of the model. ij is a vector related to the input sequence, measuring the importance of information, and a is a feedforward neural network.

[0073]

[0074] a ij =softmax(e ij )

[0075] e ij =a(v i-1 ,h j ).

[0076] The step S8 is specifically implemented as follows:

[0077] Step S81: construct a dual-branch feature fusion module by using a cascade feature fusion method combined with an attention mechanism to achieve fusion of features extracted by the first branch and the second branch;

[0078] Step S82: In Dual-STMA, the optical branch and the SAR branch are composed. Different branches extract spatiotemporal features of different levels through ResNet and STSA modules respectively. The features output by the optical branch are represented as z1 Opt 、z2 Opt 、z3 Opt , the SAR branch output feature is represented as z1 SAR 、z2 SAR 、z3 SAR , after being output by STSA, they are fused according to the corresponding scale. The features of different levels extracted after the fusion of optical and SAR data are used. In order to unify the size of different multi-scale features, 1×1 convolution and average pooling operations are used to balance the number of channels to obtain z1, z2, and z3. The fusion process is shown as follows:

[0079] z n =AvgPool(Conv(Concat(z n Opt +z n SAR )))

[0080] The n here represents different levels;

[0081] Step S83: The accumulated temperature data enters the LSTM and attention mechanism after the 2D convolution module and the corresponding dimensional transformation, and then extracts high-level features and performs dimensional matching through the Linear and FC layers, which serves as the input for subsequent fusion with satellite data.

[0082] The feature z4 extracted from the accumulated temperature data of the ALSTM model is fused with the features z3 and z2 to form z3′ and z2′, and then input into the multi-level attention module (MultiCrossAttention, MCA) with z1 respectively to integrate the contributions of different scales and capture the differences between multi-scale features.

[0083]

[0084] in, Represents the query, key, and value of the MCA mechanism respectively. Represents the outer product operation. Q and M K Q c , K c Finally, the valuable semantic information of the high-level feature z3 is added to the generated features to obtain the aggregated temperature-satellite remote sensing spatiotemporal feature z with rich multi-scale information. m .

[0085] The step S9 is specifically implemented as follows:

[0086] Step S91, training the entire network based on the minimization principle cross entropy (CE) loss function and the auxiliary loss function, wherein the minimization principle cross entropy loss is used to train the entire network to learn mapping from the input time series to the ground reference data, and the auxiliary loss is used to supervise the optimization process of multi-level attention to prevent the loss of detailed information of the intermediate network layer, so that the network architecture can learn better multi-scale spatiotemporal feature representation, thereby improving the mapping performance of crops in the semantic view;

[0087]

[0088] Step S92, the total model loss is expressed as follows: α is used to balance the trade-off between the cross entropy loss and the auxiliary loss; the auxiliary loss function is generated by minimizing the auxiliary map y generated by the CNN-based decoder aux With ground reference map The cross entropy loss function is calculated based on the minimization principle between the output crop mapping result y and the ground reference data. It is only used in the training stage so as not to affect the test stage. The reconstruction loss between the two; the multi-class minimization principle cross entropy loss of the minimization principle cross entropy and the auxiliary loss is calculated as shown in the formula of step S91 above, where N represents the number of pixels; and i Represents the unique hot encoding of the true label on the i-th image pixel and the corresponding softmax output of the model;

[0089] L All =αl CE +(1-α)L Aux ;

[0090] Step S93, divide the input data into a training set and a validation set in a ratio of 6:4; use the training set data for training, use the Adam optimizer with an initial learning rate Lr as the optimizer for model training, set the model training batch to N, set the model iteration number to R, measure the quality of model prediction based on the above loss function, and back propagate the output error to the network parameters for each batch to fit the output of the sample; use accuracy to measure the performance of the model in each batch, and save the model optimal weight file when the validation set accuracy is the highest.

[0091] The step S10 is specifically implemented as follows:

[0092] Step S101, training model weight files based on different time series lengths, performing crop prediction when stitching images cropped according to corresponding time series, and generating crop recognition results of different time series lengths;

[0093] Step S102: Use the ground truth samples to evaluate the crop recognition results of each time series, and use the incremental classification strategy for early recognition, that is, the acquisition time of the latest image captured in the time series with the shortest time series and reaching a specific accuracy is regarded as the earliest identifiable time of the crop;

[0094] The step S11 is specifically implemented as follows:

[0095] Step S111, obtaining Sentinel-1A, Sentinel-2, and ERA5 Daily temperature data in the application range area and the time span from the crop sowing period to the earliest identifiable time, and processing according to steps S1, S2, and S3 to form model input data;

[0096] Step S112: Perform early recognition on the data to be applied based on the model weight file obtained by training the corresponding time series length of the earliest identifiable time, and obtain the extraction result of early crop recognition.

[0097] The present invention proposes an early remote sensing identification method for crops that integrates effective accumulated temperature. The method is an early identification method for crops that integrates deep learning and effective accumulated temperature. It is based on a spatiotemporal multi-level attention model and a long-short time memory neural network, and integrates a crop extraction method of effective accumulated temperature and multi-source remote sensing data. By making full use of the complementary information between SAR and optical data, fusing them at the feature layer, and further fusing them with the effective accumulated temperature data, the crop environmental information is enhanced, which enables the model to better understand the correlation between the early characteristics of different crops and the environment, thereby improving the early identification performance of crops.

[0098] Compared with traditional methods, the dual-branch network of the present invention that integrates effective accumulated temperature and satellite remote sensing data has the advantage of being able to make full use of limited early crop growth images, conduct a more comprehensive analysis of the environment and characteristics of the crops themselves, and realize automatic extraction of refined early crop spatial distribution in small sample conditions. In addition, the method of the present invention takes into account the impact of multiple features and model structures on early crop recognition results. BRIEF DESCRIPTION OF THE DRAWINGS

[0099] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0100] Attached Figure 1 is a schematic diagram of a method flow chart of a preferred embodiment of the present invention;

[0101] Attached Figure 2 This is a schematic diagram of data processing in this embodiment;

[0102] Attached Figure 3 is a schematic diagram of the early identification method of this embodiment;

[0103] Attached Figure 4 Schematic diagram of the TDSTMA network model structure of this embodiment;

[0104] Attached Figure 5 Schematic diagram of the structure of the feature fusion module of this embodiment;

[0105] Attached Figure 6 This is a schematic diagram of the results obtained by extracting cotton distribution information using the proposed multi-branch network model and different branches and corresponding data in this embodiment. DETAILED DESCRIPTION

[0106] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings.

[0107] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0108] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations.

[0109] As shown in the figure, the early remote sensing identification method of crops integrated with effective accumulated temperature includes the following steps:

[0110] Step S1, obtaining a medium-resolution time-series Sentinel-1 radar SAR image of the study area, and processing it to obtain the backscatter coefficient VV and VH characteristics of the cultivated land plot level;

[0111] Step S2, obtain medium-resolution time-series Sentinel-2 optical images of the study area, calculate kNDVI, ReG_RVI, ReG_NDVI, NDWI, REP, and mNDVI705 features, and select the best optical feature combination: kNDVI, NDWI, mNDVI705, and Red through the random forest feature optimization algorithm;

[0112] Step S3: Obtain daily ERA-5 temperature data for the study area and calculate the effective accumulated temperature T cal ;

[0113] Step S4, preparing a crop sample data set based on the cultivated land plots;

[0114] Step S5, constructing a dual-branch network model framework based on deep learning semantic segmentation, wherein the first branch is a multi-level network model integrating a spatiotemporal attention mechanism, which is used to extract SAR and optical image features, and the second branch is a long short-term memory neural network model of an attention mechanism, which is used to extract effective accumulated temperature features;

[0115] Step S6, using a multi-level network model of the spatiotemporal attention mechanism as the first branch, for mining the multi-scale spatiotemporal information of SAR and optical medium-resolution images; this branch consists of two parallel multi-level feature extractors based on ResNet and a spatiotemporal attention mechanism module, SAR and optical data are input in parallel, spatial features are extracted through different receptive fields, and the spatiotemporal attention mechanism module is used to further extract multi-level spatiotemporal features, and the extracted high-level features of SAR and optical images are spliced ​​and fused according to different levels;

[0116] Step S7, using a long short-term memory neural network model (LSTM) with an attention mechanism for crop effective accumulated temperature information as the second branch, using a long short-term memory neural network to introduce an attention mechanism, screen the extracted high-temporal resolution information, extract deep-level crop accumulated temperature information, construct a relationship between the labeled crop and its environment, and establish an effective crop accumulated temperature feature;

[0117] Step S8: A dual-branch feature fusion module fuses the low- and medium-level satellite image features with the effective accumulated temperature information, and then inputs the multi-level cross attention module to integrate contributions from different scales and capture differences, ultimately obtaining aggregated effective accumulated temperature-satellite remote sensing spatiotemporal features with rich multi-scale information;

[0118] Step S9: According to the early recognition strategy of incremental classification, the training set data is used to generate data sets of different time lengths and input into the model for training; the training process is based on the Adam optimizer, the cross entropy loss function and the auxiliary loss function, and finally a classifier trained for different time series lengths is obtained;

[0119] Step S10: Use the test set to evaluate the accuracy of the results of different classifiers, identify the shortest time series and the time series that reaches a specific accuracy, and the acquisition time of the last scene image is regarded as the earliest identifiable time of the crop, so as to realize the extraction of early crop planting areas at the plot level;

[0120] Step S11: using the above method and the classifier determined as the earliest identifiable time, carry out early identification of regional crops.

[0121] The step S1 is specifically implemented as follows:

[0122] Step S11, obtaining ground range detection GRD data in the interferometric wide-band IW mode of Sentinel-1 radar SAR images in the early to mid-growth period of the target year's cover crops, preprocessing the data, including thermal noise removal, orbit file correction, radiation calibration, filter correction, and Doppler terrain correction, and finally obtaining backscattering information;

[0123] Step S12, calculating the VV and VH polarization backscattering characteristics of the time series Sentinel-1 radar SAR images, performing time series smoothing through SG filtering, and clipping them to the study area through plot constraints;

[0124] The step S2 is specifically implemented as follows:

[0125] Step S21, obtaining the Level-1C product of Sentinel-2 optical image data of the early to mid-term growth of cover crops in the target year, which is processed by radiometric calibration and atmospheric correction; in view of the fact that optical images are easily disturbed by clouds, firstly select the medium-resolution time-series Sentinel-2 optical images with cloud cover and snow cover percentage less than 20%, then use the QA60 band marker to remove clouds, mosaic the obtained cloud-free images and crop them to the target area;

[0126] Step S22: Calculate the vegetation index kNDVI, ReG_RVI, ReG_NDVI, NDWI, REP, and mNDVI705 images of the study area based on the medium-resolution time-series Sentinel-2 optical images. The calculation formula is as follows:

[0127]

[0128]

[0129] Among them, NIR represents the Sentinel-2 near-infrared band, Red represents the red band, Green represents the green band, RE2 and RE1 represent the red edge 1 and red edge 2 bands respectively, and SWIR1 represents the short-wave infrared 1 band. Finally, it is clipped to the study area through the plot constraint;

[0130] Step S23, using random forest RF to rank the extracted features by importance; using random forest RF, the feature importance is calculated based on the Gini index, and only some classification feature variables with high scores are selected to avoid interference from redundant and irrelevant features, and effectively avoid overfitting and reduce the generalization error of classification;

[0131] Assume that for a given data set D, which contains n categories, the probability that a sample belongs to the i-th category is P i , then the calculation formula of the Gini index is:

[0132]

[0133] The Gini index, also known as Gini impurity, is used to characterize the purity of a sample set. The smaller its value, the smaller the probability of sample misclassification. When the value is 0, the data set categories are consistent. If the set D is divided into k parts according to a certain feature, the Gini index after segmentation is:

[0134]

[0135] Where S j is the number of samples at child node j, S is the total number of samples; when the data set traverses all the splits, the Gini index before the split is subtracted from the Gini after the split split(D) The Gini gain value is obtained. The larger the value, the higher the importance, and the better the classification effect when splitting nodes according to the features;

[0136] Step S24: Based on the optical index and optical band that are beneficial to the early identification of crops, different feature combinations are formed by changing the number of feature variables to obtain the relationship between classification accuracy and feature quantity. The contribution rate of each phase feature is accumulated to screen out the best combination of optical features kNDVI, NDWI, mNDVI705, and Red that are beneficial to early identification.

[0137] The step S3 is specifically implemented as follows:

[0138] Step S31, obtaining the maximum and minimum ERA-5Daily temperature data from the early to mid-term growth period of the cover crop in the target year, mosaicking and synthesizing the obtained temperature data, and cropping them to the target area;

[0139] Step S32: Calculate the effective accumulated temperature of specific crops every day by using the maximum and minimum values ​​of the daily ERA-5Daily data. The calculation formula is as follows:

[0140]

[0141] Where t represents the day of the target year, and They represent the minimum and maximum temperatures from the i-th day (i=1, 2, 3…t) from the sowing day to the target year mid-day, T base It is the effective accumulated temperature range of crops, which has a fixed value according to different crops. After calculation, the image is resampled to 10m resolution to match the satellite image in space, and it is cropped to the study area through plot constraints.

[0142] When preparing the crop sample data set in step S4, the method is as follows: each SAR, optical, and effective accumulated temperature feature time series data is processed into a mat file of size H×W×T, and then the time series image is cropped into a data set of size N×128×128×T, and the label data is of size N×128×128, where N represents the number of 128×128 small blocks that the study area is cropped into, and T is the time series length. Finally, all features are merged into an npz (NumPyZip) file, in which the ratio of the training set to the test set is 6:4, and after processing, it is used as the model input data. Step S4 also specifically includes the following steps;

[0143] Step S41: Based on steps S1, S2, and S3, each feature is processed into a mat file in the form of H×W×T, and the effective accumulated temperature features with high temporal resolution are aggregated to the mid-year day obtained by satellite images;

[0144] Step S42, set the repetition rate overlay of each time series feature data to 0.5, crop it to a specific length and width, i.e., 128*128, and record it as a list, normalize it according to its own characteristics and read it as an npz file, organize it into a four-dimensional tensor T×C×H×W, where T and C represent the time series length and the corresponding number of channels, respectively, and H×W represents the length and width of the image, as input data;

[0145] Step S43, based on the historical crop distribution data set, crop samples are produced based on plots, the raster value of the crop sample data is assigned to 1, the plot vector is converted to raster data and assigned to 2, the background value is 0, firstly, raster calculation is performed to assign 1 to the cotton on the plot, and then a voting mechanism is adopted for each plot, that is, the plot with the largest proportion is judged as that type of land, and finally other crops are assigned 2, which are organized into a three-dimensional tensor Class×H×W, where Class is the category, which is used as the input label data.

[0146] The step S5 is specifically implemented as follows:

[0147] Step S51, constructing a parallel branch network model coupling multi-source remote sensing data;

[0148] Step S52: The input time series image is transmitted through the encoder, and two parallel branches are designed, the first branch is used to extract the medium-resolution multi-source satellite image features, and the second branch is used to extract the effective accumulated temperature data features;

[0149] Step S53: Fuse the different modal feature information of the two branches at the decoder front end.

[0150] The step S6 is specifically implemented as follows:

[0151] Step S61: Based on the STMA model framework, a satellite remote sensing data feature extraction network with a dual-branch ResNet-based multi-level feature extractor and a spatiotemporal attention mechanism module is constructed;

[0152] Step S62, the encoder is a multi-level feature front-end extractor based on ResNet18, including a standard 7×7 convolution block, a maximum pooling layer and four residual blocks, wherein each residual block contains two 3×3 convolution operations, batch normalization BN, a rectified linear unit ReLU and a jump connection layer consisting of a standard 1×1 convolution operation and BN; after convolution and pooling, the spatial size of the input patch cube is reduced by 1 / 2, 1 / 4 and 1 / 8 respectively, and the channel dimension gradually increases; ResNet extracts multi-scale features in multiple receptive fields, corresponding to shallow features, middle features and high-level features extracted based on different spatial resolutions, respectively, extracts local spatial features at each time in turn, and then concatenates the output features along the time dimension, thereby retaining the time dimension information and promoting the subsequent time attention module to realize time feature extraction;

[0153] Step S63: Use spatial attention position embedding before entering spatiotemporal attention, use embedded learnable functions to flexibly extract important position information related to the task, and learn complex position relationships from the feature cube extracted by the ResNet module;

[0154] x′ n =x n ×σ(DConv(x n ))

[0155] in, Represents the input feature cube and output tag embedding extracted by the CNN-based ResNet module; DConv(.) represents a deep convolution operation with a kernel of 3×3 and padding of 1, extracting spatial weights from the input sequence; deep convolution applies a convolution filter to each input channel, collects spatial information conditioned on the local neighborhood of the input tag, and maintains translation equivalence while keeping small trainable parameters. By assigning different weight coefficients, each pixel in the extracted feature cube can adaptively obtain a position embedding, preserving spatial information to the output tag embedding;

[0156] Step S64, STSA consists of spatial self-attention, temporal self-attention and feature fusion modules; spatial self-attention and temporal self-attention parts independently extract spatial and temporal information according to different flattened forms of the encoded feature cube. The feature fusion module contains convolution operations, LayerNorm layers LN and multi-layer perceptron MLP layers, so that the concatenated features are integrated to obtain robust spatiotemporal features and avoid the influence of manual intervention on the concatenation order of spatiotemporal features; first, the input tag embedding x′ n Flattened into two embedding sequences along the spatial and temporal dimensions and Then, the flattened embedding sequence is fed into multi-head self-attention MSA along the spatial and temporal axes to capture spatial and temporal information; and Represent the spatial and temporal features of the output of spatial self-attention and temporal self-attention respectively:

[0157] z S ′=MSA(LN(z s ))+z s

[0158] z T ′=MSA(LN(z T ))+z T

[0159] MSA(z)=Concat(head1,...,head h ) O

[0160]

[0161] Where h is the number of heads in MSA, W O is the linear transformation matrix used for feature space transformation; Represents the key, query, and value of each header respectively; is a linear transformation matrix whose parameters can be learned; represents the scaling factor;

[0162] The Softmax activation function provides an attention value from 0 to 1, and the self-attention matrices in the spatial and temporal dimensions are [H n W n , H n W n ] and [T, T] to reflect the weight distribution of different pixels and time; the feature fusion module realizes the aggregation of spatial and temporal features, connects the extracted features in series, and performs a feedforward network similar to the original transformer; the output of STSA is expressed in the following form:

[0163]

[0164] is the spatiotemporal feature of the output. The MLP layer consists of two fully connected layers, using Gaussian Error Linear Unit (GELU) activation. are learnable network parameters used for adaptive fusion. Represents z S ′ and z T′ is the concatenated result in the third dimension, so that the local structure information is retained in the convolution operation to avoid insufficient fusion. After STSA, the output spatiotemporal feature d is reshaped into the two-dimensional image space to generate discriminative spatiotemporal features For different multi-scale features obtained by ResNet, the STSA module can also obtain mid-level features And high-level features

[0165] The step S7 is specifically implemented as follows:

[0166] Step S71, constructing an encoder for fully extracting deep-level information of crop effective accumulated temperature, which is composed of a 2D convolution block, an attention mechanism and an LSTM. The LSTM introduces the attention mechanism as a screening mechanism for extracting high-temporal-resolution information;

[0167] Step S72, the two 2D convolution blocks of the effective accumulated temperature data encoder are responsible for dimensional compression of the data, reducing the calculation pressure and then inputting into the LSTM; Step S73, the signal output by the LSTM is passed through the attention mechanism to highlight the key information of the image by calculating the image feature weights, thereby helping the model to make more accurate decisions, and finally output the extracted features through the Linear layer and FClayer;

[0168] In step S72, the LSTM is defined as follows: Each LSTM unit includes a memory unit (C t ) and three gating mechanisms, namely the input gate t , Forget t And output gate Output t The input data of the LSTM unit at the tth time step is composed of the current unit input signal x t and the previous unit output signal h t When the input data enters the LSTM, the input gate uses the Sigmoid function to determine which information can enter the memory unit; the forget gate uses the Sigmoid function to determine which previous information needs to be discarded. The information output of the input gate and the forget gate determines the memory unit C. t The input data is multiplied by the input gate output information after passing through the hyperbolic tangent function tanh, and the forget gate output information is multiplied by the previous memory unit C t-1 Multiply them, and finally sum them to generate a new memory value C t The output gate determines the final information output through the Sigmoid function and generates the new memory value C t Through the hyperbolic tangent function, multiply the output information of the output gate to get the final output signal h t, where σ is the Sigmoid function, W is the weight, and tanh represents the hyperbolic tangent function. The specific calculation process is as follows:

[0169] Input t =σ(W i [h t-1， x t ])

[0170] Forget t =σ(W f [h t-1 , x t ])

[0171] Output t =σ(W o [h t-1， x t ])

[0172] C′ t =σ(W c [h t-1 , x t ])

[0173] C t =Multiply[Forget t , ε t-1 ]+Multiply[Input t , C′ t ]

[0174] h t =Multiply[Output t , tanh(C t )];

[0175] In step S72, the attention mechanism is defined as follows: LSTM cell unit output signal h t The attention mechanism is used to capture important temporal features related to the target object. As an information screening mechanism, the attention mechanism is located between the encoder and the decoder. It highlights the key information of the image by calculating the weight of the image features, thereby helping the model make more accurate decisions. The attention mechanism assigns a context vector v to the input sequence data, which depends on the input sequence h = {h1, h2, ...h t ) can reflect the most important information in the current time step. ijis the attention weight, which is the model output of a softmax function, indicating the importance of each time step to the model output. This weight calculation method enables the model to focus on the most important part of the input sequence, thereby improving the performance and accuracy of the model. ij is a vector related to the input sequence, measuring the importance of information, and a is a feedforward neural network.

[0176]

[0177] a ij =softmax(e ij )

[0178] e ij =a(v i-1 ,h j ).

[0179] The step S8 is specifically implemented as follows:

[0180] Step S81: construct a dual-branch feature fusion module by using a cascade feature fusion method combined with an attention mechanism to achieve fusion of features extracted by the first branch and the second branch;

[0181] Step S82: In Dual-STMA, the optical branch and the SAR branch are composed. Different branches extract spatiotemporal features of different levels through ResNet and STSA modules respectively. The features output by the optical branch are represented as z1 Opt 、z2 Opt 、z3 Opt , the SAR branch output feature is represented as z1 SAR 、z2 SAR 、z3 SAR , after being output by STSA, they are fused according to the corresponding scale. The features of different levels extracted after the fusion of optical and SAR data are used. In order to unify the size of different multi-scale features, 1×1 convolution and average pooling operations are used to balance the number of channels to obtain z1, z2, and z3. The fusion process is shown as follows:

[0182] z n =AvgPool(Conu(Concat(z n Opt +z n SAR )))

[0183] The n here represents different levels;

[0184] Step S83: The accumulated temperature data enters the LSTM and attention mechanism after the 2D convolution module and the corresponding dimensional transformation, and then extracts high-level features and performs dimensional matching through the Linear and FC layers, which serves as the input for subsequent fusion with satellite data.

[0185] The feature z4 extracted from the accumulated temperature data of the ALSTM model is fused with the features z3 and z2 to form z3′ and z2′, and then input into the multi-level attention module (Multi CrossAttention, MCA) with z1 respectively to integrate the contributions of different scales and capture the differences between multi-scale features.

[0186]

[0187] in, Represents the query, key, and value of the MCA mechanism respectively. Represents the outer product operation. Q and M K Q c , K c Finally, the valuable semantic information of the high-level feature z3 is added to the generated features to obtain the aggregated temperature-satellite remote sensing spatiotemporal feature z with rich multi-scale information. m .

[0188] The step S9 is specifically implemented as follows:

[0189] Step S91, training the entire network based on the minimization principle cross entropy (CE) loss function and the auxiliary loss function, wherein the minimization principle cross entropy loss is used to train the entire network to learn mapping from the input time series to the ground reference data, and the auxiliary loss is used to supervise the optimization process of multi-level attention to prevent the loss of detailed information of the intermediate network layer, so that the network architecture can learn better multi-scale spatiotemporal feature representation, thereby improving the mapping performance of crops in the semantic view;

[0190]

[0191] Step S92, the total model loss is expressed as follows: α is used to balance the trade-off between the cross entropy loss and the auxiliary loss; the auxiliary loss function is generated by minimizing the auxiliary map y generated by the CNN-based decoder aux With ground reference map The cross entropy loss function is calculated based on the minimization principle between the output crop mapping result y and the ground reference data. It is only used in the training stage so as not to affect the test stage. The reconstruction loss between the two; the multi-class minimization principle cross entropy loss of the minimization principle cross entropy and the auxiliary loss is calculated as shown in the formula of step S91 above, where N represents the number of pixels; and i Represents the unique hot encoding of the true label on the i-th image pixel and the corresponding softmax output of the model;

[0192] L All =αL CE +(1-α)L Aux ;

[0193] Step S93, divide the input data into a training set and a validation set at a ratio of 6:4; use the training set data for training, use the Adam optimizer with an initial learning rate Lr as the optimizer for model training, set the model training batch to N, set the model iteration number to R, measure the quality of model prediction based on the above loss function, and back-propagate the output error to the network parameters for each batch to fit the output of the sample; use accuracy to measure the performance of the model in each batch, and save the model optimal weight file when the validation set accuracy is the highest.

[0194] The step S10 is specifically implemented as follows:

[0195] Step S101, training model weight files based on different time series lengths, performing crop prediction when stitching images cropped according to corresponding time series, and generating crop recognition results of different time series lengths;

[0196] Step S102: Use the ground truth samples to evaluate the crop recognition results of each time series, and use the incremental classification strategy for early recognition, that is, the acquisition time of the latest image captured in the time series with the shortest time series and reaching a specific accuracy is regarded as the earliest identifiable time of the crop;

[0197] The step S11 is specifically implemented as follows:

[0198] Step S111, obtaining Sentinel-1A, Sentinel-2, and ERA5 Daily temperature data in the application range area and the time span from the crop sowing period to the earliest identifiable time, and processing according to steps S1, S2, and S3 to form model input data;

[0199] Step S112: Perform early recognition on the data to be applied based on the model weight file obtained by training the corresponding time series length of the earliest identifiable time, and obtain the extraction result of early crop recognition.

[0200] In this embodiment, the study area is located in Manas County and Hutubi County, Changji Hui Autonomous Prefecture, Xinjiang Uygur Autonomous Region. The data sources include Sentinel-2 optical images (2021, 10m), Sentinel-1SAR data (2021, 10m), ERA5-Daily temperature data images (2021, 0.1°), and the temperature data is resampled to 10m. This area is a major cotton producing area all year round, and the distribution of cotton planting fields has not changed much. This embodiment obtains 3672 remote sensing images with a pixel size of 128×128 and corresponding label images for model training.

[0201] like Figure 1 FIG. 1 is a schematic diagram of a method flow chart of a preferred embodiment of the present invention.

[0202] like Figure 2 The figure is a schematic diagram of data processing in this embodiment.

[0203] like Figure 3 FIG. 1 is a schematic diagram of the early identification method of this embodiment.

[0204] like Figure 4 As shown, it is the TDSTMA network model structure diagram of this embodiment.

[0205] like Figure 5 As shown, it is the structural diagram of the feature fusion module of this embodiment.

[0206] like Figure 6 As shown in the figure, this embodiment uses the proposed multi-branch network model and uses different branches and corresponding data to extract cotton distribution information. It can be seen from the figure that when only the SAR data set is used, cotton can be effectively extracted, but there are still some false extractions and omissions; when only the optical data set is used, the omissions are reduced; and when the optical + SAR data set is used, the false extraction is significantly reduced; and when the accumulated temperature data is added, the effect is the best. It can be seen that by gradually introducing multi-source satellite remote sensing images and effective accumulated temperature data, the model can make full use of the complementarity of various information to reduce false extractions.

[0207] The crop extraction method integrating effective accumulated temperature and multi-source remote sensing data of the present invention is for early crop identification. A method based on spatiotemporal multi-level attention model and long-short time memory neural network is proposed, which makes full use of the complementary information between SAR and optical data, fuses them at the feature layer, and further fuses them with effective accumulated temperature data to enhance crop environmental information, so that the model can better understand the correlation between the early characteristics of different crops and the environment, and improve the early crop identification performance. Compared with traditional methods, the advantage of this dual-branch network integrating effective accumulated temperature and satellite remote sensing data is that it can make full use of limited early crop growth images, analyze the environment and its own characteristics of crops more comprehensively, and realize refined automatic extraction of early crop spatial distribution in small sample conditions. The method of the present invention takes into account the influence of multiple features and model structure on early crop identification results.

[0208] The above are preferred embodiments of the present invention. Any changes made according to the technical solution of the present invention, as long as the resulting functions do not exceed the scope of the technical solution of the present invention, belong to the protection scope of the present invention.

Claims

1. A remote sensing identification method for early crops integrating effective accumulated temperature, characterized by: The following steps are involved: Step S1, obtaining a medium-resolution time series radar SAR image of the study area, and processing it to obtain the backscatter coefficient VV and VH characteristics of the cultivated land plot level; Step S2, obtain medium-resolution time-series optical images of the study area, calculate kNDVI, ReG_RVI, ReG_NDVI, NDWI, REP, and mNDVI705 features, and select the best optical feature combination: kNDVI, NDWI, mNDVI705, and Red through the random forest feature optimization algorithm; Step S3: Obtain daily ERA-5 temperature data for the study area and calculate the effective accumulated temperature T cal ; Step S4, preparing a crop sample data set based on the cultivated land plots; Step S5, constructing a dual-branch network model framework based on deep learning semantic segmentation, wherein the first branch is a multi-level network model integrating a spatiotemporal attention mechanism, which is used to extract SAR and optical image features, and the second branch is a long short-term memory neural network model of an attention mechanism, which is used to extract effective accumulated temperature features; Step S6, using a multi-level network model of the spatiotemporal attention mechanism as the first branch, for mining the multi-scale spatiotemporal information of SAR and optical medium-resolution images; this branch consists of two parallel multi-level feature extractors based on ResNet and a spatiotemporal attention mechanism module, SAR and optical data are input in parallel, spatial features are extracted through different receptive fields, and the spatiotemporal attention mechanism module is used to further extract multi-level spatiotemporal features, and the extracted high-level features of SAR and optical images are spliced ​​and fused according to different levels; Step S7, using the long short-term memory neural network model LSTM with the attention mechanism of crop effective accumulated temperature information as the second branch, using the long short-term memory neural network to introduce the attention mechanism, screen the extracted high-time resolution information, extract the deep-level information of crop accumulated temperature, construct the relationship between the labeled crop and its environment, and establish an effective crop accumulated temperature feature; Step S8: A dual-branch feature fusion module fuses the low- and medium-level satellite image features with the effective accumulated temperature information, and then inputs the multi-level cross attention module to integrate contributions from different scales and capture differences, ultimately obtaining aggregated effective accumulated temperature-satellite remote sensing spatiotemporal features with rich multi-scale information; Step S9: according to the early recognition strategy of incremental classification, using the training set data to generate data sets of different time lengths and input them into the model for training; The training process is based on the Adam optimizer, the cross entropy loss function, and the auxiliary loss function, and finally a classifier trained for different time series lengths is obtained; Step S10: Use the test set to evaluate the accuracy of the results of different classifiers, identify the shortest time series and the time series that reaches a specific accuracy, and the acquisition time of the last scene image is regarded as the earliest identifiable time of the crop, so as to realize the extraction of early crop planting areas at the plot level; Step S11: using the above method and the classifier determined as the earliest identifiable time, carry out early identification of regional crops.

2. The method for early crop remote sensing identification integrating effective accumulated temperature according to claim 1 is characterized in that: The image acquired in step S1 is a medium-resolution time-series Sentinel-1 radar SAR image of the study area. Step S1 is specifically implemented as follows: Step S11, obtaining ground range detection GRD data in the interferometric wide-band IW mode of Sentinel-1 radar SAR images in the early to mid-growth period of the target year's cover crops, preprocessing the data, including thermal noise removal, orbit file correction, radiation calibration, filter correction, and Doppler terrain correction, and finally obtaining backscattering information; Step S12, calculating the VV and VH polarization backscattering characteristics of the time series Sentinel-1 radar SAR images, performing time series smoothing through SG filtering, and clipping them to the study area through plot constraints; The image acquired in step S2 is a medium-resolution time-series Sentinel-2 optical image of the study area. Step S2 is specifically implemented as follows: Step S21, obtaining the Level-1C product of Sentinel-2 optical image data of the early to mid-term growth of cover crops in the target year, which is processed by radiometric calibration and atmospheric correction; in view of the fact that optical images are easily disturbed by clouds, firstly select the medium-resolution time-series Sentinel-2 optical images with cloud cover and snow cover percentage less than 20%, then use the QA60 band marker to remove clouds, mosaic the obtained cloud-free images and crop them to the target area; Step S22: Calculate the vegetation index kNDVI, ReG_RVI, ReG_NDVI, NDWI, REP, and mNDVI705 images of the study area based on the medium-resolution time-series Sentinel-2 optical images. The calculation formula is as follows: Among them, NIR represents the Sentinel-2 near-infrared band, Red represents the red band, Green represents the green band, RE2 and RE1 represent the red edge 1 and red edge 2 bands respectively, and SWIR1 represents the short-wave infrared 1 band. Finally, it is clipped to the study area through the plot constraint; Step S23, using random forest RF to rank the extracted features by importance; using random forest RF, the feature importance is calculated based on the Gini index, and only some classification feature variables with high scores are selected to avoid interference from redundant and irrelevant features, and effectively avoid overfitting and reduce the generalization error of classification; Assume that for a given data set D, which contains n categories, the probability that a sample belongs to the i-th category is P i , then the calculation formula of the Gini index is: The Gini index, also known as Gini impurity, is used to characterize the purity of a sample set. The smaller its value, the smaller the probability of sample misclassification. When the value is 0, the data set categories are consistent. If the set D is divided into k parts according to a certain feature, the Gini index after segmentation is: Where S j is the number of samples at child node j, S is the total number of samples; when the data set traverses all the splits, the Gini index before the split is subtracted from the Gini after the split split (D) The Gini gain value is obtained. The larger the value, the higher the importance, and the better the classification effect when splitting nodes according to the features; Step S24: Based on the optical index and optical band that are beneficial to the early identification of crops, different feature combinations are formed by changing the number of feature variables to obtain the relationship between classification accuracy and feature quantity. The contribution rate of each phase feature is accumulated to screen out the best combination of optical features kNDVI, NDWI, mNDVI705, and Red that are beneficial to early identification.

3. The method for early crop remote sensing identification integrating effective accumulated temperature according to claim 1 is characterized in that: The step S3 is specifically implemented as follows: Step S31, obtaining the maximum and minimum ERA-5Daily temperature data from the early to mid-term growth period of the cover crop in the target year, mosaicking and synthesizing the obtained temperature data, and cropping them to the target area; Step S32: Calculate the effective accumulated temperature of specific crops every day by using the maximum and minimum values ​​of the daily ERA-5Daily data. The calculation formula is as follows: Where t represents the day of the target year, and They represent the minimum and maximum temperatures from the sowing day to the target mid-year day, i = 1, 2, 3…t, respectively. base It is the effective accumulated temperature range of crops, which has a fixed value according to different crops. After calculation, the image is resampled to 10m resolution to match the satellite image in space, and it is cropped to the study area through plot constraints.

4. The method for early crop remote sensing identification integrating effective accumulated temperature according to claim 1 is characterized in that: Step S4: When making the crop sample data set, the method is as follows: process each SAR, optical, and effective accumulated temperature feature time series data into a mat file of size H×W×T, then crop the time series image into a data set of size N×128×128×T, and the label data is of size N×128×128, where N represents the number of 128×128 blocks that the study area is cropped into, and T is the time series length; finally, merge all features into an npz file, where the ratio of the training set to the test set is 6:4, and after processing, it is used as the model input data. Step S4 is also specific The steps include: Step S41: Based on steps S1, S2, and S3, each feature is processed into a mat file in the form of H×W×T, and the effective accumulated temperature features with high temporal resolution are aggregated to the mid-year day obtained by satellite images; Step S42, set the repetition rate overlay of each time series feature data to 0.5, crop it to a specific length and width, i.e., 128*128, and record it as a list, normalize it according to its own characteristics and read it as an npz file, organize it into a four-dimensional tensor T×C×H×W, where T and C represent the time series length and the corresponding number of channels, respectively, and H×W represents the length and width of the image, as input data; Step S43, based on the historical crop distribution data set, crop samples are produced based on plots, the raster value of the crop sample data is assigned to 1, the plot vector is converted to raster data and assigned to 2, the background value is 0, firstly, raster calculation is performed to assign 1 to the cotton on the plot, and then a voting mechanism is adopted for each plot, that is, the plot with the largest proportion is judged as that type of land, and finally other crops are assigned 2, which are organized into a three-dimensional tensor Class×H×W, where Class is the category, which is used as the input label data.

5. The method for early crop remote sensing identification integrating effective accumulated temperature according to claim 1 is characterized in that: The step S5 is specifically implemented as follows: Step S51, constructing a parallel branch network model coupling multi-source remote sensing data; Step S52: The input time series image is transmitted through the encoder, and two parallel branches are designed, the first branch is used to extract the medium-resolution multi-source satellite image features, and the second branch is used to extract the effective accumulated temperature data features; Step S53: Fuse the different modal feature information of the two branches at the decoder front end.

6. The method for early crop remote sensing identification integrating effective accumulated temperature according to claim 1, characterized in that: The step S6 is specifically implemented as follows: Step S61: Based on the STMA model framework, a satellite remote sensing data feature extraction network with a dual-branch ResNet-based multi-level feature extractor and a spatiotemporal attention mechanism module is constructed; Step S62, the encoder is a multi-level feature front-end extractor based on ResNet18, including a standard 7×7 convolution block, a maximum pooling layer and four residual blocks, wherein each residual block contains two 3×3 convolution operations, batch normalization BN, a rectified linear unit ReLU and a jump connection layer consisting of a standard 1×1 convolution operation and BN; after convolution and pooling, the spatial size of the input patch cube is reduced by 1 / 2, 1 / 4 and 1 / 8 respectively, and the channel dimension gradually increases; ResNet extracts multi-scale features in multiple receptive fields, corresponding to shallow features, middle features and high-level features extracted based on different spatial resolutions, respectively, extracts local spatial features at each time in turn, and then concatenates the output features along the time dimension, thereby retaining the time dimension information and promoting the subsequent time attention module to realize time feature extraction; Step S63: Use spatial attention position embedding before entering spatiotemporal attention, use embedded learnable functions to flexibly extract important position information related to the task, and learn complex position relationships from the feature cube extracted by the ResNet module; x′ n =x n ×σ(DConv(x n )) in, Represents the input feature cube and output tag embedding extracted by the CNN-based ResNet module; DConv(.) represents a deep convolution operation with a kernel of 3×3 and padding of 1, extracting spatial weights from the input sequence; deep convolution applies a convolution filter to each input channel, collects spatial information conditioned on the local neighborhood of the input tag, and maintains translation equivalence while keeping small trainable parameters; by assigning different weight coefficients, each pixel in the extracted feature cube can adaptively obtain a position embedding, preserving spatial information to the output tag embedding; Step S64, STSA consists of spatial self-attention, temporal self-attention and feature fusion modules; the spatial self-attention and temporal self-attention parts independently extract spatial and temporal information according to different flattened forms of the encoded feature cube; the feature fusion module includes convolution operation, LayerNorm layer LN and multi-layer perceptron MLP layer, so that the concatenated features are integrated to obtain robust spatiotemporal features and avoid the influence of manual intervention on the concatenation order of spatiotemporal features; first, the input tag embedding x′ n Flattened into two embedding sequences along the spatial and temporal dimensions and Then, the flattened embedding sequence is fed into the multi-head self-attention MSA along the spatial and temporal axes to capture the spatial and temporal information; and Represent the spatial and temporal features of the output of spatial self-attention and temporal self-attention respectively: z S ′=MSA(LN(z S ))+z S z T ′=MSA(LN(z T ))+z T MSA(z)=Concat(head1,...,head h )W O Where h is the number of heads in MSA, W O is the linear transformation matrix used for feature space transformation; Represents the key, query, and value of each header respectively; is a linear transformation matrix whose parameters can be learned; represents the scaling factor; the Softmax activation function provides an attention value from 0 to 1, and the self-attention matrices in the spatial and temporal dimensions are [H n W n , H n W n ] and [T, T] to reflect the weight distribution of different pixels and time; the feature fusion module realizes the aggregation of spatial and temporal features, connects the extracted features in series, and performs a feedforward network similar to the original transformer; the output of STSA is expressed in the following form: It is the spatiotemporal feature of the output; the MLP layer consists of two fully connected layers, activated by Gaussian error linear unit GELU; are learnable network parameters for adaptive fusion; Represents z S ′ and z T ′ is the concatenated result in the third dimension, so that the local structure information is retained in the convolution operation to avoid insufficient fusion; after STSA, the output spatiotemporal feature d is reshaped into the two-dimensional image space to generate discriminative spatiotemporal features For different multi-scale features obtained by ResNet, the STSA module can also obtain mid-level features And high-level features 7. The method for early crop remote sensing identification integrating effective accumulated temperature according to claim 1 is characterized in that: The step S7 is specifically implemented as follows: Step S71, constructing an encoder for fully extracting deep-level information of crop effective accumulated temperature, which is composed of a 2D convolution block, an attention mechanism and an LSTM. The LSTM introduces the attention mechanism as a screening mechanism for extracting high-temporal-resolution information; Step S72, two 2D convolution blocks of the effective accumulated temperature data encoder are responsible for dimensional compression of the data to reduce the calculation pressure before inputting into the LSTM; Step S73: The signal output by LSTM is used through the attention mechanism to highlight the key information of the image by calculating the weight of the image features, thereby helping the model to make more accurate decisions, and finally outputs the extracted features through the Linear layer and the FC layer; In step S72, the LSTM is defined as follows: Each LSTM unit includes a memory unit C t And three gating mechanisms, namely the input gate Input t , Forget t And output gate Output t ; The input data of the LSTM unit at the tth time step is the current unit input signal x t and the previous unit output signal h t When the input data enters the LSTM, the input gate uses the Sigmoid function to determine which information can enter the memory unit; the forget gate uses the Sigmoid function to determine which previous information needs to be discarded; the information output of the input gate and the forget gate determines the memory unit C t The input data is multiplied by the input gate output information after passing through the hyperbolic tangent function tanh, and the forget gate output information is multiplied by the previous memory unit C t-1 Multiply them, and finally sum them to generate a new memory value C t ; The output gate determines the final information output through the Sigmoid function and generates the new memory value C t Through the hyperbolic tangent function, multiply the output information of the output gate to get the final output signal h t , where σ is the Sigmoid function, W is the weight, and tanh represents the hyperbolic tangent function; the specific calculation process is as follows: Input t =σ(W i [h t-1 ,x t ]) Forget t =σ(W f [h t-1 ,x t ]) Output t =σ(W o [h t-1 ,x t ]) C′ t =σ(W c [h t-1 ,x t ]) C t =Multiply[Forget t ,C t-1 ]+Multiply[Input t ,C′ t ] h t =Multiply[Output t ,tanh(C t )]; In step S72, the attention mechanism is defined as follows: LSTM cell unit output signal h t Processed through an attention mechanism to capture important temporal features related to the target object; As an information screening mechanism, the attention mechanism is located between the encoder and the decoder. It highlights the key information of the image by calculating the weight of image features, thereby helping the model make more accurate decisions. The attention mechanism assigns a context vector v to the input sequence data, which depends on the input sequence h = {h1, h2, ...h t }, which can reflect the most important information in the current time step; a ij is the attention weight, which is the model output of a softmax function, indicating the importance of each time step to the model output. This weight calculation method enables the model to focus on the most important part of the input sequence, thereby improving the performance and accuracy of the model; ij is a vector related to the input sequence, measuring the importance of information, and a is a feedforward neural network; the ij =softmax(e ij ) e ij =a(v i-1 ,h j )。 8. The method for early crop remote sensing identification integrating effective accumulated temperature according to claim 1 is characterized in that: The step S8 is specifically implemented as follows: Step S81: construct a dual-branch feature fusion module by using a cascade feature fusion method combined with an attention mechanism to achieve fusion of features extracted by the first branch and the second branch; Step S82: In Dual-STMA, the optical branch and the SAR branch are composed. Different branches extract spatiotemporal features of different levels through ResNet and STSA modules respectively. The features output by the optical branch are represented as z1 Opt 、z2 Opt 、z3 Opt , the SAR branch output feature is represented as z1 SAR 、z2 SAR 、z3 SAR , after being output by STSA, they are fused according to the corresponding scale. The features of different levels extracted after the fusion of optical and SAR data are used. In order to unify the size of different multi-scale features, 1×1 convolution and average pooling operations are used to balance the number of channels to obtain z1, z2, and z3. The fusion process is shown as follows: z n =AvgPool(Conv(Concat(z n Opt +z n SAR ))) The n here represents different levels; Step S83: After the accumulated temperature data passes through the 2D convolution module and the corresponding dimension transformation, it enters the LSTM and attention mechanism, and then extracts high-level features and performs dimension matching through the Linear and FC layers, which serves as the input for subsequent fusion with satellite data; The feature z4 extracted from the ALSTM model accumulated temperature data is fused with the features z3 and z2 to form z3′ and z2′, and then input into the multi-level attention module MCA with z1 to integrate the contributions of different scales and capture the differences between multi-scale features; in, They represent the query, key, and value of the MCA mechanism respectively; Represents the outer product operation; M Q and M K Q c , K c Finally, the valuable semantic information of the high-level feature z3 is added to the generated features to obtain the aggregated temperature-satellite remote sensing spatiotemporal feature z with rich multi-scale information. m .

9. The method for early crop remote sensing identification integrating effective accumulated temperature according to claim 1, characterized in that: The step S9 is specifically implemented as follows: Step S91, training the entire network based on the minimization principle cross entropy CE loss function and the auxiliary loss function, the minimization principle cross entropy loss is used to train the entire network to learn mapping from the input time series to the ground reference data, and the auxiliary loss is used to supervise the optimization process of multi-level attention to prevent the loss of detailed information of the intermediate network layer, so that the network architecture can learn better multi-scale spatiotemporal feature representation, thereby improving the mapping performance of crops in the semantic view; Step S92, the total model loss is expressed as follows: α is used to balance the trade-off between the cross entropy loss and the auxiliary loss; the auxiliary loss function is generated by minimizing the auxiliary map y generated by the CNN-based decoder aux With ground reference map It is calculated based on the cross entropy loss minimization principle, which is only used in the training phase so as not to affect the test phase. Minimization principle cross entropy loss function is based on considering the output crop mapping result y and the ground reference data The reconstruction loss between the two; the multi-class minimization principle cross entropy loss of the minimization principle cross entropy and the auxiliary loss is calculated as shown in the formula of step S91 above, where N represents the number of pixels; and i Represents the unique hot encoding of the true label on the i-th image pixel and the corresponding softmax output of the model; L All <aL CE +(1−α)L Aux 4 Step S93, divide the input data into a training set and a validation set at a ratio of 6:4; use the training set data for training, use the Adam optimizer with an initial learning rate Lr as the optimizer for model training, set the model training batch to N, set the model iteration number to R, measure the quality of model prediction based on the above loss function, and back-propagate the output error to the network parameters for each batch to fit the output of the sample; use accuracy to measure the performance of the model in each batch, and save the model optimal weight file when the validation set accuracy is the highest.

10. The method for early crop remote sensing identification integrating effective accumulated temperature according to claim 1, characterized in that: The step S10 is specifically implemented as follows: Step S101, training model weight files based on different time series lengths, performing crop prediction when stitching images cropped according to corresponding time series, and generating crop recognition results of different time series lengths; Step S102: Use the ground truth samples to evaluate the crop recognition results of each time series, and use the incremental classification strategy for early recognition, that is, the acquisition time of the latest image captured in the time series with the shortest time series and reaching a specific accuracy is regarded as the earliest identifiable time of the crop; The step S11 is specifically implemented as follows: Step S11 1. Obtain Sentinel_1A, Sentinel-2, and ERA5 Daily temperature data in the application range area and with a time span from the crop sowing period to the earliest identifiable time, and process them according to steps S1, S2, and S3 to form model input data; Step S112: Perform early recognition on the data to be applied based on the model weight file obtained by training the corresponding time series length of the earliest identifiable time, and obtain the extraction result of early crop recognition.

Images