Variable temporal remote sensing crop classification method and system fusing convolution and self-attention

Abstract

The application discloses a variable time remote sensing crop classification method and system fusing convolution and self-attention, and relates to the technical field of image data processing. The method comprises the following steps: acquiring multi-period Sentinel-2 time series remote sensing reflectivity images of a crop growth cycle of a target area, and calculating a normalized difference vegetation index (NDVI); generating monthly optimal clear-sky observation reflectivity images and monthly optimal clear-sky observation NDVI through an NDVI maximum value monthly synthesis method; performing data fusion on the monthly optimal clear-sky observation reflectivity images and the monthly optimal clear-sky observation NDVI to generate a time series dataset; inputting the time series dataset into a preset variable time crop classification model to output a remote sensing crop classification result; and the preset variable time crop classification model is a variable time crop classification model constructed by fusing a dynamic time convolution and a global self-attention mechanism. The application improves the regional generalization of the crop classification model and the accuracy of crop recognition.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, specifically to a variable temporal remote sensing crop classification method and system that integrates convolution and self-attention. Background Technology

[0002] Precision agriculture is a method that utilizes advanced technologies and information technology to achieve precise management of the entire agricultural production process. Traditional remote sensing image classification methods have made significant progress in large-scale remote sensing crop classification. However, given the large-scale and high-precision requirements of smart agriculture, there is an urgent need to develop crop classification methods at the arable land plot scale. In recent years, with the improvement of satellite remote sensing sensor payloads and data availability, high-resolution remote sensing imagery has provided a rich data foundation for the precise identification of crop planting distribution. Deep learning models and their ability to automatically extract high-dimensional hierarchical features from images have made it possible to achieve precise remote sensing crop classification at the arable land plot scale, and have already demonstrated considerable potential and advantages in precise crop identification tasks.

[0003] Existing technical solutions can be categorized based on both technical implementation and data sources. In terms of data sources, they can be divided into: crop classification based on single high-resolution remote sensing imagery, crop classification methods based on hyperspectral imagery, remote sensing crop classification methods based on multi-temporal multispectral remote sensing imagery, and crop classification methods based on the fusion of Synthetic Aperture Radar (SAR) and optical imagery. In terms of technical implementation, they can be divided into: crop classification methods based on machine learning models and crop classification methods based on deep learning models. Currently, with the development and widespread adoption of remote sensing intelligent interpretation technology, these two dimensions of crop classification methods are largely being used in a cross-fertilization approach.

[0004] Due to significant regional variations in crop distribution, complex crop growth structures, and difficulties in labeling spaceborne remote sensing data caused by cloud cover and other factors, existing technologies still struggle to meet the demands of refined crop classification tasks in situations with substantial spatiotemporal differences and complex regional planting structures. Furthermore, because crop planting and growth exhibit phenological characteristics across different seasons and growth stages, their appearance and spectral properties change, requiring models to understand and adapt to these temporal variations. In addition, during multi-temporal phenological feature analysis, regional climate differences often lead to data gaps in optical remote sensing images due to cloud cover and other factors, necessitating models with the ability to generalize to handle random temporal data gaps.

[0005] In existing crop classification and identification methods, data sources mostly consist of single-period high-resolution optical remote sensing images and multi-time-series medium-resolution remote sensing images. SAR images are often used as a supplementary data source for feature extraction and are not used alone for crop classification and identification. Among these:

[0006] The existing technologies disclosed in CN113989668B and CN108932521B are based on multi-temporal satellite imagery to construct machine learning or deep learning models for crop classification. This approach effectively utilizes the temporal characteristics of crop growth; however, when affected by significant cloud and fog conditions or regional migration, the absence of one or more data periods in the temporal series renders the trained model unusable, requiring retraining with labeled samples in new regions. This approach lacks generalization ability across different regions and data sets.

[0007] Existing technologies with publication numbers CN115035413B, CN116664959B, and CN112183209B construct multi-temporal, multi-source fused time-series datasets by fusing SAR imagery and optical remote sensing time-series imagery. Leveraging the SAR sensor's immunity to cloud and fog interference, they fuse backscattering features, vegetation indices, and time-series data to improve the accuracy of crop remote sensing identification. However, this approach still fails to address the alignment issues that often arise between SAR imagery and optical data when optical time-series data is randomly missing. Optical time-series images suffer from data loss due to cloud and fog interference, while SAR remains unaffected. In practical operational use, this approach is often limited in its widespread application due to data loss.

[0008] The existing technologies disclosed in CN116091940B and CN114092832B, based on high-resolution remote sensing imagery, utilize lightweight encoders and parallel training of two- and three-dimensional convolutions to train deep learning crop classification models. These methods can leverage the high-resolution characteristics of high-resolution imagery to achieve refined crop classification at the plot scale. However, due to temporal differences in planting habits across different regions and variations in the appearance and spectral characteristics of crops at different growth stages, it is difficult to capture key phenological stages of crops based on single-period high-resolution remote sensing imagery. Furthermore, the trained models are often affected by differences in image style and regional variations, making it difficult to generalize crop classification across different regions based solely on optical image features.

[0009] The existing technology disclosed in CN110751019B constructs and trains models for different phenological stages in high-resolution images, and then uses the appropriate model matched to the phenological stage to extract crop classification results. This method addresses the problem of significant feature differences for the same crop at different phenological stages to some extent. However, when regional planting habits differ, matching the phenological stage to the corresponding model becomes a bottleneck limiting the model's recognition accuracy. Furthermore, this approach requires training models specifically for different phenological stages, placing high demands on data and samples.

[0010] In view of the above, this application is hereby submitted. Summary of the Invention

[0011] This invention aims to address two major technical shortcomings of existing crop classification methods: first, data gaps lead to inconsistent temporal lengths, limiting the model's generalization ability across different regions and time series; second, the accuracy of existing models is limited by image quality, with low-quality images resulting in low model recognition accuracy. The purpose of this invention is to provide a variable temporal remote sensing crop classification method and system that integrates convolution and self-attention. It fuses a network obtained by combining dynamic temporal convolution and global self-attention mechanisms, and extracts spatial and temporal phenological features of crops based on variable temporal medium-resolution Sentinel-2 reflectance images, combined with vegetation indices, to achieve fine-grained crop classification under complex climatic and topographical conditions. This invention effectively solves the problem of needing to retrain the model when temporal data is missing, effectively improves the model's regional generalization ability, and is an effective technical means to improve the accuracy of refined crop classification.

[0012] This invention is achieved through the following technical solution:

[0013] In a first aspect, the present invention provides a variable temporal remote sensing crop classification method that integrates convolution and self-attention, the method comprising:

[0014] Acquire remote sensing image data of the target area, wherein the remote sensing image data is multi-period Sentinel-2 time-series remote sensing reflectance images of the crop growth cycle;

[0015] Based on remote sensing image data, the Normalized Difference Vegetation Index (NDVI) is calculated; and the monthly best clear-sky reflectance image and the monthly best clear-sky NDVI are generated by combining the maximum NDVI values ​​monthly.

[0016] The monthly best clear sky observation reflectance image and the monthly best clear sky observation NDVI are fused to generate a time-series dataset;

[0017] Input the time-series dataset into the preset variable time-series crop classification model and output the remote sensing crop classification results; the preset variable time-series crop classification model is a variable time-series crop classification model constructed by integrating dynamic temporal convolution and global self-attention mechanism.

[0018] Furthermore, based on remote sensing image data, the Normalized Difference Vegetation Index (NDVI) is calculated; by combining the maximum NDVI values ​​monthly, the monthly best clear-sky reflectance image and the monthly best clear-sky NDVI are generated, including:

[0019] Based on the Sentinel-2 time-series remote sensing reflectance image, cloud masking is performed on the cloud-covered areas to obtain the processed time-series remote sensing reflectance image.

[0020] Based on the processed temporal remote sensing reflectance image, the normalized difference vegetation index (NDVI) was calculated using Sentinel-2 multiband reflectance.

[0021] By using the monthly composite method of maximum NDVI, the reflectance of each band in each time series within each month is sorted according to the maximum NDVI and compared. The maximum NDVI is selected as the representative reflectance for that month, generating the monthly best clear sky observation reflectance image and the monthly best clear sky observation NDVI.

[0022] Furthermore, the monthly best clear-sky reflectance image and the monthly best clear-sky NDVI image are fused to generate a time-series dataset, including:

[0023] The missing observation values ​​were filled based on the monthly best clear sky reflectance image and the monthly best clear sky NDVI.

[0024] The inflated monthly best clear-sky reflectance image and the monthly best clear-sky NDVI image are fused to generate a time-series dataset. Where T represents the dynamically variable length of the time series month; C represents the number of image bands; H represents the image height; and W represents the image width.

[0025] Furthermore, the pre-defined variable temporal crop classification model includes a dynamic temporal convolution module, a global self-attention module, a temporal residual enhanced feature fusion module, and a classification decision module connected in sequence. The dynamic temporal convolution module is also connected to the temporal residual enhanced feature fusion module; wherein:

[0026] The dynamic temporal convolution module extracts local features based on the dynamic temporal convolution mechanism and temporal pooling strategy. These local features include local spatial characteristics of crops under different time series lengths and key temporal patterns of phenological changes.

[0027] The global self-attention module improves the global self-attention mechanism by introducing time-aware linear transformations and decaying attention weights, enabling the model to adapt to different time lengths and extract global features; the global features are the global correlation features of key phenological information of crops.

[0028] The temporal residual enhancement feature fusion module introduces temporal residual connections to decompose and weightedly fuse local and global features to obtain fused features.

[0029] The classification decision module maps the fused features to the category space based on the fully connected layer, and normalizes the output of the fully connected layer through the Softmax activation function.

[0030] Furthermore, the convolution calculation formula for the dynamic temporal convolution mechanism is as follows:

[0031]

[0032]

[0033] In the formula, For local features; r is the time-series dimension index. For a three-dimensional convolutional kernel that includes a temporal dimension, For spatial height, For space width, The temporal length is the size of the convolutional kernel in the temporal dimension. For input channel index; The spatial height index for the convolution kernel; This is the index of the convolution kernel's spatial width. The month interval between the query vector Q and the key vector K corresponding to the time series points in the self-attention mechanism module; For output channel index; This represents the feature value with spatial coordinates (i+p, j+q), temporal dimension index r, and channel index c in the input temporal matrix M, corresponding to the original or preprocessed data of a certain pixel in a certain month and a certain band; T is a dynamically variable temporal month length parameter.

[0034] Furthermore, global features are extracted from the global self-attention module. The specific steps are as follows:

[0035] The local features are mapped to three time-series length T: a query vector Q, a key vector K, and a value vector V, through a time-aware linear transformation. ;in, Local features; These are the learnable parameter matrices for the query vector Q, key vector K, and value vector V, respectively.

[0036] Calculate the attention weight matrix A based on the query vector Q and the key vector K; the expression for the attention weight matrix A is:

[0037] ,in, To query the interval between time series points corresponding to vector Q and key vector K (e.g., the interval between time series points in the 3rd month and the 5th month is...) =2), The learnable decay coefficient (default value is 0.1); is the size of Q / K, representing the feature dimensions of Q and K; j is the index of the K time series point, used to traverse all time series points corresponding to K to ensure that the attention weight of Q with each K time series point is calculated;

[0038] The attention weight matrix is ​​Softmax normalized, and the global features are calculated and output. The expression is .

[0039] Furthermore, the feature fusion module for temporal residual enhancement decomposes and weights local and global features separately, and then fuses them. The specific steps are as follows:

[0040] Local features Decomposed into spatial components X s and time-series components X t ;

[0041] global features Decomposed into key phenological period component X p and time-related components X c ;

[0042] Using a weighted fusion method, local features are analyzed. and global features The decomposed components are fused to obtain the fused features; the expression for the weighted fusion method is: ,in, Features of fusion; They are spatial components X s Time-series component X t Key phenological period component X p and time-related components X c The corresponding learnable weights, and satisfying .

[0043] Furthermore, the loss function L used in the training process of the pre-defined variable time-series crop classification model is:

[0044]

[0045] Where N is the total number of samples and C is the total number of categories. This is the input data for the i-th sample; This is a mask function used to determine if data is missing; when x is a missing value, When x is valid data, ; and Let be the predicted output probability and the true label of the k-th class for the i-th sample, respectively. As a time-series weighting factor, it assigns high weight to the loss calculation during critical phenological periods and low weight to the loss calculation during non-critical periods.

[0046] Secondly, this invention provides a variable temporal remote sensing crop classification system that integrates convolution and self-attention, the system comprising:

[0047] The acquisition unit is used to acquire remote sensing image data of the target area, wherein the remote sensing image data is a multi-period Sentinel-2 time-series remote sensing reflectance image of the crop growth cycle.

[0048] The monthly composite unit with maximum NDVI is used to calculate the Normalized Difference Vegetation Index (NDVI) based on remote sensing image data; and to generate the monthly best clear-sky reflectance image and the monthly best clear-sky NDVI by using the monthly composite method with maximum NDVI.

[0049] The time-series dataset generation unit is used to fuse the monthly best clear sky observation reflectance image and the monthly best clear sky observation NDVI to generate a time-series dataset.

[0050] The classification unit is used to input the time-series dataset into the preset variable time-series crop classification model and output the remote sensing crop classification results; the preset variable time-series crop classification model is a variable time-series crop classification model constructed by integrating dynamic temporal convolution and global self-attention mechanism.

[0051] Furthermore, the pre-defined variable temporal crop classification model includes a dynamic temporal convolution module, a global self-attention module, a temporal residual enhanced feature fusion module, and a classification decision module connected in sequence. The dynamic temporal convolution module is also connected to the temporal residual enhanced feature fusion module; wherein:

[0052] The dynamic temporal convolution module extracts local features based on the dynamic temporal convolution mechanism and temporal pooling strategy. These local features include local spatial characteristics of crops under different time series lengths and key temporal patterns of phenological changes.

[0053] The global self-attention module improves the global self-attention mechanism by introducing time-aware linear transformations and decaying attention weights, enabling the model to adapt to different time lengths and extract global features; the global features are the global correlation features of key phenological information of crops.

[0054] The temporal residual enhancement feature fusion module introduces temporal residual connections to decompose and weightedly fuse local and global features to obtain fused features.

[0055] The classification decision module maps the fused features to the category space based on the fully connected layer, and normalizes the output of the fully connected layer through the Softmax activation function.

[0056] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0057] This invention effectively solves the problem that crop classification models cannot generalize across different regions and time series data of varying lengths when there are random gaps in the temporal data of remote sensing images in remote sensing crop classification tasks. The proposed variable temporal crop classification model (i.e., a variable temporal crop classification model that integrates dynamic temporal convolution and global self-attention mechanisms) can effectively cope with random gaps in the temporal data of remote sensing images, adapt to the temporal differences in planting habits in different regions and the characteristic changes of crops at different growth stages, thereby improving the regional generalization of the crop classification model and the accuracy of crop identification.

[0058] (1) Significant advantages in processing variable temporal data: Existing technologies often lead to model failure when faced with random missing temporal data of remote sensing images, requiring re-labeling of samples for training, and are difficult to apply when data is missing or the region is shifted. However, this invention divides monthly multi-temporal data, uses monthly synthesis of Sentinel-2 reflectivity images with the maximum NDVI value, and fills in temporal positions through specific data correspondences. Combined with a loss calculation mechanism, the model can effectively learn variable temporal features, solving the problem of model unusability due to missing data and improving the model's adaptability to different data integrity scenarios.

[0059] (2) Multi-feature fusion effectively improves classification accuracy: Although existing technologies use multi-source data fusion, they suffer from problems such as temporal feature alignment, which affects classification results. In contrast, this invention deeply integrates different crop planting conditions and characteristics to construct a multi-temporal, vegetation index combination dataset (i.e., temporal dataset), which significantly enhances the feature differences between different crops. At the same time, it integrates dynamic temporal convolutional neural networks and global self-attention mechanisms to achieve multi-feature fusion learning of spatial texture features, color features, and temporal phenological information within the crop growth cycle. Compared with existing technologies, it can extract crop features more comprehensively and accurately, thereby effectively improving the accuracy of refined crop classification.

[0060] (3) Enhanced Model Generalization Ability through Network Architecture: Existing technologies based on high-resolution remote sensing images are susceptible to the influence of image style and regional differences, resulting in poor model generalization. Furthermore, the phenological period matching model also suffers from limitations due to phenological differences, leading to incorrect matching. This invention utilizes an innovative architecture that integrates dynamic temporal convolution and global self-attention mechanisms. This architecture enables the model to adapt to temporal differences in planting habits across different regions and the characteristic changes of crops at different growth stages, effectively improving the model's generalization ability under different regional and data conditions. It eliminates the need for extensive model training for different phenological periods, reducing the requirements for data and samples. Attached Figure Description

[0061] The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and form part of this application, do not constitute a limitation thereof. In the drawings:

[0062] Figure 1 This is a flowchart of the variable temporal remote sensing crop classification method that integrates convolution and self-attention according to the present invention.

[0063] Figure 2 This is a flowchart illustrating the construction process of the time-series dataset of this invention;

[0064] Figure 3 This is a structural diagram of the variable time-series crop classification model preset in this invention;

[0065] Figure 4 This is a block diagram of the variable temporal remote sensing crop classification system that integrates convolution and self-attention according to the present invention. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0067] Currently, there is an urgent need for crop classification schemes that can still generalize even when temporal data of remote sensing images is randomly missing. This invention proposes a variable temporal remote sensing crop classification scheme that integrates convolution and global self-attention mechanisms. By using the integrated convolution and global self-attention mechanisms, and inputting remote sensing images and spectral indices within the variable temporal crop growth cycle, it achieves multi-feature fusion learning of spatial texture features, shape features, and temporal phenological features within the crop growth cycle. This effectively solves the problem of needing to retrain the model when temporal data is missing, and effectively improves the model's regional generalization ability. It is an effective technical means to improve the accuracy of refined crop classification.

[0068] Specifically, this invention proposes a fusion network that integrates dynamic temporal convolution and global self-attention, and based on variable temporal medium-resolution Sentinel-2 temporal remote sensing reflectance images, it extracts spatial and temporal phenological features of crops by combining multiple vegetation indices, thereby achieving fine classification of crops under complex climatic and topographical conditions. The core key point of this invention is:

[0069] 1) Variable Time Series Data Processing: Multi-time series data is divided into monthly segments, with the annual time series length broken down into monthly time scales. The monthly NDVI maximum value is used to synthesize and process the multi-time series data, constructing a unified time series framework. Time series positions are filled using specific data correspondences, and a variable time series loss calculation mechanism is combined to enable the model to effectively learn variable time series features.

[0070] 2) Innovative Fusion Architecture with Variable Temporal Sequence: This invention integrates dynamic temporal convolution and a global self-attention mechanism to achieve complementary advantages between spatial and temporal feature information, effectively improving crop classification accuracy. Specifically, it proposes a dynamic temporal convolution mechanism and a temporal pooling strategy, achieving matching of different temporal lengths through adaptive adjustment of the temporal dimension. Furthermore, by introducing mechanisms such as temporal-aware linear transformation, temporal decay factors for attention weights, and temporal residual connections, the model can maintain high classification accuracy even when some temporal data is missing. This invention effectively solves the problem in existing technologies where feature misalignment leads to abnormal classification results when temporal data is missing.

[0071] 3) Time-weighted loss calculation: This invention introduces a time-weighting factor for variable time-series data. Different weights are assigned to the loss calculation for critical and non-critical phenological periods, so that the loss calculation focuses on effective data and critical phenological periods, ignoring the interference of imputed values, and improving the model's learning efficiency for variable time series data.

[0072] Example 1

[0073] like Figure 1 As shown, this invention provides a variable temporal remote sensing crop classification method that integrates convolution and self-attention. The method includes:

[0074] Step 1: Acquire remote sensing image data of the target area. The remote sensing image data is multi-period Sentinel-2 time-series remote sensing reflectance images of the crop growth cycle.

[0075] In this embodiment, the present invention selects medium-resolution Sentinel-2 reflectance imagery as the data source. Considering the significant differences in phenological characteristics of different crops during their growth cycle, remote sensing images of crops at multiple time points throughout their entire growth cycle from sowing to harvest are collected to obtain rich temporal information.

[0076] Step 2: Calculate the Normalized Difference Vegetation Index (NDVI) based on remote sensing image data; generate the monthly best clear-sky reflectance image and the monthly best clear-sky NDVI by combining the maximum NDVI values ​​monthly.

[0077] In this embodiment, step 2 specifically includes:

[0078] Step 21: Based on the Sentinel-2 time-series remote sensing reflectance image, perform cloud masking on the cloud-covered areas to obtain the processed time-series remote sensing reflectance image.

[0079] Specifically, based on the Sentinel-2 time-series remote sensing reflectance image, it is determined whether there are cloud areas. If there are cloud areas, the cloud masking attribute in the Sentinel-2 L2A dataset of the Google Earth Engine cloud computing platform is used to perform cloud masking on the cloud areas, remove the cloud area pixels, and fill them with 0 values ​​as missing values.

[0080] Step 22: Based on the processed temporal remote sensing reflectance image, calculate the Normalized Difference Vegetation Index (NDVI) using Sentinel-2 multi-band reflectance; the NDVI calculation formula is as follows:

[0081]

[0082] In the formula, NIR is the reflectance in the near-infrared band, and RED is the reflectance in the red band.

[0083] Step 23: By using the monthly composite method of maximum NDVI, the reflectance of each band in each time series within each month is sorted according to the maximum NDVI and compared. The maximum NDVI is selected as the representative reflectance of the month, generating the monthly best clear sky observation reflectance image and the monthly best clear sky observation NDVI.

[0084] Specifically, the Sentinel-2 reflectance image data for each month is preprocessed and composited by stitching and cropping the images to obtain the Sentinel-2 multiband reflectance image of the target area using the monthly composite method based on the maximum NDVI value (i.e., the best clear sky observation).

[0085] Let the NDVI dataset for the i-th month be: The NDVI data set after the monthly synthesis of the maximum NDVI value is: .

[0086] The method for combining maximum reflectance images each month is as follows:

[0087]

[0088] in, This represents the maximum reflectance value in the m-th month and the b-th band. It is the set of all valid observation dates in the m-th month. This indicates the reflectivity of the date d and the band b.

[0089] This synthesis method sorts the reflectance of each band in each time series within each month according to the maximum NDVI value and compares them. The maximum NDVI value is selected as the representative reflectance of that month, generating the reflectance image of the best clear sky observation for each month.

[0090] Step 3: Perform data fusion on the monthly best clear sky observation reflectance image and the monthly best clear sky observation NDVI to generate a time series dataset;

[0091] In this embodiment, step 3 specifically includes:

[0092] Step 31: Fill in the missing observation values ​​based on the monthly best clear sky observation reflectance image and the monthly best clear sky observation NDVI.

[0093] Specifically, if any month's data is missing, this invention employs a common data imputation method, using a value of 0 as the missing value. In a 12-month time series, for months with missing data... The filling rules are as follows:

[0094]

[0095] That is, the reflectance and NDVI values ​​of all bands for the missing months are set to 0.

[0096] Step 32: Perform data fusion on the filled monthly best clear sky observation reflectance image and the monthly best clear sky observation NDVI to generate a time series dataset. Where T represents the dynamically variable length of the time series month; C represents the number of image bands; H represents the image height; and W represents the image width.

[0097] To achieve accurate crop feature extraction, this invention further integrates Sentinel-2 remote sensing reflectance images synthesized from the monthly maximum values ​​each year and the monthly best clear-sky observation NDVI, and generates a time-series dataset with variable time series. The specific steps are as follows:

[0098] The inflated monthly best clear-sky reflectance image and the monthly best clear-sky NDVI image are fused to generate a time-series dataset M(T*C*H*W); the fusion formula is:

[0099]

[0100] Where m represents the month, i represents the data channel index, and B is the band number of the Sentinel-2 reflectance image. When i is in the range of 1 to B, the maximum reflectance of the corresponding band for the corresponding month is taken; when i = B+1, the NDVI value of the corresponding month is taken.

[0101] The above processing method can effectively solve the problem of misalignment of time-series features due to missing data, and can be applied to crop feature extraction and classification tasks with any number of months (valid data months >= 1).

[0102] Step 4: Input the time series dataset into the preset variable time series crop classification model and output the remote sensing crop classification results; the preset variable time series crop classification model is a variable time series crop classification model constructed by integrating dynamic temporal convolution and global self-attention mechanism.

[0103] In this embodiment, the preset variable temporal crop classification model includes a dynamic temporal convolution module, a global self-attention module, a temporal residual enhanced feature fusion module, and a classification decision module connected in sequence. The dynamic temporal convolution module is also connected to the temporal residual enhanced feature fusion module; wherein:

[0104] The dynamic temporal convolution module extracts local features based on the dynamic temporal convolution mechanism and temporal pooling strategy. These local features include local spatial characteristics of crops under different time series lengths and key temporal patterns of phenological changes.

[0105] The global self-attention module improves the global self-attention mechanism by introducing time-aware linear transformations and decaying attention weights, enabling the model to adapt to different time lengths and extract global features; the global features are the global correlation features of key phenological information of crops.

[0106] The temporal residual enhancement feature fusion module introduces temporal residual connections to decompose and weightedly fuse local and global features to obtain fused features.

[0107] The classification decision module maps the fused features to the category space based on the fully connected layer, and normalizes the output of the fully connected layer through the Softmax activation function.

[0108] In this embodiment, the modules in the preset variable time-series crop classification model are described in detail below:

[0109] (1) Dynamic temporal convolution module:

[0110] The time-series dataset M generated in step 3 is input into the dynamic temporal convolution module. In variable temporal scenarios, differences in the temporal length of different samples may lead to incomplete local feature extraction. To address this, this invention proposes a dynamic temporal convolution mechanism: the convolution kernel employs multi-scale (3×3, 5×5) kernels in the spatial dimension, while introducing a dynamically variable time-series month length parameter T (1<=T<=12) in the temporal dimension, automatically adapting the temporal dimension size of the convolution kernel to T. The calculation formula is as follows:

[0111]

[0112] in, The temporal length refers to the size of the convolutional kernel in the temporal dimension. It is associated with T through a linear mapping function, thereby automatically adapting the kernel size to different temporal lengths. The convolution calculation formula for the dynamic temporal convolution mechanism is:

[0113]

[0114] In the formula, For local features; r is the time-series dimension index. For a three-dimensional convolutional kernel that includes a temporal dimension, For spatial height, For space width, The temporal length is the size of the convolutional kernel in the temporal dimension. For input channel index; The spatial height index for the convolution kernel; This is the index of the convolution kernel's spatial width. The month interval between the query vector Q and the key vector K corresponding to the time series points in the self-attention mechanism module; For output channel index; This represents the feature value with spatial coordinates (i+p, j+q), temporal dimension index r, and channel index c in the input temporal matrix M, corresponding to the original or preprocessed data of a certain pixel in a certain month and a certain band; T is a dynamically variable temporal month length parameter.

[0115] The above method extracts the spatial features and key temporal patterns of crops in both spatial and temporal dimensions by simultaneously sliding the spatial and temporal dimensions, thus forming local features. .

[0116] The pooling layer employs a temporal pooling strategy, which reduces spatial dimensionality while preserving key temporal nodes. For example, interval sampling max pooling is used for the temporal dimension. The calculation formula is as follows:

[0117]

[0118] Where S is the temporal pooling step size, S=2 when T>=6 (retaining a maximum value for every two time points); when T<6, S=1 (retaining the maximum value for all time points), r is the index of the temporal dimension after pooling, i is the height coordinate of the feature matrix space, representing the vertical position index of the feature map, and j is the width coordinate of the feature matrix space, representing the horizontal position index of the feature map. This strategy avoids feature loss in short-term data while reducing redundant information in long-term data.

[0119] This dynamic temporal convolution module is designed for medium-resolution multi-temporal images. It can extract local spatial features of crops at different temporal lengths and capture key temporal patterns of phenological changes, thus solving the feature extraction bias problem of conventional convolution kernels in scenarios with missing temporal data.

[0120] (2) Global Self-Attention Module

[0121] When processing fixed-time-series data, the dimensions of the query vector Q, key vector K, and value vector V are designed based on a preset time-series length. However, variable-time-series data can lead to distortion in the calculation of long-distance dependencies. This invention improves the global self-attention mechanism by introducing time-aware linear transformations and decaying attention weights, enabling the model to adapt to different time-series lengths and capture the global correlation of key phenological information of crops. Specifically, the design of this module is as follows:

[0122] Local features are transformed through time-aware linear transformation. The mapping is represented by three time-series vectors: a query vector Q, a key vector K, and a value vector V, each with a time series length of T. ;in, Local features; These are the learnable parameter matrices for the query vector Q, key vector K, and value vector V, respectively; their calculation formulas are as follows: .in, It is a mapping function learned through training. Specifically, the dimensions of the parameter matrix satisfy:

[0123] In the formula, dim() represents the dimension of the parameter matrix;

[0124] Calculate the attention weight matrix A based on the query vector Q and the key vector K; the expression for the attention weight matrix A is:

[0125] ,in, To query the interval between time series points corresponding to vector Q and key vector K (e.g., the interval between time series points in the 3rd month and the 5th month is...) =2), The learnable decay coefficient (default value is 0.1); is the size of Q / K, representing the feature dimensions of Q and K; j is the index of the K time series point, used to traverse all time series points corresponding to K to ensure that the attention weight of Q with each K time series point is calculated;

[0126] The attention weight matrix is ​​Softmax normalized, and the global features are calculated and output. The expression is .

[0127] The above technical solutions, through the design of time-series data in this invention, enable the model to pay more attention to the correlation between adjacent time series of crop growth stages when calculating long-distance dependencies, while weakening time-series interference with excessively large spans and improving robustness to missing data.

[0128] (3) Feature fusion module for temporal residual enhancement

[0129] To fully utilize the local features extracted by convolutional networks (Including spatial features and temporal patterns) and global features extracted by the global self-attention mechanism This invention redesigns the feature fusion layer, introducing temporal residual links to integrate local features. Decomposed into spatial components X s and time-series components X t ; global features Decomposed into key phenological period component X p and time-related components X c Then, a weighted fusion method is used to analyze local features. and global features The decomposed components are fused to obtain the fused features; the expression for the weighted fusion method is: ,in, Features of fusion; They are spatial components X s Time-series component X t Key phenological period component X p and time-related components X c The corresponding learnable weights, and satisfying .

[0130] The above technical solution, through dynamic weight allocation, enables the model to automatically enhance the weight of effective observations in spring and autumn when time series are missing (e.g., cloudy and rainy weather in summer), thereby making up for the feature gaps caused by missing time series.

[0131] (4) Classification Decision Module

[0132] The fused features Input classification decision layer. This layer consists of a fully connected layer and a softmax activation function. The fully connected layer maps the fused features to the class space, and the calculation formula is as follows:

[0133]

[0134] in, Let be the weight matrix of the fully connected layer, and b be the bias vector. Softmax normalizes the output of the fully connected layer to obtain the probability value corresponding to each category, as shown in the formula:

[0135]

[0136] in, Let be the i-th element of the output vector of the fully connected layer, and C be the total number of classes. Let be the probability that a sample belongs to the i-th class. By selecting the class with the highest probability, the model output is mapped to the crop type.

[0137] In this embodiment, the calculation of time-weighted loss is explained as follows:

[0138] When constructing a variable time-series dataset, this proposal synthesizes and partitions the multi-time-series data by combining the best clear-sky observation data within a month using the monthly maximum value, and embeds it into the corresponding time-series position, enabling effective alignment of the multi-time-series dataset at the monthly scale. To make the model robust and generalizable to data with missing time series during training, this proposal redesigns the loss calculation method for variable time-series data.

[0139] Let the model's predicted output be The true label is y. We use the common cross-entropy loss function as a basis. The original cross-entropy loss function is calculated as follows:

[0140]

[0141] Where C represents the total number of categories. This represents the probability of the k-th class in the true label. This represents the probability that the model predicts a sample belongs to the k-th class. A mask function is introduced to address this technique. This is used to determine if the data is missing. When x is a missing value... When x is valid data, .

[0142] This invention introduces a time-series weighting factor for variable time-series data. ( The calculation of losses during key phenological periods (such as the crop maturity period from June to August) is given a higher weight. =1.2), assigning lower weights to non-critical periods (such as the period from November to February of the following year when crops are not planted). =0.8), the improved calculation method for the loss function L, which is adaptable to variable time lengths, is as follows:

[0143]

[0144] Where N is the total number of samples and C is the total number of categories. This is the input data for the i-th sample; This is a mask function used to determine whether data is missing. and Let be the predicted output probability and the true label of the k-th class for the i-th sample, respectively. As a time-series weighting factor, it assigns high weight to the loss calculation during critical phenological periods and low weight to the loss calculation during non-critical periods.

[0145] The proposed fusion architecture of dynamic temporal convolution and global self-attention mechanism, along with the variable temporal loss calculation mechanism, can effectively process variable temporal data and fuse local spatial features, global temporal features, and phenological information from images. This enables fine-grained classification of crops even when complex climatic conditions result in missing temporal data.

[0146] Example 2

[0147] like Figure 4 As shown, the difference between this embodiment and Embodiment 1 is that this embodiment provides a variable temporal remote sensing crop classification system that fuses convolution and self-attention. This system corresponds one-to-one with the variable temporal remote sensing crop classification method that fuses convolution and self-attention in Embodiment 1. The system includes:

[0148] The acquisition unit is used to acquire remote sensing image data of the target area. The remote sensing image data is multi-period Sentinel-2 time-series remote sensing reflectance images of crops during their growth cycle.

[0149] The monthly composite unit with maximum NDVI is used to calculate the Normalized Difference Vegetation Index (NDVI) based on remote sensing image data; and to generate the monthly best clear-sky reflectance image and the monthly best clear-sky NDVI by using the monthly composite method with maximum NDVI.

[0150] The time-series dataset generation unit is used to fuse the monthly best clear sky observation reflectance image and the monthly best clear sky observation NDVI to generate a time-series dataset.

[0151] The classification unit is used to input the time-series dataset into the preset variable time-series crop classification model and output the remote sensing crop classification results; the preset variable time-series crop classification model is a variable time-series crop classification model constructed by integrating dynamic temporal convolution and global self-attention mechanism.

[0152] As a further implementation, the pre-defined variable temporal crop classification model includes a dynamic temporal convolution module, a global self-attention module, a temporal residual enhanced feature fusion module, and a classification decision module connected in sequence. The dynamic temporal convolution module is also connected to the temporal residual enhanced feature fusion module; wherein:

[0153] The dynamic temporal convolution module extracts local features based on the dynamic temporal convolution mechanism and temporal pooling strategy. These local features include local spatial characteristics of crops under different time series lengths and key temporal patterns of phenological changes.

[0154] The global self-attention module improves the global self-attention mechanism by introducing time-aware linear transformations and decaying attention weights, enabling the model to adapt to different time lengths and extract global features; the global features are the global correlation features of key phenological information of crops.

[0155] The temporal residual enhancement feature fusion module introduces temporal residual connections to decompose and weightedly fuse local and global features to obtain fused features.

[0156] The classification decision module maps the fused features to the category space based on the fully connected layer, and normalizes the output of the fully connected layer through the Softmax activation function.

[0157] The execution process of each unit can be carried out according to the steps of the variable temporal remote sensing crop classification method with fusion convolution and self-attention in Example 1, and will not be described in detail in this example.

[0158] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0159] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0160] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0161] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0162] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A variable temporal remote sensing crop classification method integrating convolution and self-attention, characterized in that, The method includes: Acquire remote sensing image data of the target area, wherein the remote sensing image data is multi-period Sentinel-2 time-series remote sensing reflectance images of the crop growth cycle; Based on the remote sensing image data, the Normalized Difference Vegetation Index (NDVI) is calculated; and the monthly best clear-sky reflectance image and the monthly best clear-sky NDVI are generated by combining the maximum NDVI values ​​monthly. The monthly best clear sky observation reflectance image and the monthly best clear sky observation NDVI are fused to generate a time-series dataset; The time-series dataset is input into a preset variable time-series crop classification model, and the remote sensing crop classification results are output; the preset variable time-series crop classification model is a variable time-series crop classification model constructed by integrating dynamic temporal convolution and global self-attention mechanism; The convolution calculation formula for the dynamic temporal convolution mechanism is as follows: ； ； In the formula, For local features; r is the time-series dimension index. For a three-dimensional convolutional kernel that includes a temporal dimension, For spatial height, For space width, The temporal length is the size of the convolutional kernel in the temporal dimension. For input channel index; For the spatial height index of the convolution kernel; This is the index of the convolution kernel's spatial width. The month interval between the query vector Q and the key vector K corresponding to the time series points in the self-attention mechanism module; For output channel index; This indicates that in the input time series matrix M, the spatial coordinates are... The feature value with time-series dimension index r and channel index c corresponds to the original or preprocessed data of a certain pixel in a certain month and a certain band; T is a dynamically variable time-series month length parameter.

2. The variable temporal remote sensing crop classification method fusing convolution and self-attention as described in claim 1, characterized in that, Calculate the Normalized Difference Vegetation Index (NDVI) based on the remote sensing image data; By using the monthly composite method of maximum NDVI, the monthly best clear-sky reflectance image and the monthly best clear-sky NDVI are generated, including: Based on the Sentinel-2 time-series remote sensing reflectance image, cloud masking is performed on the cloud-covered areas to obtain the processed time-series remote sensing reflectance image. Based on the processed temporal remote sensing reflectance image, the normalized difference vegetation index (NDVI) was calculated using Sentinel-2 multiband reflectance. By using the monthly composite method of maximum NDVI, the reflectance of each band in each time series within each month is sorted according to the maximum NDVI and compared. The maximum NDVI is selected as the representative reflectance for that month, generating the monthly best clear sky observation reflectance image and the monthly best clear sky observation NDVI.

3. The variable temporal remote sensing crop classification method fusing convolution and self-attention as described in claim 1, characterized in that, The monthly best clear-sky reflectance image and the monthly best clear-sky NDVI image are fused to generate a time-series dataset, including: Based on the monthly best clear-sky observation reflectance image and the monthly best clear-sky observation NDVI, the observation gaps are filled. The inflated monthly best clear-sky reflectance image and the monthly best clear-sky NDVI image are fused to generate a time-series dataset. Where T represents the dynamically changing length of the time series month; C represents the number of image bands; H represents the image height; and W represents the image width.

4. The variable temporal remote sensing crop classification method fusing convolution and self-attention as described in claim 1, characterized in that, The preset variable temporal crop classification model includes a dynamic temporal convolution module, a global self-attention module, a temporal residual enhanced feature fusion module, and a classification decision module connected in sequence; wherein: The dynamic temporal convolution module extracts local features based on the dynamic temporal convolution mechanism and temporal pooling strategy; the local features include local spatial features of crops under different time lengths and local key temporal patterns of phenological changes; The global self-attention module improves the global self-attention mechanism by introducing time-aware linear transformation and decaying attention weights to extract global features; the global features are the global correlation features of key phenological information of crops. The temporal residual enhancement feature fusion module introduces temporal residual connections to decompose and weight-fuse the local and global features respectively to obtain fused features; The classification decision module maps the fused features to the category space based on the fully connected layer, and normalizes the output of the fully connected layer using the Softmax activation function.

5. The variable temporal remote sensing crop classification method fusing convolution and self-attention as described in claim 4, characterized in that, The global self-attention module extracts global features, and the specific steps are as follows: The local features are mapped into three time-series month-length query vectors Q, key vector K, and value vector V through a time-aware linear transformation. ;in, Local features; These are the learnable parameter matrices for the query vector Q, key vector K, and value vector V, respectively. Calculate the attention weight matrix A based on the query vector Q and the key vector K; the expression for the attention weight matrix A is: ,in, To query the interval between the time series points corresponding to the key vector Q and the key vector K, The attenuation coefficient is a learnable factor. is the size of Q / K, representing the feature dimensions of Q and K; j is the index of the K time series point, used to traverse all time series points corresponding to K to ensure that the attention weight of Q with each K time series point is calculated; The attention weight matrix is ​​Softmax normalized, and the global features are calculated and output. The expression is .

6. The variable temporal remote sensing crop classification method fusing convolution and self-attention according to claim 4, characterized in that, The feature fusion module for temporal residual enhancement decomposes and weights the local and global features respectively, and then fuses them. The specific steps are as follows: The local features Decomposed into spatial components X s and time-series components X t ; The global features Decomposed into key phenological period component X p and time-related components X c ; The local features are processed using a weighted fusion method. and the global features The disassembled components are then fused to obtain the fused features; The expression for the weighted fusion method is: ,in, Features of fusion; They are spatial components X s Time-series component X t Key phenological period component X p and time-related components X c The corresponding learnable weights, and satisfying .

7. The variable temporal remote sensing crop classification method fusing convolution and self-attention as described in claim 1, characterized in that, The loss function L used in the training process of the preset variable time-series crop classification model is: ； Where N is the total number of samples and C is the total number of categories. This is the input data for the i-th sample; This is a mask function used to determine if data is missing; when x is a missing value, When x is valid data, ; and Let be the predicted output probability and the true label of the k-th class for the i-th sample, respectively. As a time-series weighting factor, it assigns high weight to the loss calculation during critical phenological periods and low weight to the loss calculation during non-critical periods.

8. A variable temporal remote sensing crop classification system integrating convolution and self-attention, characterized in that, The system includes: The acquisition unit is used to acquire remote sensing image data of the target area, wherein the remote sensing image data is multi-period Sentinel-2 time-series remote sensing reflectance images of crops during their growth cycle. The NDVI maximum monthly composite unit is used to calculate the Normalized Difference Vegetation Index (NDVI) based on the remote sensing image data; and to generate the monthly best clear sky observation reflectance image and the monthly best clear sky observation NDVI through the NDVI maximum monthly composite method. The time-series dataset generation unit is used to fuse the monthly best clear sky observation reflectance image and the monthly best clear sky observation NDVI to generate a time-series dataset. The classification unit is used to input the time-series dataset into a preset variable time-series crop classification model and output the remote sensing crop classification result; the preset variable time-series crop classification model is a variable time-series crop classification model constructed by integrating dynamic temporal convolution and global self-attention mechanism; The convolution calculation formula for the dynamic temporal convolution mechanism is as follows: ； ； In the formula, For local features; r is the time-series dimension index. For a three-dimensional convolutional kernel that includes a temporal dimension, For spatial height, For space width, The temporal length is the size of the convolutional kernel in the temporal dimension. For input channel index; For the spatial height index of the convolution kernel; This is the index of the convolution kernel's spatial width. The month interval between the query vector Q and the key vector K corresponding to the time series points in the self-attention mechanism module; For output channel index; This indicates that in the input time series matrix M, the spatial coordinates are... The feature value with time-series dimension index r and channel index c corresponds to the original or preprocessed data of a certain pixel in a certain month and a certain band; T is a dynamically variable time-series month length parameter.

9. The variable temporal remote sensing crop classification system fusing convolution and self-attention as described in claim 8, characterized in that, The preset variable temporal crop classification model includes a dynamic temporal convolution module, a global self-attention module, a temporal residual enhanced feature fusion module, and a classification decision module connected in sequence. The dynamic temporal convolution module is also connected to the temporal residual enhanced feature fusion module; wherein: The dynamic temporal convolution module extracts local features based on the dynamic temporal convolution mechanism and temporal pooling strategy; the local features include local spatial features of crops under different time lengths and local key temporal patterns of phenological changes; The global self-attention module improves the global self-attention mechanism by introducing time-aware linear transformation and decaying attention weights to extract global features; the global features are the global correlation features of key phenological information of crops. The temporal residual enhancement feature fusion module introduces temporal residual connections to decompose and weight-fuse the local and global features respectively to obtain fused features; The classification decision module maps the fused features to the category space based on the fully connected layer, and normalizes the output of the fully connected layer using the Softmax activation function.

Images