A time series remote sensing crop classification method and device based on CNN-ViT

Abstract

The application discloses a time series remote sensing image crop classification method based on CNN and ViT, and belongs to the technical field of remote sensing image crop classification. Firstly, time series remote sensing image data is acquired and a standardized data set is constructed. Secondly, CNN is adopted to extract image local spatial features, and the perception ability of the model to spatial details is enhanced. Then, dynamic time position coding and static spatial position coding are introduced, and the inter-frame dynamic changes of the time dimension and the feature interaction relationship of the spatial dimension are modeled in combination with a Transform module to generate unified space-time joint feature representation. Finally, the space-time features are mapped into a category probability distribution through a pixel-level segmentation head to generate a complete classification result map, and accurate prediction of crop planting categories is realized. Through combination of the local feature extraction capability of CNN and the global feature modeling capability of Transform, introduction of space-time position coding, effective capture of time and space features, and significant improvement of the precision and efficiency of time series remote sensing image crop classification are achieved.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image classification and deep learning technology, and proposes a time series remote sensing crop classification method and device based on convolutional neural network (CNN) and Vision Transformer (ViT). Background Technology

[0002] Crop classification is a crucial task in modern agricultural research, playing a vital role in crop planting area statistics, yield prediction, and precision agricultural management. With intensifying climate change, population growth, and increasingly scarce land resources, efficient and accurate monitoring of crop planting conditions has become a critical issue that urgently needs to be addressed. Remote sensing imagery technology, due to its ability to rapidly acquire surface information over a wide area, is widely used in crop classification. In particular, the introduction of time-series remote sensing data can effectively utilize spectral variations in crops at different growth stages, thereby improving classification accuracy.

[0003] Currently, intelligent crop classification technologies are mainly divided into two categories: traditional machine learning methods and deep learning methods. Traditional machine learning methods, represented by Support Vector Machines and Random Forests, manually extract spectral, texture, and shape features from remote sensing images and input them into a classifier to classify crops. However, these methods are highly dependent on manually extracted features and struggle to handle complex and diverse scenarios. In recent years, with the rapid development of deep learning, technologies such as Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Vision Transformers (ViTs) have been widely applied to crop classification tasks. CNNs, with their powerful spatial feature extraction capabilities, can efficiently model single-temporal remote sensing images; RNNs excel in modeling time-series data and are suitable for modeling multi-temporal remote sensing images; ViTs, through their self-attention mechanism, effectively model long-range spatial dependencies in images, showing great potential in image classification tasks.

[0004] Although the above methods have achieved certain results in crop classification tasks, the following problems still exist: (1) Traditional machine learning methods rely too much on handmade features, and their classification performance is limited by the quality of feature extraction, making it difficult to maintain high accuracy in complex scenarios; (2) Although deep learning methods can automatically learn features, when CNN or RNN models are used alone, it is difficult to achieve interactive modeling of spatial and temporal features, which limits the ability to model time-series remote sensing images; (3) Although the Transformer model has the ability to model global features, its high computational complexity makes it face significant computational bottlenecks when processing high-resolution multi-temporal remote sensing images; (4) Existing methods usually fail to fully integrate spatial and temporal information and ignore the dynamic characteristics of crops at different spatiotemporal scales, resulting in limited classification accuracy. Summary of the Invention

[0005] The present invention aims to overcome the above-mentioned shortcomings of the prior art and provide a time series remote sensing crop classification method and device based on CNN and ViT.

[0006] This invention effectively addresses the shortcomings of current methods, such as insufficient spatial and temporal feature modeling, limited classification accuracy, and low computational efficiency, by combining the powerful spatial local features of convolutional neural networks with the spatiotemporal interaction modeling capabilities of Vision Transformers. In processing crop classification tasks using time-series remote sensing imagery, this method not only significantly improves classification accuracy and efficiency but also provides a novel technical solution for crop monitoring and management.

[0007] To achieve the above objectives, the first aspect of the present invention relates to a time-series remote sensing crop classification method based on CNN and ViT, comprising the following steps:

[0008] Step 1: Dataset Construction;

[0009] Step 1.1: Data Acquisition. Select remote sensing image data with a spatial resolution of H×W and a channel number of C for each observation time point T to form a time series image frame set X. i :

[0010]

[0011] in, Let t be the image data of the i-th sample.

[0012] Then select and X i The corresponding pixel-level category label Y i Image dataset D:

[0013] D = {(X i ,Y i )∣i=1,2,…,N} (2)

[0014] Among them, X i For the i-th time-series remote sensing image sample, Y i Let be the pixel-level category label corresponding to the i-th sample, and N be the total number of samples in the image dataset.

[0015] Step 1.2: Data cropping. To reduce computational complexity and adapt to model input requirements, while ensuring that the sub-images contain rich spatial information, it is necessary to crop the X-ray data of each input image. i The image is cropped into a subgraph of size H'×W', forming a subgraph set X. i :

[0016]

[0017] in, This represents the k-th cropped sub-image in the t-th frame of the i-th sample, where M is the number of sub-images after each frame is divided into blocks, defined as:

[0018]

[0019] Where H′ and W′ are the height and width of the cropped sub-image.

[0020] The cropped dataset D′ can be represented as a set of cropped sub-images and their corresponding labels:

[0021]

[0022] in, Let represent the k-th subgraph after cropping. This indicates the pixel-level label corresponding to the cropped sub-image.

[0023] Step 1.3: Dataset partitioning. Divide the processed dataset D′ into the following three subsets according to the specified proportions: training set D... train Validation set D val Test set D test This is to improve the independence and generalization performance of model training and avoid overfitting.

[0024] Step 2: Convolutional feature extraction;

[0025] Step 2.1: Convolutional Feature Extraction Module Construction. Remote sensing images typically possess significant spatial structural information (such as land parcel boundaries and textures) and local feature distributions (such as crop shapes and arrangements). While Transformer excels at global feature modeling, its ability to directly model local spatial features is limited. To enhance local spatial information extraction, a convolutional feature extraction module is constructed to improve local perception capabilities. For each frame of image X in the input data X... (t) Using the lightweight convolution module f CNN Feature extraction is performed to generate convolutional feature maps. Specifically, this includes the following steps:

[0026] First, a 2D convolutional layer (Conv2D) is used to extract features from the input image, extracting local spatial structure information. Then, the feature map generated by the convolution operation is normalized to balance the data distribution, eliminate differences in magnitude between different features, accelerate model training, and stabilize gradient flow. Finally, the ReLU activation function is introduced to make the activated features sparse, improve the discriminative power of the features, and effectively avoid the gradient vanishing problem. The final generated convolutional feature map... It can provide input support for subsequent spatiotemporal feature modeling, as shown below:

[0027]

[0028] Where H′×W′ is the spatial resolution after convolution, and D is the channel dimension after convolution.

[0029] Step 2.2: Patch segmentation and linear projection, the generated feature map after convolutional feature extraction. It maintains a high resolution and number of channels. To further process the feature map, it is divided into non-overlapping patches of size h×w, each patch containing local spatial features. The total number of patches N after segmentation can be expressed as:

[0030]

[0031] For the feature map of frame t The i-th patch is denoted as P. (t,i) Each P (t,i) Convolutional features encompassing local regions preserve high-dimensional channel information within those regions:

[0032] P (t,i) ∈R h×w×D ,i=1,2,…,N (8)

[0033] To convert each patch into a token format suitable for the Transformer encoder's input format, each patch needs to be processed. (t ,i) Perform the following operations: First, set P (t,i) Expand it into a one-dimensional vector; then, through a linear projection operation, map the one-dimensional vector to P. (t,i) Corresponding TokenZ (t,i) , represented as:

[0034]

[0035] Where W is the weight matrix of the linear projection, b is the bias vector of the linear projection, and D Z For the feature dimensions of the token.

[0036] Finally, using the same method, all patches in time frame t are converted into a token sequence Z. (t) :

[0037]

[0038] Where N is the number of segments, and D Z For the feature dimensions of the token.

[0039] Step 3: Encoder stage;

[0040] Step 3.1: Spatiotemporal location encoding construction. To preserve the temporal information of each frame and the spatial location information of each patch in the time-series remote sensing image, the generated token sequence Z... (t) Add time location coding and spatial location coding.

[0041] Temporal location encoding is used to represent the temporal location information of different frames in a time series. Since the acquisition time of remote sensing images is often irregular, traditional fixed temporal location encoding methods are difficult to effectively model time intervals. Therefore, this invention introduces dynamic temporal location encoding by constructing a time lookup table P. T ∈R T′×D This embeds information about all observation times, where T′ is the number of all observation times in the data, and D is the feature dimension of the token. The feature representation Z for time frame t... (t) By finding the corresponding time location code P T [t] is added to each token to obtain the time-enhanced feature. Represented as:

[0042]

[0043] Spatial location coding is used to identify the spatial location information of each patch within an image, ensuring that the spatial structure features of the image are preserved during time series modeling. A grid coordinate coding method is used to assign coordinates (p...) to each patch. x ,p y Generate two independent vectors, encoding the row and column coordinates respectively, and calculate them as follows:

[0044]

[0045] in, The sinusoidal component encoded for row coordinate position. Encode the cosine component for the row coordinate position. The sine component in the column coordinate position encoding. The cosine component in the column coordinate position encoding.

[0046] The row coordinate encoding and column coordinate encoding are concatenated to obtain the position encoding vector P for each patch. S [i]:

[0047]

[0048] Stacking the positional encoding vectors of all patches yields the complete spatial positional encoding matrix P. S :

[0049]

[0050] Then, the spatial location code is added to the temporally location-coded token to obtain the spatiotemporally location-coded representation of the token:

[0051]

[0052] For T frames of the entire time series, the token sequence encoded by all spatiotemporal locations is represented as:

[0053]

[0054] Step 3.2: Construct a spatiotemporal encoder. The spatiotemporal encoder uses a Transformer module, consisting of a multi-head self-attention mechanism (MSA) and a feedforward network (FFN), to capture long-range dependencies in the temporal dimension and model feature interactions in the spatial dimension. In each layer, the token encoded from the input spatiotemporal location is first processed. Normalization is performed, followed by capturing long-range dependencies in the temporal dimension and modeling feature interaction relationships in the spatial dimension through a multi-head self-attention mechanism. The time series and spatial features are then updated. The calculation process is as follows:

[0055]

[0056]

[0057] Through processing by a temporal encoder and a spatial encoder, the final extracted feature representation is Z. final :

[0058]

[0059] Step 4: Decoder stage;

[0060] Step 4.1: Pixel-level segmentation head construction, through Z... final A linear projection operation is performed to convert the spatial features of each category into pixel-level prediction results. For category k and spatial location i, a multilayer perceptron (MLP) is used to map the features, transforming high-dimensional features into a low-dimensional space, resulting in the pixel-level probability distribution p of each category in the current spatial block. k,i .

[0061] Step 4.2: Spatial Reassembly and Merging. After completing the linear projection of all blocks, the pixel-level probability results of each block are stitched back to the resolution of the original input image. This is done by calculating the predicted probability p of each block. k,i The images are rearranged into the global image space to generate complete pixel-level prediction results, enabling pixel-level classification of the entire remote sensing image.

[0062] Step 5: Model Training and Usage;

[0063] During the model training and usage phase, the training set D defined in step 1.3 is used. train Validation set D val Test set D test Perform the following tasks:

[0064] Model training: The model is optimized using the training set by progressively updating the model parameters through block-based feature extraction, encoder processing, and pixel-level semantic segmentation operations of the decoder.

[0065] Model validation: Use the validation set to tune hyperparameters and evaluate the model’s performance on unseen data.

[0066] Model testing: Use the test set to perform the final performance evaluation of the model.

[0067] Finally, through the entire process of model processing, the system generates a category probability distribution for each pixel in the input time-series remote sensing image data, thereby enabling the classification and prediction of crops.

[0068] A second aspect of the present invention relates to a time-series remote sensing crop classification device based on CNN-ViT, comprising a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the time-series remote sensing crop classification method based on CNN-ViT of the present invention.

[0069] A third aspect of the invention relates to a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the CNN-ViT-based time-series remote sensing crop classification method of the present invention.

[0070] The advantages of this invention are:

[0071] 1) This invention employs a small-scale segmentation method in the segmented feature extraction stage, combined with a CNN module to further capture local spatial details and texture information, while preserving high-resolution spatial features. By introducing a spatiotemporal encoder, the model can simultaneously focus on the interaction of local and global spatiotemporal features, thereby significantly improving the accuracy of crop classification.

[0072] 2) This invention introduces dynamic temporal location coding and static spatial location coding to effectively address the modeling problems of irregular time intervals and spatial location information in time series data. Dynamic temporal location coding enhances the model's ability to model spatiotemporal correlation information, while static spatial location coding preserves spatial distribution characteristics, reduces the loss of location information, and can more comprehensively reflect the crop's growth cycle and spatial distribution characteristics. Attached Figure Description

[0073] Figure 1 This is a flowchart of the method of the present invention.

[0074] Figure 2 This is the network structure diagram of the present invention.

[0075] Figure 3(a) is a remote sensing image, Figure 3(b) is a ground truth image of crop labels, and Figure 3(c) is a prediction result of the CNN-ViT model.

[0076] Figure 4 This is a diagram of the apparatus of the present invention. Detailed Implementation

[0077] To further clarify the content of this invention, the following detailed description is provided in conjunction with embodiments of the invention to facilitate understanding and practice by those skilled in the art. It should be noted that the following embodiments are merely for demonstrating and explaining the technical solutions of this invention and do not constitute a limitation on the scope of the invention. It should be understood that not all features of all actual implementations are the same as those in this embodiment; in actual engineering projects, some implementation details may be adjusted according to specific conditions and objectives.

[0078] Example 1

[0079] This embodiment proposes a time-series remote sensing crop classification method based on CNN and ViT. Addressing the challenges of traditional deep learning models in processing time-series remote sensing image data—such as difficulty in simultaneously capturing spatiotemporal features, high computational complexity, and difficulty in ensuring classification accuracy and efficiency for large-scale crop classification tasks—a hybrid architecture combining Convolutional Neural Networks (CNN) and ViT is proposed. This method achieves accurate classification of crops in time-series remote sensing images through efficient block-based feature extraction, spatiotemporal encoding, and decoding strategies. The specific implementation of this invention is illustrated in the accompanying drawings, and the process is as follows:

[0080] This embodiment uses the XJDS and PASTIS remote sensing image datasets for parts of the Xinjiang Uygur Autonomous Region. The XJDS dataset contains 2066 time-series satellite image samples (SITS), and the PASTIS dataset contains 2433 time-series satellite image samples. Each sample is a 128×128 pixel remote sensing image with 10 spectral bands and a spatial resolution of 10m. The XJDS dataset uses 5 crop labels, and the PASTIS dataset uses 18 crop labels. Each sample in both datasets has 33 to 61 time-acquisition frames.

[0081] Step 1: Dataset Construction;

[0082] Step 1.1: Select 2066 samples from the XJDS dataset and 2400 remote sensing image samples from the PASTIS dataset with a spatial resolution of 128×128 and 10 channels, respectively, to form a time-series image frame set X. i Then extract X i The corresponding crop category label Y i Composition of image dataset D XJDS and D PASTIS .

[0083] Step 1.2: To reduce computational overhead during the experiment, each sample in the XJDS and PASTIS datasets was cropped into 24×24 blocks while retaining all time-series frames. This cropping method reduces computational complexity without losing original image information, while ensuring that the model output can be reassembled back to the original image size, ultimately yielding 60,500 24×24 time-series image samples.

[0084] Step 1.3: Dataset partitioning, dividing the 60,500 samples into: a 40,500-sample training set D train 10,000 validation sample sets D val 10,000 test sample sets D test .

[0085] Step 2: Segmented feature extraction;

[0086] Step 2.1: Construction of the convolutional feature extraction module. This module is designed for the input T frames of 24×24 image sample data X∈R. T×24×24×10 Each frame of the image has 10 bands. First, for each frame of the input data X... (t) The input is fed into 16 3×3 convolutional layers, resulting in a 24×24×16 feature map as the output. Then feature map The input is fed into 32 3×3 convolutional layers, resulting in a 24×24×32 feature map. The feature map is then obtained through a 2×2 max pooling operation.

[0087] Step 2.2: Linear mapping, after obtaining the feature map Next, the output feature map is segmented into patches. The feature map has a size of 12×12×32, which is divided into 4×4 non-overlapping patches. Each patch contains convolutional features of a local region, resulting in 9 patches. Each patch has a dimension of 4×4×32, containing spatial features and high-dimensional channel information within the region. Then, each patch is flattened into a one-dimensional vector, resulting in a dimension of 512. This flattened vector is then mapped to 128 dimensions using linear projection. The TokenZ corresponding to each patch is obtained through linear projection. (t,i) Finally, for each frame t, the tokens corresponding to all patches are concatenated in order to obtain the token sequence Z. (t) ={Z (t,1) Z (t ,2) ,...,Z (t,9)}, which serves as the input to the Transformer encoder.

[0088] Step 3: Encoder stage;

[0089] Step 3.1: Time-location encoding construction, first in the generated token sequence Z (t) Time location encoding is added. Based on the collection times of all samples in the XJDS and PASTIS datasets, a time lookup table P is constructed. T The corresponding time location encoding vector P is retrieved through the lookup table. T [t]. Then add it to the Token, as follows:

[0090]

[0091] Spatial location encoding is constructed using a grid coordinate encoding method to assign block coordinate locations (p) to each frame of imagery in the XJDS and PASTIS datasets. x ,p y Generate two independent vectors, encoding the row and column coordinates respectively, calculated as follows:

[0092]

[0093] in The sinusoidal component encoded for row coordinate position. Encode the cosine component for the row coordinate position. The sine component in the column coordinate position encoding. The cosine component in the column coordinate position encoding.

[0094] The row coordinate encoding and column coordinate encoding are concatenated to obtain the position encoding vector P of each block. S[i] Stack all the positional encoding vectors of the blocks to obtain the complete spatial positional encoding matrix P. S .

[0095] For the input token sequence Z of the encoder module (t) This combines temporal and spatial location encoding to form the final input features.

[0096]

[0097] Step 3.2: Construct a spatiotemporal encoder. The spatiotemporal encoder is computed using the Transformer module. In each layer, the token encoded from the input spatiotemporal location is first processed. After normalization, the multi-head sub-attention mechanism is computed using the MSA module to obtain the output Y. TS Subsequently, a two-layer fully connected network and activation function of MLP are used to obtain the final extracted feature representation Z. final .

[0098] Step 4: Decoder stage;

[0099] Pixel-level segmentation head construction, Z final The features are mapped to crop categories using a multilayer perceptron (MLP) to obtain the crop classification probability distribution for each pixel. Softmax normalization is then applied to all category probabilities for each pixel to calculate the classification probability for each pixel.

[0100] After completing the linear projection of all blocks, the pixel-level probability results of each block are stitched back to the original input image at a resolution of 24×24 to obtain the complete crop classification results of the remote sensing image samples.

[0101] Step 5: Model Training and Usage;

[0102] During the model training and usage phase, the training set D created in step 1.3 is used. train Validation set D val Test set D test The model was trained, validated, and tested, with the hyperparameters set as follows: Epochs = 100, Batch Size = 16, Learning Rate = 0.001. The network structure is as follows. Figure 2 As shown in Figure 3, by performing block-based feature extraction, encoder-stage processing, and pixel-level semantic segmentation on the input time-series remote sensing image data, the probability distribution of the category corresponding to each pixel is finally obtained, thus achieving the classification of crops. The results obtained by this method are shown in Figure 3, where Figure 3(a) is the remote sensing image, 3(b) is the ground truth image of crop labels, and 3(c) is the prediction result of the CNN-ViT model.

[0103] Example 2

[0104] Reference Figure 4 This embodiment relates to a time-series remote sensing crop classification device based on CNN-ViT, including a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the time-series remote sensing crop classification method based on CNN-ViT of Embodiment 1.

[0105] Example 3

[0106] This embodiment relates to a computer-readable storage medium storing a program that, when executed by a processor, implements the CNN-ViT-based time-series remote sensing crop classification method of the present invention.

[0107] The above is merely a description of embodiments of the present invention, but the scope of protection of the present invention should not be considered as limited to the specific forms described in the embodiments. The scope of protection of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the concept of the present invention.

Claims

1. A time-series remote sensing crop classification method based on CNN-ViT, characterized in that, include: Step 1: Dataset construction. By preprocessing and standardizing the time-series remote sensing image data, a spatiotemporally consistent training dataset is formed. Step 2: Convolutional feature extraction. This step utilizes a local feature extraction module to obtain spatial structure information from the image, enhancing the model's ability to perceive local details. Specifically, it includes: Step 2.1: Convolutional Feature Extraction Module Construction. Remote sensing images possess significant spatial structure information and local feature distributions. While Transformer excels at global feature modeling, its ability to directly model local spatial features is limited. To enhance local spatial information extraction, a convolutional feature extraction module is constructed to improve local perception capabilities. This module is used to process the input data... Each frame of the image Through lightweight convolution modules Feature extraction is performed to generate convolutional feature maps; specifically, the following steps are included: First, a 2D convolutional layer (Conv2D) is used to extract features from the input image, extracting local spatial structure information. Then, the feature map generated by the convolution operation is normalized to balance the data distribution, eliminate differences in magnitude between different features, accelerate model training, and stabilize gradient flow. Finally, the ReLU activation function is introduced to make the activated features sparse, improve the discriminative power of the features, and effectively avoid the gradient vanishing problem. The final generated convolutional feature map... It can provide input support for subsequent spatiotemporal feature modeling, as shown below: in, This represents the spatial resolution after convolution. The channel dimension after the convolution operation; Step 2.2: Patch segmentation and linear projection, the feature map generated after convolutional feature extraction. Maintaining high resolution and channel count; to further process the feature map, it is divided into sections of size [size missing]. Non-overlapping patches, each patch containing local spatial features; total number of segments. It can be represented as: For the Feature map of a frame , No. A Patch is represented as each Convolutional features encompassing local regions preserve high-dimensional channel information within those regions: To convert each patch into a token, the input format suitable for the Transformer encoder, each patch needs to be... Perform the following operations: First, Expand into a one-dimensional vector; then, through a linear projection operation, map the one-dimensional vector to... Corresponding Token , represented as: in, The weight matrix is ​​the linear projection. This is the bias vector for the linear projection. For the feature dimensions of a token; Finally, using the same method, the time frame was... All patches are converted into token sequences. : in, It is the number of patches in the split. For the feature dimensions of a token; Step 3: Encoder stage, combining temporal and spatial location encoding to generate a spatiotemporal joint feature representation that integrates inter-frame dynamic changes and spatial feature interactions; Step 4: Decoder stage, the joint features are decoded layer by layer, mapped to class probability distribution, and the classification result is reconstructed; Step 5: Model training and use to generate prediction and classification results for crop planting categories.

2. The time-series remote sensing crop classification method based on CNN-ViT according to claim 1, characterized in that, The dataset construction described in step 1 specifically includes: Step 1.1: Data Acquisition, Selecting Each Observation Time Point T The spatial resolution is And the number of channels is C The remote sensing image data forms a time-series image frame set. : in, For the first The first sample Frame image data; Select again and Corresponding pixel-level category labels Composition of image dataset : in, For the first A time-series remote sensing image sample, For the first The pixel-level category label corresponding to each sample N The total number of samples in the image dataset; Step 1.2: Data cropping. To reduce computational complexity and adapt to model input requirements, while ensuring that sub-images contain rich spatial information, it is necessary to crop each input image data. Cut to size The subgraphs form a set of subgraphs. : in, Indicates the first The first sample The first frame A cutout image, M The number of sub-images after each frame of image is divided into blocks is defined as: in, , The height and width of the cropped sub-image; The cropped dataset This can be represented as a set of cropped sub-images and their corresponding labels: in, Indicates the first crop after trimming. Sub-image, This indicates the pixel-level label corresponding to the cropped sub-image; Step 1.3: Dataset partitioning, dividing the processed dataset... The training set is divided into the following three subsets according to the proportions: D train Validation set D val Test set D test This is to improve the independence and generalization performance of model training and avoid overfitting.

3. The time-series remote sensing crop classification method based on CNN-ViT according to claim 2, characterized in that, Step 3, the encoder stage, specifically includes: Step 3.1: Spatiotemporal location encoding construction. To preserve the temporal information of each frame and the spatial location information of each patch in the time-series remote sensing imagery, the generated token sequence... Add time location coding and spatial location coding to it; Temporal location coding is used to represent the temporal location information of different frames in a time series. Since the acquisition time of remote sensing images is often irregular, traditional fixed temporal location coding methods are difficult to effectively model time intervals. Therefore, dynamic temporal location coding is introduced by constructing a time lookup table. All observation time information is embedded in it, This represents the number of all observation times in the data. For the feature dimensions of the token; for time frames Feature representation By finding the corresponding time location code Add it to each token to obtain the time-enhanced feature. , represented as: Spatial location coding is used to identify the spatial location information of each patch within an image, ensuring that the spatial structure features of the image are preserved during time series modeling; a grid coordinate coding method is used to define the coordinate position of each patch. Two independent vectors are generated, encoding the row and column coordinates respectively, and the calculation method is as follows: in, The sinusoidal component encoded for row coordinate position. Encode the cosine component for the row coordinate position. The sine component in the column coordinate position encoding. The cosine component in the column coordinate position encoding; The row coordinate encoding and column coordinate encoding are concatenated to obtain the position encoding vector of each patch. : Stack the positional encoding vectors of all patches to obtain the complete spatial positional encoding matrix. : Then, the spatial location code is added to the temporal location code-encoded token to obtain the spatiotemporal location-encoded representation of the token: For the entire time series The frame, represented as a sequence of tokens encoded from all spatiotemporal locations, is as follows: Step 3.2: Construct a spatiotemporal encoder. The spatiotemporal encoder uses a Transformer module, consisting of a multi-head self-attention mechanism (MSA) and a feedforward network (FFN), to capture long-range dependencies in the temporal dimension and model feature interactions in the spatial dimension. In each layer, the token encoded from the input spatiotemporal location is first processed. Normalization is performed, followed by capturing long-range dependencies in the temporal dimension and modeling feature interaction relationships in the spatial dimension through a multi-head self-attention mechanism. The time series and spatial features are then updated. The calculation process is as follows: Through processing by the temporal encoder and spatial encoder, the final extracted features are represented as follows: .

4. The time-series remote sensing crop classification method based on CNN-ViT according to claim 3, characterized in that, The decoder stage described in step 4 specifically includes: Step 4.1: Pixel-level segmentation head construction, through... Perform a linear projection operation to convert the spatial features of each category into pixel-level prediction results; for each category... and spatial location By mapping features through a multilayer perceptron (MLP), high-dimensional features are mapped to a low-dimensional space, resulting in the pixel-level probability distribution of each category in the current spatial block. ; Step 4.2: Spatial Reassembly and Merging. After completing the linear projection of all blocks, the pixel-level probability results of each block are stitched back to the resolution of the original input image; this is achieved by adjusting the predicted probabilities of each block. The images are rearranged into the global image space to generate complete pixel-level prediction results, enabling pixel-level classification of the entire remote sensing image.

5. The time-series remote sensing crop classification method based on CNN-ViT according to claim 4, characterized in that, Step 5, model training and usage, specifically includes: During the model training and usage phase, the training set defined in step 1.3 is used. D train Validation set D val Test set D test Perform the following tasks: Model training: The model is optimized using the training set by progressively updating the model parameters through block-based feature extraction, encoder processing, and pixel-level semantic segmentation operations of the decoder. Model validation: Use the validation set to tune hyperparameters and evaluate the model's performance on unseen data; Model testing: Use the test set to perform the final performance evaluation of the model; Finally, through the entire process of model processing, the system generates a category probability distribution for each pixel in the input time-series remote sensing image data, thereby enabling the classification and prediction of crops.

6. A time-series remote sensing crop classification device based on CNN-ViT, characterized in that, The method includes a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the time series remote sensing crop classification method based on CNN-ViT as described in any one of claims 1-5.

7. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements the time-series remote sensing crop classification method based on CNN-ViT as described in any one of claims 1-5.

Images