A Temporal SAR Crop Classification Method Based on Lightweight Linear Attention

By constructing a time-series SAR crop classification method based on a lightweight linear attention mechanism using U-Net, the problems of high computational complexity and numerous parameters of deep learning models in remote sensing data processing are solved, achieving efficient crop classification, especially in long-term SAR imagery, improving the accuracy and efficiency of classification.

CN119131470BActive Publication Date: 2026-11-10UNIV OF ELECTRONICS SCI & TECH OF CHINA
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411153955.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-21
Publication Date
2026-11-10
Estimated Expiration
2044-08-21

AI Technical Summary

Technical Problem

Existing deep learning crop classification methods suffer from high computational complexity and numerous model parameters when processing large-scale remote sensing data. In particular, they are prone to problems such as limited receptive field, vanishing gradient, and exploding gradient in long-term series data. Furthermore, the computational complexity of the dot product of traditional attention mechanisms is proportional to the square of the data volume, which limits their efficiency in practical applications.

Method used

A time-series SAR crop classification method based on lightweight linear attention is adopted. By constructing a U-Net encoder-decoder structure and combining it with a lightweight linear temporal attention encoder module (LLTAEBlock) to extract spatiotemporal features, an attention mechanism with linear computational complexity is implemented using FAVOR+ to reduce model parameters and computational cost, and detailed information is preserved through skip connections.

Benefits of technology

It effectively captures long-term temporal relationships, improves the efficiency and practicality of crop classification, reduces memory requirements, and enhances the accuracy and speed of the model in long-term SAR image classification.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119131470B_ABST
    Figure CN119131470B_ABST
Patent Text Reader

Abstract

This invention belongs to the field of remote sensing monitoring technology, specifically a time-series SAR crop classification method based on lightweight linear attention. This invention utilizes the spatiotemporal characteristics of time-series SAR imagery. First, it efficiently extracts spatial features from each time step of the imagery using a weight-sharing U-Net encoder. Then, it uses LLTABlock to extract spatiotemporal features from the high-level spatial features of the time series, achieving linear computational complexity with FAVOR+. A shared query matrix serves as a learnable parameter, reducing computation and parameter count, and enabling global information interaction across the time dimension to model complex phenological relationships in crops. The decoder retains deep spatiotemporal features and fuses them with shallow temporal spatial features to obtain a prediction result with the same spatial scale as the input data. Under the constraint of the loss function, an optimizer adjusts the model to obtain the best-fitting crop classification model. This invention is particularly suitable for crop classification of long-term time-series SAR imagery, reducing memory requirements and improving the efficiency of crop classification.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of remote sensing monitoring technology, specifically a time-series SAR crop classification method based on lightweight linear attention. Background Technology

[0002] Crop classification aims to identify and differentiate different crop types using remote sensing technology. Accurate crop classification is crucial for monitoring crop growth, optimizing resource allocation, and promoting sustainable agricultural development.

[0003] In terms of remote sensing data sources, crop classification largely relies on optical remote sensing imagery, classifying crops by analyzing color, texture, and shape features. However, optical remote sensing is susceptible to weather conditions, making it difficult to obtain clear images in areas with frequent cloud cover and fog. In contrast, Synthetic Aperture Radar (SAR) offers all-weather, day-and-night monitoring capabilities, enabling the acquisition of remote sensing images of crops under various weather conditions. Furthermore, SAR is sensitive to the shape, structure, and dielectric constant of crop scatterers, allowing it to identify crop types that are difficult to distinguish using optical imagery. During crop growth, different crops exhibit different spectral characteristics and structural changes at different growth stages. Extracting spatiotemporal features from stable crop growth monitoring data provided by SAR can improve the accuracy of crop classification.

[0004] In crop classification, traditional methods such as maximum likelihood classification, support vector machines (SVM), and random forests (RF) rely heavily on handcrafted features. These methods have limited effectiveness in handling complex temporal variations and are sensitive to salt-and-pepper noise in SAR images. In recent years, deep learning methods have demonstrated significant advantages in crop classification. Unlike traditional methods, deep learning models can automatically extract spatiotemporal features from temporal SAR, effectively handling complex temporal variations and salt-and-pepper noise. Commonly used temporal crop classification models include convolutional neural networks (CNN), recurrent neural networks (RNN), and their variants. These models can capture the dynamic characteristics of crop growth, but they suffer from problems such as limited receptive field, vanishing gradients, and exploding gradients when processing long-term data.

[0005] To avoid this problem, self-attention mechanisms capture long-distance dependencies by calculating the correlation between each basic unit in the sequence and other basic units. Dot-product attention is one implementation of self-attention, measuring the similarity between tokens (the basic units of the input sequence) by calculating the dot product between the query matrix and the key matrix. However, the computational complexity of dot-product attention is proportional to the square of the number of tokens, resulting in high memory requirements when processing large-scale data, which limits its efficiency in practical applications. Summary of the Invention

[0006] To address the aforementioned problems and shortcomings and to overcome the limitations of existing deep learning crop classification methods, this invention provides a time-series SAR crop classification method based on lightweight linear attention. This method effectively captures long-term temporal relationships with linear computational complexity, avoiding the quadratic complexity problem of traditional attention mechanisms. Simultaneously, a lightweight strategy is employed to reduce model parameters, improving the efficiency and practicality of crop classification using time-series SAR images.

[0007] A time-series SAR crop classification method based on lightweight linear attention includes the following steps:

[0008] Step 1: Dataset Construction: The samples in the target region dataset (e.g., PASTIS-R) are divided into non-overlapping training and testing sets according to a set ratio for training and testing the constructed neural network model. Each sample includes a temporal image patch. and a corresponding tag block Where h represents the height of the array block, w represents the width of the array block, c represents the number of features in the array block, and t represents the temporal length of the temporal image block.

[0009] Step 2: Construct a neural network model based on the U-Net encoder-decoder structure:

[0010] Step 2-1: Constructing the encoder: Within the U-Net framework, a weight-sharing U-Net encoder is used to extract spatial features from the image data at each time step, avoiding parameter redundancy caused by using separate encoders for each time step image data; multi-scale, local spatial features are obtained by stacking L-layer downsampling modules.

[0011] F i =EnConvBlock i (F i-1 )

[0012]

[0013] EnConvBlock i For the weight-shared downsampling module corresponding to the i-th scale, i∈[1,L]; This represents the temporal spatial feature corresponding to the i-th scale, containing the spatial features corresponding to a total of t time steps.

[0014] Each downsampling module contains R convolutional blocks. The output of the current downsampling module is used as the input of the next downsampling module.

[0015] Each convolutional block includes convolution, batch normalization (BN), and an activation function, used to extract and integrate spatial features of the image and enhance the semantic information of the features. BN is batch normalization, where the output of the current convolutional block is used as the input of the next convolutional block. The convolution operation extracts local spatial features by applying a convolution kernel to the input features. Batch normalization (BN) reduces the impact of internal covariate shifts by standardizing the feature distribution within each batch, thereby accelerating the model training process and making the model more robust to different input batches. The activation function alleviates the gradient vanishing problem while introducing nonlinear features.

[0016] Step 2-2: Extracting Temporal Features: A lightweight linear temporal attention encoder module, LLTAEBlock, is constructed using a linear attention mechanism combined with a lightweight strategy. This module extracts the temporal high-level spatial features F obtained from the encoder constructed in Step 2-1. L Extracting spatiotemporal features F L′ :

[0017] F L′ =LLTAEBlock(F L )

[0018] Represents high-level spatial features of time series, h L w L c L These represent the height, width, and number of features corresponding to the Lth scale, respectively. To extract the spatiotemporal features, h′ L w′ L c′ L These represent the height, width, and number of features corresponding to the Lth scale, respectively; LLTAEBlock is a lightweight linear temporal attention encoder module, which includes LN, a lightweight linear temporal attention encoder LLTAE, LN, and a fully connected layer. The output of LLTAE is connected to the output of the fully connected layer in a skip connection as the output of LLTAEBlock.

[0019] Layer normalization (LN) eliminates covariance bias between different features in the input data by applying LN to the features of each sample, enabling more efficient gradient propagation between different layers and contributing to greater model stability during training. LLTAE uses a linearly computationally complex attention mechanism to achieve global information interaction in the time dimension, extracting long-range dependencies and modeling complex phenological relationships in crops. It defines the query matrix as neural network model parameters to reduce the number of parameters and computational cost. Fully connected layers further process and manipulate features through linear transformations. Skip connections allow gradient information to propagate to deeper layers of the network, avoiding problems caused by vanishing and exploding gradients in deep networks.

[0020] Steps 2-3: Constructing the decoder: The decoder restores the spatial scale by stacking multiple layers of upsampling modules and performs semantic-level segmentation for each pixel.

[0021] The temporal shallow spatial features F in the encoder i Taking the mean in the time dimension and the deep spatiotemporal features F i+1′ After upsampling, skip connections are performed.

[0022] F i′ =UPBlock i (F i+1′ mean(F) i ))

[0023] The mean(F) represents the deep spatiotemporal feature corresponding to the (i+1)th scale. i ) refers to the temporal shallow spatial features F corresponding to the i-th scale. i Taking the average over the time dimension, the array shape starts from h. i ×w i ×c i ×t becomes h i ×w i ×c i ;UPBlock i This represents the upsampling module corresponding to the i-th scale. The upsampling module includes upsampling the deep spatiotemporal features, concatenating the shallow spatial features and the deep spatiotemporal features in the feature dimension to form a skip connection, so as to fully preserve the detailed information, and inputting the result of the skip connection into the convolutional block.

[0024] Upsampling includes deconvolution, batch normalization (BN), and activation functions; skip connections require averaging the temporal shallow spatial features in the encoder over time and then performing skip connections with the upsampled results of the deep spatiotemporal features; convolutional blocks contain convolution, batch normalization (BN), and activation functions, used to change the number of features in the array after skip connections.

[0025] Step 3: Train the neural network model constructed in Step 2: During the forward propagation of the input image patch, the neural network model obtains crop classification prediction results with the same spatial scale as the input image patch. The loss function is used to measure the difference between the prediction results of the neural network model and the label patch (LOSS). During the backward propagation, the gradient of LOSS with respect to the parameters of the neural network model is calculated and the optimizer is used to adjust the weights of the neural network model so that LOSS gradually decreases, thereby enabling the neural network model constructed in Step 2 to better fit the training set.

[0026] Step 3.1: Use a loss function to measure the difference between the prediction results of the neural network model constructed in Step 2 and the label blocks.

[0027] LOSS = LossFunction(F 1′ (label)

[0028] The prediction results of the neural network model are constructed for step 2, where h′1 = h, w′1 = w, and c′1 are equal to the number of crop classification categories nu in the model; LossFunction is the loss function that measures the difference between the prediction results of the neural network model and the labels.

[0029] Step 3.2, Other parameter settings: Set the initial learning rate and specify the maximum number of iterations; then use the optimizer to adjust the weights of the neural network model so that the model has less loss on the training set and converges to the optimal solution.

[0030] Step 4: Input the neural network model trained in Step 3 into the test set to obtain the crop classification results.

[0031] Furthermore, the LLTAEBlock comprises an LN (Leadership Node), a lightweight linear temporal attention encoder (LLTAE), another LN, and a fully connected layer. The output of the LLTAE is connected to the output of the fully connected layer via a skip connection. The LN ensures that the mean of each feature dimension is 0 and the variance is 1, which helps stabilize the training process, reduce gradient vanishing or exploding, and accelerate model convergence. The LLTAE implements attention operations with linear computational complexity, employing two lightweight strategies to reduce computation and parameter count: using the query matrix as a learnable parameter and sharing the query matrix with different tokens (i.e., the basic units of the input sequence). The fully connected layer maps features to a high-dimensional feature space, further extracting features through linear transformations and activation functions to enhance the model's expressive power. The skip connection fuses features extracted at different levels, helping to preserve detailed information and capture structural information. The specific construction process of the LLTAEBlock is as follows:

[0032] Step 2-2-1: Use LN to normalize the input data along the feature dimension:

[0033] X norm =LayerNorm(F L )

[0034] LayerNorm represents LN, X norm Representing temporal high-level spatial features F L The output after LN.

[0035] Step 2-2-2: Construct an LLTAE, using FAVOR+ to provide scalable, low-variance, and unbiased estimation of the attention mechanism, implementing an attention mechanism with linear complexity. Two lightweight strategies are employed to reduce computation and parameter count: using the query matrix as a learnable parameter and sharing the query matrix among different tokens. The LLTAE is then used to extract spatiotemporal features.

[0036] Computational paradigm of attention mechanism:

[0037] Assume the input is NL is the length of the input sequence; the number of features of the token is C, and each head in the multi-head self-attention MSA is represented as:

[0038] Q = xW Q K = xW K V = xW V

[0039]

[0040] is the projection matrix; Q, K, and V represent the query matrix, key matrix, and value matrix, respectively; d is the number of features for each head in multi-head attention; Sim is the similarity function; o m Represents the m-th query vector Q m The result after global interaction with the key matrix and weighting;

[0041] when When, the above formula O m The computation process becomes SoftmaxAttention, which forces the calculation of the similarity between all query matrices Q and key matrices K, resulting in O(N) time complexity. 2 The computational complexity of linear attention is less than that of Q and K; linear attention applies the mapping function Sim to Q and K respectively. The computation order is changed based on the associative law of matrix multiplication to achieve O(N) complexity; the Gaussian kernel function is approximated using FAVOR+, eliminating the need to explicitly construct an attention matrix of quadratically increasing size; given FAVOR+:

[0042]

[0043] dx refers to the number of features corresponding to vector x; the result of feature mapping. Approximate Gaussian kernel function;

[0044] Calculate the normalization factor:

[0045]

[0046] Calculate the weighted sum:

[0047]

[0048] Calculate the final output:

[0049]

[0050] ∈ represents a constant (e.g., 10). -8 To avoid denominators being zero during computation, the above is a general paradigm for implementing linear complexity attention using FAVOR+. When initializing the query matrix Q, the query matrix is ​​used as a model parameter, rather than the result of the input sequence linear layer, and a shared query matrix is ​​used for different tokens.

[0051] Using LLTAE to normalize the vector X norm Extracting spatiotemporal features X lltae :

[0052] X lltae =LLTAE(X norm )

[0053] Step 2-2-3: Use LN to normalize the feature dimensions of the array:

[0054] X lltaenorm =LayerNorm(X lltae )

[0055] Step 2-2-4: Use fully connected layers to enhance nonlinear mapping capabilities and improve the model's ability to express complex relationships.

[0056] X mlp =MLP(X lltaenorm )

[0057] MLP stands for Fully Connected Module, which consists of a fully connected layer, an activation function (such as GELU), another fully connected layer, and a Dropout layer. The first fully connected layer maps features to a higher dimension; the activation function (such as GELU) enhances the model's representational power; the second fully connected layer reduces the feature dimension to match the number of features in the input data; the Dropout layer randomly discards neurons during training to improve the model's generalization ability.

[0058] Step 2-2-5: Use residual connections to improve training stability and model performance:

[0059] F L′ =X lltae +X mlp

[0060] Furthermore, steps 2-3, which involve constructing the decoder, include the following steps:

[0061] Step 2-3-1: Analyze the deep spatiotemporal features F i+1′ Upsampling is performed to restore the spatial scale:

[0062]

[0063] CovTransBlock i This represents the upsampling corresponding to the i-th scale, including deconvolution, batch normalization (BN), and activation functions (such as ReLU). Representing deep spatiotemporal features F i+1′ The upsampling results.

[0064] Step 2-3-2: Process the temporal shallow spatial features F in the encoder. i Take the average over the time dimension:

[0065]

[0066] For F i Taking the average over the time dimension, the shape changes from h. i ×w i ×c i ×t becomes h i ×w i ×c i .

[0067] Step 2-3-3: Upsampling results of deep spatiotemporal features Mean results of shallow spatial features in time dimension Make jump connections:

[0068]

[0069] A connection refers to concatenating two arrays along their feature dimensions to reduce information loss during scaling and retain useful information. In this case, h′ i =h i w′ i =w i , It is a feature after skip connections.

[0070] Steps 2-3-4: Analyze the InderF feature after the skip connections. i′ Change the number of features:

[0071] F i′ =DeConvBlock i (InderF i′ )

[0072] DeConvBlock iFor the convolutional block corresponding to the i-th scale, it contains convolution, BN, and activation functions; This is the output of the i-th scale upsampling module; through the stacking of multiple upsampling modules, the spatial scale is gradually restored, and finally the prediction result of the neural network model is obtained. c1 is equal to the number of crop classification categories nu.

[0073] Furthermore, the activation function is ReLU.

[0074] Furthermore, the loss function is cross-entropy.

[0075] Furthermore, the optimizer is Adam.

[0076] This invention proposes a deep learning crop classification model based on a U-Net encoder-decoder architecture. The encoder part of the model extracts multi-scale, local spatial features by stacking multiple downsampling modules. Each downsampling module contains multiple convolutional blocks, and each convolutional block contains a convolutional layer, batch normalization (BN), and an activation function. The convolutional blocks are used to extract and integrate the spatial features of the image, enhancing the semantic information of the features.

[0077] For high-level temporal spatial features, a lightweight strategy and a linear temporal attention mechanism are combined to extract spatiotemporal features. The dot product operation in traditional attention is replaced with a Gaussian kernel function, and then FAVOR+ is used to approximate the Gaussian kernel function, thus achieving an attention mechanism with linear computational complexity. A shared query matrix is ​​used as a learnable parameter, and the query matrix is ​​initialized with a normal distribution. LLTAEBlock is used to model complex phenological relationships in crops, extracting spatiotemporal features with long-range dependencies.

[0078] The decoder uses an upsampling module to progressively restore the spatial scale of the extracted spatiotemporal features, bringing the deep spatiotemporal features back to the original input spatial scale. Simultaneously, a skip connection is made between the temporal shallow spatial features extracted from the encoder and the upsampled deep spatiotemporal features, preserving crop details and structural information. Ultimately, a prediction result with the same spatial scale as the input remote sensing image patch is obtained.

[0079] In summary, this invention fully utilizes the temporal and spatial features of time-series SAR images, extracting multi-scale, local spatial features based on the U-Net framework; it uses LLTABlock to extract spatiotemporal features from high-level temporal spatial features; and it restores the spatial scale by stacking multiple upsampling modules on deep spatiotemporal features. First, spatial features are efficiently extracted from the images at each time step in the time-series image using a weight-sharing U-Net encoder; LLTABlock is used to extract spatiotemporal features from high-level temporal spatial features, FAVOR+ achieves linear computational complexity, and the shared query matrix serves as a learnable parameter to reduce computation and parameter count, enabling global information interaction in the temporal dimension and modeling complex phenological relationships of crops; the decoder retains deep spatiotemporal features and fuses shallow temporal spatial features to ultimately obtain prediction results with the same spatial scale as the input data. Under the constraint of the loss function, the optimizer adjusts the weights of the neural network model to obtain the crop classification model that best fits the training set. This invention is particularly suitable for crop classification of long-term time-series SAR images, reducing memory requirements and improving the efficiency of crop classification. Attached Figure Description

[0080] Figure 1 This is a schematic diagram of the process of the present invention;

[0081] Figure 2 This is a schematic diagram of the neural network model architecture in step 2 of the present invention;

[0082] Figure 3 This is a schematic diagram of the structure of the convolution block of the present invention;

[0083] Figure 4 This is a schematic diagram of the lightweight linear time attention encoder module LLTAEBlock of the present invention;

[0084] Figure 5 This is a schematic diagram of the lightweight linear time attention encoder (LLTAE) of the present invention.

[0085] Figure 6 This is a schematic diagram of the upsampling module of the present invention;

[0086] Figure 7 The crop classification results are shown in the example diagram. Detailed Implementation

[0087] The present invention will now be described in further detail with reference to the embodiments and accompanying drawings.

[0088] This invention provides a time-series SAR crop classification method based on lightweight linear attention. It utilizes a U-Net encoder-decoder structure, employing a linearly computationally efficient attention mechanism and a lightweight strategy for spatiotemporal feature extraction. First, a training set and a test set are partitioned. The training samples are then fed into a weight-shared U-Net encoder to extract spatial feature maps of time length t with L scales from a time-series image patch X. Then the temporal high-level spatial features F L Inputting data into LLTAEBlock extracts spatiotemporal features, capturing high-level temporal spatial features. Long-range dependencies over time are used to model complex phenological patterns to obtain F. L′ The decoder uses an upsampling module to extract deep spatiotemporal features F i+1′ Upsampling with F i Same spatial scale The temporal shallow spatial features F in the encoder i The average value is obtained by taking the time dimension. and F is obtained by concatenating features along the feature dimension through skip connections and by changing the number of features through convolutional blocks. i′ By stacking multiple upsampling blocks, the final prediction result F of the neural network model at the same spatial scale as the input image is obtained. 1′ Under the constraints of the loss function, the weights of the neural network model are adjusted by the optimizer to fit the training set, and the trained neural network model is tested on the test set to obtain the crop classification results.

[0089] As an example of implementation, this invention can realize an automated operation process based on computer software and hardware.

[0090] This embodiment uses the PyTorch open-source framework to build and train the model. PyTorch features a dynamic computation graph and convenient debugging capabilities. To accelerate the model training process, this embodiment utilizes NVIDIA's CUDA parallel computing platform. CUDA distributes computational tasks to the GPU, significantly improving the efficiency of model training. Furthermore, to process and transform the data, this embodiment uses the NumPy library. NumPy is a fundamental Python library for handling multidimensional arrays and matrix operations, converting image data into tensor formats that PyTorch can process.

[0091] The specific configuration is as follows: Python version 3.9.18, PyTorch version is the PyTorch-CUDA version compatible with CUDA 11.7, and Numpy version is 1.26.4.

[0092] PASTIS-R contains four different Sentinel images of the French metropolitan area, covering a wide range of climates and cultural distributions. It includes Sentinel-1 images acquired between September 2018 and November 2019. The Sentinel images span 100×100 km with a spatial resolution of 10 m. Each image is subdivided into 1.28×1.28 km (i.e., 128×128 pixels, 10 m spatial resolution) square blocks. Ten percent of these blocks, totaling 2433 smaller blocks and 33917 radar images, were selected. These smaller blocks tend to contain rare crop types to reduce class imbalance in the dataset. Each Sentinel-1 image includes three features: vertical polarization, horizontal polarization, and the ratio of vertical to horizontal polarization.

[0093] Because the incident angle of the spaceborne radar significantly affects the echo signal, PASTIS-R separates the observations obtained from the Sentinel-1 satellite's ascending and descending orbits into two different time series. This embodiment uses the observations obtained from the ascending orbit. The label data used in PASTIS-R originates from the French land parcel identification system, provided by the French surveying agency IGN, and has an annotation accuracy of over 98% for crop types. PASTIS-R labels include 18 crop categories, a non-agricultural background category, and a mixed category containing multiple rare crop types. The label values, category values, and corresponding category names are shown in Table 1. The Sentinel-1 ascending orbit observations and labels provided by the PASTIS-R dataset are divided into training and test sets in a 3:1 ratio without overlap.

[0094] Table 1:

[0095]

[0096]

[0097] The number of crop categories is nu(20), and there are a total of S(2433) samples (each sample includes a segmented temporal SAR sequence and a corresponding label block). The shape of the temporal image block X is 128×128×3×71 (for samples with a temporal length not equal to 71, the temporal length is padded to 71), and the shape of the label block label is 128×128. The 2433 samples are divided into training and test sets in a 3:1 ratio without overlap.

[0098] Step 2: Construct a neural network model based on the U-Net encoder-decoder structure (e.g., Figure 2 As shown):

[0099] Step 2-1: Within the U-Net framework, a shared-weight encoder is used to extract multi-scale, local spatial features at each time step. There are a total of L(3) spatial scales, and each downsampling module contains R(3) stacked convolutional blocks. The stride of the convolution operation in the first convolutional block is set to 2, thus obtaining the corresponding temporal spatial features at each scale. F 1 F 2 F 3 The shapes are 128*128*64*71, 64*64*128*71, and 32*32*256*71 respectively.

[0100] F i =EnConvBlock i (F i-1 )

[0101] The downsampling module EnConvBlock corresponding to the i-th scale i It contains 3 convolutional blocks, and the convolutional blocks (such as...) Figure 3 (As shown) includes convolution, BN, and ReLU activation functions.

[0102] Step 2-2: Extract spatiotemporal features: Utilize the constructed LLTABlock (e.g., Figure 4 As shown, for temporal high-level spatial features F L (F 3 Extracting spatiotemporal features, LLTAE locks include LN, LLTAE (such as...) Figure 5 As shown), LLTAE, a fully connected layer, and a jump connection are made between the results of LLTAE and the results of the fully connected layer. LLTAElock can model complex phenological relationships and spatiotemporal features F. L′ (F 3′ The shape is 32*32*256.

[0103] F L′ =LLTAEBlock(F L )

[0104] Step 2-3: Using the spatiotemporal features obtained in step 2-2 as input to the decoder, the spatial scale is recovered using an upsampling module (e.g., ...). Figure 6 As shown, the average of the temporal shallow spatial features obtained from the encoder along the time dimension is skipped and connected with the upsampled results of the deep spatiotemporal features. This connection serves as the input to the next upsampling module. The shapes of the output arrays of the decoder's multi-layer upsampling modules are 64*64*128 and 128*128*20, respectively. The number of crop classification categories predicted by the neural network model is 20.

[0105] F i′ =UPBlock i(F i+1′ mean(F) i ))

[0106] Step 3: Train the neural network model built in Step 2. Use the cross-entropy loss function to measure the loss between the neural network model's prediction and the label value. Use the Adam optimizer to adjust the neural network model weights. The Adam optimizer combines the advantages of momentum gradient descent and RMSprop algorithm, using first-order moment estimation and second-order moment estimation to dynamically adjust the learning rate. An adaptive learning rate is calculated for each parameter to accelerate convergence and alleviate gradient oscillation problems. The initial learning rate is set to 0.001, the maximum number of iterations is set to 40, and the number of samples per batch is set to 2.

[0107] LOSS = LossFunction(F 1′ (label)

[0108] Cross-entropy loss function:

[0109]

[0110] y g and p g The true class and predicted probability corresponding to the g-th category, where nu is the number of categories.

[0111] Quantitative evaluation metrics were selected: Overall Accuracy (OA), kappa coefficient, and Intersection over Union (IoU) to quantitatively evaluate the segmentation performance of the model.

[0112]

[0113] Among them TP g FP represents the number of pixels that correctly predict the positive class as positive for the g-th class model; g TN represents the number of pixels that, for the g-th class, are incorrectly predicted as positive instead of negative; g FN represents the number of pixels that correctly predict the negative class as negative for the g-th class model; g This represents the number of pixels that, for the g-th class, are incorrectly predicted as negative instead of positive.

[0114] FLOPS is the number of floating-point operations (FLOPS) during the forward propagation process, which represents the amount of computation required by the model to process the input data. In this embodiment, FLOPS = 442.44G, and the number of parameters is 3.51M.

[0115] Step 4: Input the test set into the neural network model trained in Step 3 to obtain the crop classification results. The crop classification results in this embodiment are as follows: Figure 7 As shown, the OA, kappa coefficient, and IOU on the test set are 77.9%, 69.5%, and 44.7%, respectively. This demonstrates that the technical effectiveness and accuracy of the present invention meet industry requirements.

[0116] As can be seen from the above embodiments, this invention fully utilizes the temporal and spatial features of time-series SAR images. It efficiently extracts local spatial features from the images corresponding to each time step in the time-series image using a weight-sharing U-Net encoder; it uses LLTABlock to extract spatiotemporal features from the high-level spatial features of the time series; FAVOR+ achieves linear computational complexity; and the shared query matrix serves as a learnable parameter to reduce computational and parameter counts, enabling global information interaction in the temporal dimension and modeling complex phenological relationships of crops. Then, the decoder upsamples the deep spatiotemporal features to recover the spatial scale, and takes the average of the shallow temporal spatial features in the encoder in the temporal dimension. The average result is then connected to the upsampled result via a skip connection, ultimately obtaining a prediction result with the same spatial scale as the input data. Under the constraint of the loss function, the optimizer adjusts the weights of the neural network model to obtain the crop classification model that best fits the training set. This invention is particularly suitable for crop classification of long-term time-series SAR images, reducing memory requirements and improving the efficiency of crop classification.

Claims

1. A temporal SAR crop classification method based on lightweight linear attention, characterized in that, Includes the following steps: Step 1: Dataset Construction: Divide the samples in the target region dataset into non-overlapping training and test sets according to a certain ratio for the training and testing of the constructed neural network model; each sample includes a time-series image patch. and a corresponding tag block Where h represents the height of the array block, w represents the width of the array block, c represents the number of features in the array block, and t represents the temporal length of the temporal image block; Step 2: Construct a neural network model based on the U-Net encoder-decoder structure: Step 2-1: Constructing the encoder: Within the U-Net framework, a weight-sharing U-Net encoder is used to extract spatial features from the image data corresponding to each time step. Multi-scale, local spatial features are obtained by stacking L-layer downsampling modules. F i =EnConvBlock i (F i-1 ) EnConvBlock i For the weight-shared downsampling module corresponding to the i-th scale, i∈[1,L]; This represents the temporal spatial feature corresponding to the i-th scale, containing the spatial features corresponding to a total of t time steps. j∈[1,t]; Each downsampling module contains R convolutional blocks. r∈[1,R], the output of the current downsampling module is used as the input of the next layer downsampling module; Each convolutional block includes convolution, batch normalization (BN), and an activation function to extract and integrate spatial features of the image, enhancing the semantic information of the features. BN is batch normalization, where the output of the current convolutional block serves as the input to the next convolutional block. The convolution operation extracts local spatial features by applying a convolution kernel to the input features. Batch normalization (BN) reduces the impact of internal covariate shifts by standardizing the feature distribution within each batch, thereby accelerating the model training process and making the model more robust to different input batches. The activation function alleviates the gradient vanishing problem while introducing non-linear features. Step 2-2: Extracting Temporal Features: A lightweight linear temporal attention encoder module, LLTAEBlock, is constructed using a linear attention mechanism combined with a lightweight strategy. This module extracts the temporal high-level spatial features F obtained from the encoder constructed in Step 2-1. L Extracting spatiotemporal features F L′ : F L′ =LLTAEBlock(F L ) Represents high-level spatial features of time series, h L W L c L These represent the height, width, and number of features corresponding to the Lth scale, respectively. To extract the spatiotemporal features, h ′ L w L ′ c L ′ These represent the height, width, and number of features corresponding to the Lth scale, respectively; LLTAEBlock is a lightweight linear temporal attention encoder module, which includes LN, a lightweight linear temporal attention encoder LLTAE, LN, and a fully connected layer. The output of LLTAE is connected to the output of the fully connected layer in a skip connection as the output of LLTAEBlock. LN stands for Layer Normalization. By applying LN to the features of each sample, it eliminates the covariance bias between different features in the input data. LLTAE uses an attention mechanism with linear computational complexity to achieve the interaction of global information in the time dimension, extract long-range dependencies, and model complex phenological relationships of crops. It defines the query matrix as the parameters of the neural network model to reduce the parameters and computational cost of the neural network model. Fully connected layers further process and manipulate features through linear transformations. Skip connections allow gradient information to be passed to deeper layers of the network, avoiding problems caused by gradient vanishing and gradient exploding in deep networks. Steps 2-3: Constructing the decoder: The decoder restores the spatial scale by stacking multiple layers of upsampling modules and achieves semantic-level segmentation for each pixel; The temporal shallow spatial features F in the encoder i Taking the mean in the time dimension and the deep spatiotemporal features F i+1′ Skip connections are performed after upsampling; F i′ =UPBlock i (F i+1′ ,mean(F i )) The mean(F) represents the deep spatiotemporal feature corresponding to the (i+1)th scale. i ) refers to the temporal shallow spatial features F corresponding to the i-th scale. i Taking the average over the time dimension, the array shape starts from h. i ×w i ×c i ×t becomes h i ×w i ×c i ;UPNlock i This represents the upsampling module corresponding to the i-th scale. The upsampling module includes upsampling the deep spatiotemporal features, concatenating the shallow spatial features and the deep spatiotemporal features in the feature dimension to form a skip connection, and inputting the result of the skip connection into the convolutional block. Upsampling includes deconvolution, batch normalization (BN), and activation functions; skip connections require averaging the temporal shallow spatial features in the encoder over time and then making skip connections with the upsampled results of the deep spatiotemporal features; convolutional blocks contain convolution, batch normalization (BN), and activation functions, used to change the number of features in the array after skip connections; Step 3: Train the neural network model constructed in Step 2: During the forward propagation of the input image patch, the neural network model obtains crop classification prediction results with the same spatial scale as the input image patch. The loss function is used to measure the difference between the prediction results of the neural network model and the label patch (LOSS). During the backward propagation, the gradient of LOSS with respect to the parameters of the neural network model is calculated and the optimizer is used to adjust the weights of the neural network model so that LOSS gradually decreases, thereby enabling the neural network model constructed in Step 2 to better fit the training set. Step 3.1: Use a loss function to measure the difference between the prediction results of the neural network model constructed in Step 2 and the label blocks. LOSS=LossFunction(F 1′ ,label) The prediction results of the neural network model are constructed for step 2, where h′1 = h, w′1 = w, and c′1 are equal to the number of crop classification categories nu in the model; LossFunction is the loss function that measures the difference between the prediction results of the neural network model and the labels. Step 3.2, Other parameter settings: Set the initial learning rate and specify the maximum number of iterations; then use the optimizer to adjust the weights of the neural network model so that the model has less loss on the training set and converges to the optimal solution; Step 4: Input the neural network model trained in Step 3 into the test set to obtain the crop classification results.

2. The time-series SAR crop classification method based on lightweight linear attention as described in claim 1, characterized in that: The LLTAEBlock comprises an LN (Leadership Node), a lightweight linear temporal attention encoder (LLTAE), another LN, and a fully connected layer. The outputs of the LLTAE and the fully connected layer are connected via a skip connection. The LN ensures that the mean of each feature dimension is 0 and the variance is 1. The LLTAE implements attention operations with linear computational complexity, employing two lightweight strategies to reduce computation and parameter count: using the query matrix as a learnable parameter and sharing the query matrix among different tokens. The fully connected layer maps features to a high-dimensional feature space, further extracting features through linear transformations and activation functions to enhance the model's expressive power. The skip connection fuses features extracted at different levels. The specific construction process of LLTAEBlock is as follows: Step 2-2-1: Use LN to normalize the input data along the feature dimension: X norm =LayerNorm(F L ) LayerNorm represents LN, X norm Representing temporal high-level spatial features F L The output after LN; Step 2-2-2: Construct an LLTAE, use FAVOR+ to provide scalable, low-variance, unbiased estimation of the attention mechanism, implement an attention mechanism with linear complexity, and use two lightweight strategies to reduce computation and parameter count: using the query matrix as a learnable parameter and sharing the query matrix with different tokens. And LLTAE was used to extract spatiotemporal features; Computational paradigm of attention mechanism: Assume the input is NL is the length of the input sequence; the number of features of the token is C, and each head in the multi-head self-attention MSA is represented as: w=xW Q ,K=xW K ,V=xW V W Q W K , is the projection matrix; Q, K, and V represent the query matrix, key matrix, and value matrix, respectively; d is the number of features for each head in multi-head attention; Sim is the similarity function; O m Represents the m-th query vector Q m The result after global interaction with the key matrix and weighting; when When, the above formula O m The computation process becomes SoftmaxAttention, which forces the calculation of the similarity between all query matrices Q and key matrices K, resulting in O(N) time complexity. 2 The computational complexity of ) LinearAttention applies the mapping function Sim to Q and K respectively. The computation order is changed based on the associative law of matrix multiplication to achieve O(N) complexity; the Gaussian kernel function is approximated using FAVOR+, eliminating the need to explicitly construct an attention matrix whose size increases quadratically. Given FAVOR+: dx refers to the number of features corresponding to vector x; the result of feature mapping. Approximate Gaussian kernel function; Calculate the normalization factor: Calculate the weighted sum: Calculate the final output: ∈ represents a constant to avoid denominators of 0 during calculation; the above is the general paradigm for implementing linear complexity attention using FAVOR+; when initializing the query matrix Q, the query matrix is ​​used as a model parameter, rather than the result of the linear layer of the input sequence, and a shared query matrix is ​​used for different tokens. Using LLTAE to normalize the vector X norm Extracting spatiotemporal features X lltae : X lltae =LLTAE(X norm ) Step 2-2-3: Use LN to normalize the feature dimensions of the array: X lltaenorm =LayerNorm(X lltae ) Step 2-2-4: Use fully connected modules to enhance nonlinear mapping capabilities and improve the model's ability to express complex relationships. X mlp =MLP(X lltaenorm ) MLP stands for Fully Connected Module, which consists of a fully connected layer, an activation function, another fully connected layer, and a Dropout layer. The first fully connected layer maps features to a higher dimension. Activation functions enhance the representational power of neural network models; the second fully connected layer reduces the feature dimension to match the number of features in the input data; the Dropout layer randomly discards neurons during training to improve the model's generalization ability. Step 2-2-5: Use residual connections to improve training stability and model performance: F L′ =X lltae +X mlp 。 3. The time-series SAR crop classification method based on lightweight linear attention as described in claim 1, characterized in that: Steps 2-3, which construct the decoder, include the following steps: Step 2-3-1: Analyze the deep spatiotemporal features F i+1′ Upsampling is performed to restore the spatial scale: CovTransBlock i This represents the upsampling corresponding to the i-th scale, including deconvolution, batch normalization (BN), and activation functions. Representing deep spatiotemporal features F i+1′ The upsampling results; Step 2-3-2: Process the temporal shallow spatial features F in the encoder. i Take the average over the time dimension: For F i Taking the average over the time dimension, the shape changes from h. i ×w i ×c i ×t becomes h i ×w i ×c i ; Step 2-3-3: Upsampling results of deep spatiotemporal features Mean results of shallow spatial features in time dimension Make jump connections: A connection refers to concatenating two arrays along their feature dimensions to reduce information loss during scaling and retain useful information. In this case, h... i i =h i w i ′ =w i , These are features resulting from skip connections; Steps 2-3-4: Analyze the InderF feature after the skip connections. i′ Change the number of features: F i′ =DeConvBlock i (InderF i′ ) DeConvBlock i For the convolutional block corresponding to the i-th scale, it contains convolution, BN, and activation functions; This is the output of the i-th scale upsampling module; through the stacking of multiple upsampling modules, the spatial scale is gradually restored, and finally the prediction result of the neural network model is obtained. c1 is equal to the number of crop classification categories nu.

4. The time-series SAR crop classification method based on lightweight linear attention as described in claim 1, characterized in that: The activation function is ReLU.

5. The time-series SAR crop classification method based on lightweight linear attention as described in claim 1, characterized in that: The loss function is cross-entropy.

6. The temporal SAR crop classification method based on lightweight linear attention as described in claim 1, characterized in that: The optimizer is Adam.

7. The time-series SAR crop classification method based on lightweight linear attention as described in claim 2, characterized in that: The ∈ = 10 -8 .

Citation Information

Patent Citations

  • Crop classification method based on optical image and SAR image time sequence feature fusion

    CN116310805A

  • SAR image change detection method and device based on space attention model

    CN117392187A