Sequential SAR crop classification method based on lightweight linear attention

Abstract

The invention belongs to the technical field of remote sensing monitoring, and particularly relates to a sequential SAR crop classification method based on lightweight linear attention. The method comprises the following steps of: firstly, efficiently extracting spatial characteristics of an image of each time step in a time sequence image by utilizing a weight-shared U-Net encoder by utilizing spatial and temporal characteristics of the time sequence SAR image; lLTAEBlock is used for extracting spatiotemporal features from time sequence advanced spatial features, FAVOR + is used for achieving linear calculation complexity, a shared query matrix serves as a learnable parameter, the calculation amount and the parameter amount are reduced, global information interaction in the time dimension is achieved, and modeling is conducted on the complex phenological relation of crops; and the decoder retains the deep spatial-temporal features and fuses the time sequence shallow spatial features to finally obtain a prediction result with the same spatial scale as the input data. And under the constraint of the loss function, adjusting through an optimizer to obtain a crop classification model with the best fitting. The method is especially suitable for crop classification of long-time-sequence SAR images, reduces the need of a memory, and improves the crop classification efficiency.

Description

Technical Field

[0001] The invention belongs to the technical field of remote sensing monitoring, and in particular is a time-series SAR crop classification method based on lightweight linear attention. Background Art

[0002] Crop classification aims to identify and distinguish different crop types using remote sensing technology. Accurate crop classification is of great significance for monitoring crop growth conditions, optimizing resource allocation and promoting sustainable agricultural development.

[0003] In terms of remote sensing data sources, crop classification mostly relies on optical remote sensing images, which are classified by analyzing color, texture, and shape features. However, optical remote sensing is easily affected by weather conditions, and it is difficult to obtain clear images in areas with frequent clouds and fog. In contrast, synthetic aperture radar SAR has all-day and all-weather monitoring capabilities and can obtain remote sensing images of crops under various weather conditions. In addition, SAR is sensitive to the shape, structure, and dielectric constant of crop scatterers, and can identify crop types that are difficult to distinguish with optical images. During the growth process of crops, different crops will show different spectral characteristics and structural changes at different growth stages. The temporal and spatial characteristics of the stable crop growth monitoring data provided by SAR are extracted to improve the accuracy of crop classification.

[0004] In terms of crop classification methods, traditional methods such as maximum likelihood classification, support vector machine (SVM), and random forest (RF) machine learning methods mostly rely on manual features. These methods have limited effects when dealing with complex time series changes and are sensitive to salt and pepper noise in SAR images. In recent years, deep learning methods have shown significant advantages in crop classification. Unlike traditional methods, deep learning models can automatically extract spatiotemporal features from time series SAR and effectively deal with complex time series changes and salt and pepper noise. Commonly used time series crop classification models include convolutional neural networks (CNN), recurrent neural networks (RNN), and their variant networks. These models can capture the dynamic characteristics of crop growth, but there are problems such as limited receptive field, gradient disappearance, and gradient explosion when dealing with long time series data.

[0005] To avoid this problem, the self-attention mechanism captures long-range dependencies by calculating the correlation between each basic unit in the sequence and other basic units. Dot product attention is an implementation of the self-attention mechanism, which measures the similarity between tokens, i.e., the basic units of the input sequence, by calculating the dot product between the query matrix and the key matrix. However, the computational complexity of dot product attention is proportional to the square of the number of tokens, and the high memory requirements when processing large-scale data limit its efficiency in practical applications. Summary of the invention

[0006] In view of the above-mentioned problems or shortcomings, and to solve the limitations of existing deep learning crop classification methods, the present invention provides a time-series SAR crop classification method based on lightweight linear attention. The linear attention mechanism is used to effectively capture long-term time series relationships under the premise of linear computational complexity, thereby avoiding the quadratic complexity problem of the traditional attention mechanism. At the same time, a lightweight strategy is used to reduce model parameters, thereby improving the efficiency and practicability of crop classification for time-series SAR images.

[0007] A method for temporal SAR crop classification based on lightweight linear attention includes the following steps:

[0008] Step 1: Dataset construction: Divide the samples in the target area dataset (such as PASTIS-R) into non-overlapping training sets and test sets according to the proportion for training and testing of the constructed neural network model. Each sample includes a time series image block and a corresponding label block Among them, h represents the height of the array block, w represents the width of the array block, c represents the number of features of the array block, and t represents the temporal length of the temporal image block.

[0009] Step 2: Build a neural network model based on the U-Net encoder-decoder structure:

[0010] Step 2-1, build the encoder: Under the U-Net framework, use the weight-sharing U-Net encoder to extract spatial features from the image data corresponding to each time step, avoiding parameter redundancy caused by using encoders for image data at each time step; stack L layers of downsampling modules to obtain multi-scale, local spatial features:

[0011] F i =EnConvBlock i (F i-1 )

[0012]

[0013] EnConvBlock i is the weight-shared downsampling module corresponding to the i-th scale, i∈[1,L]; Represents the temporal spatial features corresponding to the i-th scale, including the spatial features corresponding to a total of t time steps

[0014] Each downsampling module contains R convolutional blocks. The output of the current downsampling module serves as the input of the next downsampling module.

[0015] Each convolution block includes convolution, BN and activation function, which are used to extract and integrate the spatial features of the image and enhance the semantic information of the features; BN is batch normalization, and the output of the current convolution block is used as the input of the next convolution block; the convolution operation extracts local spatial features by applying the convolution kernel to the input features; batch normalization BN standardizes the feature distribution within each batch to reduce the impact of internal covariate shift, thereby accelerating the model training process and making the model more robust to different input batches; the activation function alleviates the gradient vanishing problem while introducing nonlinear features.

[0016] Step 2-2, extracting temporal features: Using the linear attention mechanism combined with the lightweight strategy to build a lightweight linear temporal attention encoder module LLTAEBlock, the encoder constructed in step 2-1 obtains the temporal high-level spatial features F L Extract spatiotemporal features F L′ :

[0017] F L′ =LLTAEBlock(F L )

[0018] represents the temporal high-level spatial features, h L 、w L 、c L They represent the height, width, and number of features of the spatial features corresponding to the Lth scale respectively; is the extracted spatiotemporal features, h′ L , w′ L , c′ L They represent the height, width, and number of features of the spatiotemporal features corresponding to the Lth scale respectively; LLTAEBlock is a constructed lightweight linear temporal attention encoder module, including LN, lightweight linear temporal attention encoder LLTAE, LN, and a fully connected layer. The output of LLTAE is skip-connected with the output of the fully connected layer as the output of LLTAEBlock;

[0019] LN stands for layer normalization. By performing layer normalization LN on the features of each sample, the covariance offset between different features in the input data is eliminated, so that the model can perform more effective gradient transfer between different layers, which helps the model to be more stable during training. LLTAE uses an attention mechanism with linear computational complexity to realize the interaction of global information in the time dimension, extract long-range dependencies, model the complex phenological relationships of crops, and define the query matrix as a neural network model parameter to reduce the parameters and computational complexity of the neural network model. The fully connected layer further processes and handles the features through linear transformation. The jump connection allows the gradient information to be passed to deeper layers of the network, avoiding problems caused by gradient vanishing and gradient exploding in deep networks.

[0020] Step 2-3, build a decoder: The decoder restores the spatial scale by stacking multiple layers of upsampling modules and achieves semantic level segmentation for each pixel.

[0021] The temporal shallow spatial features F in the encoder i Take the mean and deep spatiotemporal features F in the time dimension i+1′ The upsampling is followed by skip connections.

[0022] F i′ =UPBlock i (F i+1′ ,mean(F i ))

[0023] represents the deep spatiotemporal features corresponding to the i+1th scale; mean(F i ) refers to the temporal shallow spatial feature F corresponding to the i-th scale i Take the mean in the time dimension, the array shape changes from h i × i ×c i ×t becomes h i × i ×c i ;UPBlock i It represents the upsampling module corresponding to the i-th scale. The upsampling module includes upsampling the deep spatiotemporal features, concatenating the shallow spatial features with the deep spatiotemporal features in the feature dimension for jump connection to fully retain the detail information, and inputting the jump connection result into the convolution block.

[0024] Upsampling includes deconvolution, BN and activation function; the jump connection needs to first take the average of the shallow spatial features of the temporal sequence in the encoder in the time dimension, and make a jump connection with the upsampling result of the deep spatiotemporal features; the convolution block contains convolution, BN and activation function, which is used to change the number of features of the array after the jump connection.

[0025] Step 3, training the neural network model constructed in step 2: in the process of forward propagation, the input image block obtains the crop classification prediction result of the same spatial scale as the input image block through the neural network model, and uses the loss function to measure the difference LOSS between the prediction result of the neural network model and the label block. In the process of backward propagation, the gradient of LOSS to the parameters of the neural network model is calculated and the weight of the neural network model is adjusted using the optimizer to gradually reduce LOSS, so that the neural network model constructed in step 2 can better fit the training set;

[0026] Step 3.1: Use the loss function to measure the difference between the prediction result of the neural network model constructed in step 2 and the label block:

[0027] LOSS=LossFunction(F 1′ , label)

[0028] Construct the prediction result of the neural network model for step 2, where h′ 1 =h, w′ 1 =w, c′ 1 It is equal to the number of crop classification categories nu of the model; LossFunction is the loss function that measures the difference between the prediction result of the neural network model and the label.

[0029] Step 3.2, other parameter settings: set the initial learning rate and specify the maximum number of iterations; then use the optimizer to adjust the weights of the neural network model so that the model has a smaller loss on the training set and converges to the optimal solution.

[0030] Step 4: Input the test set into the neural network model trained in step 3 to obtain the crop classification results.

[0031] Furthermore, the LLTAEblock includes LN, a lightweight linear temporal attention encoder LLTAE, LN and a fully connected layer, and the output of the LLTAE is jump-connected to the output of the fully connected layer. LN makes the mean of each feature dimension 0 and the variance 1, which helps to stabilize the training process, reduce gradient vanishing or gradient explosion, and accelerate model convergence; LLTAE implements attention operations with linear computational complexity, using two lightweight strategies of query matrix as learnable parameters and different Tokens (i.e., the basic unit of the input sequence) to share the query matrix to reduce the amount of calculation and the amount of parameters; the fully connected layer maps the features to a high-dimensional feature space, further extracts features through linear transformations and activation functions, and enhances the expressiveness of the model; the jump connection fuses the extracted features at different levels, which helps to retain detail information and capture structural information. The construction process of LLTAEBlock is specifically as follows:

[0032] Step 2-2-1: Use LN to normalize the input data in the feature dimension:

[0033] X norm =LayerNorm(F L )

[0034] LayerNorm means LN,X norm Represents the temporal high-level spatial feature F L Output through LN.

[0035] Step 2-2-2, build LLTAE, use FAVOR+ to provide scalable low-variance and unbiased estimation of the attention mechanism, implement the attention mechanism with linear complexity, use the query matrix as a learnable parameter and two lightweight strategies of different tokens sharing the query matrix to reduce the amount of calculation and parameters. And use LLTAE to extract spatiotemporal features.

[0036] Computational paradigm of attention mechanism:

[0037] Assume the input is NL is the length of the input sequence; the number of features of the token is C, and each head in the multi-head self-attention MSA is expressed as:

[0038] Q = xW Q , K = xW K , V = xW V

[0039]

[0040] is the projection matrix; Q, K, V represent the query matrix, key matrix, and value matrix respectively; d is the number of features of each head in the multi-head attention; Sim is the similarity function; o m represents the mth query vector Q m The result after global interaction and weighting with the bond matrix;

[0041] when When m The calculation process becomes SoftmaxAttention, which forces the calculation of the similarity between all query matrices Q and key matrices K, resulting in O(N 2 ) computational complexity; LinearAttention applies the mapping function Sim to Q and K respectively, According to the associative law of matrix multiplication, the calculation order is changed to achieve O(N) complexity; FAVOR+ is used to approximate the Gaussian kernel function, without explicitly constructing the attention matrix whose size grows quadratically; FAVOR+ is known:

[0042]

[0043] dx refers to the number of features corresponding to the x vector; use the feature mapping result Approximate Gaussian kernel function;

[0044] Calculate the normalization factor:

[0045]

[0046] Compute the weighted sum:

[0047]

[0048] Calculate the final output:

[0049]

[0050] ∈ represents a constant (such as 10 -8 ), to avoid the denominator being 0 during the calculation process; the above is the general paradigm for using FAVOR+ to achieve linear complexity attention; when initializing the query matrix Q, the query matrix is ​​used as a model parameter instead of the result of the linear layer of the input sequence, and a shared query matrix is ​​used for different Tokens

[0051] Use LLTAE to normalize the vector X norm Extract spatiotemporal features X lltae :

[0052] X lltae =LLTAE(X norm )

[0053] Step 2-2-3: Use LN to normalize the feature dimension of the array:

[0054] X lltaenorm =LayerNorm(X lltae )

[0055] Step 2-2-4: Use the fully connected layer to enhance the nonlinear mapping capability and the model's ability to express complex relationships:

[0056] X mlp =MLP(X lltaenorm )

[0057] MLP stands for fully connected module, which includes fully connected layer, activation function (such as GELU), fully connected layer and Dropout layer in sequence. The first fully connected layer maps features to higher dimensions; activation function (such as GELU) enhances the representation ability of the model; the second fully connected layer reduces the feature dimension to keep consistent with the number of features of the input data; Dropout layer randomly discards neurons during training to improve the generalization ability of the model.

[0058] Step 2-2-5: Use residual connections to improve training stability and model performance:

[0059] F L′ =X lltae +X mlp

[0060] Furthermore, the step 2-3 constructs a decoder, comprising the following steps:

[0061] Step 2-3-1: Deep spatiotemporal features F i+1′ Upsample to restore the spatial scale:

[0062]

[0063] CovTransBlock i Indicates the upsampling corresponding to the i-th scale, including deconvolution, BN, and activation function (such as ReLU). Represents the deep spatiotemporal features F i+1′ The upsampling result.

[0064] Step 2-3-2: Temporal shallow spatial features F in the encoder i Taking the mean over the time dimension:

[0065]

[0066] F i Taking the mean in the time dimension, the shape changes from h i × i ×c i ×t becomes h i × i ×c i .

[0067] Step 2-3-3: Upsampling results of deep spatiotemporal features The average result of shallow spatial features of time series in the time dimension To make a jump connection:

[0068]

[0069] Connection refers to the concatenation of two arrays in the feature dimension to reduce the information loss caused by the scale transformation process and retain valid information. At this time, h′ i =h i , w′ i =w i , It is the feature after skip connection.

[0070] Step 2-3-4: InderF after skip connection i′ Change the number of features:

[0071] F i′ =DeConvBlock i (InderF i′ )

[0072] DeConvBlock iIt is the convolution block corresponding to the i-th scale, including convolution, BN, and activation function; is the output of the i-th scale upsampling module; after stacking multiple layers of upsampling modules, the spatial scale is gradually restored, and finally the prediction result of the neural network model is obtained c 1 Equal to the number of crop classification categories nu.

[0073] Furthermore, the activation function is ReLU.

[0074] Furthermore, the loss function is cross entropy.

[0075] Furthermore, the optimizer is Adam.

[0076] The present invention proposes a deep learning crop classification model based on the U-Net encoder-decoder structure. The encoder part of the model extracts multi-scale, local spatial features by stacking multiple downsampling modules, each of which contains multiple convolutional blocks, each of which contains a convolutional layer, BN and an activation function. The convolutional block is used to extract and integrate the spatial features of the image and enhance the semantic information of the features.

[0077] For high-level temporal spatial features, a lightweight strategy is combined with a linear time attention mechanism to extract spatiotemporal features. The dot product operation in traditional attention is replaced with a Gaussian kernel function, and then the Gaussian kernel function is approximated using FAVOR+ to implement an attention mechanism with linear computational complexity; the shared query matrix is ​​used as a learnable parameter, and the query matrix is ​​initialized with a normal distribution. LLTAEBlock is used to model the complex phenological relationship of crops and extract spatiotemporal features with long-range dependencies.

[0078] The decoder gradually restores the spatial scale of the extracted spatiotemporal features through the upsampling module, and restores the deep spatiotemporal features to the spatial scale of the original input. At the same time, the temporal shallow spatial features extracted in the encoder are jump-connected with the upsampled deep spatiotemporal features to retain the details and structural information of the crop. Finally, the prediction result with the same spatial scale as the input remote sensing image block is obtained.

[0079] In summary, the present invention makes full use of the features of the time and space dimensions of time series SAR images, extracts multi-scale, local spatial features based on the U-Net framework; uses LLT AEBlock to extract spatiotemporal features from high-level spatial features of the time series; and stacks multi-layer upsampling modules for deep spatiotemporal features to restore the spatial scale. First, the U-Net encoder with weight sharing is used to efficiently extract spatial features from the images of each time step in the time series image; LLT AEBlock is used to extract spatiotemporal features from high-level spatial features of the time series, FAVOR+ achieves linear computational complexity, and the shared query matrix is ​​used as a learnable parameter to reduce the amount of calculation and parameter, realize global information interaction in the time dimension, and model the complex phenological relationship of crops; the decoder retains the deep spatiotemporal features and fuses the shallow spatial features of the time series to finally obtain the prediction result with the same spatial scale as the input data. Under the constraint of the loss function, the weights of the neural network model are adjusted by the optimizer to obtain the crop classification model that best fits the training set. The present invention is particularly suitable for crop classification of long time series SAR images, reduces the need for memory, and improves the efficiency of crop classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 It is a schematic diagram of the process of the present invention;

[0081] Figure 2 This is a schematic diagram of the architecture of the neural network model in step 2 of the present invention;

[0082] Figure 3 Schematic diagram of the structure of the convolution block of the present invention;

[0083] Figure 4 It is a structural diagram of the lightweight linear time attention encoder module LLTAEBlock of the present invention;

[0084] Figure 5 Schematic diagram of the structure of the lightweight linear time attention encoder LLTAE of the present invention;

[0085] Figure 6 It is a structural schematic diagram of the up-sampling module of the present invention;

[0086] Figure 7 This is a crop classification result diagram of the embodiment. DETAILED DESCRIPTION

[0087] The present invention is further described in detail below in conjunction with embodiments and drawings.

[0088] The present invention provides a time-series SAR crop classification method based on lightweight linear attention, which extracts spatiotemporal features based on the U-Net encoder-decoder structure, the attention mechanism with linear computational complexity and the lightweight strategy. First, the training set and the test set are divided; the training samples are sent to the weight-sharing U-Net encoder, and the spatial feature map with a time sequence length of t and L scales is extracted for the time sequence image block X. Then the temporal high-level spatial feature F L Input into LLTAEBlock to extract spatiotemporal features and capture high-level spatial features of time series The long-range dependencies in the temporal dimension are used to model complex phenological patterns and obtain F L′ ; The decoder uses the upsampling module to transform the deep spatiotemporal features F i+1′ Upsample to F i The same spatial scale And the temporal shallow spatial feature F in the encoder i Taking the average value in the time dimension and Through skip connections, the feature dimension is concatenated, and the number of features is changed through convolution blocks to obtain F. i′ , through the stacking of multiple layers of upsampling blocks, the neural network model prediction result F with the same spatial scale as the input image is finally obtained 1′ Under the constraints of the loss function, the weights of the neural network model are adjusted through the optimizer, the training set is fitted, and the trained neural network model is tested on the test set to obtain the crop classification results.

[0089] As an implementation case, the present invention can realize automatic operation process based on computer software and hardware.

[0090] This embodiment implements the construction and training of the model based on the PyTorch open source framework, which has dynamic calculation graphs and convenient debugging functions. In order to accelerate the training process of the model, this embodiment uses NVIDIA's CUDA parallel computing platform, which assigns computing tasks to the GPU, significantly improving the efficiency of model training. In addition, in order to process and convert data, this embodiment uses the Numpy library, which is the basic library for processing multidimensional arrays and matrix operations in Python, to convert image data into a tensor format that can be processed by PyTorch.

[0091] The specific configuration is as follows: Python version is 3.9.18, PyTorch version is PyTorch-CUDA version compatible with CUDA11.7, and Numpy version is 1.26.4.

[0092] PASTIS-R contains 4 different Sentinel images of the French metropolitan area, covering a wide range of climates and cultural distributions, including Sentinel-1 images acquired between September 2018 and November 2019. Sentinel images span 100×100km with a spatial resolution of 10m. Each image is subdivided into square blocks of 1.28×1.28km (i.e. 128×128 pixels, spatial resolution 10m). 10% of them, i.e. 2433 small blocks, a total of 33917 radar images, are selected. These small blocks tend to be rare crop types to reduce the class imbalance of the dataset. Each Sentinel-1 image includes three features: vertical polarization, horizontal polarization, and the ratio of vertical to horizontal polarization.

[0093] Because the incident angle of the satellite-borne radar will significantly affect the echo signal, PASTIS-R divides the observation results obtained from the ascending orbit and the descending orbit of the Sentinel-1 satellite into two different time series. This embodiment uses the observation results obtained from the ascending orbit. The label data used in PASTIS-R is derived from the French land parcel identification system and provided by the French mapping agency IGN. The annotation accuracy of crop types exceeds 98%. PASTIS-R labels include 18 crop classes, a background class for non-agricultural crops, and a mixed class containing a variety of rare crop types. The label value category value and the corresponding category name are shown in Table 1. The Sentinel-1 ascending orbit observation results and labels provided by the PASTIS-R dataset are divided into training sets and test sets in a non-overlapping ratio of 3:1.

[0094] Table 1:

[0095]

[0096]

[0097] The number of crop categories is nu(20), with a total of S(2433) samples (each sample includes a time-series SAR sequence after segmentation and a corresponding label block), where the shape of the time-series image block X is 128×128×3×71 (for samples whose time-series length is not equal to 71, the time-series length is padded to 71), and the shape of the label block label is 128×128. The 2433 samples are divided into training and test sets in a non-overlapping ratio of 3:1.

[0098] Step 2: Build a neural network model based on the U-Net encoder-decoder structure (such as Figure 2 shown):

[0099] Step 2-1: In the U-Net framework, a shared weight encoder is used to extract multi-scale, local spatial features for each time step. There are L(3) spatial scales in total, and each downsampling module contains R(3) stacked convolution blocks. The stride of the convolution operation in the first convolution block is set to 2, and the corresponding temporal spatial features are obtained at each scale. F 1 , F 2 , F 3 The shapes are 128*128*64*71, 64*64*128*71, and 32*32*256*71 respectively.

[0100] F i =EnConvBlock i (F i-1 )

[0101] The downsampling module EnConvBlock corresponding to the i-th scale i Contains 3 convolution blocks, convolution blocks (such as Figure 3 As shown) includes convolution, BN, and ReLU activation functions.

[0102] Step 2-2, extracting spatiotemporal features: using the constructed LLTAEBlock (such as Figure 4 As shown in the figure, the temporal high-level spatial feature F L (F 3 ) extracts spatiotemporal features, LLTAEBlock includes LN, LLTAE (such as Figure 5 As shown), LN, fully connected layer, the results of LLTAE are skipped and connected with the results of the fully connected layer. LLTAEBlock can model complex phenological relationships, and the spatiotemporal features F L′ (F 3′ )The shape is 32*32*256:

[0103] F L′ =LLTAEBlock(F L )

[0104] Step 2-3: Use the spatiotemporal features obtained in step 2-2 as the input of the decoder, and use the upsampling module to restore the spatial scale (such as Figure 6 As shown in the figure, the shallow spatial features of the temporal sequence obtained in the encoder are averaged in the time dimension and jump-connected with the upsampling results of the deep spatiotemporal features as the input of the next upsampling module. The shapes of the output arrays of the multi-layer upsampling module of the decoder are: 64*64*128 and 128*128*20. The number of crop classification categories predicted by the neural network model is 20.

[0105] F i′ =UPBlock i(F i+1′ ,mean(F i ))

[0106] Step 3: Train the neural network model constructed in step 2, use the cross entropy loss function to measure the difference LOSS between the prediction result of the neural network model and the label value, and use the Adam optimizer to adjust the weight of the neural network model. The Adam optimizer combines the advantages of the momentum gradient descent and RMSprop algorithms, and uses the first-order moment estimation and second-order moment estimation to dynamically adjust the learning rate. The adaptive learning rate is calculated for each parameter to accelerate convergence and alleviate the gradient oscillation problem. The initial learning rate is set to 0.001, the maximum number of iterations is set to 40, and the number of batch processing samples is set to 2:

[0107] LOSS=LossFunction(F 1′ , label)

[0108] Cross entropy loss function:

[0109]

[0110] y g and p g The true class and predicted probability corresponding to the g-th category, nu is the number of categories.

[0111] Select quantitative evaluation indicators: Overall Accuracy (OA), kappa coefficient, and Intersection over Union (IoU) to perform quantitative accuracy evaluation on the model segmentation effect.

[0112]

[0113] TP g Indicates the number of pixels that the g-th model correctly predicts as positive; FP g Indicates the number of pixels for which the g-th model incorrectly predicts the negative class as the positive class; TN g FN represents the number of pixels that the g-th model correctly predicts as negative. g Represents the number of pixels for which the g-th class model incorrectly predicts the positive class as the negative class.

[0114] FLOPS calculates the number of floating point operations (FLOPS) during the forward propagation process, that is, the amount of calculation required by the model when processing input data. For this embodiment, FLOPS=442.44G, and the number of parameters is 3.51M.

[0115] Step 4: Input the test set into the neural network model trained in step 3 to obtain the crop classification result. The crop classification result of this embodiment is as follows: Figure 7 As shown, the 0A, kappa coefficient, and IOU on the test set are 77.9%, 69.5%, and 44.7%, respectively. It can be seen that the technical effect and accuracy of the present invention meet the industry requirements.

[0116] It can be seen from the above embodiments that the present invention makes full use of the features of the time and space dimensions of the time series SAR images, and efficiently extracts local spatial features by using a weight-sharing U-Net encoder for the images corresponding to each time step in the time series images; LLT AEBlock is used to extract spatiotemporal features from the time series high-level spatial features, FAVOR+ realizes linear computational complexity, and the shared query matrix is ​​used as a learnable parameter to reduce the amount of calculation and the amount of parameters, so as to realize global information interaction in the time dimension and model the complex phenological relationship of crops; then the decoder upsamples the deep spatiotemporal features to restore the spatial scale, takes the average of the shallow spatial features of the time series in the encoder in the time dimension, and jumps the average result with the upsampling result, and finally obtains the prediction result with the same spatial scale as the input data. Under the constraint of the loss function, the weights of the neural network model are adjusted by the optimizer to obtain the crop classification model that best fits the training set. The present invention is particularly suitable for crop classification of long time series SAR images, reduces the need for memory, and improves the efficiency of crop classification.

Claims

1. A method for temporal SAR crop classification based on lightweight linear attention, characterized in that: The following steps are involved: Step 1: Dataset construction: Divide the samples in the target area dataset into non-overlapping training sets and test sets according to the proportion for training and testing the constructed neural network model; each sample includes a time series image block and a corresponding label block Among them, h represents the height of the array block, w represents the width of the array block, c represents the number of features of the array block, and t represents the temporal length of the temporal image block; Step 2: Build a neural network model based on the U-Net encoder-decoder structure: Step 2-1, build the encoder: Under the U-Net framework, use the weight-sharing U-Net encoder to extract spatial features from the image data corresponding to each time step, and stack L layers of downsampling modules to obtain multi-scale, local spatial features: F i =EnConvBlock i (F i-1 ) EnConvBlock i is the weight-shared downsampling module corresponding to the i-th scale, i∈[1,L]; Represents the temporal spatial features corresponding to the i-th scale, including the spatial features corresponding to a total of t time steps j∈[1,t]; Each downsampling module contains R convolutional blocks. r∈[1,R], the output of the current downsampling module is used as the input of the next downsampling module; Each convolution block includes convolution, BN and activation function, which are used to extract and integrate the spatial features of the image and enhance the semantic information of the features; BN is batch normalization, and the output of the current convolution block is used as the input of the next convolution block; the convolution operation extracts local spatial features by applying the convolution kernel to the input features; batch normalization BN standardizes the feature distribution within each batch to reduce the impact of internal covariate shift, thereby accelerating the model training process and making the model more robust to different input batches; the activation function introduces nonlinear features while alleviating the gradient vanishing problem; Step 2-2, extracting temporal features: Using the linear attention mechanism combined with the lightweight strategy to build a lightweight linear temporal attention encoder module LLTAEBlock, the encoder constructed in step 2-1 obtains the temporal high-level spatial features F L Extract spatiotemporal features F L′ : F L′ =LLTAEBlock(F L ) represents the temporal high-level spatial features, h L , W L 、c L They represent the height, width, and number of features of the spatial features corresponding to the Lth scale respectively; is the extracted spatiotemporal features, h ′ L 、w L ′ 、c L ′ They represent the height, width, and number of features of the spatiotemporal features corresponding to the Lth scale respectively; LLTAEBlock is a constructed lightweight linear temporal attention encoder module, including LN, lightweight linear temporal attention encoder LLTAE, LN, and a fully connected layer. The output of LLTAE is skip-connected with the output of the fully connected layer as the output of LLTAEBlock; LN is layer normalization. By performing layer normalization LN on the features of each sample, the covariance offset between different features in the input data is eliminated. LLTAE uses an attention mechanism with linear computational complexity to realize the interaction of global information in the time dimension, extract remote dependencies, model the complex phenological relationship of crops, and define the query matrix as a neural network model parameter to reduce the parameters and computation of the neural network model. The fully connected layer further processes and handles the features through linear transformation. Skip connections allow gradient information to be passed to deeper layers of the network, avoiding problems caused by gradient vanishing and gradient exploding in deep networks. Step 2-3: Construct a decoder: The decoder restores the spatial scale by stacking multiple layers of upsampling modules and implements semantic level segmentation for each pixel; The temporal shallow spatial features F in the encoder i Take the mean and deep spatiotemporal features F in the time dimension i+1′ Skip connection is performed after upsampling; F i′ =UPBlock i (F i+1′ ,mean(F i )) represents the deep spatiotemporal features corresponding to the i+1th scale; mean(F i ) refers to the temporal shallow spatial feature F corresponding to the i-th scale i Take the mean in the time dimension, the array shape changes from h i × i ×c i ×t becomes h i × i ×c i ;UPNlock i represents the upsampling module corresponding to the i-th scale. The upsampling module includes upsampling the deep spatiotemporal features, concatenating the shallow spatial features with the deep spatiotemporal features in the feature dimension for jump connection, and inputting the jump connection result into the convolution block; Upsampling includes deconvolution, BN and activation function; skip connection requires first taking the average of the temporal shallow spatial features in the encoder in the time dimension, and making a skip connection with the upsampling result of the deep spatiotemporal features; convolution block includes convolution, BN and activation function, which is used to change the number of features of the array after the skip connection; Step 3, training the neural network model constructed in step 2: in the process of forward propagation, the input image block obtains the crop classification prediction result of the same spatial scale as the input image block through the neural network model, and uses the loss function to measure the difference LOSS between the prediction result of the neural network model and the label block. In the process of backward propagation, the gradient of LOSS to the parameters of the neural network model is calculated and the weight of the neural network model is adjusted using the optimizer to gradually reduce LOSS, so that the neural network model constructed in step 2 can better fit the training set; Step 3.1: Use the loss function to measure the difference between the prediction result of the neural network model constructed in step 2 and the label block: LOSS=LossFunction(F 1′ ,label) Construct the prediction result of the neural network model for step 2, where h′1=h, w′1=w, c′1 is equal to the number of model crop classification categories nu; LossFunction is the loss function that measures the difference between the prediction result of the neural network model and the label; Step 3.2, other parameter settings: set the initial learning rate and specify the maximum number of iterations; then use the optimizer to adjust the weights of the neural network model so that the model has a smaller loss on the training set and converges to the optimal solution; Step 4: Input the test set into the neural network model trained in step 3 to obtain the crop classification results.

2. The method for crop classification of time-series SAR based on lightweight linear attention as claimed in claim 1, characterized in that: The LLTAE block includes LN, a lightweight linear time attention encoder LLTAE, LN and a fully connected layer, and the output of LLTAE is skip-connected with the output of the fully connected layer; LN makes the mean of each feature dimension 0 and the variance 1; LLTAE implements attention operations with linear computational complexity, and uses two lightweight strategies of using the query matrix as a learnable parameter and sharing the query matrix with different tokens to reduce the amount of calculation and the amount of parameters; the fully connected layer maps the features to a high-dimensional feature space, further extracts the features through linear transformation and activation function, and enhances the expression ability of the model; the skip connection fuses the extracted features at different levels; the construction process of LLTAE block is as follows: Step 2-2-1: Use LN to normalize the input data in the feature dimension: X norm =LayerNorm(F L ) LayerNorm means LN,X norm Represents the temporal high-level spatial feature F L Output through LN; Step 2-2-2, build LLTAE, use FAVOR+ to provide scalable low-variance and unbiased estimation of the attention mechanism, implement the attention mechanism with linear complexity, and use two lightweight strategies: query matrix as learnable parameter and different tokens share query matrix to reduce the amount of calculation and parameter; And use LLTAE to extract spatiotemporal features; Computational paradigm of attention mechanism: Assume the input is NL is the length of the input sequence; the number of features of the token is C, and each head in the multi-head self-attention MSA is expressed as: w=xW Q ,K=xW K ,V=xW V W Q , W K , is the projection matrix; Q, K, and V represent the query matrix, key matrix, and value matrix, respectively; d is the number of features of each head in the multi-head attention; Sim is the similarity function; O m represents the mth query vector Q m The result after global interaction and weighting with the bond matrix; when When m The calculation process becomes SoftmaxAttention, which forces the calculation of the similarity between all query matrices Q and key matrices K, resulting in O(N 2 )’s computational complexity; LinearAttention applies the mapping function Sim to Q and K respectively. The computation order is changed according to the associative law of matrix multiplication to achieve O(N) complexity; FAVOR+ is used to approximate the Gaussian kernel function, without explicitly constructing the attention matrix whose size grows quadratically; Known FAVOR+: dx refers to the number of features corresponding to the x vector; use the feature mapping result Approximate Gaussian kernel function; Calculate the normalization factor: Compute the weighted sum: Calculate the final output: ∈ represents a constant to avoid the denominator being 0 during the calculation process; the above is the general paradigm for using FAVOR+ to implement linear complexity attention; when initializing the query matrix Q, the query matrix is ​​used as a model parameter instead of the result of the linear layer of the input sequence, and a shared query matrix is ​​used for different Tokens Use LLTAE to normalize the vector X norm Extract spatiotemporal features X lltae : X lltae =LLTAE(X norm ) Step 2-2-3: Use LN to normalize the feature dimension of the array: X lltaenorm =LayerNorm(X lltae ) Step 2-2-4: Use the fully connected module to enhance the nonlinear mapping capability and the model's ability to express complex relationships: X mlp =MLP(X lltaenorm ) MLP stands for fully connected module, which includes fully connected layer, activation function, fully connected layer and Dropout layer in sequence; the first fully connected layer maps the features to a higher dimension; The activation function enhances the representation ability of the neural network model; the second fully connected layer reduces the feature dimension to be consistent with the number of features of the input data; the Dropout layer randomly discards neurons during training to improve the generalization ability of the model; Step 2-2-5: Use residual connections to improve training stability and model performance: F L′ =X lltae +X mlp 。 3. The method for crop classification of time-series SAR based on lightweight linear attention as claimed in claim 1, characterized in that: The step 2-3 constructs a decoder, comprising the following steps: Step 2-3-1: Deep spatiotemporal features F i+1′ Upsample to restore the spatial scale: CovTransBlock i Indicates the upsampling corresponding to the i-th scale, including deconvolution, BN, and activation function. Represents the deep spatiotemporal features F i+1′ The upsampling result of Step 2-3-2: Temporal shallow spatial features F in the encoder i Taking the mean over the time dimension: F i Taking the mean in the time dimension, the shape changes from h i × i ×c i ×t becomes h i × i ×c i ; Step 2-3-3: Upsampling results of deep spatiotemporal features The average result of shallow spatial features of time series in the time dimension To make a jump connection: Connection refers to the concatenation of two arrays in the feature dimension to reduce the information loss caused by the scale transformation process and retain valid information. At this time, h i i =h i 、w i ′ =w i , It is the feature after skip connection; Step 2-3-4: InderF after skip connection i′ Change the number of features: F i′ =DeConvBlock i (InderF i′ ) DeConvBlock i It is the convolution block corresponding to the i-th scale, including convolution, BN, and activation function; is the output of the i-th scale upsampling module; after stacking multiple layers of upsampling modules, the spatial scale is gradually restored, and finally the prediction result of the neural network model is obtained c1 is equal to the number of crop classification categories nu.

4. The method for crop classification of time-series SAR based on lightweight linear attention as claimed in claim 1, characterized in that: The activation function is ReLU.

5. The method for crop classification of time-series SAR based on lightweight linear attention as claimed in claim 1, characterized in that: The loss function is cross entropy.

6. The method for temporal SAR crop classification based on lightweight linear attention as claimed in claim 1, characterized in that: The optimizer is Adam.

7. The method for crop classification of time-series SAR based on lightweight linear attention as claimed in claim 2, characterized in that: ∈=10 -8 .

Images