Rice identification method based on optical and SAR image fusion

Abstract

The invention relates to a rice identification method based on optical and SAR image fusion, and belongs to the technical field of remote sensing image processing. The method comprises the following steps: synchronously acquiring SAR and optical time sequence data, respectively preprocessing the SAR and optical time sequence data, and extracting characteristic parameters; the method comprises the following steps: constructing a double-flow Vision Transform model, wherein the double-flow Vision Transform model comprises a double-branch feature extraction module, a space-time Transform aggregation module, a gating feature fusion module and a decoder; the double-branch feature extraction module extracts SAR and optical features according to the SAR and optical feature parameters; inputting the extracted SAR features and optical features into a space-time Transform aggregation module, and carrying out time sequence dependence modeling through a self-attention mechanism; the gating feature fusion module performs weighted fusion on the optical and SAR features; and the decoder generates a rice distribution probability graph according to a weighted fusion result. According to the method, the rice identification precision under the multi-temporal and multi-environment conditions can be improved.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image processing technology, and relates to a method for identifying rice based on the fusion of optical images and SAR images. Background Technology

[0002] As a major global food crop, the accurate and efficient monitoring of rice planting area is crucial for food security. Remote sensing technology, due to its wide coverage and periodic observation capabilities, has become the primary means of large-scale rice identification. Currently, rice identification methods based on remote sensing imagery are mainly divided into three categories: methods based on optical imagery, methods based on synthetic aperture radar (SAR) data, and methods based on the fusion of optical and SAR data.

[0003] Optical images provide rich spectral information and texture features, and have good characterization capabilities for different types of crops. However, methods based on optical images also have certain limitations, mainly reflected in the following aspects: 1) Due to the observation characteristics of the optical band itself, data acquisition depends on daytime lighting conditions, making it impossible to achieve 24-hour monitoring; 2) They are greatly affected by weather. Under cloudy, rainy, foggy, aerosol, and dusty weather conditions, image quality deteriorates significantly, and data loss is severe; 3) Optical data is sensitive to the occlusion and projection of ground objects, resulting in low accuracy when classifying complex ground objects and backgrounds; 4) Due to the limitations of payload design conditions, it is difficult to balance the spatiotemporal resolution of optical data. Long-term resolution data can accurately depict the crop growth process, but insufficient spatial resolution makes it impossible to characterize the details of crop distribution in space. While high spatial resolution data can depict more detailed information, the revisit period is long, making it easy to miss the monitoring of key growth periods; 5) Some crops have similar spectral characteristics in the optical band, which can easily cause confusion and affect the accuracy of classification. Therefore, although optical remote sensing data has made some progress in rice identification, there are still problems such as insufficient spatial resolution, susceptibility to weather, lack of dynamic monitoring capabilities, classification accuracy affected by vegetation index, and difficulty in obtaining samples. These problems limit its widespread application in the field of rice identification and monitoring.

[0004] Compared to traditional real-time radar systems, SAR can generate high-resolution radar images for imaging and monitoring of ground targets. It features high resolution, strong penetration, and all-weather monitoring capabilities, and has been widely used in fields such as geological and topographical exploration, environmental monitoring, agriculture, urban planning, and disaster monitoring, becoming an important data source for rice remote sensing monitoring.

[0005] However, rice identification based on SAR data also faces many problems. First, speckle noise, which is common in SAR data, reduces the clarity of the image and affects the classification accuracy, making data processing and analysis more complicated. Second, SAR data requires complex preprocessing steps, such as speckle noise removal, radiometric correction and geometric correction, which increases the processing difficulty and time cost. Third, the backscattering characteristics of different land cover in SAR images are relatively small, especially in vegetation-covered areas, making it difficult to distinguish between different crop types and growth stages. Rice identification based on SAR data using machine learning / deep learning methods has received increasing attention, but this method requires a large number of high-quality training samples. However, obtaining these samples in a wide range and diverse environment is both difficult and time-consuming

[18] . In terms of time series analysis, processing a large amount of time series data requires high storage and computing resources, which is also a major challenge. Effective classification results also require verification and correction using ground ground data, but obtaining these data in practice is often challenging. In addition, the applicability of SAR data in different regions and different crop types may vary, and some models and methods may not be applicable in specific regions, limiting their widespread promotion. Therefore, these issues collectively limit the application of SAR data-based rice identification in large-scale and diverse environments. Crop classification methods that fuse optical and SAR data can simultaneously utilize the spatiotemporal information of optical data and the physical characteristics of SAR data, combining their respective advantages to achieve dynamic monitoring and change analysis of rice growth stages, thereby improving the accuracy and robustness of rice identification. Furthermore, the application of novel data fusion algorithms and models has brought higher accuracy and efficiency to rice identification technology, promoting the further development of rice remote sensing monitoring technology and providing more reliable technical support for agricultural production and food security. However, fusion identification methods also have some shortcomings: First, the mismatch and inconsistency between optical and SAR data during acquisition, preprocessing, and fusion may affect the final identification effect, requiring further research into data fusion methods to improve data consistency and quality; second, the algorithms and models for optical and SAR data fusion need more refined optimization and adjustment to fully leverage the advantages of both types of data while overcoming their limitations, such as the influence of weather conditions and vegetation cover on optical data, and the interference of speckle noise on SAR data; in addition, the processing methods for optical and SAR data fusion need to consider the differences in different regions and environments to adapt to different rice planting conditions and growth conditions, which requires further in-depth research and verification.

[0006] In summary, existing technologies have not yet been able to achieve a high-precision, robust rice identification scheme with low dependence on labeled samples in tropical cloudy and rainy regions where optical data is severely lacking. Summary of the Invention

[0007] In view of this, the purpose of this invention is to provide a rice identification method based on optical and SAR image fusion, which can achieve high-precision and high-robust rice identification in tropical cloudy and rainy areas.

[0008] To achieve the above objectives, the present invention provides the following technical solution: A rice identification method based on optical and SAR image fusion, comprising: Simultaneously acquire SAR time-series data and optical time-series data, and extract feature parameters after preprocessing them respectively; A dual-stream Vision Transformer model is constructed, which includes a dual-branch feature extraction module, a spatiotemporal Transformer aggregation module, a gated feature fusion module, and a decoder. The extracted SAR and optical feature parameters are input into the dual-branch feature extraction module to extract SAR and optical features respectively; the extracted SAR and optical features are input into the spatiotemporal Transformer aggregation module to perform temporal dependency modeling through a self-attention mechanism; the gated feature fusion module performs weighted fusion of optical and SAR features; the decoder generates a rice distribution probability map based on the weighted fusion result and outputs a binary rice field segmentation result.

[0009] Furthermore, the acquired SAR time-series data are ground distance detection time-series data collected by the Sentinel-1 satellite in interferometric wide-swath mode and dual-polarized C-band, and the acquired optical time-series data are Sentinel-2 data with cloud cover of less than 5% during the same period.

[0010] Furthermore, the preprocessing of SAR time series data includes radiometric calibration and terrain correction of SAR time series data based on the GEE platform, and then extracting the VV polarization and VH polarization backscattering coefficients. Preprocessing of optical time-series data includes cloud masking and then calculating three indices: NDVI, NDWI, and NIR.

[0011] Furthermore, the dual-branch feature extraction module includes a SAR branch for extracting SAR features and an optical branch for extracting optical features; wherein, the input of the SAR branch is the VV polarization and VH polarization backscattering coefficients; and the input of the optical branch is the indices NDVI, NDWI and NIR.

[0012] Furthermore, the spatiotemporal Transformer aggregation module includes a multi-head self-attention layer, a feedforward network, a position encoder, and a residual normalization structure. The position encoder converts the encoder output at each time step into a token sequence and adds temporal and spatial position encoding. The token sequence is input into the multi-head self-attention layer, which models temporal dependencies through a self-attention mechanism to capture long-range dependencies between different time steps and finally outputs a feature sequence. The output of the multi-head self-attention layer is input into the feedforward network after residual connection and layer normalization for higher-dimensional nonlinear transformation and mapping to extract deeper semantic information. The output of the feedforward network is then processed through residual connection and layer normalization to obtain SAR temporal features and optical temporal features.

[0013] Furthermore, the gated feature fusion module includes a feature compression unit, a weight prediction unit, and a weighted fusion unit. The feature compression unit compresses the input optical time-series features and SAR time-series features to a unified dimension through 1×1 convolution. The weight prediction unit calculates the fusion weights based on the compressed optical time-series features and SAR time-series features. The weighted fusion unit performs weighted fusion on the compressed optical time-series features and SAR time-series features according to the calculated fusion weights.

[0014] Furthermore, the method also includes training the constructed two-stream Vision Transformer model using a loss function, which is expressed as:

[0015] In the formula, For the sample size, For the true category distribution, This represents the Softmax probability output by the model.

[0016] The beneficial effects of this invention are as follows: This invention designs a spatiotemporally aware multimodal fusion network architecture, effectively solving the heterogeneity problem between optical and SAR data in terms of spatiotemporal resolution and physical mechanisms. Addressing the problem that traditional fusion methods neglect the mismatch between the two data sources in the spatiotemporal dimension, directly leading to a decrease in fusion performance, this invention proposes a dual-stream temporal Transformer network. This network uses a spatiotemporal alignment module to handle the temporal inconsistency caused by different revisit periods and employs an improved dynamic time warping algorithm to establish an elastic mapping relationship between optical and SAR temporal sequences. At the feature fusion level, this invention introduces a cross-attention mechanism to achieve bidirectional information interaction. When optical features are used as query vectors, complementary structural information can be extracted from SAR features; conversely, when SAR features are used as query vectors, spectral context information can be obtained from optical features. This bidirectional interaction mechanism fully leverages the synergistic effect between the spectral sensitivity of optical data and the structural awareness capability of SAR data, significantly improving the recognition accuracy under complex planting structures, especially demonstrating excellent performance in distinguishing complex planting patterns such as double-cropping rice and triple-cropping rice.

[0017] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0018] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 This is a schematic diagram of a rice identification method based on optical and SAR image fusion according to an embodiment of the present invention; Figure 2 Preprocessing procedures for Sentinel-1 and Sentinel-2 time series data; Figure 3 This is a schematic diagram of the dual-stream Vision Transformer model structure. Detailed Implementation

[0019] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0020] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0021] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0022] An embodiment of this invention provides a rice identification method based on optical and SAR image fusion. This method mainly includes: acquiring Sentinel-1 SAR time-series data and Sentinel-2 optical time-series data simultaneously from GEE using multi-source data collaborative acquisition technology to construct a complete crop growth cycle observation sequence; employing an intelligent data preprocessing workflow to perform radiometric and geometric correction on the original remote sensing data, and identifying invalid pixels in the optical data based on an advanced cloud and fog detection algorithm, while reconstructing the optical features of missing time points using spatiotemporal interpolation technology to ensure the continuity and integrity of the time-series data; in the multimodal feature extraction stage, calculating vegetation index features such as NDVI, NDWI, and EVI from the preprocessed optical data, and extracting physical structure parameters such as backscattering coefficient and polarization decomposition features from the SAR data to form a complementary feature expression system; improving the dual-stream fusion network architecture of Vision Transformer, constructing SAR time-series feature extraction branches and optical time-series feature extraction branches respectively, and utilizing a cross-attention mechanism to achieve deep interaction and fusion of the two modal features, fully exploring the intrinsic correlation between the spectral temporal variation law of optical data and the structural response characteristics of SAR data; finally, combining Focal... The Loss function effectively solves the problem of extreme class imbalance in rice identification. It outputs a high-precision spatial distribution map of rice through an end-to-end deep learning framework, providing reliable technical support for monitoring rice planting area in tropical cloudy and rainy areas.

[0023] Please see Figures 1 to 3 The rice identification method provided in this embodiment is as follows: 1. Synchronously acquire Sentinel-1 SAR time-series data and Sentinel-2 optical time-series data from GEE and extract feature parameters, such as... Figure 2 As shown.

[0024] Specifically, ground range detection (GRD) time-series data were acquired by the Sentinel-1 satellite in interferometric wide swath (IW) mode and dual-polarized (VH and VV) C-band. This time-series data spans from January to August 2022, covering the main rice growing season, a total of eight months. Then, based on the Google Earth Engine (GEE) platform, the collected time-series dual-polarized SAR imagery was preprocessed, primarily including radiometric calibration and terrain correction. Simultaneously, the monthly SAR data averages were calculated, ultimately yielding the monthly average of the dual-polarized backscattering coefficients during the rice growing season, resulting in 16 bands.

[0025] Sentinel-2 data with cloud cover less than 5% during the same period were collected using the GEE platform. Cloud masking was applied to all images using the GEE platform, and the optical indices of rice were calculated. Finally, the median values ​​of the three optical indices during the rice growth period were calculated to obtain the results for NDVI, NDWI, and NIR, resulting in a total of three bands.

[0026] The NDVI indicator is calculated using the following formula:

[0027] In the formula, This indicates the reflectance value of the red band (B4 band of Sentinel-2).

[0028] The NDWI index is calculated using the following formula:

[0029] In the formula, This indicates the reflectance value of the green band (B3 band of Sentinel-2).

[0030] The backscattering coefficients of VH and VV dual-polarization radar can reflect structural features such as surface roughness and water content in paddy fields. The three optical index bands NDVI, NIR, and NDWI can effectively characterize vegetation growth and water accumulation. The two types of data have similar spatial resolution, making them suitable for medium- to high-precision crop identification tasks.

[0031] 2. Construct a dual-stream Vision Transformer model to extract features from optical and SAR data. Because optical and SAR data differ significantly in physical properties and information representation, direct channel superposition can easily lead to problems such as feature interference and difficulty in model convergence. Therefore, unlike the traditional direct superposition of six channels (RGBN + VV + VH), this invention avoids mutual interference between multi-source data in the early stages of feature extraction. Instead, it constructs SAR feature branches and optical feature branches for independent feature learning. This strategy can reduce feature coupling in the early extraction stage of the model and improve the relevance and stability of feature representation.

[0032] This embodiment proposes using a dual-stream temporal Vision Transformer to replace ConvGRU for multi-source remote sensing rice identification. In the dual-stream temporal Vision Transformer, SAR / optical spatial features are first extracted independently by a dual-branch U-Net encoder. The temporal and modal relationships are then modeled by the dual-stream Transformer encoder, and deep modal complementarity is achieved through bidirectional cross-attention. Subsequently, a pixel-level gating mechanism adaptively fuses the features and feeds them into the U-Net decoder to output a binary segmentation map of the rice paddy. The dual-stream temporal Vision Transformer takes into account both spatial texture and long-range temporal dependencies, which can improve the identification accuracy and generalization ability in multi-temporal and multi-source environments.

[0033] like Figure 2 As shown, the dual-stream Vision Transformer model provided in this embodiment includes a dual-branch feature extraction module, a spatiotemporal Transformer aggregation module, a gated feature fusion module (Fuse Gate), and a decoder.

[0034] (1) Dual-branch feature extraction module This module establishes independent encoders for Sentinel-1 and Sentinel-2 data respectively. Each branch uses an improved U-Net structure as the feature extraction network, and utilizes multi-scale convolutional blocks to extract spatial semantic features at different scales.

[0035] The SAR branch takes VV and VH polarimetric radar backscattering coefficients as inputs to capture paddy field structure and water content characteristics; the optical branch takes NDVI, NIR, and NDWI spectral index channels as inputs to learn vegetation growth and water accumulation characteristics.

[0036] Each branch consists of the following sub-units: ① Multi-scale Convolution Block, which contains parallel convolutional layers with 3×3 and 5×5 convolutional kernels, followed by BatchNorm and ReLU.

[0037] Multi-scale convolutional coding units are used to capture spatial textures and scattering / spectral features at different scales, as shown below:

[0038] in, Indicates the first The input feature map of the layer, Indicates the first The feature map output by the layer is the result after multi-scale convolution processing.

[0039] ② Down-sampling Block: Downsampling is achieved using convolutions with a stride of 2.

[0040] Downsampling units can increase the receptive field while preserving the spatial structure.

[0041] ③ Skip Connections are used to preserve shallow spatial details and assist subsequent decoders in reconstructing the boundaries of the paddy fields.

[0042] The data processing flow of the dual-branch feature extraction module is as follows: The input parameters are then fed into each branch accordingly: Local spatial semantics are extracted using multi-scale convolutional blocks: Deep features at different scales are formed through layer-by-layer downsampling: SAR feature sequences and optical feature sequences are formed.

[0043] (2) Spatiotemporal Transformer Aggregation Module To overcome the limitations of traditional ConvGRU in long sequence modeling, this embodiment employs a pure Transformer structure for global dependency modeling in the temporal dimension. This module consists of a multi-head self-attention layer, a feed-forward network, a positional encoder, and a residual normalization structure. Its processing is as follows: ① Tokenization and Location Encoding For each time step, the output of the dual-branch feature extraction module is patched or pixel-embedded to convert the two-dimensional feature map into a token sequence. → Furthermore, it incorporates temporal and spatial location encoding, enabling the Transformer to simultaneously perceive both temporal progression and spatial location information. This is the transformed token sequence (embedded vector sequence). The number of tokens (usually determined by H*W or the number of blocks). This refers to the embedding dimension or hidden layer dimension of the Transformer.

[0044] ② Multi-head self-attention layer captures long-range dependencies between different time steps:

[0045] in, , .

[0046] ③ Output feature sequence It also includes: the growth curve characteristics of paddy fields over time, the structural differences of SAR / optics in the time dimension, and long-range dependency information (such as the influence of heading period on maturity classification).

[0047] Furthermore, to enhance the model's nonlinear expressive power and ensure the training stability of deep networks, the feature sequences are sequentially processed through a residual normalization structure and a feedforward network. The specific processing flow is as follows: First, residual connections and layer normalization are used to process the output of the multi-head self-attention layer. Specifically, the input and output of the self-attention layer are added together to preserve the original information flow and prevent gradient vanishing. Then, layer normalization is applied to the added features to unify the feature distribution within a stable range, thereby accelerating model convergence. The process can be represented as follows:

[0048] in, For layer normalization operation, This is the output of the multi-head self-attention mechanism.

[0049] Next, the normalized features are fed into the Feed Forward Network (FFN). The feed forward network consists of two fully connected layers and a non-linear activation function (GELU), used to perform higher-dimensional non-linear transformations and mappings on the features, extracting deeper semantic information.

[0050] in, , This is the weight matrix of the fully connected layer. is the activation function for the Gaussian error linear unit.

[0051] Finally, the input and output of the feedforward network are fused and normalized again through a residual connection and layer normalization structure to obtain the final aggregated features for this time step:

[0052] This stacked structure of "attention mechanism - residual normalization - feedforward network - residual normalization" ensures that the model captures long-range temporal dependencies while possessing strong feature fitting capabilities and training robustness. Due to the adoption of a dual-stream network architecture, the aforementioned spatiotemporal Transformer aggregation module is applied separately and in parallel to the SAR branch feature sequences and the optical branch feature sequences. Therefore, the final output of this module contains two sets of features: the SAR temporal features after temporal aggregation (denoted as...). Optical temporal features after temporal aggregation (denoted as...) These two sets of features retain the global dependencies of their respective modalities in the time dimension and are mapped to the same feature dimension space. This dual-stream output format directly serves as the input to the subsequent gated feature fusion module, providing an aligned and complementary feature foundation for subsequent calculation of fusion weights and execution of weighted fusion operations.

[0053] (3) Gating feature fusion module (Fuse Gate) To fully leverage the complementarity of optical and radar data, a gated fusion module adaptively learns the importance weights of the two feature streams at the pixel level. During the fusion process, the model dynamically adjusts the contribution ratio of each source feature, allowing for greater reliance on SAR features under cloudy or shadowy conditions, while fully utilizing optical features during clear weather. This fusion method maintains model stability and robustness under complex observation conditions.

[0054] The gating feature fusion module consists of three sub-units: ① The feature compression unit (Channel Compression) uses 1×1 convolutions to compress features from two modalities to a uniform dimension, ensuring additivity and cross-modal alignment. The compressed temporal features are as follows: , .

[0055] ② The weight prediction unit (Gate Unit) consists of two MLP layers or 1×1 convolutions and calculates the fusion weights:

[0056] in, Soft weights for each pixel.

[0057] ③ Weighted Fusion Block, used to perform fusion operations:

[0058] in, The weights are the convolution kernel weights of the weight prediction unit.

[0059] The data processing flow of the gating feature fusion module is as follows: Input the SAR and optical temporal features output by the spatiotemporal Transformer aggregation module; The features of the two modalities are concatenated and input into the weight prediction unit; Calculate the weight map for each pixel ; Adaptive fusion based on weights; The final fused features are output for decoding and segmentation.

[0060] (4) Decoding and Output Layer The fused multi-source spatiotemporal features are upsampled and decoded to generate a rice distribution probability map that is consistent with the input image space, and then output as a single-channel binary paddy field segmentation result.

[0061] 3. During the model training phase, the Cross Entropy Loss function and Dice loss are used for joint optimization to measure the difference between the predicted results and the real samples, thereby promoting the model's rapid convergence and boundary detail optimization in binary segmentation tasks.

[0062] The loss function is expressed as:

[0063] in, For the sample size, For the true category distribution, This represents the Softmax probability output by the model.

[0064] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A rice identification method based on optical and SAR image fusion, characterized in that, The method includes: Simultaneously acquire SAR time-series data and optical time-series data, and extract feature parameters after preprocessing them respectively; A dual-stream Vision Transformer model is constructed, which includes a dual-branch feature extraction module, a spatiotemporal Transformer aggregation module, a gated feature fusion module, and a decoder. The extracted SAR and optical feature parameters are input into the dual-branch feature extraction module to extract SAR and optical features respectively; the extracted SAR and optical features are input into the spatiotemporal Transformer aggregation module to perform temporal dependency modeling through a self-attention mechanism; the gated feature fusion module performs weighted fusion of optical and SAR features; the decoder generates a rice distribution probability map based on the weighted fusion result and outputs a binary rice field segmentation result.

2. The method according to claim 1, characterized in that, The acquired SAR time series data are ground distance detection time series data collected by the Sentinel-1 satellite in interferometric wide swath mode and dual-polarized C-band, and the acquired optical time series data are Sentinel-2 data with cloud cover of less than 5% during the same period.

3. The method according to claim 2, characterized in that, Preprocessing of SAR time series data includes radiometric calibration and terrain correction of SAR time series data based on the GEE platform, and then extracting the VV polarization and VH polarization backscattering coefficients. Preprocessing of optical time-series data includes cloud masking and then calculating three indices: NDVI, NDWI, and NIR.

4. The method according to claim 1, characterized in that, The dual-branch feature extraction module includes a SAR branch for extracting SAR features and an optical branch for extracting optical features; the inputs of the SAR branch are the VV polarization and VH polarization backscattering coefficients; the inputs of the optical branch are the indices NDVI, NDWI and NIR.

5. The method according to claim 1, characterized in that, The spatiotemporal Transformer aggregation module includes a multi-head self-attention layer, a feedforward network, a position encoder, and a residual normalization structure. The position encoder converts the encoder output at each time step into a token sequence and adds temporal and spatial position encoding. The token sequence is input into the multi-head self-attention layer, which models temporal dependencies through a self-attention mechanism to capture long-range dependencies between different time steps and finally outputs a feature sequence. The output of the multi-head self-attention layer is fed into the feedforward network through residual connections and layer normalization for higher-dimensional nonlinear transformations and mappings to extract deeper semantic information. The output of the feedforward network is then processed through residual connections and layer normalization to obtain SAR temporal features and optical temporal features.

6. The method according to claim 1, characterized in that, The gated feature fusion module includes a feature compression unit, a weight prediction unit, and a weighted fusion unit. The feature compression unit compresses the input optical temporal features and SAR temporal features to a unified dimension through 1×1 convolution. The weight prediction unit calculates the fusion weights based on the compressed optical temporal features and SAR temporal features. The weighted fusion unit performs weighted fusion of compressed optical time-series features and SAR time-series features based on the calculated fusion weights.

7. The method according to claim 1, characterized in that, The method also includes training the constructed two-stream Vision Transformer model using a loss function, which is expressed as: In the formula, For the sample size, For the true category distribution, This represents the Softmax probability output by the model.

Images