A cross-scale spatio-temporal fusion feature classification method based on a double-branch architecture

Abstract

The application belongs to the technical field of artificial intelligence and satellite remote sensing, and particularly relates to a cross-scale spatio-temporal fusion ground feature classification method based on a double-branch architecture, which comprises the following steps: preprocessing a remote sensing image; obtaining high-resolution optical images and multi-period medium-resolution time-series images, generating time-series data and labeling through processing, and cutting a data set; constructing a double-branch network, wherein the network comprises a spatial and temporal feature extraction branch and a fusion and decoding module; training and optimizing a model by using a weight optimizer and a loss function with dynamically adjusted class weights; post-processing a classification result, and outputting vector data. The method realizes the fusion of heterogeneous information, solves problems such as the same object with different spectra, improves the feature utilization rate and the recognition accuracy of a minority class, is convenient to design in an end-to-end mode, has strong generalization ability, and is fast in reasoning.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence and satellite remote sensing, and particularly relates to a cross-scale spatio-temporal fusion ground feature classification method based on a double-branch architecture. BACKGROUND

[0002] Remote sensing image ground feature classification is a core link of application of satellite remote sensing technology, which realizes division of land use / cover types by extracting and identifying features of different ground features in remote sensing images, and provides data support for agricultural production regulation (such as crop planting area statistics, pest and disease monitoring), natural resource management (such as forest range definition, water resource distribution evaluation) and the like. With the development of remote sensing sensor technology, the acquisition cost of multi-source remote sensing data (covering image data of different resolutions, different time dimensions and different wave bands) is significantly reduced, and how to efficiently fuse spatio-temporal information in multi-source data and break through the precision bottleneck of traditional classification methods has become a research hotspot in the field.

[0003] Existing ground feature classification technologies are mainly divided into two categories, but they all have technical defects that are difficult to avoid: one category is a single-source data classification method, which depends on image data obtained by a single sensor, and realizes classification by extracting spectral features or simple spatial features, typical schemes including traditional machine learning algorithms and deep learning algorithms based on single-period image:

[0004] Traditional machine learning algorithm: such as the semi-supervised interval two fuzzy C means algorithm (SS-AIT2FCM) disclosed in the patent with the publication number CN108830297, which is aimed at multi-spectral remote sensing image, initializes clustering centroids by labeled samples, constructs membership interval by fuzzy distance measurement, and dynamically adjusts membership function by combining adaptive factor to improve the robustness of large-scale image classification. However, this kind of method only relies on spectral reflectance features, and cannot describe the spatial texture (such as the tree canopy layer structure of forest land and the regular texture of film of greenhouse) and context association relationship of ground features, and in the scenes of "same object different spectrum" (such as different seasonal farmland showing different spectral features due to different crop growth states) and "different objects same spectrum" (such as water body and shadow showing highly similar spectral features in visible light wave band), the classification accuracy is significantly reduced.

[0005] Single-period image deep learning algorithms, such as the improved DeepLab network proposed in patent CN109255334, enhance the input by fusing multi-scale texture features (two-dimensional entropy, roughness) and structural features (Canny edges), and expand the network's receptive field by combining porous convolution to improve the classification accuracy of high-resolution images. Although this type of method can extract the spatial geometric details of ground features, it lacks the utilization of temporal dynamic information (such as the NDVI index changes of crop growth cycle and the seasonal fluctuations of greenhouse vegetation cover), and it is difficult to distinguish ground features with similar spatial morphology but significantly different phenological characteristics (such as paddy fields and natural ponds during irrigation season, both of which show a "sheet-like water body" morphology in single-period high-resolution images, but the NDVI temporal curve of paddy fields fluctuates significantly with crop planting / harvesting, while the NDVI temporal curve of ponds tends to be stable).

[0006] In addition, some single-source data classification methods attempt to incorporate temporal features, such as the Liaohe River Estuary land cover classification method disclosed in patent CN118155069. This method is based on multispectral temporal images from the Sentinel-2 sensor, extracting multi-temporal temporal features and single-temporal spectral / texture / geometric features through object-oriented segmentation, and combining these with classifiers such as random forests to achieve land cover classification. However, this method is still limited to single-source temporal data, lacks sufficient mining of high-resolution spatial detail information, and its feature selection is strongly tied to a specific region (Liaohe River Estuary), exhibiting weak generalization ability for other scenarios with complex terrain and diverse land cover types.

[0007] Another type is multi-source data fusion classification methods. To compensate for the information limitations of single-source data, existing technologies are gradually shifting towards multi-source data fusion, but two core shortcomings still exist:

[0008] Limited Data Resolution in Data Fusion: Existing solutions often focus on fusing low- to medium-resolution data (such as the dual-channel convolutional ladder network disclosed in patent CN107273938, which fuses Landsat-8 and Sentinel-2 medium-resolution data). These methods extract features from each source data using a shared-parameter encoder and optimize the model using cross-entropy and reconstruction loss. However, these methods can only capture macroscopic phenological patterns and cannot utilize the microscopic geometric details in high-resolution images (such as the granular texture of orchards, the canopy texture of woodlands, and the distribution of field paths). This results in significant errors in classification when identifying fine land features (such as distinguishing between woodlands and orchards).

[0009] The fusion and classification tasks are disconnected: some methods only stay at the feature extraction stage and do not build a decoding mechanism that is deeply bound to the classification task. For example, the multi-source remote sensing time series data fusion feature extraction method disclosed in patent CN115527122 fuses the spatial features of optical and SAR time series data and generates spatiotemporal fusion features through a shared time encoder, but does not directly apply the features to land cover classification. It requires the design of an additional classification model to process the fused features, resulting in low feature utilization and difficulty in end-to-end optimization.

[0010] Meanwhile, existing multi-source fusion methods generally fail to solve two key practical problems: first, the problem of cross-resolution feature matching (e.g., the resolution difference between 10-meter time-series data and 0.75-meter spatial data is as high as 13 times, and direct stitching can easily lead to feature misalignment); second, the problem of imbalanced samples (e.g., greenhouses and other land features usually account for less than 5% in images, and traditional loss functions can easily cause the model to favor predicting land feature categories with a high proportion, resulting in extremely low recognition accuracy for a few land feature categories).

[0011] In summary, existing land cover classification technologies have significant shortcomings in terms of spatial detail, collaborative utilization of temporal dynamic information, cross-scale feature matching, deep coupling of classification tasks and fusion mechanisms, and imbalanced sample adaptation. There is an urgent need for a land cover classification method that can overcome these bottlenecks. Summary of the Invention

[0012] The purpose of this invention is to provide a cross-scale spatiotemporal fusion land cover classification method based on a dual-branch architecture, in order to solve the problems of single information dimension, mismatch of cross-scale features, and disconnect between fusion and classification tasks in the prior art mentioned in the background.

[0013] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0014] A cross-scale spatiotemporal fusion land cover classification method based on a dual-branch architecture includes the following steps: Step S1, remote sensing image preprocessing: acquire single-period high-resolution optical images and multi-period medium-resolution optical images with a preset duration before the reference time phase, achieve spatiotemporal alignment through cloud removal, gap filling and vegetation index extraction, and generate continuous time-series vegetation index data, generate pixel-level labels based on high-resolution image annotation of land cover, and divide the two types of images into geographically corresponding spatial slices and time-series slices, and divide them into training set, validation set and test set;

[0015] Step S2: Construct a dual-branch cross-scale spatiotemporal fusion network: The network includes a spatial feature extraction branch, a temporal feature extraction branch, a cross-level spatiotemporal fusion module, and a progressive spatiotemporal decoder;

[0016] Step S3, Model Training and Optimization: An optimizer with weight decay is used to validate the model using classification evaluation metrics; the model is optimized using a cross-entropy loss function with class weights, and the class weights are dynamically adjusted based on the proportion of each class sample in the training set.

[0017] Step S4, post-processing of classification results: Morphological optimization is performed on the pixel-level land cover classification probability map to eliminate noise and holes; the processed classification results are stitched together into a complete image and converted into vector data; boundary smoothing is performed on the vector data to output vectorized land cover classification results.

[0018] According to the above technical solution, in step S1, the resolution of the high-resolution optical image is 0.5-1 meter, and the resolution of the medium-resolution optical image is 5-15 meters; the preset duration is 10-14 months, the vegetation index is the Normalized Difference Vegetation Index (NDVI), and the number of periods for the continuous time series vegetation index data is 10-14 periods.

[0019] According to the above technical solution, in step S2, the spatial feature extraction branch specifically involves: using a lightweight convolutional network as the backbone network, inputting spatial slices, and extracting multi-level spatial features from shallow textures to deep semantics through multi-stage downsampling;

[0020] The temporal feature extraction branch specifically employs a bidirectional temporal convolutional network as its core, inputs temporal slices, captures dynamic features of ground cover and phenology, and generates temporal features that match the multi-level feature scale of the spatial branch through progressive upsampling.

[0021] The cross-level spatiotemporal fusion module specifically employs a hierarchical misalignment fusion strategy, which performs channel splicing and feature compression on the features of different stages in the spatial branch and the corresponding adaptive stage features in the temporal branch, and introduces residual connections to output multi-level fused features.

[0022] The progressive spatiotemporal decoder specifically involves upsampling and fusing multi-level fused features at each level, and outputting a pixel-level land cover classification probability map through a classification head.

[0023] According to the above technical solution, the lightweight convolutional network is either MobileNetV3 or EfficientNet-Lite; the number of stages in the multi-stage downsampling is 3-5, and the downsampling step size is 2; the number of channels of each level of spatial features increases with the downsampling stage.

[0024] According to the above technical solution, in step S2, the bidirectional temporal convolutional network is either bidirectional ConvLSTM or bidirectional gated recurrent unit Bi-GRU; the number of progressive upsampling stages is 3-5, and after each upsampling stage, the features are refined by 3×3 convolution, and the difference between the number of channels of the temporal features and the number of channels of the corresponding stage spatial features is less than or equal to 20%.

[0025] According to the above technical solution, in step S2, the specific correspondence of the hierarchical misalignment fusion strategy is as follows: the feature of the k-th stage of the spatial branch is fused with the feature of the (M+1−k)-th stage of the temporal branch, where M is the total number of downsampling stages in the spatial branch; feature compression is achieved through 1×1 convolution, and the number of feature channels after compression is consistent with the number of feature channels in the corresponding stage of the spatial branch.

[0026] According to the above technical solution, in step S2, the progressive upsampling is achieved by any one of transposed convolution or bilinear interpolation; after each upsampling, the decoded features and the corresponding stage fusion features are concatenated by channel, and the number of channels of the concatenated features is compressed by 40%-60% by 1×1 convolution; the classification head outputs channel features consistent with the number of land cover categories by 1×1 convolution, and generates a pixel-level land cover classification probability map by Softmax and Argmax operations.

[0027] According to the above technical solution, the dimensions of channel splicing are: Where H and W are the height and width of the feature map, This represents the number of channels in the spatial feature map. The number of channels in the temporal feature map; the convolution is a 1×1 convolution, compressing the number of channels after concatenation to [number missing]. .

[0028] According to the above technical solution, in step S3, the optimizer with weight decay is either AdamW or SGD with weight decay; the classification evaluation index includes at least one of Intersection over Union (IoU) and F1 score; in the cross-entropy loss function with class weights, the class weights are calculated by normalizing the total number of samples by the number of samples in the corresponding class.

[0029] According to the above technical solution, in step S4, morphological optimization adopts morphological closing operation, and the convolution kernel size of the closing operation is 3×3 to 7×7 to eliminate noise patches with an area of ​​less than 5 to 15 pixels; boundary smoothing processing adopts Douglas-Peucker algorithm, and the algorithm simplification threshold is 0.5-2 meters. Boundary smoothing is achieved by retaining vertices that deviate from the fitted straight line by more than the threshold.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] This invention is the first to achieve heterogeneous fusion of high-resolution static spatial information and medium-resolution dynamic temporal information. It extracts the microscopic geometric details of ground objects through spatial branching and captures macroscopic phenological dynamics through temporal branching, effectively solving the problems of different spectra for the same object and the same spectra for different objects.

[0032] This invention also achieves precise alignment of 1024×1024 spatial data with 78×78 temporal data through progressive upsampling and dynamic resolution matching mechanism of temporal branch, avoiding feature distortion caused by direct scaling. The misaligned fusion strategy of spatial high-level and temporal low-level enables complementary enhancement of features at different abstraction levels (deep semantics and shallow dynamics), and the utilization rate of fused features is greatly improved compared with traditional splicing fusion.

[0033] In this invention, the cross-entropy loss function with inverse class frequency weights significantly improves the IoU of classification for features such as greenhouses compared to traditional loss functions. This significantly reduces the problems of missed and false detections of minority class features, meeting practical needs such as greenhouse area statistics in agricultural monitoring and small-area feature identification in resource management. Furthermore, this invention features an end-to-end design with strong practicality and generalization ability. The entire process chain, from multi-source data preprocessing to vector result output, eliminates the need for additional classification models to process fusion features, directly outputting vector results usable in GIS systems. Its ease of operation is significantly superior to existing segmented methods. Simultaneously, through lightweight networks and adaptive optimization strategies, the model achieves an inference speed of 8.42 frames per second on an NVIDIA H800 GPU, and the classification accuracy fluctuation is less than 5% across different terrain regions (plains, hills, and mountains), demonstrating strong generalization ability. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the dual-branch network model structure of the present invention. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] Example 1

[0037] This invention provides a cross-scale spatiotemporal fusion land cover classification method based on a dual-branch architecture. The core idea is to decouple and synergistically utilize the spatial features of high-resolution imagery and the temporal features of medium-resolution temporal imagery (e.g., through a complete process design of "preprocessing, dual-branch fusion network, model training, and post-processing") by designing a whole-process system. Figure 1 As shown in the figure, the deep fusion and accurate classification of cross-scale spatiotemporal information are achieved. The specific technical solution is as follows:

[0038] Step 1: Remote sensing image preprocessing.

[0039] The core of remote sensing image preprocessing is to transform multi-source remote sensing data into structured inputs that can be directly processed by neural networks, ensuring that the spatial details of high-resolution images correspond precisely to the phenological dynamics information of medium-resolution time-series images. The process begins with data acquisition and spatiotemporal alignment. Specifically, a single-period high-resolution optical image is selected as the reference time-phase data. The preferred high-resolution optical image is an RGB three-band image with a resolution of 0.5-1 meter (such as a 0.75-meter resolution image from the Jilin-1 satellite). The acquisition time is recorded as the reference time-phase. At the same time, multiple periods of medium-resolution optical images are selected as time-series data. The preferred medium-resolution optical images are multi-band images with a resolution of 8-12 meters (such as a 10-meter resolution image from the Sentinel-2 satellite). Monthly composite data from 10-14 months (preferably 12 months) before the reference time-phase are selected, and the red band and near-infrared band are extracted. The time-series data was preprocessed on the Google Earth Engine (GEE) platform. A cloud-covered area was removed using a cloud probability-based declouding algorithm. Data gaps were filled using linear interpolation. The Normalized Difference Vegetation Index (NDVI) was calculated based on red and near-infrared bands, resulting in 12 consecutive NDVI time-series data periods. This data provides support for capturing dynamic changes in land cover and phenology, such as the "sowing, growth, and harvesting" cycle of crops in cultivated land and the seasonal adjustment of greenhouse vegetation cover. At the same time, geographic coordinate matching (based on the latitude and longitude information of the image) and time benchmark alignment (with the acquisition time of high-resolution imagery as the core, anchoring the time range of the time-series data) ensured that the high-resolution imagery and the NDVI time-series data corresponded one-to-one in terms of geographic range and time dimension.

[0040] After data acquisition and spatiotemporal alignment are completed, the sample labeling and tile segmentation stage begins. Based on the high-resolution optical image, land features are manually labeled according to requirements, with at least two types of land features labeled (preferably six types, including cultivated land, forest land, orchard, water body, greenhouse, and others). Pixel-level labels are generated that are completely consistent with the size of the high-resolution image (each pixel corresponds to a unique land feature category). Subsequently, tiles are segmented according to the neural network input requirements, dividing the high-resolution image into 1024×1024 pixel spatial tiles (adapting to the input layer size of the subsequent spatial feature extraction branch). Simultaneously, the NDVI time series data is divided into slices of corresponding sizes according to spatial location. The 1024 slices with a resolution of 0.75 meters correspond to the time series slices with a resolution of 10 meters and a size of 76-78×76-78 pixels (preferably 78×78 pixels). Geographic coordinate mapping ensures that the spatial slices and time series slices correspond to the same geographical area. Finally, the slices and their corresponding labels are divided into training set (for model parameter learning), validation set (for adjusting hyperparameters such as learning rate and number of iterations) and test set (for evaluating the generalization ability of the model) in a ratio of 8:1:1.

[0041] Step 2: Construction of a dual-branch cross-scale spatiotemporal fusion network.

[0042] The dual-branch cross-scale spatiotemporal fusion network adopts a four-order architecture of "spatial branch - temporal branch - cross-level fusion module - progressive decoder" to achieve accurate matching and deep fusion of multi-scale features. The functions and collaborative mechanisms of each module are closely connected to jointly support the land cover classification task. Among them, the spatial feature extraction branch uses a lightweight convolutional neural network (preferably MobileNetV3) as the backbone network. It progressively extracts high-resolution spatial features of land cover (from shallow texture to deep semantics) through multi-stage deep separable convolution and attention mechanism. The input of this branch is a 1024×1024×3 RGB spatial slice (3 represents the three RGB bands) and sets 4 downsampling stages (stride=2 to achieve downsampling): Stage 1 outputs a 512×512×16 feature map, which consists of 3 layers of deep separable convolution. The first layer uses a 3×3 convolution (dilation rate 1) to extract the gradient features of land cover edges. The last two layers use 1×1 convolution to achieve channel dimensionality upscaling. The feature extraction process satisfies the formula:

[0043]

[0044] in, This represents the feature map output from the second stage of the spatial branch. The spatial coordinates of the feature map This represents a 1×1 convolution, used to adjust the channel dimension. This indicates a 3×3 depthwise separable convolution, which preserves spatial features while reducing computational cost; σ is the Swish activation function, and the output features include shallow spatial information such as the edges and textures of ground features; To input RGB images, The output includes edge gradients and texture features.

[0045] Stage 2: Output a 256×256×24 feature map. After a 3×3 depthwise convolution, a squeeze-encouragement (SE) attention module is introduced. Global average pooling compresses the features into channel-dimensional vectors. Two fully connected layers generate channel attention weights, and the weights are redistributed to different channel features to enhance the response of land cover features with significant texture features, such as woodlands and greenhouses. The feature mapping process satisfies the formula:

[0046]

[0047] in, The function represents element-wise multiplication, and its output focuses on the medium-scale structural information of ground features (such as tree canopy clusters and contiguous greenhouse distributions). SE(·) generates channel attention weights through global average pooling and two fully connected layers, focusing on capturing medium-scale structures. This represents the feature map output from the second stage of the spatial branch. These are the spatial coordinates of the feature map, representing the enhanced spatial feature representation. Represents the first-stage feature map of spatial branching. Perform a 3×3 convolution operation to extract local spatial context features (such as the edges of ground features and detailed information about textures).

[0048] Stage 3: Output a 128×128×40 feature map, using stacked residual blocks (containing two 3×3 convolutional layers). Skip connections (directly stacking the input to the output) alleviate the gradient vanishing problem in deep networks and enhance the transfer of semantic features. The feature generation process satisfies the formula:

[0049]

[0050] in, This represents the feature map output from the third stage of the spatial branch. The spatial coordinates of the feature map represent the spatial feature representation after residual enhancement. The output features contain semantic association information of land features (such as the adjacency relationship between farmland and field paths). This represents the feature map output from the second stage of the spatial branch. Represents the second-stage feature map of spatial branching. Perform a 3×3 convolution operation.

[0051] Stage 4: Output a 64×64×80 feature map. After depthwise convolution, a spatial attention module is introduced. A spatial weight map is generated through 7×7 global pooling. Feature enhancement is performed on key regions in the image (such as water body boundaries and greenhouse apex). The feature extraction process satisfies the formula:

[0052]

[0053] in, This represents the feature map output from the fourth stage of the spatial branch. The coordinates of the feature map are given by , and SA(·) is the spatial attention operator that outputs deep semantic features fused with the global context. This indicates element-wise multiplication.

[0054] The temporal feature extraction branch uses a bidirectional recurrent neural network (preferably a single-layer bidirectional ConvLSTM) as its core, combined with a progressive upsampling strategy to achieve temporal feature extraction and cross-scale matching. The input to this branch is a 12-period 78×78×1 NDVI time-series slice (1 represents a single NDVI band, and 12 represents the period). It captures the phenological dynamics of land cover through a bidirectional ConvLSTM. The forward branch (time sequence: t→t+1) captures the "growth and maturation" process of land cover (e.g., the NDVI increase-decrease trend of cultivated crops from seedling to harvest), while the reverse branch (time sequence: t←t+1) captures the "decline-initiation" process (e.g., the NDVI change from leaf fall to bud break in forest land). The cell state update of the ConvLSTM satisfies the formula:

[0055]

[0056] Where * represents a convolution operation. For element-wise multiplication, These are the input gate, forget gate, and output gate, respectively. For cell state and hidden layer output, forward branch output Reverse branch output , The input data is at time t. The hidden state at time t-1 The cell state at time t-1.

[0057] It is input To the input gate The weight, It is in a hidden state. To the input gate The weight, cellular state To the input gate The weight matrix, It is the bias of the input gate; For input To the Gate of Oblivion The weight matrix, Hidden state To the Gate of Oblivion The weight matrix cellular state To the Gate of Oblivion The weight matrix, Forgotten Gate The bias term, For input Weight matrix for candidate cell state information Hidden state Weight matrix for candidate cell state information This is a bias term for candidate cell state information. For input To the output gate The weight matrix, Hidden state To the output gate The weight matrix, cellular state To the output gate The weight matrix, For output gate The bias term, For activation function, The hyperbolic tangent activation function is used, and the resulting data is concatenated to form the initial temporal feature map. (Dimensions are 78×78×64, i.e., 32 channels × 2 directions) Among them, This is the fused feature map after splicing at time t. This is a feature splicing operation.

[0058] To address the resolution difference between the initial temporal feature map and the spatial branch feature map (512×512×16 to 64×64×80), a four-level progressive upsampling strategy is designed. Each upsampling level uses transposed convolution to improve resolution. After upsampling, features are refined using 3×3 convolution (reducing feature blurring caused by the checkerboard effect). The upsampling process satisfies dynamic resolution matching, i.e., the resolution of the feature maps at each stage of the spatial branch is statistically analyzed (stage 1: 512×512, stage 2: 256×256, stage 3: 128×128, stage 4: 64×64), and the upsampling ratio of the temporal features is dynamically adjusted to generate temporal feature maps that match the scale of each stage of the spatial branch in sequence: 64×64×80 (matching spatial stage 4), 128×128×40 (matching spatial stage 3), 256×256×24 (matching spatial stage 2), and 512×512×16 (matching spatial stage 1).

[0059] The cross-level spatiotemporal fusion module adopts a staggered fusion strategy of "high-level spatial layer - low-level temporal layer" to achieve complementary enhancement of features at different scales and abstract levels. The fusion hierarchy corresponds to the fusion of the feature map at stage k of the spatial branch (k=1,2,3,4) and the feature map at stage (5-k) of the temporal branch, forming four sets of cross-scale complementary feature pairs. The feature fusion operations are channel splicing, channel compression, and residual connection: Channel splicing performs channel dimension splicing on each complementary feature pair, resulting in a feature dimension of [missing information]. Where H and W are the height and width of the feature map, respectively. These represent the number of spatial / temporal feature channels, respectively, and the formula is:

[0060]

[0061] in, This is the fused feature map after cross-scale stitching; For feature splicing operations, The feature map for the k-th stage of the spatial branch. The feature map of the 5-k stage of the spatial branch, and Form cross-scale feature pairs.

[0062] Channel compression uses 1×1 convolution to compress the number of channels after concatenation to the original number of channels in the spatial features. To reduce computational load and fuse cross-modal information, the formula is:

[0063]

[0064] in To compress convolution weights; This is the compressed fused feature map. For 1×1 convolution operations, This is the fused feature map after cross-scale stitching.

[0065] Residual connections superimpose compressed fused features with the original features of spatial branches, preserving spatial feature gradients and avoiding dilution of spatial details by temporal features. The formula is as follows:

[0066]

[0067] in, The output feature map after the first stage of fusion is the result of integrating multi-source features; It is a linear rectified function. This is the compressed fused feature map. This is the feature map of the k-th stage of the spatial branch.

[0068] The final output consists of 4 sets of cross-scale fused feature maps. (512×512×16) (256×256×24) (128×128×40) (64×64×80).

[0069] The progressive spatiotemporal decoder restores cross-scale fused features to the original image resolution through progressive upsampling and feature fusion to output high-precision land cover classification results. This decoder uses cross-scale fused feature maps... to As input, features are fused from the deepest layer. (64×64×80) The process begins with "upsampling-stitching-channel compression" and proceeds step by step: the first stage of decoding uses transposed convolution to... Upsampled to 128×128, and (128×128×40) concatenation, followed by 1×1 convolution to compress the number of channels by 50% (from 80 to 40), yielding the decoding features. (128×128×40); Level 2 decoding will Upsampled to 256×256×24, and (256×256×24) concatenation, compressed to 24 channels by 1×1 convolution, yields (256×256×24); Level 3 decoding will Upsampled to 512×512×16, and (512×512×16) concatenation, compressed to 16 channels via 1×1 convolution, yields (512×512×16); Level 4 decoding will Upsampled to 1024×1024, and then adjusted to 6 channels using a 1×1 convolution to obtain the final decoded features. (1024×1024×6). The classification output stage uses a classification head (containing a 1×1 convolutional layer) to... The mapping to the probability of land cover categories is expressed by the following formula:

[0070]

[0071] in, This is the output pixel-level probability map, where (x, y) are the spatial coordinates of the feature map; For normalized activation functions, For 1×1 convolution operations, This is the final output feature map of the decoder.

[0072] Output a 1024×1024×6 probability map (6 corresponds to 6 land cover categories), and then process the probability map using the Argmax function to generate pixel-level land cover classification results (each pixel corresponds to a unique land cover category).

[0073] Step 3, Model Training.

[0074] The model training employs a cross-entropy loss function with class weights and an adaptive optimizer to achieve efficient optimization of model parameters and adaptation to sample imbalance. The optimizer used is the AdamW optimizer (Adam optimizer with weight decay), with an initial learning rate of 0.0001 and a weight decay coefficient of 1e-5. The learning rate is dynamically adjusted using a cosine annealing strategy (the learning rate decays according to a cosine curve after each iteration to avoid overfitting in later stages). To address the problem of imbalanced ground cover samples (e.g., greenhouse samples accounting for less than 5%), this invention designs a cross-entropy loss function with inverse class frequency weights, the specific formula of which is as follows:

[0075]

[0076] Where C is the number of land cover categories (in this example, C=6). Let N be the weight of class C, and N be the total number of samples in the training set. c Let C be the number of samples in category C. The numerator represents the sum of the total number of samples in all 6 categories divided by the number of samples in each category, which is used to normalize the weights.

[0077] Let i be the true label (0 or 1) of pixel i for category c, where i represents the sample index (traversing all N samples) and c represents the category index (traversing 6 land cover categories, such as farmland, greenhouse, water body, etc.). This function assigns a higher weight to categories (e.g., greenhouses) to predict the probability that pixel i belongs to category c. This makes the model training process more focused on the classification error of minority land cover categories, improving overall classification balance. During training, the training set is input into the model in batches (batchsize=8), and iterative training is performed for 100 rounds. After each round, the model performance is evaluated using a validation set. The model with the highest IoU (Intersection over Union) on the validation set is saved as the optimal model. Simultaneously, IoU (Intersection over Union) and F1-score (Harmonic Average of Precision and Recall) are used as core evaluation metrics to comprehensively reflect the model's classification accuracy across different land cover categories.

[0078] Step 4, post-processing.

[0079] The post-processing stage refines the pixel-level classification results output by the model to generate vector results that meet the needs of practical applications. The process begins with raster result optimization, using morphological closing operations (5×5 rectangular convolution kernels) to fill small holes (area < 50 pixels) inside features such as farmland and forest land, avoiding damage to the integrity of features caused by model prediction errors. At the same time, noise patches with an area smaller than a preset threshold (e.g., 80 pixels) (mostly isolated pixels misclassified by the model) are deleted to improve the spatial continuity of the classification results. After the raster results are optimized, the optimized raster classification results (pixel level) are converted into vector polygons (based on the geographic coordinate system) to achieve the mapping from "pixel" to "feature entity". Then, the Douglas-Peucker algorithm is used to simplify the boundaries of the vector polygons, setting a simplification threshold of 1 meter to remove redundant vertices on the boundaries and smooth the jagged boundaries into natural curves, ensuring that the vector results are easy to use for subsequent GIS (Geographic Information System) analysis and applications.

[0080] Compared with the prior art, the present invention has the following significant advantages:

[0081] Firstly, the information utilization is more comprehensive and the classification accuracy is significantly improved. This invention is the first to achieve heterogeneous fusion of "high-resolution static spatial information + medium-resolution dynamic temporal information". It extracts the microscopic geometric details of land features (such as greenhouse boundaries and field paths) through spatial branches and captures macroscopic phenological dynamics (such as crop growth cycles and vegetation cover changes) through temporal branches, effectively solving the problems of "same object, different spectrum" and "different objects, same spectrum". Experimental verification shows that the average IoU of this invention reaches 84.6% in 6 types of land feature classification tasks, which is 8.1% higher than the single-source high-resolution image classification method (average IoU 76.5%) and higher than the medium-resolution temporal data classification method (average IoU 76.5%). The first improvement is a 12.3% increase from 72.3%. Secondly, cross-scale matching is more accurate and feature fusion efficiency is higher. Through progressive upsampling and dynamic resolution matching mechanisms using time branches, precise alignment of 1024×1024 spatial data with 78×78 temporal data is achieved, avoiding feature distortion caused by direct scaling. The "high-level spatial - low-level temporal" misalignment fusion strategy enhances the complementary use of features at different abstraction levels (deep semantics and shallow dynamics), improving the utilization rate of fused features by more than 40% compared to traditional splicing fusion. Thirdly, it exhibits strong sample imbalance adaptability and high minority class recognition accuracy. The cross-entropy loss function with inverse class frequency weights improves the classification IoU of features such as greenhouses. The loss function efficiency has been improved from 52.1% to 78.3%, significantly improving the problems of "missed detection" and "false detection" of a few types of land features, and meeting the practical needs of greenhouse area statistics in agricultural monitoring and small-area land feature identification in resource management. Fourth, the end-to-end full-process design has strong practicality and generalization ability. The entire process technology chain from "multi-source data preprocessing" to "vector result output" does not require additional design of classification models to process fusion features. It directly outputs vector results that can be used in GIS systems. The ease of operation is significantly better than the existing segmented methods. At the same time, through the lightweight network (MobileNetV3) and adaptive optimization strategy, the model achieves an inference speed of 2 images / second (1024×1024 image) on a regular GPU (NVIDIA RTX 3090). Moreover, the classification accuracy fluctuation in different terrain areas (plains, hills, and mountains) is less than 5%, and the generalization ability is strong.

[0082] Example 2

[0083] This embodiment selects an agricultural demonstration area (approximately 100 km²) in Sichuan Province as the study area. For data sources, the high-resolution imagery is the 0.75-meter resolution RGB three-band imagery acquired by the Jilin-1 satellite in May 2025, and the time-series data is 12 periods of 10-meter resolution imagery from the Sentinel-2 satellite from May 2024 to April 2025. The red (B4) and near-infrared (B8) bands were extracted and cloud removal and linear interpolation were performed on the GEE platform. In the sample labeling stage, six types of land features were labeled in Sichuan Province (35% arable land, 20% forest land, 15% orchard land, 10% water bodies, 5% greenhouses, and 15% others), generating 1024×1024 pixel-level labels. Subsequently, 10,000 spatial slices (1024×1024) and corresponding temporal slices (78×78) were divided into a training set (8,000), a validation set (1,000), and a test set (1,000) in a ratio of 8:1:1.

[0084] In terms of network parameter settings, the spatial branch uses MobileNetV3-small version with 3 input channels and outputs 4-level feature maps (dimensions consistent with the technical solution); the temporal branch uses a single-layer bidirectional ConvLSTM with 32 hidden layer channels and transposed convolution (stride 2) for upsampling; the number of 1×1 convolutional kernels in the fusion module is consistent with the number of spatial feature channels, and the activation function is ReLU; the transposed convolution stride in the decoder is 2, and the 1×1 convolution output channel of the classification head is 6. Regarding training parameters, the optimizer is AdamW with an initial learning rate of 0.0001, weight decay of 1e-5, batch size of 8, 100 iterations, and the loss function is inverse class frequency-weighted cross-entropy, with weights for each class as follows: 0.06 for cultivated land, 0.10 for forest land, 0.13 for orchard land, 0.19 for water bodies, 0.39 for greenhouses, and 0.13 for others. In the post-processing parameters, the morphological closing operation uses a 5×5 convolution kernel and iterates once, the noise patch threshold is set to 50 pixels, and the Douglas-Peucker algorithm simplifies the threshold to 1 meter.

[0085] Results and Analysis of the Example: This example evaluates the classification performance of the model using a test set. The IoU and F1-score data for the six land cover types are as follows: Cultivated land IoU 88.5%, F1-score 93.9%; Forest land IoU 86.3%, F1-score 92.7%; Orchard IoU 84.7%, F1-score 91.7%; Water body IoU 90.2%, F1-score 94.8%; Greenhouse IoU 72.3%, F1-score 83.8%; Other land cover IoU 80.2%, F1-score 89.6%. The average IoU reaches 84.6%, and the average F1-score reaches 90.8%. The results show that the present invention achieves high-precision classification for all types of land cover, especially for land cover such as greenhouses, where the IoU is significantly higher than that of existing technologies (the average of traditional methods is 72.3%), verifying the effectiveness of the imbalanced sample solution.

[0086] In the key scenario verification phase, for the scenario of distinguishing between paddy fields and ponds, by using the temporal characteristics of NDVI (the NDVI of paddy fields reaches 0.6-0.9 during the crop growing season and drops to 0.2-0.3 after harvest, while the NDVI of ponds is stable at -0.2-0.2), the confusion rate between the two types of land cover was reduced from 35% to 8% using traditional methods. For the scenario of distinguishing between greenhouses and cultivated land, by using high-resolution spatial characteristics (greenhouses have regular boundaries and are distributed in a rectangular shape) and temporal characteristics (the NDVI of greenhouses is blocked by the film and is stable at 0.2-0.5 throughout the year, lower than the 0.7-0.9 of cultivated land during the growing season), the confusion rate was reduced from 28% to 5%, which fully demonstrates the advantages of this invention in solving the problem of classifying land cover with "similar spatial morphology but different phenological characteristics".

[0087] This embodiment verifies the technical feasibility of the present invention using actual satellite remote sensing data. The results show that the present invention achieves deep fusion of cross-scale spatiotemporal information through a dual-branch architecture, effectively solving the problems of single information dimension, difficulty in cross-scale matching, and sample imbalance in existing land cover classification methods. The classification accuracy and practicality are significantly better than existing technologies, and can meet the high-precision land cover classification needs in fields such as agricultural monitoring and resource management.

[0088] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0089] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A cross-scale spatiotemporal fusion land cover classification method based on a dual-branch architecture, characterized in that: Includes the following steps: Step S1, remote sensing image preprocessing: acquire single-period high-resolution optical images and multi-period medium-resolution optical images with a preset duration before the reference time phase, achieve spatiotemporal alignment through cloud removal, gap filling and vegetation index extraction, and generate continuous time-series vegetation index data, generate pixel-level labels based on high-resolution image annotation of ground features, and divide the two types of images into geographically corresponding spatial slices and time-series slices, and divide them into training set, validation set and test set. Step S2, construct a dual-branch cross-scale spatiotemporal fusion network: the network includes a spatial feature extraction branch, a temporal feature extraction branch, a cross-level spatiotemporal fusion module, and a progressive spatiotemporal decoder; the cross-level spatiotemporal fusion module specifically adopts a hierarchical misalignment fusion strategy, performs channel splicing and feature compression on the features of different stages of the spatial branch and the corresponding adapted stage features of the temporal branch, and introduces residual connections to output multi-level fused features; The specific correspondence of the hierarchical misalignment fusion strategy is as follows: the feature of the k-th stage of the spatial branch is fused with the feature of the (M+1−k)-th stage of the temporal branch, where M is the total number of downsampling stages in the spatial branch; Step S3, Model Training and Optimization: An optimizer with weight decay is used to validate the model using classification evaluation metrics; the model is optimized using a cross-entropy loss function with class weights, and the class weights are dynamically adjusted based on the proportion of each class sample in the training set. Step S4, post-processing of classification results: Morphological optimization is performed on the pixel-level land cover classification probability map to eliminate noise and holes; the processed classification results are stitched together into a complete image and converted into vector data; boundary smoothing is performed on the vector data to output vectorized land cover classification results.

2. The cross-scale spatiotemporal fusion land cover classification method based on a dual-branch architecture according to claim 1, characterized in that: In step S1, the resolution of the high-resolution optical image is 0.5-1 meter, and the resolution of the medium-resolution optical image is 5-15 meters; the preset duration is 10-14 months, the vegetation index is the Normalized Difference Vegetation Index (NDVI), and the number of periods for the continuous time series vegetation index data is 10-14 periods.

3. The cross-scale spatiotemporal fusion land cover classification method based on a dual-branch architecture according to claim 1, characterized in that: In step S2, the spatial feature extraction branch specifically involves: using a lightweight convolutional network as the backbone network, inputting spatial slices, and extracting multi-level spatial features from shallow texture to deep semantics through multi-stage downsampling; The temporal feature extraction branch specifically employs a bidirectional temporal convolutional network as its core, inputs temporal slices, captures dynamic features of ground cover and phenology, and generates temporal features that match the multi-level feature scale of the spatial branch through progressive upsampling. The progressive spatiotemporal decoder specifically involves upsampling and fusing multi-level fused features at each level, and outputting a pixel-level land cover classification probability map through a classification head.

4. The cross-scale spatiotemporal fusion land cover classification method based on a dual-branch architecture according to claim 3, characterized in that: The lightweight convolutional network can be either MobileNetV3 or EfficientNet-Lite; the number of stages in the multi-stage downsampling is 3-5, and the downsampling stride is 2; the number of channels of spatial features at each stage increases with the downsampling stage.

5. The cross-scale spatiotemporal fusion land cover classification method based on a dual-branch architecture according to claim 4, characterized in that: In step S2, the bidirectional temporal convolutional network is either bidirectional ConvLSTM or bidirectional gated recurrent unit Bi-GRU; the progressive upsampling stage has 3-5 stages, and after each upsampling stage, the features are refined by 3×3 convolution. The difference between the number of channels of the temporal features and the number of channels of the corresponding stage spatial features is less than or equal to 20%.

6. The cross-scale spatiotemporal fusion land cover classification method based on a dual-branch architecture according to claim 5, characterized in that: In step S2, feature compression is achieved through 1×1 convolution, and the number of feature channels after compression is consistent with the number of feature channels in the corresponding stage of the spatial branch.

7. The cross-scale spatiotemporal fusion land cover classification method based on a dual-branch architecture according to claim 6, characterized in that: In step S2, the progressive upsampling is achieved through either transposed convolution or bilinear interpolation. After each upsampling stage, the decoded features are concatenated with the corresponding stage fused features, and the number of channels of the concatenated features is compressed by 40%-60% through 1×1 convolution. The classification head outputs channel features consistent with the number of land cover categories through 1×1 convolution, and generates a pixel-level land cover classification probability map through Softmax and Argmax operations.

8. The cross-scale spatiotemporal fusion land cover classification method based on a dual-branch architecture according to claim 7, characterized in that: The dimensions of channel splicing are Where H and W are the height and width of the feature map, This represents the number of channels in the spatial feature map. The number of channels in the temporal feature map; the convolution is a 1×1 convolution, compressing the number of channels after concatenation to [number missing]. .

9. The cross-scale spatiotemporal fusion land cover classification method based on a dual-branch architecture according to claim 1, characterized in that: In step S3, the optimizer with weight decay is either AdamW or SGD with weight decay; the classification evaluation index includes at least one of Intersection over Union (IoU) and F1 score; in the cross-entropy loss function with class weights, the class weights are calculated by normalizing the total number of samples by the number of samples in the corresponding class.

10. The cross-scale spatiotemporal fusion land cover classification method based on a dual-branch architecture according to claim 1, characterized in that: In step S4, morphological optimization employs morphological closing operations with convolution kernel sizes ranging from 3×3 to 7×7 to eliminate noise patches with areas smaller than 5 to 15 pixels. Boundary smoothing uses the Douglas-Peucker algorithm with a simplified threshold of 0.5-2 meters, achieving boundary smoothing by retaining vertices that deviate from the fitted line by more than the threshold.

Images