A remote sensing image classification method and system based on change perception and spatio-temporal fusion

Abstract

The application discloses a remote sensing image classification method and system based on change perception and space-time fusion, relates to the technical field of image data processing, and takes a multi-temporal remote sensing image sequence as input, extracts multi-level semantic features by adopting a hierarchical visual Transformer backbone network, and extracts multi-level change prior features by a change prior branch based on start and end phase difference images; the change prior features are scale-aligned by a multi-level feature time sequence fusion module, and are guided and fused into a semantic feature sequence in a cross-time sequence injection manner, multi-level fusion features are generated by performing time sequence pooling, change information representation is enhanced, and interference caused by seasonal phenology difference is reduced; finally, based on the multi-level fusion features, a multi-level decoder containing a guided fusion unit is adopted to reconstruct high-resolution features step by step, so that boundary expression consistency and spatial detail fidelity are improved, and the accuracy of the remote sensing image classification result is improved.

Description

Technical Field

[0001] This application relates to the field of image data processing technology, specifically to a remote sensing image classification method and system based on change perception and spatiotemporal fusion. Background Technology

[0002] Multi-temporal remote sensing image classification is typically implemented using deep learning architectures such as convolutional neural networks (CNNs) or transformer models. CNN methods, represented by U-Net (U-shaped network) and DeepLabV3+ (semantic segmentation network), mainly rely on convolution operators to extract local features. However, limited by the local receptive field and the ability to express multi-scale context, they often struggle to fully model long-distance spatial dependencies in areas with complex terrain or fragmented landmasses, easily leading to problems such as discontinuous classification boundaries or missing details. On the other hand, medium-resolution optical remote sensing images are significantly affected by seasonal, illumination, and phenological changes, easily producing phenomena such as different spectra for the same object or the same spectra for different objects, making it difficult to effectively distinguish between "real ground feature changes" and "seasonal / phenological driven pseudo-changes" in the feature space in multi-temporal classification tasks. Existing temporal modeling methods mostly focus on direct fusion of multi-temporal features or attention modeling, but lack a guiding mechanism to explicitly introduce change cues into the feature extraction and fusion process. This can weaken change-related information in deep representations, thus affecting the ability to identify subtle changes and boundary areas. Furthermore, in scenarios where the class distribution is unbalanced and the sample space is scattered, the features of classes with a low sample proportion (such as grassland, bare land, etc.) are more easily interfered with by background classes or similar classes, which can easily lead to class confusion and affect the overall classification accuracy and stability.

[0003] Therefore, there is an urgent need to provide a multi-temporal remote sensing image classification method and system that can utilize multi-temporal information while introducing an explicit guidance mechanism oriented towards change cues. This would enable the effective fusion and aggregation of change-related information and semantic information at multi-scale feature levels, thereby reducing the interference of pseudo-changes caused by factors such as seasonal phenological differences, improving the ability to express boundary details in complex backgrounds, and enhancing the identification stability of categories with low sample proportions. This would improve the overall accuracy and robustness of multi-temporal remote sensing image classification. Summary of the Invention

[0004] The purpose of this application is to provide a remote sensing image classification method and system based on change perception and spatiotemporal fusion, which solves the problems of existing technologies in multi-temporal remote sensing image classification, such as sensitivity to seasonal differences, insufficient extraction of change features, and difficulty in identifying rare categories.

[0005] This application is achieved through the following technical solution:

[0006] The first aspect of this application provides a remote sensing image classification method based on change perception and spatiotemporal fusion, including:

[0007] Obtain multi-temporal remote sensing image sequences corresponding to the region to be classified, and preprocess the multi-temporal remote sensing image sequences to obtain standardized model input data;

[0008] Based on the standardized model input data, multi-level semantic features containing spatiotemporal information are extracted through a hierarchical visual Transformer backbone network; and multi-level variable prior features are extracted through variable prior branches.

[0009] After the multi-level change prior features and multi-level semantic features are scale-aligned by the multi-level feature temporal fusion module, the aligned change prior features are explicitly injected into each phase of the multi-level semantic features using a cross-temporal broadcasting mechanism, and temporal pooling is performed to generate multi-level fused features.

[0010] Based on the last-level features extracted by the hierarchical visual Transformer backbone network and the multi-level fused features, a multi-level decoder containing a guided fusion unit is used to reconstruct and decode the features step by step to obtain the remote sensing image classification results.

[0011] In one possible implementation, a multi-temporal remote sensing image sequence corresponding to the region to be classified is acquired, and the multi-temporal remote sensing image sequence is preprocessed to obtain standardized model input data, including:

[0012] Obtain multi-temporal remote sensing image sequences corresponding to the region to be classified;

[0013] The multi-temporal remote sensing image sequences corresponding to the region to be classified are resampled, cropped, and / or normalized to obtain standardized model input data.

[0014] In one possible implementation, based on the standardized model input data, multi-level semantic features containing spatiotemporal information are extracted through a hierarchical visual Transformer backbone network, including:

[0015] The standardized model input data is used as the input of the hierarchical visual Transformer backbone network to perform feature extraction in four stages through the hierarchical visual Transformer backbone network, thereby obtaining four levels of semantic features containing spatiotemporal information.

[0016] In one possible implementation, multi-level change prior features are extracted through change prior branches, including:

[0017] Based on the standardized model input data, obtain the initial and final phase difference images;

[0018] Initial processing features are obtained by performing initial processing on the initial convolutional layer and two-dimensional max pooling layer in the aforementioned change prior branch to perform initial processing on the initial and final phase difference images.

[0019] The initial processing features are processed by the first CPM encoder to obtain the first-level change prior features; where CPM represents the change prior module.

[0020] The first-level change prior features are processed by the second CPM encoder to obtain the second-level change prior features;

[0021] The second-level change prior features are processed by the third CPM encoder to obtain the third-level change prior features.

[0022] In one possible implementation, a cross-temporal broadcasting mechanism is used to explicitly inject aligned change prior features into each phase of the multi-level semantic features, and temporal pooling is performed to generate multi-level fused features, including:

[0023] The first feature temporal fusion module injects the first-level change prior features as guiding information into the semantic feature sequence corresponding to the first-level semantic features, and performs temporal pooling to obtain the first-level fused features.

[0024] The second feature temporal fusion module injects the second-level change prior features as guiding information into the semantic feature sequence corresponding to the second-level semantic features, and performs temporal pooling to obtain the second-level fused features.

[0025] The third-level change prior features are injected as guiding information into the semantic feature sequence corresponding to the third-level semantic features through the third feature temporal fusion module, and temporal pooling is performed to obtain the third-level fused features.

[0026] In one possible implementation, the first feature time-series fusion module, the second feature time-series fusion module, and the third feature time-series fusion module operate on the same principle and all include:

[0027] Bilinear interpolation upsampling and convolution mapping are performed on the static change prior features to make their spatial resolution and channel dimension fully aligned with the temporal semantic features of the corresponding level, thus obtaining the aligned change prior features.

[0028] The aligned prior features are copied along the time dimension and injected into each time step of the temporal semantic feature sequence via broadcast addition to obtain enhanced temporal features.

[0029] Temporal pooling is performed on the enhanced temporal features to compress the time dimension and obtain fused features.

[0030] In one possible implementation, a multi-level decoder including a guided fusion unit is used for step-by-step feature reconstruction and decoding, including:

[0031] Temporal pooling is performed on the last-level features output by the hierarchical visual Transformer backbone network to obtain bottleneck features. These bottleneck features are then input together with the third-level fusion features into the fourth GFU decoder for processing. Here, GFU stands for Guided Fusion Unit.

[0032] The output features of the fourth GFU decoder and the second-level fused features are input together into the third GFU decoder for processing;

[0033] The first-level fusion feature is used as the skip connection guiding feature and input into the second GFU decoder and the first GFU decoder respectively.

[0034] The output features of the third GFU decoder are fused and decoded with the first-level fusion features by the second GFU decoder to output the second GFU decoded features.

[0035] The second GFU decoding feature is guided to be fused and refined with the first-level fusion feature by the first GFU decoder, and the first GFU decoding feature is output.

[0036] The first GFU decoded features are upsampled and convolutionally processed to obtain the final remote sensing image classification result.

[0037] In one possible implementation, the first GFU decoder, the second GFU decoder, the third GFU decoder, and the fourth GFU decoder all employ a spatial gating weighting mechanism for feature-guided fusion, wherein the spatial gating weighting mechanism includes:

[0038] The two features input to the GFU decoder are concatenated to obtain the initial fused features; the two features include the previous level decoding features and the fused features and / or skip connection features of the corresponding level.

[0039] The initial fusion features are convolutionally mapped and processed by the Sigmoid activation function to generate a spatial gated weight map that characterizes pixel-level importance.

[0040] The spatial gating weight map is multiplied element-wise with the initial fusion feature to spatially weight the initial fusion feature and perform feature recalibration to obtain the weighted refined feature;

[0041] The weighted refined features are subjected to at least one convolutional process, followed by normalization and activation processing, to achieve the fusion of spatial details and deep semantics, and output decoded features.

[0042] In one possible implementation, the method further includes: constructing an adaptive weight joint loss function;

[0043] Auxiliary supervision heads are set at the outputs of the first GFU decoder and the second GFU decoder respectively to calculate the first auxiliary loss and the second auxiliary loss. Combined with the main loss calculated from the final classification result, the joint loss function is constructed as follows:

[0044] L=α·L main +β·L aux1 +γ·L aux2 ;

[0045] Where L is the joint loss function, L main Main loss, L aux1 As the first auxiliary loss, L aux2 The second auxiliary loss is α, the first weight parameter is β, the second weight parameter is β, and the third weight parameter is γ.

[0046] Based on the same inventive concept, this application provides a remote sensing image classification system based on change perception and spatiotemporal fusion, comprising:

[0047] The data standardization module is used to acquire multi-temporal remote sensing image sequences corresponding to the region to be classified, and to preprocess the multi-temporal remote sensing image sequences to obtain standardized model input data.

[0048] The multi-level feature extraction module is used to extract multi-level semantic features containing spatiotemporal information based on the standardized model input data through a hierarchical visual Transformer backbone network; and to extract multi-level variable prior features through variable prior branches.

[0049] The multi-level feature fusion module is used to perform scale alignment between the multi-level change prior features and the multi-level semantic features through the multi-level feature temporal fusion module, and then use a cross-temporal broadcasting mechanism to explicitly inject the aligned change prior features into each temporal phase of the multi-level semantic features, and perform temporal pooling to generate multi-level fused features.

[0050] The step-by-step decoding module is used to reconstruct and decode the features step-by-step based on the last-level features extracted by the hierarchical visual Transformer backbone network and the multi-level fused features, and to obtain the remote sensing image classification results.

[0051] Compared with the prior art, this application has the following advantages and beneficial effects:

[0052] This application provides a remote sensing image classification method and system based on change perception and spatiotemporal fusion. It extracts multi-level semantic features containing spatiotemporal information through a hierarchical visual Transformer backbone network and extracts multi-level change prior features through change prior branches. It can effectively utilize the initial and final phase differences to effectively distinguish between real classification changes and seasonal phenological differences, thereby improving the overall classification accuracy. At the same time, with the explicit prior injection mechanism, it can improve the recognition accuracy of rare categories with fewer samples. Finally, the multi-level decoder effectively suppresses artifacts and noise, making the boundaries of the classification results clearer and more coherent, and preserving more spatial details. Attached Figure Description

[0053] To more clearly illustrate the technical solutions of the exemplary embodiments of this application, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of this application and should not be considered as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort. In the drawings:

[0054] Figure 1 A flowchart illustrating a remote sensing image classification method based on change perception and spatiotemporal fusion, provided for embodiments of this application;

[0055] Figure 2 A schematic diagram of a classification network provided in an embodiment of this application;

[0056] Figure 3 The images provided in this application are Landsat 8 original remote sensing images and classification results for four time phases; (a) is a spring Landsat 8 remote sensing image provided in this application; (b) is a summer Landsat 8 remote sensing image provided in this application; (c) is an autumn Landsat 8 remote sensing image provided in this application; (d) is a winter Landsat 8 remote sensing image provided in this application; and (e) is the Landsat 8 remote sensing image classification result corresponding to the technical solution shown in this application.

[0057] Figure 4 A schematic diagram illustrating feature extraction and fusion provided in an embodiment of this application;

[0058] Figure 5 This is a schematic diagram illustrating the working principle of the feature temporal fusion module provided in the embodiments of this application;

[0059] Figure 6 This is a schematic diagram of step-by-step decoding provided in an embodiment of this application;

[0060] Figure 7A schematic diagram illustrating feature-guided fusion during the decoding process provided in an embodiment of this application;

[0061] Figure 8 Comparison of Landsat 8 remote sensing image classification results based on different deep learning methods provided in this application embodiment; wherein (a) is the Landsat 8 remote sensing image to be classified provided in this application embodiment; (b) is the classification label corresponding to the Landsat 8 remote sensing image to be classified provided in this application embodiment; (c) is the classification result corresponding to the technical solution shown; (d) is the classification result of the VT-Unet method provided in this application embodiment; (e) is the classification result of the 3D-Unet method provided in this application embodiment; (f) is the classification result of the Deeplabv3+ method provided in this application embodiment; (g) is the classification result of the Segformer method provided in this application embodiment; (h) is the classification result of the Swin-Unet method provided in this application embodiment; (j) is the classification result of the Unet method provided in this application embodiment;

[0062] Figure 9 A schematic diagram of the structure of a remote sensing image classification system based on change perception and spatiotemporal fusion provided in an embodiment of this application;

[0063] The attached diagram shows the markings and corresponding component names:

[0064] 901 - Data Standardization Module, 902 - Multi-level Feature Extraction Module, 903 - Multi-level Feature Fusion Module, 904 - Step-by-Step Decoding Module. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the embodiments and accompanying drawings. The illustrative embodiments and descriptions of this application are only for explaining this application and are not intended to limit this application.

[0066] like Figure 1 As shown in the embodiments of this application, a remote sensing image classification method based on change perception and spatiotemporal fusion is provided, including:

[0067] S101. Obtain the multi-temporal remote sensing image sequence corresponding to the region to be classified, and preprocess the multi-temporal remote sensing image sequence to obtain standardized model input data.

[0068] For example, multi-temporal remote sensing image sequences of the area to be classified can be acquired and preprocessed (such as resampling, cropping, and normalization) to form standardized model input data. For instance, Landsat 8 remote sensing images covering all four seasons in 2020 can be selected, with bands B2-B7, and after preprocessing, they can be cropped into 256×256 pixel samples, totaling 8116 images.

[0069] S102. Based on the standardized model input data, extract multi-level semantic features containing spatiotemporal information through the hierarchical vision transformer (Hierarchical Vision Transformer) backbone network; and extract multi-level change prior features through change prior branches.

[0070] like Figure 2 As shown, the hierarchical visual Transformer backbone network uses a hierarchical visual Transformer (such as the Swing Transformer, a layered visual transformer) to process the input spatiotemporal image cube (that is, to construct a multi-temporal image cube containing spring, summer, autumn and winter from the multi-temporal remote sensing image sequence) and extract semantic features containing spatiotemporal information.

[0071] The Change Prior Branch includes a Change Prior Module (CPM). The input to this module is the beginning-end time-phase difference image from the beginning and end time-phase remote sensing images in a multi-temporal remote sensing image sequence. The CPM module sequentially encodes the features of the beginning-end time-phase difference image through convolutional layers, pooling layers, and a multi-level CPM encoder, extracting multi-scale change prior features to explicitly represent ground feature change cues and provide guiding information for subsequent multi-level feature fusion. For example, the beginning-end time-phase difference image can be obtained by subtracting a winter image (i.e., the end time-phase remote sensing image) from a spring image (i.e., the beginning time-phase remote sensing image).

[0072] S103. After scaling the multi-level change prior features and the multi-level semantic features through the multi-level feature temporal fusion module, the aligned change prior features are explicitly injected into each temporal phase of the multi-level semantic features using the cross-temporal broadcasting (CB) mechanism, and temporal pooling (TP) is performed to generate multi-level fused features.

[0073] For example, a channel adapter (e.g., a 1×1 convolution) can be used to adjust the dimensions of the changing prior features, broadcast them to each time step of the backbone semantic feature sequence for element-wise summation, thereby explicitly injecting the static changing prior features into the dynamic semantic features. Then, temporal pooling is performed along the time dimension to compress the dynamic temporal information into a stable static feature map, so as to suppress temporal redundancy information and retain global statistical features, thereby obtaining fused features.

[0074] S104. Based on the last-level features extracted by the hierarchical visual Transformer backbone network and the multi-level fused features, a multi-level decoder containing a guided fusion unit is used to reconstruct and decode the features step by step to obtain the remote sensing image classification results.

[0075] The hierarchical visual Transformer backbone network can be specifically implemented as a Swing Transformer. In the embodiments of this application, the hierarchical visual Transformer backbone network can adopt a Tiny version of the Swing Transformer (Swin-T).

[0076] For example, a decoder containing Guided Fusion Units (GFUs) can be constructed through guided decoding and deep supervised classification to progressively restore spatial resolution and output remote sensing image classification results. For instance, the GFU decoder can receive upsampled features from the previous level decoder and skip connection features from the feature temporal fusion module (the skip connection features refer to first-level, second-level, or third-level fusion features). Through a built-in Spatial Attention Gating (SAG) mechanism, a spatial gating weight map is generated using the sigmoid activation function to weight and recombine the skip connection features, suppressing background noise in invariant regions and highlighting target boundaries. The final remote sensing image classification result output by the multi-level decoder is shown in the following figure. Figure 3 As shown in (a) to (e) in the diagram. An auxiliary supervisory head can also be set in the intermediate layer of the multi-level decoder to calculate the auxiliary loss and perform joint optimization with the main loss. This provides multi-scale supervisory signals, enhancing the discriminative ability for multi-scale features.

[0077] Assuming there are four GFU decoders, with the first GFU decoder outputting the final result and the second GFU decoder being the previous stage decoder of the first GFU decoder, auxiliary losses can be introduced into both the first and second GFU decoders. The total loss function consists of a weighted sum of the main loss and the two auxiliary losses. The AdamW optimizer is used for training, with an initial learning rate of 5×10⁻⁶. -5 The weight decays to 5×10-4 The batch size was set to 8, and the total number of training rounds was 100. Testing showed that the overall accuracy (OA) of this application was 89.23%, the Kappa coefficient was 0.8480, and the mean intersection-over-union (mIoU) was 64.29%. Compared with Swin-Unet (a U-shaped network based on Swin Transformer), the overall accuracy was improved by 1.31%, and the mean IoU was improved by 1.98%.

[0078] As shown in Table 1, the qualitative results show that the technical solution provided by the embodiments of this application can better maintain the consistency of details, especially in major land types such as cultivated land, forest land, water bodies and construction land, showing clearer boundary delineation and stronger contour coherence; at the same time, it also shows stable discrimination ability in small fragmented land types such as grassland and other land uses that are fragmented and scattered.

[0079] Table 1. Comparison of accuracy of various deep learning methods in Landsat 8 remote sensing image classification.

[0080]

[0081] like Figure 4 As shown, based on the technical solution described in the embodiments of this application, a schematic diagram of feature extraction and fusion is provided. The change prior branch takes the initial and final phase difference images as input, and sequentially passes them through the initial convolutional layer, the two-dimensional max pooling layer, and the multi-level CPM encoder to generate multi-scale change prior features (i.e., first-level change prior features to third-level change prior features). These features are then injected into the semantic features output at the corresponding stage of the hierarchical visual Transformer backbone network for fusion, forming spatiotemporal joint features, resulting in first-level fusion features to third-level fusion features.

[0082] To facilitate understanding of the network architecture and data flow process provided in the embodiments of this application, the symbols and dimensional parameters involved in Figure 4 are explained as follows:

[0083] B: Represents the batch size, which is the number of samples the model processes in a single training or inference cycle.

[0084] T: Represents the time dimension or the number of time phases. In this embodiment, it can correspond to the four time phases of spring, summer, autumn, and winter, for example, T=4.

[0085] C in : Indicates the initial number of channels in the input data, which is composed of the original spectral bands and their corresponding derivative rational index features.

[0086] H and W represent the original height and width of the input remote sensing image in spatial dimensions, respectively.

[0087] K0, K1, K2, K3: represent the number of feature channels output by the initial convolutional layer and each level of the variable prior encoder in the variable prior branch (CPM).

[0088] C1, C2, C3, C4: represent the number of semantic feature channels output by the hierarchical visual Transformer backbone network in four consecutive feature extraction stages.

[0089] H / n, W / n: represent the downsampling scale ratio of the feature map relative to the original input image in the spatial dimension.

[0090] H / 4 and W / 4 indicate that the feature map size is 1 / 4 of the original size, corresponding to the first-level feature processing stage of the network.

[0091] H / 8 and W / 8 indicate that the feature map size is 1 / 8 of the original size, corresponding to the second-level feature processing stage of the network.

[0092] H / 16 and W / 16 indicate that the feature map size is 1 / 16 of the original size, corresponding to the third-level feature processing stage of the network.

[0093] H / 32 and W / 32 indicate that the feature map size is 1 / 32 of the original size, corresponding to the last bottleneck feature extracted by the backbone network.

[0094] In one possible implementation, a multi-temporal remote sensing image sequence corresponding to the region to be classified is acquired, and the multi-temporal remote sensing image sequence is preprocessed to obtain standardized model input data, including:

[0095] Obtain multi-temporal remote sensing image sequences corresponding to the region to be classified;

[0096] The multi-temporal remote sensing image sequences corresponding to the region to be classified are resampled, cropped, and / or normalized to obtain standardized model input data.

[0097] In one possible implementation, based on the standardized model input data, multi-level semantic features containing spatiotemporal information are extracted through a hierarchical visual Transformer backbone network, including:

[0098] The standardized model input data is used as the input of the hierarchical visual Transformer backbone network to perform feature extraction in four stages through the hierarchical visual Transformer backbone network, thereby obtaining four levels of semantic features containing spatiotemporal information.

[0099] For example, standardized model input data can be first constructed into a multi-temporal image cube, and then fed into a hierarchical visual Transformer backbone network for processing to obtain four levels of semantic features containing spatiotemporal information. Traditional remote sensing classification is in a black-box learning state, relying solely on the instantaneous spectral values ​​of the original bands. Influenced by phenology, it is highly susceptible to confusion between different spectra of the same object or the same spectrum of different objects (e.g., the spectra of withered vegetation in winter are extremely similar to those of bare land). Furthermore, multi-temporal data often introduces artificial noise due to seasonal gaps or inconsistent preprocessing, leading to the misidentification of data defects as changes in land cover. Therefore, this application's embodiments, by constructing a multi-temporal image cube, achieve a paradigm shift from instantaneous spectral identification to geophysical phenological pattern discrimination. Utilizing the seasonal range channel suppresses seasonal pseudo-change interference from the source, significantly improving the model's robustness in complex scenarios, especially providing strong global representation support for rare land types.

[0100] Remote sensing imagery data for the same geographic location at multiple time phases can be acquired according to a preset time sequence, resulting in a multi-temporal remote sensing image sequence. This sequence includes remote sensing images for spring, summer, autumn, and winter, comprehensively depicting the phenological evolution of ground features on an annual scale. Subsequently, spatial registration, resampling, and uniform cropping are performed on the remote sensing images for each season to eliminate spatial displacement errors introduced by differences in sensor imaging angles or satellite orbital offsets. This ensures that the remote sensing images from different seasons maintain a high degree of consistency in pixel coordinates, spatial resolution, and geographic coverage, conforming to the preset spatial dimensions (H, W). This establishes a unified and stable spatial reference for subsequent pixel-by-pixel temporal feature modeling. Here, H represents height, and W represents width.

[0101] After unifying the spatial reference, feature construction and channel expansion processing are performed on the remote sensing images corresponding to each season. For example, the original spectral bands of the remote sensing image can be used as the basic features, and derivative physical index features corresponding to the spectral bands can be further introduced. By performing a stitching operation in the channel dimension, a single-season image can be constructed to contain C in The multi-channel integrated feature map of each channel realizes the transformation from single spectral information to multi-dimensional joint representation of the physical attributes of ground objects, effectively enhancing the ability to identify complex ground object types and their differences.

[0102] After feature enhancement of remote sensing images for each season, the multi-channel integrated feature maps corresponding to the four seasons (spring, summer, autumn, and winter) are stacked in chronological order to construct a four-dimensional spatiotemporal feature tensor (T, Cin, H, W) containing temporal, channel, and spatial dimensions, where T represents the seasonal temporal dimension. Furthermore, to adapt to the parallel feature extraction structure of the hierarchical visual Transformer backbone network, the batch dimension B and the temporal dimension T are merged through a tensor reshaping operation to form a unified image cube input format (B×T, Cin, H, W). in The process involves using H, W) to obtain a multi-temporal image cube, which serves as the standardized input for subsequent hierarchical spatiotemporal semantic feature extraction.

[0103] In one possible implementation, multi-level change prior features are extracted through change prior branches, including:

[0104] Based on the standardized model input data, the beginning and end phase difference images are obtained; for example, assuming that the standardized model input data includes images of the four seasons of spring, summer, autumn and winter, then the beginning and end phase difference images can be obtained by obtaining the difference images of the corresponding images of winter and spring.

[0105] Initial processing features are obtained by performing initial processing on the initial convolutional layer and two-dimensional max pooling layer in the aforementioned change prior branch to perform initial processing on the initial and final phase difference images.

[0106] The initial processing features are processed by the first CPM encoder to obtain the first-level change prior features; where CPM represents the change prior module.

[0107] The first-level change prior features are processed by the second CPM encoder to obtain the second-level change prior features;

[0108] The second-level change prior features are processed by the third CPM encoder to obtain the third-level change prior features.

[0109] In existing technologies, standard U-shaped networks generally employ symmetrical connections with strict hierarchical correspondence. However, due to the irreversible loss of high-frequency geometric cues (such as boundaries and textures) during continuous downsampling of remote sensing images, symmetrical structures, when recovering mesoscale features, often suffer from broken or overly smoothed boundaries of fragmented targets such as field ridges and narrow paths due to the lack of guidance from high-resolution original details. Therefore, this application proposes an asymmetric misaligned connection topology. In the decoding stage, it breaks the limitations of same-level connections, performing long-distance cross-level reuse of the highest-resolution original detail features (64x64 resolution) captured in the first stage. This feature is not only input to the first-level decoder but also reused and synchronously injected into an earlier intermediate decoding layer (i.e., the second-level decoder). This achieves early explicit compensation for high-frequency geometric details. This non-standard topology forces intermediate decoding layers to obtain edge detail cues earlier, significantly improving the boundary connectivity and contour integrity of fragmented landmasses and small-scale linear targets, and enhancing the spatial identification of sparsely distributed features.

[0110] In one possible implementation, a cross-temporal broadcasting mechanism is used to explicitly inject aligned change prior features into each phase of the multi-level semantic features, and temporal pooling is performed to generate multi-level fused features, including:

[0111] The first feature temporal fusion module injects the first-level change prior features as guiding information into the semantic feature sequence corresponding to the first-level semantic features, and performs temporal pooling to obtain the first-level fused features.

[0112] The second feature temporal fusion module injects the second-level change prior features as guiding information into the semantic feature sequence corresponding to the second-level semantic features, and performs temporal pooling to obtain the second-level fused features.

[0113] The third-level change prior features are injected as guiding information into the semantic feature sequence corresponding to the third-level semantic features through the third feature temporal fusion module, and temporal pooling is performed to obtain the third-level fused features.

[0114] In existing technologies, when fusing static change cues with dynamic time-series sequences, traditional stitching methods not only suffer from dimensional mismatch but also lead to the rapid dilution of prior signals during feature transfer, making it impossible to apply consistent directional constraints to each time step and reducing the sensitivity to real-world changes. In contrast, this application's embodiment uses a feature time-series fusion module to broadcast static prior features along the time axis and explicitly injects them into each time step of the semantic features in an element-wise addition manner, achieving directional physical calibration of features across all time phases. This mechanism forces the model to have a global change reference when processing remote sensing images from any season, solving the problem of prior signal dilution with extremely low computational cost, significantly enhancing the ability to identify real changes, and effectively filtering out false changes caused by seasonal fluctuations.

[0115] In one possible implementation, the first feature time-series fusion module, the second feature time-series fusion module, and the third feature time-series fusion module operate on the same principle and all include:

[0116] Bilinear interpolation upsampling and convolution mapping are performed on the static change prior features to make their spatial resolution and channel dimension fully aligned with the temporal semantic features of the corresponding level, thus obtaining the aligned change prior features.

[0117] The aligned prior features are copied along the time dimension and injected into each time step of the temporal semantic feature sequence via broadcast addition to obtain enhanced temporal features.

[0118] Temporal pooling is performed on the enhanced temporal features to compress the time dimension and obtain fused features.

[0119] like Figure 5 As shown, the feature temporal fusion module mainly includes two core processes: change prior fusion and temporal pooling. By injecting static change prior features into multi-temporal semantic features, the prior information is utilized to maximize the model's perception ability and finally obtain a static feature map that integrates spatiotemporal information, thus obtaining fused features.

[0120] To facilitate understanding of the data interaction and transformation process within the feature temporal fusion module (the i-th stage of feature processing), the symbols and dimensional parameters involved in Figure 5 are explained below:

[0121] B: Represents the batch size, which is the number of samples that the model processes in parallel during a single training or inference cycle.

[0122] T: Represents the time step or temporal dimension, used to characterize the length of the time dimension of a multi-temporal remote sensing image sequence.

[0123] C i : Represents the number of channels in the feature map of stage i, used to characterize the multidimensional semantic attribute features of ground objects.

[0124] H i W i : These represent the height and width of the feature map in the i-th stage in terms of spatial dimensions, respectively.

[0125] (B, T, C i H i W i ): Represents the five-dimensional tensor dimensions of the multi-temporal semantic feature sequence, corresponding to batch, time, channel, height and width respectively.

[0126] (B×T, C i H i W i ): This represents the four-dimensional tensor dimension resulting from merging the batch dimension and the time dimension, used for parallel processing and feature extraction of features at each time phase in the backbone network.

[0127] (B, C i H i W i ): This represents the static feature tensor obtained by performing global average pooling along the time dimension on the fused feature sequence. This operation is called temporal pooling (TP), in which the time dimension T has been compressed.

[0128] Dimension restoration: refers to the operation of decomposing the merged dimension (B×T) back into batch dimension B and time dimension T in order to restore the structure and temporal logic of the time series feature sequence.

[0129] Cross-time broadcasting refers to the operation of copying and expanding static, changing prior features along the time dimension so that their time length is consistent with the semantic feature sequence, and then injecting them into each time step.

[0130] Element-wise addition fusion operator: refers to adding the broadcast changed prior features and the semantic features of the corresponding time step element-wise at the same spatial position to achieve explicit injection and fusion of changed priors.

[0131] In one possible implementation, a multi-level decoder including a guided fusion unit is used for step-by-step feature reconstruction and decoding, including:

[0132] Temporal pooling is performed on the last-level features output by the hierarchical visual Transformer backbone network to obtain bottleneck features. These bottleneck features are then input together with the third-level fusion features into the fourth GFU decoder for processing. Here, GFU stands for Guided Fusion Unit.

[0133] The output features of the fourth GFU decoder and the second-level fused features are input together into the third GFU decoder for processing;

[0134] The first-level fusion feature is used as the skip connection guiding feature and input into the second GFU decoder and the first GFU decoder respectively.

[0135] The output features of the third GFU decoder are fused and decoded with the first-level fusion features by the second GFU decoder to output the second GFU decoded features.

[0136] The second GFU decoding feature is guided to be fused and refined with the first-level fusion feature by the first GFU decoder, and the first GFU decoding feature is output.

[0137] The first GFU decoded features are upsampled and convolutionally processed to obtain the final remote sensing image classification result.

[0138] In one possible implementation, the method further includes: constructing an adaptive weight joint loss function;

[0139] Auxiliary supervision heads are set at the outputs of the first GFU decoder and the second GFU decoder respectively to calculate the first auxiliary loss and the second auxiliary loss. Combined with the main loss calculated from the final classification result, the joint loss function is constructed as follows:

[0140] L=α·L main +β·L aux1 +γ·L aux2 ;

[0141] Where L is the joint loss function, L main Main loss, L aux1 As the first auxiliary loss, L aux2 The second auxiliary loss is α, the first weight parameter is β, the second weight parameter is β, and the third weight parameter is γ.

[0142] Both the main loss and auxiliary loss employ a weighted combination of CE (cross-entropy loss) and Focal (focus loss). The first to third weight parameters are learnable and can be adaptively updated based on gradients through backpropagation during training to dynamically adjust the contribution ratio of the main loss and each level of auxiliary loss. It is worth noting that the first, second, and third weight parameters can also be preset to fixed values.

[0143] like Figure 6 As shown, the step-by-step upsampling process in the decoding stage is illustrated, as well as the connection method that introduces auxiliary loss (AL) in intermediate layers (such as the first GFU decoder and the second GFU decoder) for multi-scale depth supervision.

[0144] In one possible implementation, the first GFU decoder, the second GFU decoder, the third GFU decoder, and the fourth GFU decoder all employ a spatial gating weighting mechanism for feature-guided fusion, wherein the spatial gating weighting mechanism includes:

[0145] The two features input to the GFU decoder are concatenated to obtain the initial fused features; the two features include the previous level decoding features and the fused features and / or skip connection features of the corresponding level.

[0146] The initial fusion features are convolutionally mapped and processed by the Sigmoid activation function to generate a spatial gated weight map that characterizes pixel-level importance.

[0147] The spatial gating weight map is multiplied element-wise with the initial fusion feature to spatially weight the initial fusion feature and perform feature recalibration to obtain the weighted refined feature;

[0148] The weighted refined features are subjected to at least one convolutional process, followed by normalization and activation processing, to achieve the fusion of spatial details and deep semantics, and output decoded features.

[0149] like Figure 7 As shown, the GFU decoder first performs feature concatenation and attention-weighted fusion, and then completes feature refinement through convolution and batch normalization.

[0150] Traditional skip connections often employ linear concatenation, indiscriminately introducing cloud shadows, noise, and non-ground texture interference from the shallow layers of the encoder into the decoder. Unfiltered fusion of these low-level features leads to speckle noise (salt-and-pepper noise), inter-class adhesion, and blurred edges in the final classification result. The GFU decoder provided in this application employs a spatial gating weighting mechanism. It utilizes deep upsampled features to guide shallow skip connection features to generate a pixel-level spatial gating weight map, and performs pixel-by-pixel recalibration (weight allocation) on the fused initial features, thereby achieving feature refinement. This achieves pixel-level feature artifact removal and preservation. This mechanism suppresses noise responses in non-target regions through adaptive filters, strengthens real edges and structural responses, making the boundaries of the classification results sharper and more coherent, and significantly improving the spatial detail fidelity in complex background scenes.

[0151] like Figure 8 (a) to (a)j As shown in the figure, this is a comparison chart of Landsat 8 remote sensing image classification results based on different deep learning methods provided in this application embodiment. Circular annotations highlight areas where significant differences exist in the Landsat 8 remote sensing image classification results from different deep learning methods. This is primarily used to assist readers in visual comparison and spatial detail recognition and does not have specific physical or statistical meaning. It can be seen that the classification results generated by the technical solution described in this application embodiment have clearer and more coherent boundaries. Among them, Figure 8 (a) in the image is an optical image; Figure 8 (b) in the image is the label image; Figure 8 (c) in the table represents the classification result of this application; Figure 8 In the diagram, (d) represents the classification result of VT-Unet (a U-shaped network based on visual Transformer); Figure 8 In the diagram, (e) represents the classification result of 3D-Unet (a three-dimensional U-shaped convolutional neural network). Figure 8 In the table, (f) represents the classification result of Deeplabv3+ (a semantic segmentation network based on dilated convolutional spatial pyramid pooling); Figure 8 In the diagram, (g) represents the classification result of Segformer (a semantic segmentation network based on Transformer); Figure 8 In the table, (h) represents the classification result of Swin-Unet (a U-shaped network based on SwinTransformer); Figure 8 In the table, (j) represents the classification result of the U-net.

[0152] Compared with the prior art, this application has the following advantages:

[0153] Strong resistance to seasonal interference: This application uses the beginning and end phase difference explicitly through the CPM decoder, which can effectively distinguish between real classification changes and seasonal phenological differences, and the overall classification accuracy reaches 89.23% in complex scenarios.

[0154] Significantly improves the recognition rate of rare categories: This application, with its explicit prior injection mechanism, significantly outperforms existing mainstream models in the recognition accuracy of rare categories with fewer samples; for example, the recognition accuracy in grassland and other categories reaches 67.01% and 60.45%, respectively.

[0155] More refined boundary characterization: The guided fusion unit provided in this application, combined with a deep supervision strategy, effectively suppresses artifacts and noise, making the boundaries of the classification results clearer and more coherent, and preserving more spatial details.

[0156] High computational efficiency: This application adopts a design combining Swin-T with lightweight CPM coding, which achieves higher accuracy with a lower number of parameters compared to heavy backbone networks such as ResNet50, thus achieving a good balance between accuracy and efficiency.

[0157] Mechanism Synergy Advantages: This application, based on the injection of explicit change priors, further combines... Figure 5 The temporal pooling (global average pooling along the time axis) shown performs time-dimensional mean aggregation on the fused multi-temporal feature sequences, forming a lightweight temporal mean aggregation mechanism. This effectively suppresses redundant temporal noise while preserving the statistical characteristics of change, achieving a synergistic optimization of preserving change and seeking stability.

[0158] Improved accuracy in rare land cover identification: This application benefits from the annual reference provided by the phenological range constraint and the detailed compensation provided by the cross-level topology, which provides strong discrimination support for rare land cover with scattered spatial distribution and small sample size, effectively solving the problem that such targets are easily submerged by large categories in the feature space (e.g., grassland accuracy is improved to 67.01%).

[0159] Engineering determinism and reproducibility: This application has clear data flow and physical topology constraints. Combined with adaptive loss weights and underlying repair for mixed precision training conflicts, it ensures extremely fast convergence and high reproducibility accuracy on large-scale multi-temporal data.

[0160] like Figure 9 As shown, based on the same inventive concept, embodiments of this application provide a remote sensing image classification system based on change perception and spatiotemporal fusion, comprising:

[0161] The data standardization module 901 is used to acquire the multi-temporal remote sensing image sequence corresponding to the region to be classified, and to preprocess the multi-temporal remote sensing image sequence to obtain standardized model input data.

[0162] The multi-level feature extraction module 902 is used to extract multi-level semantic features containing spatiotemporal information based on the standardized model input data through a hierarchical visual Transformer backbone network; and to extract multi-level variable prior features through variable prior branches.

[0163] The multi-level feature fusion module 903 is used to perform scale alignment between the multi-level change prior features and the multi-level semantic features through the multi-level feature temporal fusion module, and then use a cross-temporal broadcasting mechanism to explicitly inject the aligned change prior features into each temporal phase of the multi-level semantic features, and perform temporal pooling to generate multi-level fused features.

[0164] The step-by-step decoding module 904 is used to reconstruct and decode the features step-by-step based on the last-level features extracted by the hierarchical visual Transformer backbone network and the multi-level fused features, and to obtain the remote sensing image classification results.

[0165] The remote sensing image classification system based on change perception and spatiotemporal fusion provided in this application embodiment can execute the above-mentioned method and technical solution. Its principle and beneficial effects are similar, and will not be repeated here.

[0166] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present invention can be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0167] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0168] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0169] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0170] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0171] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A remote sensing image classification method based on change perception and spatiotemporal fusion, characterized in that, include: Obtain multi-temporal remote sensing image sequences corresponding to the region to be classified, and preprocess the multi-temporal remote sensing image sequences to obtain standardized model input data; Based on the standardized model input data, multi-level semantic features containing spatiotemporal information are extracted through a hierarchical visual Transformer backbone network; and multi-level variable prior features are extracted through variable prior branches. After the multi-level change prior features and multi-level semantic features are scale-aligned by the multi-level feature temporal fusion module, the aligned change prior features are explicitly injected into each phase of the multi-level semantic features using a cross-temporal broadcasting mechanism, and temporal pooling is performed to generate multi-level fused features; the multi-level fused features include a first-level fused feature, a second-level fused feature, and a third-level fused feature generated sequentially; Based on the last-level features extracted by the hierarchical visual Transformer backbone network and the multi-level fused features, a multi-level decoder containing guided fusion units is used to reconstruct and decode the features step by step, resulting in remote sensing image classification results, including: Temporal pooling is performed on the last-level features output by the hierarchical visual Transformer backbone network to obtain bottleneck features. These bottleneck features are then input together with the third-level fusion features into the fourth GFU decoder for processing. Here, GFU stands for Guided Fusion Unit. The output features of the fourth GFU decoder and the second-level fused features are input together into the third GFU decoder for processing; The first-level fusion feature is used as the skip connection guiding feature and input into the second GFU decoder and the first GFU decoder respectively. The output features of the third GFU decoder are fused and decoded with the first-level fusion features by the second GFU decoder to output the second GFU decoded features. The first GFU decoder guides the fusion and refinement of the second GFU decoding feature and the first-level fusion feature, and outputs the first GFU decoding feature. The first GFU decoded features are upsampled and convolutionally processed to obtain the final remote sensing image classification result.

2. The remote sensing image classification method based on change perception and spatiotemporal fusion according to claim 1, characterized in that, Obtain multi-temporal remote sensing image sequences corresponding to the region to be classified, and preprocess the multi-temporal remote sensing image sequences to obtain standardized model input data, including: Obtain multi-temporal remote sensing image sequences corresponding to the region to be classified; The multi-temporal remote sensing image sequences corresponding to the region to be classified are resampled, cropped, and / or normalized to obtain standardized model input data.

3. The remote sensing image classification method based on change perception and spatiotemporal fusion according to claim 1, characterized in that, Based on the standardized model input data, a hierarchical visual Transformer backbone network is used to extract multi-level semantic features containing spatiotemporal information, including: The standardized model input data is used as the input of the hierarchical visual Transformer backbone network to perform feature extraction in four stages through the hierarchical visual Transformer backbone network, thereby obtaining four levels of semantic features containing spatiotemporal information.

4. The remote sensing image classification method based on change perception and spatiotemporal fusion according to claim 3, characterized in that, Multi-level change prior features are extracted through change prior branches, including: Based on the standardized model input data, obtain the initial and final phase difference images; Initial processing features are obtained by performing initial processing on the initial convolutional layer and two-dimensional max pooling layer in the aforementioned change prior branch to perform initial processing on the initial and final phase difference images. The initial processing features are processed by the first CPM encoder to obtain the first-level change prior features; where CPM represents the change prior module. The first-level change prior features are processed by the second CPM encoder to obtain the second-level change prior features; The second-level change prior features are processed by the third CPM encoder to obtain the third-level change prior features.

5. The remote sensing image classification method based on change perception and spatiotemporal fusion according to claim 4, characterized in that, A cross-temporal broadcasting mechanism is employed to explicitly inject aligned change prior features into each temporal phase of the multi-level semantic features, and temporal pooling is performed to generate multi-level fused features, including: The first feature temporal fusion module injects the first-level change prior features as guiding information into the semantic feature sequence corresponding to the first-level semantic features, and performs temporal pooling to obtain the first-level fused features. The second feature temporal fusion module injects the second-level change prior features as guiding information into the semantic feature sequence corresponding to the second-level semantic features, and performs temporal pooling to obtain the second-level fused features. The third feature temporal fusion module injects the third-level change prior features as guiding information into the semantic feature sequence corresponding to the third-level semantic features, and performs temporal pooling to obtain the third-level fused features.

6. The remote sensing image classification method based on change perception and spatiotemporal fusion according to claim 5, characterized in that, The first feature time-series fusion module, the second feature time-series fusion module, and the third feature time-series fusion module operate on the same principle and all include: Bilinear interpolation upsampling and convolution mapping are performed on the static change prior features to make their spatial resolution and channel dimension fully aligned with the temporal semantic features of the corresponding level, thus obtaining the aligned change prior features. The aligned prior features are copied along the time dimension and injected into each time step of the temporal semantic feature sequence via broadcast addition to obtain enhanced temporal features. Temporal pooling is performed on the enhanced temporal features to compress the time dimension and obtain fused features.

7. The remote sensing image classification method based on change perception and spatiotemporal fusion according to claim 5, characterized in that, The first GFU decoder, the second GFU decoder, the third GFU decoder, and the fourth GFU decoder all employ a spatial gating weighting mechanism for feature-guided fusion. This spatial gating weighting mechanism includes: The two features input to the GFU decoder are concatenated to obtain the initial fused features; the two features include the previous level decoding features and the fused features and / or skip connection features of the corresponding level. The initial fusion features are convolutionally mapped and processed by the Sigmoid activation function to generate a spatial gated weight map that characterizes pixel-level importance. The spatial gating weight map is multiplied element-wise with the initial fusion feature to spatially weight the initial fusion feature and perform feature recalibration to obtain the weighted refined feature; The weighted refined features are subjected to at least one convolutional process, followed by normalization and activation processing, to achieve the fusion of spatial details and deep semantics, and output decoded features.

8. The remote sensing image classification method based on change perception and spatiotemporal fusion according to claim 5, characterized in that, The method further includes: constructing an adaptive weight joint loss function; Auxiliary supervision heads are set at the outputs of the first GFU decoder and the second GFU decoder respectively to calculate the first auxiliary loss and the second auxiliary loss. Combined with the main loss calculated from the final classification result, the joint loss function is constructed as follows: L=α·L main +β·L aux1 +γ·L aux2 ; Where L is the joint loss function, L main Main loss, L aux1 As the first auxiliary loss, L aux2 The second auxiliary loss is α, the first weight parameter is β, the second weight parameter is β, and the third weight parameter is γ.

9. A remote sensing image classification system based on change perception and spatiotemporal fusion, wherein the remote sensing image classification system based on change perception and spatiotemporal fusion is used to execute the remote sensing image classification method based on change perception and spatiotemporal fusion as described in any one of claims 1 to 8, characterized in that, include: The data standardization module is used to acquire multi-temporal remote sensing image sequences corresponding to the region to be classified, and to preprocess the multi-temporal remote sensing image sequences to obtain standardized model input data. The multi-level feature extraction module is used to extract multi-level semantic features containing spatiotemporal information based on the standardized model input data through a hierarchical visual Transformer backbone network; and to extract multi-level variable prior features through variable prior branches. The multi-level feature fusion module is used to perform scale alignment between the multi-level change prior features and the multi-level semantic features through the multi-level feature temporal fusion module, and then use a cross-temporal broadcasting mechanism to explicitly inject the aligned change prior features into each temporal phase of the multi-level semantic features, and perform temporal pooling to generate multi-level fused features. The step-by-step decoding module is used to reconstruct and decode the features step-by-step based on the last-level features extracted by the hierarchical visual Transformer backbone network and the multi-level fused features, and to obtain the remote sensing image classification results.

Images