A landscape pattern classification method that combines global and local multi-level shallow and deep features

Abstract

This invention discloses a landscape pattern classification method that combines global and local multi-level deep and shallow features, belonging to the field of photogrammetric data processing technology. Firstly, a deep and shallow feature cross-fusion module is introduced between the encoder and decoder to alleviate the semantic gap between features of different levels generated by the Swin Transformer. Then, by introducing three global-local attention modules in the decoder to aggregate global contextual information and local detail information, the method strengthens the identification of intra-class feature consistency and inter-class feature differences. This invention fully utilizes the advantages of both CNN and Swin Transformer in extracting local and global information, aggregating global contextual information and local fine-grained information, thus improving the problem of spectral feature similarity between land cover categories and intra-class differences in complex landscape pattern scenarios, which is of great significance for dynamic monitoring of ecosystems.

Description

Technical Field

[0001] This invention belongs to the field of photogrammetric data processing technology, specifically involving a landscape pattern classification method that combines global and local multi-level shallow and deep features. Background Technology

[0002] Ecosystems play a vital role in the global carbon cycle and are crucial for degrading pollution, conserving water resources, regulating climate, and protecting biodiversity. However, due to the impacts of human activities and natural disasters, ecosystems face problems such as ecological function degradation and biodiversity loss. Therefore, protecting ecosystems is of paramount importance. Obtaining accurate long-term landscape pattern information and analyzing its evolutionary characteristics through dynamic monitoring of ecosystem landscape patterns is essential for the protection and restoration of ecosystem resources.

[0003] With the rapid development of remote sensing technology and the advancement of Earth observation systems, a large amount of observational data has been provided for dynamic monitoring of landscape patterns. Remote sensing technology has the advantages of fast acquisition speed, wide coverage, and large information content, making it an important means of landscape pattern monitoring. However, the mixed pixel problem in medium- and low-resolution image data has affected the accuracy of landscape pattern information extraction to some extent. Therefore, the classification of ecosystem landscape patterns using high-resolution remote sensing imagery has become a current research hotspot, achieving high classification accuracy.

[0004] Landscape pattern classification using manual interpretation is time-consuming and labor-intensive. Machine learning methods, such as Random Forest (RF), Classification and Regression Tree (CART), and Support Vector Machine (SVM), have been widely used in landscape pattern mapping, but their accuracy still needs improvement. With the development of artificial intelligence, deep learning-based methods have achieved significant results in remote sensing image segmentation. Convolutional Neural Network (CNN) methods perform well in mapping complex landscapes, but have limitations in capturing global contextual information. The Transformer architecture, through global self-attention mechanism, models global context and shows superiority in image segmentation tasks. However, the Transformer still has shortcomings in capturing local information. Combining CNN and Transformer methods can fully leverage the advantages of both in extracting local and global information. However, these hybrid methods tend to overlook semantic differences between features when integrating features from different levels, leading to information loss or conflict.

[0005] How to utilize deep learning technology to obtain more local and global information and optimize hybrid structure methods to improve the accuracy of landscape pattern classification is a major challenge in the field of landscape pattern classification at present. Summary of the Invention

[0006] The purpose of this invention is to overcome the aforementioned problems in traditional technologies and provide a landscape pattern classification method that combines global and local multi-level shallow and deep features to improve the accuracy of landscape pattern classification.

[0007] To achieve the above-mentioned technical objectives and effects, the present invention is implemented through the following technical solution:

[0008] This invention provides a landscape pattern classification method that combines global and local multi-level shallow and deep features, comprising the following steps:

[0009] Step 1: Building the Swing Transformer Module

[0010] The Swin Transformer-CNN hybrid model (STCNet) uses four-stage Swin Transformer blocks as the encoder, employing a hierarchical construction method for multi-scale hierarchical feature extraction. A Deep-Shallow Feature Cross-fusion (DSFCF) module is introduced between the encoder and decoder to establish a connection, performing cross-fusion between adjacent features in the encoder output to establish feature complementarity and alleviate semantic confusion caused by the fusion of features from different levels.

[0011] Step 2: Construction of the Deep and Shallow Feature Cross-Fusion Module (DSFCF)

[0012] There is often a semantic gap between deep and shallow features generated in the encoder stage. Simple skip connections can easily lead to semantic confusion when feature fusion. To alleviate this semantic gap, this invention introduces a Deep-Shallow Feature Cross-fusion Module (DSFCF) between the encoder and decoder to establish a connection. Cross-fusion is performed between adjacent features output by the encoder to aggregate multi-scale features and establish complementary feature information between neighboring feature layers, effectively alleviating the semantic confusion problem caused by the fusion of features at different levels.

[0013] Step 3: Construction of the Global-Local Attention Module (GLAB)

[0014] In remote sensing images, landscape patterns and land cover distribution are discrete, detailed information is redundant, and different categories are intertwined, which easily leads to the problem of similar features between categories and differences in features within categories. Integrating global and local features can better perceive the spatial correlation of information in the image and identify the differences in features at multiple scales. Therefore, this invention introduces three Global-Local Attention Modules (GLABs) in the decoder section. The different levels of features output by the DSFCF module are fused with the features of the decoder upsampled and restored resolution and then sent to the Global-Local Attention Modules (GLABs) to obtain a hierarchical representation of features, aggregating global contextual semantic information and local detailed information at different scales.

[0015] Step 4: Model Accuracy Verification and Analysis

[0016] Experiments were conducted using high-resolution image datasets to compare and verify the present invention with other semantic segmentation models, and to perform quantitative evaluation.

[0017] The beneficial effects of this invention are:

[0018] 1. This invention utilizes a convolutional neural network (CNN) and a Swin Transformer to construct a hybrid network model, SwinTransformer-CNN. First, a deep and shallow feature cross-fusion module is introduced between the encoder and decoder to alleviate the semantic gap between features generated by different levels by SwinTransformer. By introducing three global-local attention modules (GLAB) in the decoder part to aggregate global contextual information and local detail information, the recognition of intra-class feature consistency and inter-class feature differences is strengthened.

[0019] 2. This invention fully utilizes the advantages of both CNN and Swin Transformer in extracting local and global information, aggregating global contextual information and local fine-grained information, improving the problem of spectral feature similarity between land cover categories and intra-category differences in complex landscape scenarios, improving the accuracy of landscape pattern classification, and is of great significance for dynamic monitoring of ecosystems. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a diagram illustrating the overall framework of the Swin Transformer-CNN hybrid model.

[0022] Figure 2 Here is a block structure diagram of the Swing Transformer;

[0023] Figure 3 The structure diagram of DSFCF is shown below: (a) and (c) represent the process of two boundary branches intersecting and merging with adjacent feature layers, and (b) represents the process of two intermediate branches intersecting and merging with adjacent feature layers.

[0024] Figure 4 This is a diagram of the GLAB framework.

[0025] Figure 5 This is a comparison chart of the extraction results of the method of this invention with other model methods; Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] This invention provides a deep learning method: The purpose of this invention is to provide a landscape pattern classification method that combines global and local multi-level shallow and deep features, and to design targeted deep learning algorithms to achieve high-precision and high-efficiency classification of landscape patterns under multi-temporal remote sensing imagery. Guided by this idea, as... Figure 1-5 As shown, a landscape pattern classification method combining global and local multi-level shallow and deep features is designed, including the following steps:

[0028] Step 1: Building the Swing Transformer Module

[0029] The standard Transformer block uses a multi-head self-attention (MSA) mechanism to perform global self-attention computation on all tokens in the image, which has high computational complexity. The Swing Transformer block replaces the traditional MSA with window-based Transformer blocks (W-MSA) and sliding window-based Transformer blocks (SW-MSA) to improve computational efficiency. It segments the image into non-overlapping windows, performs self-attention computation within local windows, and establishes information connections between windows through sliding windows, thus improving global feature modeling capabilities. For example... Figure 2 As shown, the Swin Transformer block designed in this invention consists of a window-based Transformer block and a window-sliding Transformer block. The window-based Transformer block includes W-MSA, a Multilayer Perceptron (MLP), and Layer Normalization (LN), while the window-sliding Transformer block also includes SW-MSA, a Multilayer Perceptron (MLP), and Layer Normalization (LN). Both internally apply residual connections. The calculation process of the Swin Transformer block is expressed by the following formula:

[0030] (1)

[0031] (2)

[0032] (3)

[0033] (4)

[0034] in, The input features representing W-MSA and This indicates the output characteristics of W-MSA and SW-MSA. and This represents the output characteristics after passing through LN and MLP.

[0035] An encoder is constructed using four stages of Swin Transformer blocks to progressively extract higher-level abstract features. Specifically, the input remote sensing image is first segmented into non-overlapping blocks of size 4×4. The feature dimension of a flattened block is 4×4×3=48, and the feature dimension of a processed image is H / 4×W / 4×48. A linear embedding layer projects the H / 4×W / 4×48 tensor onto dimension C and then feeds it into the Swin Transformer block, keeping the number of output tokens at H / 4×W / 4. The linear embedding layer and the Swin Transformer block constitute the first stage of the encoder, and the output feature dimension is H / 4×W / 4×C. As the network deepens, a block merge is applied before the Swing Transformer blocks in stages 2, 3, and 4. By merging the blocks, the number of tokens is reduced to 1 / 4 of the original (the feature resolution is reduced to 1 / 2 of the original), which increases the feature dimension of the output. The output feature dimensions of stages 2, 3, and 4, which consist of block merging and Swing Transformer blocks, are H / 8×W / 8×2C, H / 16×W / 16×4C, and H / 32×W / 32×8C, respectively.

[0036] Step 2: Construction of the Deep and Shallow Feature Cross-Fusion Module (DSFCF)

[0037] The feature levels extracted at different stages of the Swin Transformer encoder reflect different feature information. Shallow features have a smaller receptive field and typically provide fine-grained information such as boundaries, shape contours, textures, and positions, while deep features have a larger receptive field and contain more abstract contextual semantic information. During feature fusion, the significant semantic gap between deep and shallow features can lead to some semantic confusion. Therefore, this invention designs a deep-shallow feature cross-module to integrate multi-scale features and reduce the semantic gap.

[0038] First, the feature maps output by the four levels of the Swin Transformer block are aligned through channel alignment, adjusting the dimension and distribution of the feature channels. This allows features from different levels to be effectively fused in the same space and reduces computational overhead, decreasing the number of channels in the four-level feature maps from 128, 256, 512, and 1024 to 32 each. This can be expressed by the formula:

[0039] (5)

[0040] in, For the output characteristics of the encoder, Output features aligned to the CA channel.

[0041] Then, downsampling and upsampling operations are performed on the preceding and following branches of the second and third level branches respectively, converting them into the features of the current branch. Figure 1 To achieve consistent dimensions, upsampling and downsampling operations are performed on the next branch of the first-level branch and the previous branch of the fourth-level branch at the boundary, respectively. This process can be expressed by the formula:

[0042] (6)

[0043] Where i represents the i-th level feature, Down represents the downsampling operation, and Up represents a 2x bilinear interpolation upsampling operation. and These represent the output features of the preceding and following branches of the current branch, aligned via the CA channel. and These represent the output features after downsampling and upsampling operations, respectively. Next, the features from the previous and next branches are concatenated with the features from the current branch to complete cross-fusion. Then, the cross-fused features from each level undergo a 3×3 convolution operation to further extract fine-grained information. Finally, a coordinate attention mechanism with spatial location awareness is used to capture long-range dependencies in space. This process can be expressed by the following formula:

[0044] (7)

[0045] in, Indicates the characteristics of the current branch. These represent the feature maps of the preceding and following layers, respectively. Concat is the feature concatenation operation, CBR is a 3×3 convolution, BN, and ReLU operation, and A represents the coordinate attention mechanism. This represents the result of cross-fusion of features from different levels.

[0046] Step 3: Construction of the Global-Local Attention Module (GLAB)

[0047] A Global-Local Attention (GLAB) module is introduced in the decoder part of the model. GLAB consists of local branches composed of multi-scale convolutions and a global branch based on a window-based multi-head self-attention mechanism, extracting local detail information and global spatial dependency semantic features, respectively. The local branches use convolutional groups with kernel sizes of 5×5, 3×3, and 1×1 to capture spatial and spectral features at different scales, thereby fully extracting fine-grained local features. The formula is as follows:

[0048] (8)

[0049] in, This represents the output characteristics of a local branch.

[0050] In the global branch, a 1×1 convolution is first used to transform the input feature map. To reduce the computational complexity of the self-attention module, the dimension C is increased by three times. A window segmentation operation is performed on the input features, dividing the feature map into several non-overlapping windows. The self-attention mechanism is executed within each window to reduce overall computational overhead while preserving global contextual information. Within each window, the input sequence is... Mapped to respectively A vector can be represented as:

[0051] (9)

[0052] Where X is the representation of the input sequence, , , It is a learnable full-weight matrix. Attention weights are obtained by calculating the dot product between Q and K. The dot product result is normalized using the Softmax function to generate attention weights. Multiplying the attention weights by V yields the score for each attention, which can be represented as:

[0053] (10)

[0054] in, It is the channel dimension of the K vector. This is a scaling factor used to alleviate the vanishing gradient problem. The Q, K, and V vectors are divided into h heads (i.e., different subspaces), and the self-attention of each head is calculated separately. This can be represented as:

[0055] (11)

[0056] in, The output is the projection of Q, K, and V onto the i-th attention head. The results from all attention heads are concatenated, and a linear transformation is applied to obtain the final multi-head self-attention output, which can be expressed as:

[0057]

[0058] in, It is the linear transformation matrix of the science department.

[0059] By fusing global context information obtained from global branches with local features extracted from local branches, a comprehensive representation of multi-scale information is achieved, which can be expressed as:

[0060] (12)

[0061] in, This indicates the result of merging global and local branches.

[0062] GLAB internally uses residual joins, and its process can be represented as follows:

[0063] (13)

[0064] (14)

[0065] Here, GLA represents the global-local attention component, which consists of global and local branches. Represents the input features of GLA and BN. This represents the output characteristics of MLP and BN.

[0066] Step 4: Model Accuracy Verification and Analysis

[0067] The accuracy of the classification results of the method of the present invention is verified using three evaluation indicators: F1, OA, and IoU. The specific formulas are as follows:

[0068] (15)

[0069] (16)

[0070] (17)

[0071] In the formula, TP, FP, FN, and TN represent the number of true positive predictions, false positive predictions, false negative predictions, and true negative predictions, respectively.

[0072] Specifically, this invention provides an implementation example. The Swin Transformer-CNN proposed in this invention, along with seven other semantic segmentation models (DeepLabV3+, SegNet, SegFormer, HRFormer, UNetFormer, BANet, and DCSwin), were experimentally tested on the Shengjinhu high-resolution image dataset. Figure 5Table 1 lists the overall performance of each model and the evaluation indicators for each category.

[0073] On the Shengjinhu high-resolution dataset, Swin Transformer-CNN achieved average IoU, average F1, and average OA of 78.12%, 87.05%, and 93.23%, respectively. In the segmentation results of each category, Swin Transformer-CNN, except for the forest category, had significantly higher F1 scores and mIoU than SegNet.

[0074] This invention employs a deep and shallow feature cross-fusion strategy and a global-local attention mechanism to more effectively improve the extraction of semantic information of the whole and details in the image. Compared with DeepLabV3+, the mIoU is 10.23% higher and the F1 score is 7.89% higher.

[0075] SegFormer, HRFormer, UNetFormer, and DCSwin outperform the previous CNN-based methods in three metrics. However, the segmentation accuracy for different categories is still lower than that of Swin Transformer-CNN. Compared to the suboptimal method DCSwin, the method of this invention has an average F1 score that is 0.46% higher, an mIoU that is 0.57% higher, and an OA that is 0.15% higher.

[0076] Table 1 Comparison of results from different model methods

[0077]

Claims

1. A landscape pattern classification method that combines global and local multi-level shallow and deep features, characterized in that, Includes the following steps: Step 1: Construction of the Swin Transformer module, including: using four stages of Swin Transformer blocks to form an encoder, firstly segmenting the input remote sensing image into non-overlapping blocks, with a block size of 4×4. The feature dimension of a block after flattening is 4×4×3=48, and the feature dimension of a processed image is H / 4×W / 4×48. The H / 4×W / 4×48 tensor is projected onto dimension C using a linear embedding layer, and then fed into the Swin Transformer block. Step 2: Establish a connection between the encoder and decoder by introducing a deep and shallow feature cross-fusion module. This includes: First, aligning the feature maps output by the four-level Swin Transformer blocks through channel alignment to adjust the dimension and distribution of the feature channels, enabling effective fusion of features from different levels in the same space. Then, downsampling and upsampling operations are performed on the preceding and following branches of the second and third level branches to convert them to the same size as the current branch feature map. Upsampling and downsampling operations are also performed on the following branch of the first level branch and the preceding branch of the fourth level branch at the boundary. Next, the features of the preceding and following branches are concatenated with the features of the current branch to complete the cross-fusion. The cross-fused features from each level are then subjected to a 3×3 convolution operation to further extract fine-grained information. Finally, a coordinate attention mechanism with spatial location awareness is used to capture long-distance dependencies in space. Step 3: Introduce a global-local attention module in the decoder part of the model, which consists of a local branch composed of multi-scale convolutions and a global branch based on a window-based multi-head self-attention mechanism, which respectively extract local detail information and global spatial dependency semantic features. Step 4: Model accuracy verification and analysis.

2. The landscape pattern classification method based on combined global-local multi-level shallow and deep features according to claim 1, characterized in that, The Swin Transformer block in step one consists of a window-based Transformer block and a window-sliding Transformer block. The Swin Transformer block uses the window-based Transformer block and the window-sliding Transformer block to perform global self-attention calculation on all tokens of the image, divides the image into non-overlapping windows, performs self-attention calculation within local windows, and establishes information connections between windows by sliding the window. The window-based Transformer block includes W-MSA, multilayer perceptron, and layer normalization, and the window-sliding Transformer block includes SW-MSA, multilayer perceptron, and layer normalization, all of which internally apply residual connections. The calculation process of the Swin Transformer block is expressed by the following formula: （1） （2） （3） （4） in, The input features representing W-MSA and This indicates the output characteristics of W-MSA and SW-MSA. and This represents the output characteristics after passing through LN and MLP.

3. The landscape pattern classification method based on combined global-local multi-level shallow and deep features according to claim 2, characterized in that, Step one also includes: maintaining the number of output tokens at H / 4×W / 4, the linear embedding layer and the SwinTransformer block form the first stage of the encoder, and the output feature dimension is H / 4×W / 4×C. As the network deepens, a block merging is applied before the Swin Transformer block in stages 2, 3, and 4. By merging the block, the number of tokens is reduced to 1 / 4 of the original. The output feature dimensions of stages 2, 3, and 4, which consist of Patch Merging and Swin Transformer Block, are H / 8×W / 8×2C, H / 16×W / 16×4C, and H / 32×W / 32×8C, respectively.

4. The landscape pattern classification method based on combined global-local multi-level shallow and deep features according to claim 3, characterized in that, Step two also includes: reducing the number of channels in the four-level feature maps from 128, 256, 512, and 1024 to 32 respectively; and aligning the feature maps output by the four-level Swin Transformer blocks through a channel alignment operation, using the following formula: （5） in, For the output characteristics of the encoder, Output features aligned to CA channels; For the second and third level branches in the middle, downsampling and upsampling operations are performed on the preceding and following branches, respectively, using the following formulas: （6） Where i represents the i-th level feature, Down represents the downsampling operation, and Up represents a 2x bilinear interpolation upsampling operation. and These represent the output features of the preceding and following branches of the current branch, aligned via the CA channel. and These represent the output features after downsampling and upsampling operations, respectively; Long-distance dependencies within space are captured using a coordinate attention mechanism with spatial location awareness, as shown in the formula: （7） in, Indicates the characteristics of the current branch. These represent the feature maps of the preceding and following layers, respectively. Concat is the feature concatenation operation, CBR is a 3×3 convolution, BN, and ReLU operation, and A represents the coordinate attention mechanism. This represents the result of cross-fusion of features from different levels.

5. The landscape pattern classification method based on combined global-local multi-level shallow and deep features according to claim 4, characterized in that, Step three further includes: using convolutional groups with kernel sizes of 5×5, 3×3, and 1×1 in the local branches to capture spatial and spectral features at different scales, thereby fully extracting local fine-grained features, as shown in the formula: （8） in, This represents the output characteristics of a local branch; In the global branch, a 1×1 convolution is first used to transform the input feature map. To reduce the computational complexity of the self-attention module by tripling the dimension C, a window segmentation operation is performed on the input features, dividing the feature map into several non-overlapping windows. The self-attention mechanism is then executed within each window, reducing overall computational overhead and preserving global contextual information. Within each window, the input sequence is... Mapped to respectively A vector is represented as: （9） Where X is the representation of the input sequence, , , It is a learnable weight matrix. Attention weights are obtained by calculating the dot product between Q and K. The dot product result is normalized using the Softmax function to generate attention weights. Multiplying the attention weights by V yields the score for each attention, expressed as: （10） in, It is the channel dimension of the K vector. This is a scaling factor used to alleviate the vanishing gradient problem. The Q, K, and V vectors are divided into h heads, and the self-attention of each head is calculated, expressed as: （11） in, It is the projection of Q, K, and V onto the i-th attention head. The results of all attention heads are concatenated and a linear transformation is applied to obtain the final multi-head self-attention output, expressed as: ； in, It is the linear transformation matrix of the science department.

6. The landscape pattern classification method based on combined global-local multi-level shallow and deep features according to claim 5, characterized in that, By fusing global context information obtained from global branches with local features extracted from local branches, a comprehensive representation of multi-scale information is achieved, expressed as follows: （12） in, This represents the result of merging global and local branches; The global-local attention module internally applies residual connections, and the process is represented as follows: （13） （14） Here, GLA represents the global-local attention component, which consists of global and local branches. Represents the input features of GLA and BN. This represents the output characteristics of MLP and BN.

7. The landscape pattern classification method based on combined global-local multi-level shallow and deep features according to claim 6, characterized in that, In step four, the accuracy of the classification results is verified using three evaluation metrics: F1, OA, and IoU. The specific formulas are as follows: （15） （16） （17） In the formula, TP, FP, FN, and TN represent the number of true positive predictions, false positive predictions, false negative predictions, and true negative predictions, respectively.

Images