Urban Scene Classification Method Based on High-Resolution Multi-View Remote Sensing Imagery

Abstract

This invention relates to a method for urban scene classification based on high-resolution multi-view remote sensing imagery, belonging to the field of intelligent extraction technology for urban scene information. The method includes: acquiring satellite remote sensing image data, including multispectral and multi-view images, and preprocessing the data; extracting pixel values ​​from the multi-view images to construct multi-angle stereo features; establishing an LCZ sample set for a local climate zone LCZ scene classification system; establishing an MSVTNet urban scene classification model that integrates multi-scale pyramid attention and visual Transformer; training the urban scene classification model, and using the trained model for real-time LCZ scene classification of remote sensing imagery. This invention utilizes multi-view images to extract variations in ground feature height and vertical structure features, effectively supplementing planar features. Simultaneously, the proposed network model can effectively extract and fuse global and local image features, significantly improving scene classification accuracy.

Description

Technical Field

[0001] This invention belongs to the field of intelligent extraction technology of urban scene information, and relates to a method for urban scene classification based on high-resolution multi-view remote sensing images. Background Technology

[0002] Remote sensing image scene classification involves a holistic analysis of all land features within a specific area of ​​an image. By combining the natural attributes and social functional characteristics of the main land features in that area, semantic tags are assigned to achieve a high-level semantic understanding of the remote sensing image. Local Climate Zones (LCZs) are a detailed scenario classification framework proposed for studying urban climate-related issues. Based on building height and density, urban functions, and natural cover, it divides the land surface into 17 categories: LCZ1 (compact high-rise building zone), LCZ2 (compact mid-rise building zone), LCZ3 (compact low-rise building zone), LCZ4 (open high-rise building zone), LCZ5 (open mid-rise building zone), LCZ6 (open low-rise building zone), LCZ7 (lightweight building zone), LCZ8 (large low-rise building zone), LCZ9 (very open low-rise building zone), LCZ10 (heavy industrial zone), LCZA (dense tree zone), LCZB (sparse tree zone), LCZC (shrubland zone), LCZD (low vegetation zone), LCZE (hardened ground zone), LCZF (bare sand zone), and LCZG (water zone). LCZs have promising applications in urban planning and management, disaster risk assessment, air quality and environmental monitoring, and climate change research.

[0003] In LCZ classification, medium-resolution satellite imagery (such as Landsat and Sentinel-2) is commonly used. While it offers advantages in spatial coverage and temporal frequency, its limited spatial resolution, providing only a single-view image, makes it difficult to capture the three-dimensional structural features of ground features. This limitation makes distinguishing LCZ types difficult, especially when differentiating between building types finely categorized based on height and density. In complex urban scenarios, detailed, accurate, and large-scale LCZ mapping remains a significant challenge. Currently, there is no universally accepted solution for effectively improving classification accuracy in such scenarios.

[0004] Existing remote sensing image scene classification methods can be divided into those based on manually designed features and those based on deep features. Methods based on manually designed features mainly rely on extracting features such as color, texture, and shape from the image to provide decision support for classification. These methods perform well when processing images with similar texture structures or simple scenes, but their representational power is often limited when facing complex scenes, affecting classification accuracy. To overcome these shortcomings, deep learning-based methods have gradually become mainstream. Deep learning models can more effectively capture multidimensional information in complex scenes by automatically extracting high-level features. This automated feature extraction method breaks through the limitations of traditional methods and opens up new directions for remote sensing image scene classification. In recent years, various deep learning methods have become research hotspots. Transfer learning-based methods mainly apply models trained on public datasets (such as ImageNet) directly or indirectly to other image classification tasks, but for complex scenes, it is difficult to learn and apply features from other data. Methods that enhance features using attention mechanisms embed attention mechanisms into existing network architectures to strengthen the local semantic information of remote sensing images and enhance the network's ability to extract fine-grained features; however, many studies lack the acquisition and utilization of global information. Scene classification methods based on graph convolutional neural networks establish the spatial contextual dependencies of different targets in remote sensing images, improving the network's adaptability to targets. However, they easily overlook the detailed representation of local information. With the advent of the Transformer, the natural image recognition capabilities exhibited by the visual Transformer have significant potential in remote sensing image scene classification.

[0005] Current remote sensing image scene classification mainly relies on planar features extracted from images from a single viewpoint, ignoring the important role of three-dimensional spatial attributes in extracting urban information; it also lacks detailed feature descriptions of local images for each unique scene, and the overall semantic connections are insufficient, making it difficult to provide effective classification capabilities.

[0006] The three-dimensional features provided by multi-view images can supplement the two-dimensional planar information acquired from a single viewpoint, thereby further improving classification performance. This invention aims to address the problems of existing methods not fully considering stereo attributes, unclear representation of local-global features in images, and low scene classification accuracy. Summary of the Invention

[0007] In view of this, the purpose of this invention is to address the problems in current urban scene classification, such as the lack of consideration for three-dimensional information, unclear representation of local and global features of images, and low model classification accuracy. By utilizing high-resolution multi-view remote sensing images, this invention proposes an efficient and high-precision scene classification method and applies it to the LCZ classification task to verify the effectiveness of the proposed method.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] A method for classifying urban scenes based on high-resolution multi-view remote sensing imagery, comprising the following steps:

[0010] S1. Acquire satellite remote sensing image data, including multispectral and multi-view images, and preprocess the data;

[0011] S2. Extract pixel values ​​from multi-view images to construct multi-angle stereo features;

[0012] S3. Establish an LCZ classification sample set for the preprocessed multispectral and multi-view images;

[0013] S4. Establish MSVTNet, an urban scene classification model that integrates multi-scale pyramid attention and visual Transformer;

[0014] S5. Train the urban scene classification model, and use the trained model for scene classification of real-time remote sensing images.

[0015] S4 includes the following sub-steps:

[0016] S41. Establish multispectral image branch, multi-view image branch and multi-angle feature branch, which are used to process multispectral images, multi-view images and multi-angle features respectively.

[0017] S42. The image input of each branch is weighted by importance through an adaptive selector;

[0018] S43. Input the importance-weighted features into the convolutional branch and the visual Transformer branch respectively for local feature extraction and global feature extraction;

[0019] S44. The feature fusion module uses a two-branch bidirectional fusion strategy to dynamically fuse the local and global features obtained from the two branches.

[0020] S45. Combine the results of local branch features and global branch features to obtain the final classification value.

[0021] Furthermore, in step S1, the acquired multi-view images refer to images including forward-looking, front-view, and back-view perspectives; the preprocessing operations for the acquired remote sensing image data include: orthorectification, image registration, and image fusion.

[0022] Furthermore, in step S2, the extracted pixel values ​​include: Difference Multi-Angle Index (DMAI), Ratio Multi-Angle Index (RMAI), Normalized Multi-Angle Index (NMAI), and Maximum Normalized Multi-Angle Index (MNMAI), the calculation formulas of which are as follows:

[0023]

[0024]

[0025]

[0026]

[0027] in, These represent the maximum value, minimum value, mean, and variance, respectively. This represents the pixel value of the nth viewpoint image.

[0028] Furthermore, in step S3, using Google Earth and Gaode architectural data, samples are selected through visual interpretation and manual annotation, and on-site sampling is conducted to construct an LCZ classification sample set. The constructed LCZ classification sample set is as follows: Compact high-rise building area LCZ1, Compact mid-rise building area LCZ2, Compact low-rise building area LCZ3, Open high-rise building area LCZ4, Open mid-rise building area LCZ5, Open low-rise building area LCZ6, Lightweight building area LCZ7, Large low-rise building area LCZ8, Very open low-rise building area LCZ9, Heavy industrial area LCZ10, Dense tree area LCZA, Sparse tree area LCZB, Shrub area LCZC, Low vegetation area LCZD, Hardened ground area LCZE, Bare sandy soil area LCZF, and Water area LCZG.

[0029] Furthermore, in step S42, the adaptive selector uses the torch.nn.Parameter() method to add learnable adaptive weights to the three sets of inputs, the process of which is as follows:

[0030] The weights of the three branches are initialized based on their contribution and effectiveness in the classification of remote sensing image scenes.

[0031] The initial weights are trained and optimized. During training, the Softmax function is used to normalize the weights, and the specific calculation formula is as follows:

[0032]

[0033] in This represents the initial weight of the i-th branch. It is the normalized weight of the i-th branch;

[0034] The image of each branch is multiplied by its corresponding weight to obtain a weighted input image with assigned importance, which is then passed to different feature extraction branches for subsequent operations.

[0035] Furthermore, in the convolution branch of step S43, the input features are subjected to convolution and pooling operations, and the calculation formula is as follows:

[0036]

[0037] In the formula, For input features, For convolution operations, For batch normalization, For activation function, This is a max pooling operation;

[0038] Then, the convolutional pooled features are input into the Bottleneck module and the MPAM module for multi-scale feature extraction, where,

[0039] In the Bottleneck module, image features first enter a 3x3 convolutional layer Conv, and filters are applied to the input data using a sliding window approach.

[0040] Then, the output of the previous layer is normalized by the batch normalization (BN) layer; then, a ReLU activation function is used to set all negative values ​​to 0, while positive values ​​remain unchanged.

[0041] Next, a 3x3 convolutional layer is used to further extract features, and batch normalization is performed again. At the same time, a 1x1 convolutional layer is used to perform feature transformation without changing the spatial dimension, and batch normalization is performed again.

[0042] Then, an addition operation is performed to add the input to the output after the above convolution and normalization. Finally, the result is passed through a ReLU activation function and then output.

[0043] In the MPAM module, the features output by the Bottleneck module first undergo an adaptive average pooling operation, which calculates the average value by dynamically dividing the region.

[0044] Then, the features obtained in each step are interpolated using bilinear interpolation to align feature maps of different sizes and fuse them at the same scale. The specific calculation formula is as follows:

[0045]

[0046] in, It is the top-left pixel in the input feature map; It is a relative position offset; These are the interpolated pixel values;

[0047] Fourier denoising is added to the first two pooling results, i.e., image-to-frequency domain transformation is performed to filter high-pass noise, transforming the matrix domain into the frequency domain, and the weight matrix of each dimension of the feature block is obtained using a Gaussian filter:

[0048]

[0049] For filtering kernel, Represents a two-dimensional Gaussian function on an image. Let x and y represent the one-dimensional Gaussian functions in the x and y directions, respectively. The weight matrix obtained from the two-dimensional Gaussian function above is summed and averaged. After filtering out low-frequency components, it is inversely converted into the matrix domain output.

[0050] Then all feature results, i.e. multi-scale features, will be concatenated.

[0051] Finally, the concatenated multi-scale feature results are processed through the channel attention mechanism CAM. The CAM module consists of multiple Conv, normalized BN, linear activation ReLU, and sigmoid operations. The specific calculation formula is as follows:

[0052]

[0053] Finally, the Out result is output to the external FC layer, which is then introduced as the bottleneck layer.

[0054] Furthermore, in the visual Transformer branch of step S43, word embedding and position encoding operations are performed on each input image patch to obtain the E vector, calculated as follows:

[0055]

[0056] in, Indicates the first A patch embedding is defined, where PE represents the positional encoding with the same dimension as the input sequence, F is the feature map of the input, and p represents the patch size. These represent the height and width of the entire image, respectively.

[0057] Then E is input into the Transformer Encoder module to obtain global features. The Transformer Encoder module includes a normalization Norm, a multi-head attention mechanism (MLI), and a multilayer perceptron (MLP). The Norm normalizes all input features.

[0058] Multi-head attention mechanisms calculate the relevance of each patch in the entire sequence through self-attention mechanisms, using the following formula:

[0059]

[0060] Where Q, K, and V represent the query vector, key vector, and value vector generated from the input features through different projection matrices, respectively, and T represents the transpose operation. Indicates the dimension of the key vector;

[0061] The multilayer perceptron (MLP) is implemented through a series of linear, ReLU, and dropout operations.

[0062] Finally, the results of the multi-head attention mechanism are passed through the MLP_Head layer, which consists of two linear layers, to obtain the final result cls_token_final.

[0063] Furthermore, in step S44, the feature fusion module includes a self-attention feature fusion mechanism module (SACA), a convolutional branch local feature processing module, and a ViT branch global feature processing module, wherein...

[0064] The SACA module includes a global average pooling module (GAP), a soft selection mechanism, and a dynamic weight generation module (DWG). First, GAP compresses each feature to obtain a feature summary:

[0065]

[0066] in, This represents the feature of the i-th channel in two branches;

[0067] Then we enter the Soft selection mechanism, which uses FC operations and includes multiple Linear layers, ReLU layers, and Sigmoid layers. The implementation formula is as follows:

[0068]

[0069] Then, dynamic weights are generated using the DWG module. The specific formula is as follows:

[0070]

[0071] Final dynamic weights The result is obtained by weighting the input features. :

[0072]

[0073] By dynamically acquiring the different contributions of the branch features of both, the results of the dual-branch fusion strategy are generated in an auxiliary manner.

[0074] In the local feature processing module of the convolution branch, the two-dimensional shape of the convolution branch features is first preserved, and then Conv, Flatten stretching, and fusion operations are performed. The results are calculated as follows:

[0075]

[0076] in and Represents convolutional branch features and ViT branch features. This represents the Conv and Flatten operations in the convolution feature branches. This indicates a feature fusion attention mechanism;

[0077] In the global feature processing module of the ViT branch, a residual fully connected layer Linear operation is performed on the ViT branch features, and the result is calculated as follows:

[0078]

[0079] in and Represents convolutional branch features and ViT branch features. This represents the Conv and Flatten operations for convolutional branch features. This represents the Linear operation of the ViT branch feature residual fully connected layer. This represents a feature fusion attention mechanism.

[0080] Furthermore, in step S45, the calculation method for the fusion result of local branch features and global branch features is as follows:

[0081]

[0082] Finally, the fusion results will be... The final classification value is obtained by performing a fully connected layer (FC) and a softmax function. The fully connected layer (FC) maps the output feature vector to the category space, which helps to obtain the category probability distribution of the subsequent softmax function. The category with the highest probability is taken as the LCZ category to which the image belongs.

[0083] Furthermore, in step S5, the network parameters for training the urban scene classification model include configuration parameters, loss function, and accuracy evaluation metrics. The configuration parameters include batch size, learning rate, and number of iterations.

[0084] The loss function used is the cross-entropy loss function. :

[0085]

[0086] Where N is the number of samples and M is the number of classification categories; This indicates whether the category and sample i are the same; 1 indicates the same and 0 indicates different. This represents the predicted probability that observed sample i belongs to category j;

[0087] Accuracy evaluation metrics include overall accuracy (OA), cartographic accuracy (PA), user accuracy (UA), and harmonic mean score (F1-Score).

[0088] The beneficial effects of this invention are as follows:

[0089] (1) Based on high-resolution multi-view remote sensing imagery, this invention proposes a deep learning network, MSVTNet, that integrates multi-scale pyramid attention (MPAM) and visual transformer (ViT) to fully exploit planar and stereoscopic spatial information for urban scene classification. Experimental results show that this method utilizes multi-view imagery to characterize variations in ground feature height and vertical structural features, effectively supplementing planar features, strengthening inter-class differences, and significantly improving LCZ classification accuracy, thus proving the effectiveness of the proposed method for remote sensing imagery scene classification. In particular, by introducing an adaptive selector, the network can adaptively adjust weights, thereby more effectively utilizing multi-view, multispectral imagery, and multi-angle stereoscopic features.

[0090] (2) The design of the Multi-Scale Pyramid Attention Module (MPAM) in this invention enhances the network's ability to capture image features in local details. Adaptive average pooling allows the network to dynamically adjust the size of the output feature map according to the size of the input data, increasing the flexibility of the model. At the same time, Fourier transform is introduced to achieve high-precision noise suppression. Channel attention mechanism is added to the final fused features to filter out irrelevant or noisy features, making the network more focused on key features and increasing inter-class differences.

[0091] (3) The design of the Visual Transformer (ViT) module in this invention enhances the network's ability to capture image features in a global structure. Based on the representation of image patches, it can effectively capture long-distance dependencies in the image. At the same time, the Transformer has a built-in dense self-attention mechanism that continuously focuses on the most relevant parts of the image, improving the model's ability to capture contextual relationships and global information.

[0092] (4) The dual-branch bidirectional fusion strategy in this invention utilizes a self-attention feature fusion mechanism to dynamically enhance image category-related information, suppress irrelevant information, and strengthen the model's ability to distinguish categories. At the same time, different processing methods are used for the convolutional branch and the ViT branch to enrich feature diversity and provide sufficient features for subsequent classification.

[0093] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0094] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0095] Figure 1 This is a schematic diagram of the structure of MSVTNet, the urban scene classification model of the present invention that integrates multi-scale pyramid attention and visual Transformer;

[0096] Figure 2 This is a schematic diagram of the structure of the Bottleneck module in the convolution branch of the MSVTNet model of the present invention;

[0097] Figure 3 This is a schematic diagram of the MPAM module in the convolution branch of the MSVTNet model of the present invention;

[0098] Figure 4 This is a schematic diagram of the visual Transformer branch of the MSVTNet model of the present invention;

[0099] Figure 5 This is a schematic diagram of the Transformer Encoder module in the visual Transformer branch of the MSVTNet model of the present invention.

[0100] Figure 6 This is a schematic diagram of the structure of the MLP module in the visual Transformer branch of the MSVTNet model of the present invention.

[0101] Figure 7 This is a schematic diagram of the feature fusion module of the MSVTNet model of the present invention;

[0102] Figure 8 This is a schematic diagram of the experimental process in the example embodiment;

[0103] Figure 9 This is a schematic diagram of the experimental results under the example, wherein, Figure 9 (a) is a real image. Figure 9 (b) shows the classification results. Detailed Implementation

[0104] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0105] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0106] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0107] Please see Figures 1-9 This paper proposes an efficient and high-precision scene classification method based on high-resolution multi-view remote sensing imagery. It addresses the problems in current urban scene classification, such as the lack of consideration for 3D information, unclear local-global feature representation, and low model classification accuracy. The method is applied to the LCZ classification task to verify its effectiveness.

[0108] Example

[0109] This embodiment provides a detailed implementation process of the urban scene classification method based on high-resolution multi-view remote sensing imagery of the present invention, which includes the following steps:

[0110] S1. Acquire satellite remote sensing image data, including multispectral and multi-view images, and preprocess the data.

[0111] This embodiment primarily acquires remote sensing image data from the ZY-3 satellite. ZY-3 is one of China's high-resolution stereo mapping satellites, whose main missions are land resource surveys, monitoring, and management, as well as providing updated data for the national basic geographic information database. The acquired multi-view images refer to images from forward-looking, front-looking, and back-looking perspectives. Preprocessing operations on the acquired remote sensing image data include orthorectification, image registration, and image fusion.

[0112] S2. Extract pixel values ​​from multi-view images to construct multi-angle stereo features;

[0113] Specifically, the extracted pixel values ​​include: Difference Multi-Angular Index (DMAI), Ratio Multi-Angular Index (RMAI), Normalized Multi-Angular Index (NMAI), and Maximum Normalized Multi-Angular Index (MNMAI), and their calculation formulas are as follows:

[0114]

[0115]

[0116]

[0117]

[0118] in, These represent the maximum value, minimum value, mean, and variance, respectively. This represents the pixel value of the nth viewpoint image. For the ZY-3 dataset, it includes forward-looking, backward-looking, and front-looking images.

[0119] S3. Establish an LCZ classification sample set for the preprocessed multispectral and multi-view images.

[0120] Specifically, this embodiment utilizes Google Earth and Gaode Maps building data, selects samples through visual interpretation and manual annotation, and ensures sample reliability through on-site sampling to construct an LCZ classification sample set. The constructed LCZ classification sample set consists of 17 categories: LCZ1 (compact high-rise building area), LCZ2 (compact mid-rise building area), LCZ3 (compact low-rise building area), LCZ4 (open high-rise building area), LCZ5 (open mid-rise building area), LCZ6 (open low-rise building area), LCZ7 (lightweight building area), LCZ8 (large low-rise building area), LCZ9 (very open low-rise building area), LCZ10 (heavy industrial area), LCZA (dense tree area), LCZB (sparse tree area), LCZC (shrub area), LCZD (low vegetation area), LCZE (hardened ground area), LCZF (bare sand area), and LCZG (water area).

[0121] S4. Establish MSVTNet, an urban scene classification model that integrates multi-scale pyramid attention and visual Transformer.

[0122] Specifically, such as Figure 1 The overall architecture of the MSVTNet model shown integrates a multi-scale pyramid attention module (MPAM) and a Vision Transformer (ViT) module. Its specific processing steps include the following:

[0123] S41. First, establish a multispectral image branch, a multi-view image branch, and a multi-angle feature branch, which are used to process multispectral images, multi-view images, and multi-angle features, respectively. The multispectral image branch contains data from multispectral sensors and is used to extract image spectral information. The multi-view image branch contains images taken from different perspectives, providing spatial coverage diversity. The multi-angle stereo feature branch contains angular difference variation information extracted from remote sensing images from different perspectives.

[0124] S42. The image input for each branch is first weighted by an adaptive selector. The adaptive selector uses the `torch.nn.Parameter()` method to add learnable adaptive weights to the three sets of inputs, assigning different importance information to the convolution and ViT branches respectively. Previous studies have shown that spectral features are indispensable in scene classification, and extracting multi-angle stereo features using high-resolution multi-view remote sensing images has a significant effect on LCZ classification. Therefore, in remote sensing image scene classification, the initial weights are divided according to the proportion of [0.3, 0.3, 0.4] based on their contribution and effectiveness to the classification. The weights will then be optimized during training.

[0125] During training, the Softmax function is used to normalize the weights, ensuring the sum is 1 and preventing imbalance caused by excessively large weights in any branch. The specific calculation formula is as follows:

[0126]

[0127] in This represents the initial weight of the i-th branch. These are the normalized weights.

[0128] The image of each branch is multiplied by its corresponding weight to obtain a weighted input image with assigned importance, which is then passed to different feature extraction branches for subsequent operations.

[0129] S43. Input the importance-weighted features into the convolution branch and the visual Transformer branch respectively for local feature extraction and global feature extraction.

[0130] Specifically, in the convolution branch, the input features are first subjected to a 7-kernel convolution plus pooling operation, and the specific calculation formula is as follows:

[0131]

[0132] In the formula, For input features, For convolution operations, For batch normalization, For activation function, This is a max pooling operation.

[0133] Then, the Bottleneck+MPAM module is used for multi-scale feature extraction. The two modules are combined, with Bottleneck helping to reduce the amount of computation for extraction and MPAM improving the quality of extracted features, thus ensuring efficient extraction of multi-scale features.

[0134] Figure 2The diagram illustrates the structure of the Bottleneck module, a common network module in deep learning. Combined with ResNet, it generates a Bottleneck module with residuals. The image enters a 3x3 convolutional layer (Conv), where filters are applied to the input data using a sliding window. It then enters a batch normalization (BN) layer to normalize the output of the previous layer, making the output distribution more stable. Next, a ReLU activation function sets all negative values ​​to 0, leaving positive values ​​unchanged, increasing the network's non-linearity to learn more complex features. Following this is another 3x3 convolutional layer to further extract features, followed by another batch normalization. Simultaneously, a 1x1 convolutional layer performs feature transformation without changing the spatial dimensions, and then performs another batch normalization. An addition operation is then performed, adding the input (before the 1x1 convolution) to the output after the convolution and normalization. Finally, the result is passed through a ReLU activation function to add non-linearity to the network.

[0135] Figure 3 The diagram illustrates the structure of the MPAM module. First, it undergoes an AdaptiveAvgPool operation. Unlike the standard AvgPool operation, AdaptiveAvgPool has a fixed output size; the pooling window size and stride are automatically calculated and dynamically adapted based on the user-specified target size. Furthermore, unlike AvgPool, which calculates the average within a fixed window, AdaptiveAvgPool calculates the average by dynamically dividing the region.

[0136] Subsequently, the features obtained in each step are interpolated using bilinear interpolation to align feature maps of different sizes, enabling them to be fused at the same scale. The specific calculation formula is as follows:

[0137]

[0138] in, It is the top-left pixel in the input feature map; It is a relative position offset; These are the interpolated pixel values.

[0139] Additionally, a Fourier denoising (D_FFT) operation is added to the first two pooling results. This involves performing an image-to-frequency domain transformation, filtering out high-pass noise, converting the matrix domain to the frequency domain, and using a Gaussian filter to obtain the weight matrix for each dimension of the feature block.

[0140]

[0141] For filtering kernel, Represents a two-dimensional Gaussian function on an image. Let x and y represent one-dimensional Gaussian functions in the x and y directions, respectively. In many implementations, a two-dimensional Gaussian filter is decomposed into two one-dimensional Gaussian filters. The weight matrix obtained from the two-dimensional Gaussian function above is summed and averaged. After filtering out low-frequency components, it is inversely converted to the matrix domain output.

[0142] Then, all feature results, i.e., multi-scale features, are concatenated. Finally, the concatenated multi-scale feature results are processed through the Channel Attention (CAM) mechanism. The CAM module consists of multiple Conv, normalized BN, linear activation ReLU, and Sigmoid operations. The specific calculation formula is as follows:

[0143]

[0144] Finally, the output is fed to the outer fully connected (FC) layer, which is used as a bottleneck layer. This not only reduces the number of parameters but also forces the model to learn more compact and representative feature representations.

[0145] Then, in the Visual Transformer branch, the structural diagram of the ViT branch module is as follows: Figure 4 As shown, each input image patch undergoes word embedding and position encoding to obtain an E vector, enabling subsequent Transformer processing to capture semantic relationships and contextual information between words. The specific calculation formula is as follows:

[0146]

[0147] in, This represents the embedding of the i-th image patch, where PE represents the positional encoding with the same dimension as the input sequence, F is the feature map of the input, and p represents the patch size. These represent the height and width of the entire image, respectively.

[0148] Then E is fed into the Transformer Encoder to extract global features, such as... Figure 5 The diagram shows the structure of the Transformer Encoder, which mainly consists of a normalization Norm, a multi-head attention mechanism, and a multilayer perceptron (MLP). The Norm normalizes all the input features.

[0149] Multi-Head Attention, through a self-attention mechanism, calculates the relevance of each patch within the entire sequence, capturing long-distance dependencies between feature blocks. The specific calculation formula is as follows:

[0150]

[0151] Where Q, K, and V represent the query vector, key vector, and value vector generated from the input features through different projection matrices, respectively. T represents the transpose operation. Indicates the dimension of the key vector, used for scaling.

[0152] For a multilayer perceptron (MLP), its structure is as follows: Figure 6 As shown, this module is implemented through a series of linear, ReLU, and dropout operations. Finally, the result of the multi-head attention mechanism is passed through the MLP_Head layer, which consists of two linear layers, to obtain the final result cls_token_final.

[0153] S44. The feature fusion module uses a two-branch bidirectional fusion strategy to dynamically fuse the local and global features obtained from the two branches.

[0154] The structural diagram of the feature fusion module is shown in Figure 7. It includes a self-attention convolutional adaptive fusion (SACA) module, a local feature processing module for the convolutional branch, and a global feature processing module for the ViT branch. The SACA module includes a global average pooling (GAP) module, a soft selection mechanism, and a dynamic weight generation (DWG) module.

[0155] First, GAP compresses each feature to obtain a feature summary:

[0156]

[0157] in, This represents the feature of the i-th channel in the two branches.

[0158] Then comes the Soft selection mechanism, which allows the model to smoothly select or combine different components or paths by assigning weights to reflect the importance of different options. When applied to feature channels, it can dynamically emphasize certain channels while suppressing others, thereby improving the model's performance.

[0159] The Soft selection mechanism used in this embodiment, namely the FC operation, includes multiple Linear layers, ReLU layers, and Sigmoid layers. The specific implementation formula is as follows:

[0160]

[0161] Then, dynamic weights are generated using the DWG module. The specific formula is as follows:

[0162]

[0163] Final dynamic weights The result is obtained by weighting the input features. :

[0164]

[0165] This method can dynamically obtain the different contributions of the two branch features, and its addition can more effectively assist in the generation of results of the dual-branch fusion strategy.

[0166] The local feature processing module for convolutional branches first preserves the two-dimensional shape of the convolutional branch features, then performs Conv, Flatten stretching, and fusion operations. The results are calculated as follows:

[0167]

[0168] in and Represents convolutional branch features and ViT branch features. This represents the Conv and Flatten operations in the convolution feature branches. This represents a feature fusion attention mechanism.

[0169] The ViT branch global feature processing module performs residual fully connected layer Linear operations on the ViT branch features, and the results are calculated as follows:

[0170]

[0171] in and Represents convolutional branch features and ViT branch features. This represents the Conv and Flatten operations for convolutional branch features. This represents the Linear operation of the ViT branch feature residual fully connected layer. This represents a feature fusion attention mechanism.

[0172] S45. Then, the results of the local branch features and the global branch features are combined to obtain the final classification value; the fusion result is calculated as follows:

[0173]

[0174] Finally, the fusion results will be... The final classification value is obtained by performing a fully connected layer (FC) and a softmax function. The role of FC is to map the previously output feature vector to the category space, which helps to obtain the category probability distribution of the subsequent softmax function. The category with the highest probability is taken as the LCZ category to which the image belongs.

[0175] S5. Train the MSVTNet urban scene classification model, and use the trained model for scene classification of real-time remote sensing images.

[0176] Network parameter settings include various parameters, loss functions, and accuracy evaluation metrics that need to be adjusted and configured when training a neural network.

[0177] Various parameters include: batch size of 32, cross-entropy loss function, Adam optimizer on the dataset, and learning rate of 2*10. -4 The number of iterations is 300 rounds.

[0178] The loss function used is the cross-entropy loss function. (Cross-Entropy Loss), the calculation formula is as follows:

[0179]

[0180] Where N is the number of samples and M is the number of classification categories; This indicates whether the category and sample i are the same; 1 indicates the same and 0 indicates different. This represents the predicted probability that observed sample i belongs to category j.

[0181] Accuracy evaluation includes calculating accuracy metrics by comparing the consistency between the model's predictions and actual values ​​using a confusion matrix. These accuracy metrics include overall accuracy (OA), cartographic accuracy (PA), user accuracy (UA), and harmonic mean score (F1-Score), calculated using the following formulas:

[0182]

[0183]

[0184]

[0185]

[0186] Where P is the total number of samples. , , It is the value in the i-th row and j-th column of the confusion matrix.

[0187] This embodiment trains a classification model under the parameter conditions with the best accuracy evaluation results, performs LCZ classification result prediction on the image, and obtains the final city LCZ classification mapping result.

[0188] This embodiment uses the LCZ classification dataset from Resource 3, which contains 16,309 125*125m sample images covering five typical cities in China, to conduct experiments on the above process. The experimental procedure is as follows: Figure 8 As shown, the result is as follows Figure 9 The example of LCZ classification results shown below, where, Figure 9 (a) is a real image. Figure 9 (b) shows the classification results.

[0189] Existing research has achieved a maximum overall accuracy (OA) of 82.7% on the Resource 3 dataset. The method proposed in this invention improves the OA to 86.06%, verifying the effectiveness of MSVTNet for LCZ classification.

[0190] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for classifying urban scenes based on high-resolution multi-view remote sensing imagery, characterized in that: The method includes the following steps: S1. Acquire satellite remote sensing image data, including multispectral and multi-view images, and preprocess the data; S2. Extract pixel values ​​from multi-view images to construct multi-angle stereo features; S3. Establish an LCZ classification sample set for the preprocessed multispectral and multi-view images; S4. Establish MSVTNet, an urban scene classification model that integrates multi-scale pyramid attention and visual Transformer; S5. Train the urban scene classification model, and use the trained model for scene classification of real-time remote sensing images. S4 includes the following sub-steps: S41. Establish multispectral image branch, multi-view image branch and multi-angle feature branch, which are used to process multispectral images, multi-view images and multi-angle features respectively. S42. The image input of each branch is weighted by importance through an adaptive selector; S43. The importance-weighted features are input into the convolution branch and the visual Transformer branch respectively for local feature extraction and global feature extraction; the input convolution branch includes a multi-scale pyramid attention module, which includes adaptive average pooling operation, interpolation operation, Fourier denoising operation, feature concatenation and channel attention mechanism CAM; S44. The feature fusion module uses a two-branch bidirectional fusion strategy to dynamically fuse the local and global features obtained from the two branches. The feature fusion module includes a self-attention feature fusion mechanism module (SACA), a convolutional branch local feature processing module, and a ViT branch global feature processing module. S45. Combine the results of local branch features and global branch features to obtain the final classification value.

2. The urban scene classification method based on high-resolution multi-view remote sensing imagery according to claim 1, characterized in that: In step S1, the acquired multi-view images refer to images including forward-looking, front-view, and back-view perspectives; the preprocessing operations for the acquired remote sensing image data include: orthorectification, image registration, and image fusion.

3. The urban scene classification method based on high-resolution multi-view remote sensing imagery according to claim 1, characterized in that: In step S2, the extracted pixel values ​​include: Difference Multi-Angle Index (DMAI), Ratio Multi-Angle Index (RMAI), Normalized Multi-Angle Index (NMAI), and Maximum Normalized Multi-Angle Index (MNMAI), the calculation formulas of which are as follows: in, These represent the maximum value, minimum value, mean, and variance, respectively. This represents the pixel value of the nth viewpoint image.

4. The urban scene classification method based on high-resolution multi-view remote sensing imagery according to claim 1, characterized in that: In step S3, using Google Earth and Gaode building data, samples are selected through visual interpretation and manual annotation, and on-site sampling is conducted to construct an LCZ classification sample set. The constructed LCZ classification sample set is as follows: Compact high-rise building area LCZ1, Compact mid-rise building area LCZ2, Compact low-rise building area LCZ3, Open high-rise building area LCZ4, Open mid-rise building area LCZ5, Open low-rise building area LCZ6, Lightweight building area LCZ7, Large low-rise building area LCZ8, Very open low-rise building area LCZ9, Heavy industrial area LCZ10, Dense tree area LCZA, Sparse tree area LCZB, Shrub area LCZC, Low vegetation area LCZD, Hardened ground area LCZE, Bare sandy soil area LCZF, and Water area LCZG.

5. The urban scene classification method based on high-resolution multi-view remote sensing imagery according to claim 1, characterized in that: In step S42, the adaptive selector uses the torch.nn.Parameter() method to add learnable adaptive weights to the three sets of inputs. The process is as follows: The weights of the three branches are initialized based on their contribution and effectiveness in the classification of remote sensing image scenes. The initial weights are trained and optimized. During training, the Softmax function is used to normalize the weights, and the specific calculation formula is as follows: in This represents the initial weight of the i-th branch. It is the normalized weight of the i-th branch; The image of each branch is multiplied by its corresponding weight to obtain a weighted input image with assigned importance, which is then passed to different feature extraction branches for subsequent operations.

6. The urban scene classification method based on high-resolution multi-view remote sensing imagery according to claim 5, characterized in that: In the convolution branch of step S43, the input features are subjected to convolution and pooling operations, and the calculation formula is as follows: In the formula, For input features, For convolution operations, For batch normalization, For activation function, This is a max pooling operation; Then, the convolutional pooled features are input into the Bottleneck module and the multi-scale pyramid attention module for multi-scale feature extraction, where, In the Bottleneck module, image features first enter a 3x3 convolutional layer Conv, and filters are applied to the input data using a sliding window approach. Then, the output of the previous layer is normalized by the batch normalization (BN) layer; then, a ReLU activation function is used to set all negative values ​​to 0, while positive values ​​remain unchanged. Next, a 3x3 convolutional layer is used to further extract features, and batch normalization is performed again. At the same time, a 1x1 convolutional layer is used to perform feature transformation without changing the spatial dimension, and batch normalization is performed again. Then, an addition operation is performed to add the input to the output after the above convolution and normalization. Finally, the result is passed through a ReLU activation function and then output. In the multi-scale pyramid attention module, the features output by the Bottleneck module first undergo an adaptive average pooling operation, which calculates the average value by dynamically dividing the region. Then, the features obtained in each step are interpolated using bilinear interpolation to align feature maps of different sizes and fuse them at the same scale. The specific calculation formula is as follows: in, It is the top-left pixel in the input feature map; It is a relative position offset; These are the interpolated pixel values; Fourier denoising is added to the first two pooling results, i.e., image-to-frequency domain transformation is performed to filter high-pass noise, transforming the matrix domain into the frequency domain, and the weight matrix of each dimension of the feature block is obtained using a Gaussian filter: For filtering kernel, Represents a two-dimensional Gaussian function on an image. Let x and y represent the one-dimensional Gaussian functions in the x and y directions, respectively. The weight matrix obtained from the two-dimensional Gaussian function above is summed and averaged. After filtering out low-frequency components, it is inversely converted into the matrix domain output. Then all feature results, i.e. multi-scale features, will be concatenated. Finally, the concatenated multi-scale feature results are processed through the channel attention mechanism CAM. The CAM module consists of multiple Conv, normalized BN, linear activation ReLU, and sigmoid operations. The specific calculation formula is as follows: Finally, the Out result is output to the external FC layer, which is then introduced as the bottleneck layer.

7. The urban scene classification method based on high-resolution multi-view remote sensing imagery according to claim 6, characterized in that: In the visual Transformer branch of step S43, word embedding and position encoding are performed on each input image patch to obtain the E vector, calculated as follows: in, Indicates the first A patch embedding is defined, where PE represents the positional encoding with the same dimension as the input sequence, F is the feature map of the input, and p represents the patch size. These represent the height and width of the entire image, respectively. Then E is input into the Transformer Encoder module to obtain global features. The Transformer Encoder module includes a normalization Norm, a multi-head attention mechanism (MLP), and a multilayer perceptron (MLP). The Norm normalizes all input features. Multi-head attention mechanisms calculate the relevance of each patch in the entire sequence through self-attention mechanisms, using the following formula: Where Q, K, and V represent the query vector, key vector, and value vector generated from the input features through different projection matrices, respectively, and T represents the transpose operation. Indicates the dimension of the key vector; The multilayer perceptron (MLP) is implemented through a series of linear, ReLU, and dropout operations. Finally, the results of the multi-head attention mechanism are passed through the MLP_Head layer, which consists of two linear layers, to obtain the final result cls_token_final.

8. The urban scene classification method based on high-resolution multi-view remote sensing imagery according to claim 7, characterized in that: In step S44, the SACA module includes a global average pooling module (GAP), a soft selection mechanism, and a dynamic weight generation module (DWG). First, GAP compresses each feature to obtain a feature summary: in, This represents the feature of the i-th channel in the two branches; Then we enter the Soft selection mechanism, which uses FC operations and includes multiple Linear layers, ReLU layers, and Sigmoid layers. The implementation formula is as follows: Then, dynamic weights are generated using the DWG module. The specific formula is as follows: Final dynamic weight The result is obtained by weighting the input features. : By dynamically acquiring the different contributions of the branch features of both, the results of the dual-branch fusion strategy are generated in an auxiliary manner. In the local feature processing module of the convolution branch, the two-dimensional shape of the convolution branch features is first preserved, and then Conv, Flatten stretching, and fusion operations are performed. The results are calculated as follows: in and Represents convolutional branch features and ViT branch features. This represents the Conv and Flatten operations in the convolution feature branches. This indicates a feature fusion attention mechanism; In the global feature processing module of the ViT branch, a residual fully connected layer Linear operation is performed on the ViT branch features, and the result is calculated as follows: in and Represents convolutional branch features and ViT branch features. This represents the Linear operation of the ViT branch feature residual fully connected layer. This represents a feature fusion attention mechanism.

9. A method for classifying urban scenes based on high-resolution multi-view remote sensing imagery according to claim 8, characterized in that: In step S45, the calculation method for the fusion result of local branch features and global branch features is as follows: Finally, the fusion results will be... The final classification value is obtained by performing a fully connected layer (FC) and a softmax function. The fully connected layer (FC) maps the output feature vector to the category space, which helps to obtain the category probability distribution of the subsequent softmax function. The category with the highest probability is taken as the LCZ category to which the image belongs.

10. A method for classifying urban scenes based on high-resolution multi-view remote sensing imagery according to claim 1, characterized in that: In step S5, the network parameters for training the urban scene classification model include configuration parameters, loss function, and accuracy evaluation metrics. The configuration parameters include batch size, learning rate, and number of iterations. The loss function used is the cross-entropy loss function. : Where N is the number of samples and M is the number of classification categories; This indicates whether the category and sample i are the same; 1 indicates the same and 0 indicates different. This represents the predicted probability that observed sample i belongs to category j; Accuracy evaluation metrics include overall accuracy (OA), cartographic accuracy (PA), user accuracy (UA), and harmonic mean score (F1-Score).

Images