A local climate zone classification method based on multi-source data fusion

Abstract

The present invention relates to the field of remote sensing image processing technology, and specifically to a local climate zone classification method based on multi-source data fusion, comprising: building a multi-source feature extraction and fusion network, including two self-attention calculation branches and a cross-attention fusion module; the two self-attention calculation branches each include a series of self-attention modules, respectively performing different levels of feature extraction on multispectral remote sensing image features and SAR remote sensing image features; the cross-attention fusion module accepts different levels of heterogeneous data features extracted by the two self-attention calculation branches and fuses them; building a local climate zone classification model based on the multi-source feature extraction and fusion network, and performing training and testing; and classifying multi-source remote sensing images based on the local climate zone classification model to obtain local climate zone classification results. The present invention can more accurately capture the semantic associations between multi-source data to more accurately reflect urban microclimate characteristics.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing, and more particularly to a local climate zone classification method based on multi-source data fusion. Background Art

[0002] Local Climate Zone (LCZ) classification provides a globally applicable method for understanding and assessing the microclimate characteristics of cities and their surrounding areas, contributing to urban planning and environmental science research. By providing a standardized urban environmental classification, LCZ remote sensing image classification enables researchers to compare the climate characteristics of different cities, while helping planners design suitable living environments to effectively manage the urban heat island effect and improve energy efficiency. Multi-source remote sensing data has complementary and multi-dimensional feature information. For example, Synthetic Aperture Radar (SAR) has the characteristics of cloud penetration and all-weather coverage, while Multispectral Remote Sensing Imagery (MSI) has rich spectral information and surface material identification capabilities. Therefore, making full use of multi-source remote sensing data is an effective way to improve the accuracy of LCZ classification. Although current convolutional neural networks (CNNs) are good at extracting image features, they have difficulty fully capturing the associations between different data sources, and thus cannot extract more comprehensive semantic information from multi-source data. Therefore, using more appropriate technologies to capture the semantic associations between multi-source data can not only improve the performance of the LCZ classification model in complex environments, but also more accurately reflect the characteristics of urban microclimates, thereby improving the scientificity and effectiveness of urban planning and environmental management. Summary of the Invention

[0003] In view of this, the present invention provides a local climate zone classification method based on multi-source data fusion, which can more accurately capture the semantic associations between multi-source data to more accurately reflect the characteristics of urban microclimate.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions:

[0005] A local climate zone classification method based on multi-source data fusion includes the following steps:

[0006] Build a multi-source feature extraction and fusion network; the multi-source feature extraction and fusion network includes two self-attention calculation branches and a cross-attention fusion module; the two self-attention calculation branches each include a series of self-attention modules, which respectively perform feature extraction at different levels on multispectral remote sensing image features and SAR remote sensing image features; the cross-attention fusion module includes multiple cross-attention modules, and the multiple cross-attention modules respectively accept different levels of heterogeneous data features extracted by the two self-attention calculation branches to perform feature fusion, and the same cross-attention module accepts paired heterogeneous data features from the same level;

[0007] Building a local climate zone classification model based on the multi-source feature extraction and fusion network;

[0008] The local climate zone classification model is trained and tested using a dataset consisting of multispectral remote sensing images and SAR remote sensing images;

[0009] The multi-source remote sensing images are classified based on the trained and tested local climate zone classification model to obtain the local climate zone classification results.

[0010] Furthermore, the calculation process of the self-attention calculation branch on the multispectral remote sensing image features or the SAR remote sensing image features includes:

[0011] The first self-attention module in the self-attention calculation branch receives the embedded features X of the multispectral remote sensing image or the SAR remote sensing image E As input, and perform layer normalization operation;

[0012] Input the normalized vector into the multi-head attention mechanism for attention calculation;

[0013] Output and embedding features X of the multi-head attention mechanism E After performing residual connection and layer normalization, the first-level output of the self-attention module is obtained; after passing the first-level output through a multi-layer perceptron, the second-level output of the self-attention module is obtained; the first-level output and the second-level output are residually connected and then layer normalization is performed to obtain the final output of the self-attention module;

[0014] The final output of the self-attention module is used as the input of the next self-attention module or the input of the cross-attention module at the corresponding level.

[0015] Furthermore, the calculation process of the multi-head attention mechanism is expressed as:

[0016] MultiHead(X E )=COncat(head1,head2,…,headh )W O

[0017]

[0018] Among them, MultiHead(X E ) represents the output of the multi-head attention mechanism, head i represents the output of the i-th head, Q i represents the query vector of the i-th head, K i represents the key vector of the i-th head, V i represents the value vector of the i-th head, d k represents the dimension of the key vector, W O is a learnable weight matrix, and Concat represents a concatenation operation.

[0019] Furthermore, the calculation process of the first self-attention module is expressed as:

[0020] Output=LayerNorm(X E +MultiHead(X E ))

[0021] SelfAttentionOutput=LayerNorm(Output+MLP(Output))

[0022] Among them, LayerNorm represents the layer normalization operation, MultiHead(X E ) represents the output of the multi-head attention mechanism, MLP represents the multi-layer perceptron, and SelfAttentionOutput represents the final output of the self-attention module;

[0023] The complete calculation process of the self-attention module is expressed as:

[0024] SAO=SAM(X E )

[0025] Among them, SAM represents the calculation of the complete self-attention module, X E Represents the embedded features of the input image; SAO = SelfAttentionOutput, represents the output of the self-attention module, which has the same shape as the input.

[0026] Furthermore, the cross attention module includes two computational branches CAM SAR With CAM MSI , where the two computational branches CAMM SAR With CAM MSI The calculation process is similar to the self-attention module, except that:

[0027] Compute Branch CAM SAR Key vector using SAR remote sensing image features With value vector And use the query vector of multispectral remote sensing image features The calculation process is expressed as:

[0028]

[0029] Compute Branch CAM MSI Using multispectral remote sensing images, the key vector is also positive With value vector And use the query vector of SAR remote sensing image features The calculation process is expressed as:

[0030]

[0031] After the two computation branches perform multi-head attention calculations, they perform the same computation process as the self-attention module, and their outputs are expressed as follows:

[0032]

[0033] Among them, CAO SAR and CAO MSI Represents the calculation branch CAM SAR With CAM MSI The output, and Represents the final output of the two self-attention calculation branches after passing through j self-attention modules, and j represents the number of self-attention modules passed.

[0034] Furthermore, for each cross-attention module, the two computation branches CAM SAR With CAM MSI The output of is connected to obtain the final output CAO of the cross attention module, which is expressed as:

[0035] CAO=CAO SAR +CAO MSI .

[0036] Furthermore, the cross-attention fusion module includes four cross-attention modules, which respectively accept paired heterogeneous data features that have passed through 2, 4, 10, and 12 self-attention modules, expressed as:

[0037]

[0038] Among them, CAM represents the complete calculation process of a cross attention module. represents the SAR data features calculated by j self-attention modules, Represents the multispectral data features calculated by j self-attention modules;

[0039] After all cross-attention modules are calculated, the four outputs are concatenated and the feature dimension is adjusted through a convolutional layer. The specific calculation formula is expressed as:

[0040] CAFO=Conv 1×1 (Concat(CAO1,CAO2,CAO3,CAO4))

[0041] Among them, CAFO represents the final output of the cross attention fusion module, Concat represents the splicing operation, Conv 1×1 Represents a 1×1 convolution operation.

[0042] Furthermore, the final output of the SAR remote sensing image features after calculation by a series of self-attention modules is The final output of multispectral remote sensing image features after calculation by a series of self-attention modules The final output calculated by the cross-attention fusion module is concatenated, and then passed through a convolutional layer for adjusting the dimension and integrating the semantics. The result is used as the final multi-source fusion feature of the multi-source feature extraction and fusion network. The specific calculation process is expressed as follows:

[0043]

[0044] Among them, MSF represents the multi-source fusion feature finally output by the multi-source feature extraction and fusion network.

[0045] Furthermore, the local climate zone classification model includes a data feature preprocessing network, the multi-source feature extraction and fusion network, and a classification network;

[0046] The data feature preprocessing network receives SAR images and multispectral images, performs image segmentation and linear embedding on the two images, and obtains the embedding features of the SAR remote sensing image. and embedded features of multispectral remote sensing images

[0047] The multi-source feature extraction and fusion network embeds features of the two images and Perform self-attention calculation and fusion to obtain multi-source fusion feature MSF;

[0048] The classification network sequentially passes the multi-source fusion feature MSF through a linear rectification function ReLU, a 1×1 convolution operation and a softmax function, and outputs the predicted probability of each local climate zone category.

[0049] Furthermore, when training the local climate zone classification model, the loss function adopted is a cross entropy loss function.

[0050] It can be seen from the above technical solutions that compared with the prior art, the present invention has the following beneficial effects:

[0051] The proposed local climate zone classification model based on multi-source remote sensing data fusion leverages the attention mechanism to not only capture global semantic associations within the same data source but also capture semantic associations across data sources by calculating attention weights between different data sources. This enables the model to more comprehensively extract multi-source data features and understand image content when performing local climate zone classification, thereby achieving accurate LCZ classification in remote sensing images. This entire process requires no human intervention, resulting in higher prediction efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0053] Figure 1 The calculation flow chart of the multi-head attention mechanism provided by the present invention;

[0054] Figure 2 A schematic diagram of the structure of the self-attention module provided by the present invention;

[0055] Figure 3 A schematic diagram of the structure of the cross attention module provided by the present invention;

[0056] Figure 4 A schematic diagram of the structure of the cross attention fusion module provided by the present invention;

[0057] Figure 5 This is a schematic structural diagram of the local climate zone classification model provided by the present invention. DETAILED DESCRIPTION

[0058] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0059] The embodiment of the present invention discloses a local climate zone classification method based on multi-source data fusion, comprising the following steps:

[0060] Build a multi-source feature extraction and fusion network; the multi-source feature extraction and fusion network includes two self-attention calculation branches and a cross-attention fusion module; the two self-attention calculation branches each include a series of self-attention modules, which perform feature extraction at different levels on multispectral remote sensing image features and SAR remote sensing image features respectively; the cross-attention fusion module includes multiple cross-attention modules, which each accepts heterogeneous data features at different levels extracted by the two self-attention calculation branches and performs feature fusion, and the same cross-attention module accepts paired heterogeneous data features from the same level;

[0061] Building a local climate zone classification model based on multi-source feature extraction and fusion network;

[0062] The local climate zone classification model was trained and tested using a dataset consisting of multispectral remote sensing images and SAR remote sensing images.

[0063] Based on the trained and tested local climate zone classification model, multi-source remote sensing images are classified to obtain the local climate zone classification results.

[0064] Next, the construction process of the local climate zone classification model is further explained.

[0065] (1) Build a self-attention module.

[0066] For SAR remote sensing images and multispectral images, two self-attention calculation branches are designed, and each self-attention calculation branch contains a series of self-attention modules (SAMs). The self-attention modules are used to calculate the global features and long-distance dependencies of the same data, and their input is only one data source.

[0067] like Figure 2 The following is a schematic diagram of the structure of a single self-attention module.

[0068] The calculation process of the self-attention calculation branch for multispectral remote sensing image features or SAR remote sensing image features includes:

[0069] 1) The first self-attention module in the self-attention calculation branch receives the embedded features X of the multispectral remote sensing image or SAR remote sensing image E As input, and perform layer normalization operation to obtain X E =LayerNorm(X E );

[0070] 2) Input the normalized vector into the multi-head attention mechanism for attention calculation; Figure 1 As shown in the figure, in the multi-head attention mechanism, each "head" represents an independent attention calculation process, and multiple heads are processed in parallel. This parallelization allows the model to extract input features from different perspectives and ultimately integrate information from these different perspectives to form a richer and more comprehensive output.

[0071] For each head i, the input vector X E Through three independent fully connected layers (Linear layers), they are mapped into query vector (Query, Q), key vector (Key, K), and value vector (Value, V). The specific calculation formula is as follows:

[0072]

[0073] in, Represents the independent learnable weight matrix in each head, and i represents a different head. In the attention mechanism, the attention score is obtained by calculating the dot product of Q and K, scaling it, and normalizing it. The normalized attention score is then multiplied by the corresponding V to complete further calculations. The entire attention calculation process can be formally expressed as follows:

[0074]

[0075] Among them, d k Represents the dimension of the key vector. This scaling operation helps avoid producing too large values ​​before the softmax operation. The softmax function is a normalized exponential function that can be used to convert the attention score of each row into a probability distribution, which represents the degree of attention each output element pays to each element in the input sequence. According to the calculated probability distribution, the value vector is weighted and summed to generate the final output of the self-attention mechanism. This step can be understood as extracting and combining key feature information from the input elements according to the calculated attention weights. Subsequently, the outputs of all heads are spliced ​​and transformed through a linear layer to integrate the information extracted by different heads. The calculation of this process can be formally expressed as follows:

[0076] MultiHead(X E)=Concat(head1,head2,…,head h )W O

[0077] Among them, MultiHead(X E ) represents the output of the multi-head attention mechanism, head i represents the output of the i-th head, Q i represents the query vector of the i-th head, K i represents the key vector of the i-th head, V i represents the value vector of the i-th head, d k represents the dimension of the key vector, W O is a learnable weight matrix, and Concat represents a concatenation operation.

[0078] 3) Output and embedding features X of the multi-head attention mechanism E After performing residual connection and layer normalization, the first-level output of the self-attention module is obtained; after passing the first-level output through a multi-layer perceptron, the second-level output of the self-attention module is obtained; the first-level output and the second-level output are residually connected and then layer normalization is performed to obtain the final output of the self-attention module;

[0079] The calculation process of the first self-attention module is expressed as:

[0080] Output=LayerNorm(X E +MultiHead(X E ))

[0081] SelfAttentionOutput=LayerNorm(Output+MLP(Output))

[0082] Among them, LayerNorm represents the layer normalization operation, MultiHead(X E ) represents the output of the multi-head attention mechanism, MLP represents the multi-layer perceptron, and SelfAttentionOutput represents the final output of the self-attention module.

[0083] At this point, the self-attention calculation process is described and can be summarized as follows:

[0084] SAO=SAM(X E )

[0085] Among them, SAM represents the calculation of the complete self-attention module, X ERepresents the input image embedding vector, SAO = SelfAttentionOutput represents the output of the self-attention module, which has the same shape as the input and serves as the input of the next self-attention module or the input of the cross-attention module of the corresponding level.

[0086] The calculation process of each subsequent self-attention module is the same as that of the first self-attention module. Different data sources have independent self-attention calculation branches. For example, the output of the self-attention calculation of SAR data can be expressed as The calculation result of the self-attention calculation branch of multispectral data can be expressed as Where j represents the number of self-attention modules passed.

[0087] (2) Construct a cross-attention module.

[0088] The calculation of the Cross-attention Module (CAM) is to calculate the cross-source fusion features and the feature dependencies between two heterogeneous data. Therefore, its input is two heterogeneous data. Here, the data sources are SAR remote sensing images and multispectral remote sensing images. In this module structure, the cross-attention module takes the output of the self-attention module as input, which is and

[0089] like Figure 3 As shown, the cross attention module includes two computational branches CAM similar to the self-attention module SAM SAR With CAM MSI , where two computation branches CAM SAR With CAM MSI The calculation process is similar to that of the self-attention module, except that:

[0090] Compute Branch CAM SAR Key vector using SAR remote sensing image features With value vector And use the query vector of multispectral remote sensing image features The calculation process is expressed as:

[0091]

[0092] Compute Branch CAM MSI Using multispectral remote sensing images, the key vector is also positive With value vector And use the query vector of SAR remote sensing image features The calculation process is expressed as:

[0093]

[0094] After the two computation branches perform multi-head attention calculations, they perform the same calculation process as the self-attention module, and their outputs are expressed as follows:

[0095]

[0096] Among them, CAO SAR and CAO MSI Represents the calculation branch CAM SAR With CAM MSI The output, and Represents the final output of the two self-attention calculation branches after passing through j self-attention modules, and j represents the number of self-attention modules passed.

[0097] Finally, for each crisscross attention module, the two computation branches CAM SAR With CAM MSI The output of the cross-attention module is connected to obtain the final output CAO (Cross-attention Output) of the cross-attention module, which is expressed as:

[0098] CAO=CAO SAR +CAO MSI .

[0099] In summary, the entire calculation of the cross-attention module can be expressed as:

[0100]

[0101] Among them, CAM represents the complete calculation process of the cross attention module, which includes CAM SAR Branch and CAM MSI At this point, through the cross-attention calculation of SAR data and multispectral data, this module realizes the effective fusion of multi-source data.

[0102] (3) Build a cross-attention fusion module.

[0103] like Figure 4 As shown in the figure, the cross-attention fusion module includes four cross-attention modules to perform cross-attention calculations on SAR and multispectral features. The four cross-attention modules accept features from different levels as input, and the same cross-attention module accepts paired heterogeneous data features from the same level. Specifically, the cross-attention module accepts paired heterogeneous data features that have passed through 2, 4, 10, and 12 self-attention modules, respectively, which are expressed as:

[0104]

[0105] Among them, CAM represents the complete calculation process of a cross attention module. represents the SAR data features calculated by j self-attention modules, Represents the multispectral data features calculated by j self-attention modules; and They represent the SAR and multispectral data features calculated by 10 self-attention modules respectively. The subscript of CAO represents the sequence number of the cross-attention module, such as CAO3 represents the output of the third cross-attention module.

[0106] After all cross-attention modules are calculated, the four outputs are concatenated and the feature dimension is adjusted through a convolutional layer. The specific calculation formula is expressed as:

[0107] CAFO=Conv 1×1 (Concat(CAO1,CAO2,CAO3,CAO4))

[0108] Among them, CAFO (Cross-attention Fusion Output) represents the final output of the cross-attention fusion module, Concat represents the splicing operation, Conv 1×1 Represents a 1×1 convolution operation.

[0109] (4) Build a multi-source feature extraction and fusion network.

[0110] like Figure 5 As shown in the figure, the multi-source feature extraction and fusion network includes a series of continuous self-attention modules for extracting features from SAR data and multispectral data respectively, and a cross-attention fusion module for fusing the heterogeneous features extracted by the self-attention modules.

[0111] For SAR data, 12 consecutive self-attention modules are used for feature extraction, and the intermediate features after the 2nd, 4th, and 10th self-attention modules are And finally the features after 12 self-attention modules It will be sent to the cross attention fusion module for fusion calculation.

[0112] Similarly, the multispectral data is feature extracted through 12 consecutive self-attention modules and The cross-attention module is sent to perform cross-attention calculation with the SAR data features of the corresponding level to extract the semantic association between cross-source data, thereby realizing the effective fusion of SAR features and multispectral features and obtaining the fused feature CAFO.

[0113] In order to more comprehensively extract the key features after fusion and retain the key features of the original data, the final output of the SAR remote sensing image features after calculation by a series of self-attention modules is The final output of multispectral remote sensing image features after calculation by a series of self-attention modules The final output calculated by the cross-attention fusion module is concatenated, and then a convolution layer is used to adjust the dimension and integrate the semantics. The result is used as the final multi-source fusion feature of the multi-source feature extraction and fusion network. The specific calculation process is expressed as follows:

[0114]

[0115] Among them, MSF (Multi-source Features) represents the multi-source fusion features finally output by the multi-source feature extraction and fusion network.

[0116] (5) Build a local climate zone classification model.

[0117] The local climate zone classification model includes a data feature preprocessing network, a multi-source feature extraction and fusion network, and a classification network;

[0118] The data feature preprocessing network receives multi-source remote sensing data as input, namely SAR images and multispectral images, and performs image segmentation and linear embedding on the two images to obtain the embedded features of the SAR remote sensing image. and embedded features of multispectral remote sensing images The image block operation is to divide the image into blocks. The size of the image blocks is determined by the hyperparameter p. There is no overlapping area between these image blocks. The image block operation is completed through a convolution operation. Subsequently, linear embedding maps these image blocks into image embedding vectors. The specific operation can be expressed as:

[0119]

[0120] Among them, Reshape represents the linear embedding operation, X SAR With X MSI represent SAR images and multispectral images respectively.

[0121] Multi-source feature extraction and fusion network for embedding features of two images and Perform self-attention calculation and feature fusion to obtain multi-source fusion feature MSF.

[0122] The classification network takes the multi-source fusion features (MSFs) as input and generates the final LCZ classification results. Specifically, the linear rectification function (ReLU) is applied to the multi-source fusion features. A 1×1 convolution operation is then performed to adjust the feature depth and fuse the feature channels, thereby transforming complex features into higher-level feature representations directly relevant to the classification task. Finally, a softmax function is used to output the predicted probability of each local climate zone category. The computation of this process can be formally expressed as follows:

[0123] y=softmax(Conv 1×1 (ReLU(MSF)))

[0124] Among them, y represents the final LCZ classification result.

[0125] (6) Model training and testing.

[0126] After constructing the local climate zone classification model based on multi-source remote sensing data fusion, the cross-entropy loss function (Cross-entropy Loss, L CE ) trains the model for LCZ multi-source remote sensing image classification tasks, L CE It can be formally expressed as

[0127]

[0128] Where M represents the total number of LCZ categories, y c represents the true value label, The model predicts the probability of belonging to category c. When the model is trained until the loss function no longer decreases significantly, it means that the model has reached a stable state. At this time, the training is completed, and the trained remote sensing image LCZ classification model based on multi-source remote sensing data fusion is obtained.

[0129] Subsequently, the trained LCZ classification model for remote sensing images based on multi-source remote sensing data fusion is used to perform LCZ classification tests on the test images. The process can be formally expressed as follows:

[0130]

[0131] in (Multi-source Fusion Features Model) represents the trained LCZ classification model of remote sensing images based on multi-source remote sensing data fusion, x Test represents the LCZ multi-source remote sensing image to be tested, Indicates the LCZ classification results obtained by the test.

[0132] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0133] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A local climate zone classification method based on multi-source data fusion, characterized in that: The following steps are involved: Build a multi-source feature extraction and fusion network; the multi-source feature extraction and fusion network includes two self-attention calculation branches and a cross-attention fusion module; the two self-attention calculation branches each include a series of self-attention modules, which respectively perform feature extraction at different levels on multispectral remote sensing image features and SAR remote sensing image features; the cross-attention fusion module includes multiple cross-attention modules, and the multiple cross-attention modules respectively accept different levels of heterogeneous data features extracted by the two self-attention calculation branches to perform feature fusion, and the same cross-attention module accepts paired heterogeneous data features from the same level; Building a local climate zone classification model based on the multi-source feature extraction and fusion network; The local climate zone classification model is trained and tested using a dataset consisting of multispectral remote sensing images and SAR remote sensing images; Classifying the multi-source remote sensing images based on the trained and tested local climate zone classification model to obtain local climate zone classification results; The cross attention module includes two computation branches CAM SAR With CAM MSI , calculate branch CAM SAR Key vector using SAR remote sensing image features With value vector And use the query vector of multispectral remote sensing image features Compute Branch CAM MSI Key vectors using multispectral remote sensing image features With value vector And use the query vector of SAR remote sensing image features After the two calculation branches perform multi-head attention calculation, they perform the same calculation process as the self-attention module to obtain the output CAO SAR and CAO MSI ; For each crisscross attention module, the two computation branches CAM SAR With CAM MSI The output of is connected to obtain the final output CAO of the cross attention module, which is expressed as: HIGH=HIGH SAR +HIGH MSI ; After all cross-attention modules are calculated, the outputs are concatenated and the feature dimensions are adjusted through a convolutional layer to obtain the final output of the cross-attention fusion module.

2. The local climate zone classification method based on multi-source data fusion according to claim 1 is characterized in that: The calculation process of the self-attention calculation branch on the multispectral remote sensing image features or the SAR remote sensing image features includes: The first self-attention module in the self-attention calculation branch receives the embedded feature X of the multispectral remote sensing image or the SAR remote sensing image E As input, and perform layer normalization operation; Input the normalized vector into the multi-head attention mechanism for attention calculation; Output and embedding features X of the multi-head attention mechanism E Perform residual connection and layer normalization to obtain the first-level output of the self-attention module; pass the first-level output through a multi-layer perceptron to obtain the second-level output of the self-attention module; perform residual connection on the first-level output and second-level output, and then perform layer normalization to obtain the final output of the self-attention module; The final output of the self-attention module is used as the input of the next self-attention module or the input of the cross-attention module at the corresponding level.

3. The local climate zone classification method based on multi-source data fusion according to claim 2 is characterized in that: The calculation process of the multi-head attention mechanism is expressed as: MultiHead(X E )=Concat(head1,head2,…,head h )W O Among them, MultiHead(X E ) represents the output of the multi-head attention mechanism, head i represents the output of the i-th head, Q i represents the query vector of the i-th head, K i represents the key vector of the i-th head, V i represents the value vector of the i-th head, d k represents the dimension of the key vector, W O is a learnable weight matrix, and Concat represents a concatenation operation.

4. The local climate zone classification method based on multi-source data fusion according to claim 3 is characterized in that: The calculation process of the first self-attention module is expressed as: Output=LayerNorm(X E +MultiHead(X E )) SelfAttentionOutput=LayerNorm(Output+MLP(Output)) Among them, LayerNorm represents the layer normalization operation, MultiHead(X E ) represents the output of the multi-head attention mechanism, MLP represents the multi-layer perceptron, and SelfAttentionOutput represents the final output of the self-attention module; The complete calculation process of the self-attention module is expressed as: SAO=SAM(X E ) Among them, SAM represents the calculation of the complete self-attention module, X E Represents the embedded features of the input image; SAO = SelfAttentionOutput, represents the output of the self-attention module, which has the same shape as the input.

5. The local climate zone classification method based on multi-source data fusion according to claim 4 is characterized in that: Two computational branches CAM SAR With CAM MSI The difference from the calculation process of the self-attention module is that: Compute Branch CAM SAR The calculation process is expressed as: Compute Branch CAM MSI The calculation process is expressed as: The outputs of the two calculation branches are expressed as: Among them, CAO SAR and CAO MSI Represents the calculation branch CAM SAR With CAM MSI The output, and Represents the final output of the two self-attention calculation branches after passing through j self-attention modules, and j represents the number of self-attention modules passed.

6. The local climate zone classification method based on multi-source data fusion according to claim 1 is characterized in that: The cross-attention fusion module includes four cross-attention modules, which accept paired heterogeneous data features that have passed through 2, 4, 10, and 12 self-attention modules respectively, expressed as: Among them, CAM represents the complete calculation process of a cross attention module. represents the SAR data features calculated by j self-attention modules, Represents the multispectral data features calculated by j self-attention modules; The final output of the cross-attention fusion module is expressed as: CAFO=Conv 1×1 (Concat(CAO1,CAO2,CAO3,CAO4)) Among them, CAFO represents the final output of the cross attention fusion module, Concat represents the splicing operation, Conv 1×1 Represents a 1×1 convolution operation.

7. The local climate zone classification method based on multi-source data fusion according to claim 6 is characterized in that: The final output after the SAR remote sensing image features are calculated by a series of self-attention modules The final output of multispectral remote sensing image features after calculation by a series of self-attention modules The final output calculated by the cross-attention fusion module is concatenated, and then passed through a convolutional layer for adjusting the dimension and integrating the semantics. The result is used as the final multi-source fusion feature of the multi-source feature extraction and fusion network. The specific calculation process is expressed as follows: Among them, MSF represents the multi-source fusion feature finally output by the multi-source feature extraction and fusion network.

8. The local climate zone classification method based on multi-source data fusion according to claim 1 is characterized in that: The local climate zone classification model includes a data feature preprocessing network, a multi-source feature extraction and fusion network, and a classification network; The data feature preprocessing network receives SAR images and multispectral images, performs image segmentation and linear embedding on the two images, and obtains the embedding features of the SAR remote sensing image. and embedded features of multispectral remote sensing images The multi-source feature extraction and fusion network embeds features of the two images and Perform self-attention calculation and fusion to obtain multi-source fusion feature MSF; The classification network sequentially passes the multi-source fusion feature MSF through a linear rectification function ReLU, a 1×1 convolution operation and a softmax function, and outputs the predicted probability of each local climate zone category.

9. The local climate zone classification method based on multi-source data fusion according to claim 1, characterized in that: When training the local climate zone classification model, the loss function adopted is the cross entropy loss function.

Images