Multi-modal LCZ classification method based on text fusion

Abstract

The invention discloses a multi-modal LCZ classification method based on text fusion, and the method comprises the steps: constructing and training a LCZ classification model based on text fusion, and obtaining an optimized LCZ classification model; inputting a to-be-processed LCZ remote sensing image into the optimized LCZ classification model to obtain a corresponding LCZ classification result; the S2 comprises the following steps: processing an LCZ remote sensing image to be processed through a feature extraction network to obtain text-enhanced multi-modal fusion features; and obtaining an LCZ classification result through a classification network based on the text enhanced multi-modal fusion feature. Through fusion of text and image data and combination of self-attention and cross attention mechanisms, the accuracy and robustness of classification are significantly improved, full-process automation of classification tasks is realized, the application efficiency is improved, and a more scientific and reliable microclimate analysis basis is provided for urban planning.

Description

Technical Field

[0001] The present invention relates to the field of LCZ classification, and more specifically to a multimodal LCZ classification method based on text fusion. Background Art

[0002] The Local Climate Zones (LCZ) classification is a general method that aims to understand and evaluate the microclimate characteristics of cities and their surrounding areas. This classification system is of great significance to urban planning and environmental science research. It can help researchers compare the climate characteristics of different cities, thereby supporting urban planners to design a more suitable living environment. With the acceleration of urbanization, problems such as urban heat island effect, energy consumption and environmental pollution have become increasingly prominent. The LCZ classification provides a scientific basis for the effective management of these problems.

[0003] However, existing LCZ classification methods mainly rely on remote sensing image data to extract and analyze image features through deep learning models such as convolutional neural networks (CNN) and Transformer. This method has obvious limitations in capturing complex urban microclimate characteristics. The urban environment is highly heterogeneous and dynamic, and it is difficult to fully reflect subtle and complex differences such as vegetation cover, building density, and human activity patterns by relying solely on image data. These factors are crucial to understanding local climate conditions, but it is often impossible to deeply explore this information, resulting in classification results that may not be accurate enough, especially when dealing with areas that are visually similar but actually have very different climate characteristics.

[0004] In addition, traditional LCZ classification methods have limited ability to capture deep semantic information. Although remote sensing images can provide rich visual information, they are unable to provide deep semantic descriptions such as specific land use, building types, and material properties. This type of information usually needs to be combined with geographic information system (GIS) data or text descriptions to obtain a complete understanding. Existing methods lack effective integration of non-image data sources such as text and fail to fully utilize these contextual knowledge that can supplement and enrich image interpretation, thereby limiting the model's depth of understanding of the environmental background and affecting the overall performance of the classification task.

[0005] Most of the current LCZ classification technologies are based on a single-modality data source, remote sensing images, which to some extent restricts their application scope and development potential. Single-modality methods are easily limited by specific data types. For example, factors such as weather conditions and lighting conditions may significantly affect image quality, thereby interfering with classification accuracy. In addition, ignoring other potentially valuable sources of information, such as text descriptions and meteorological observation records, makes it difficult for the model to fully grasp the actual situation of the target area and cannot meet the growing needs of urban management and environmental protection.

[0006] Therefore, how to design a multimodal LCZ classification method based on text fusion, which can capture the subtle differences in complex urban environments by integrating remote sensing images and descriptive text information, and improve the accuracy and efficiency of LCZ classification is an urgent problem that technicians in this field need to solve. Summary of the invention

[0007] In view of this, the present invention provides a multimodal LCZ classification method based on text fusion, which not only needs to be able to make full use of the spectral and spatial information in the image, but also needs to be able to integrate the semantic information in the text into the classification process. It can show significant advantages in capturing subtle differences in complex urban environments, and provide a more scientific and reliable basis for urban planning, thereby effectively addressing the urban heat island effect and other environmental problems.

[0008] In order to achieve the above object, the present invention adopts the following technical solution:

[0009] A multimodal LCZ classification method based on text fusion, including:

[0010] S1. Build and train a LCZ classification model based on text fusion to obtain an optimized LCZ classification model;

[0011] S2, inputting the LCZ remote sensing image to be processed into the optimized LCZ classification model to obtain the corresponding LCZ classification result; S2 includes:

[0012] S21, processing the LCZ remote sensing image to be processed through a feature extraction network to obtain multimodal fusion features for text enhancement;

[0013] S22. Based on the multimodal fusion features of the text enhancement, an LCZ classification result is obtained through a classification network.

[0014] Furthermore, in S1, the cross entropy loss function L is used CE Perform model training; the cross entropy loss function L CE It is expressed as:

[0015]

[0016] Where M represents the total number of LCZ categories, y c represents the true value label, represents the probability that the model predicts that it belongs to category c.

[0017] Furthermore, in S21, the LCZ remote sensing image to be processed is processed by a feature extraction network, including:

[0018] Based on the input remote sensing image Img, the image embedding features are obtained through the multi-layer perceptron MLP The image embedding feature It is expressed as:

[0019]

[0020] Embedding images into features Input 12 layers of self-attention modules to obtain the second layer of image attention features Layer 4 image attention features Layer 10 image attention features and the 12th layer image attention features

[0021] Furthermore, in S21, the LCZ remote sensing image to be processed is processed by a feature extraction network, and further includes:

[0022] Based on the input remote sensing image Img, a descriptive text Text corresponding to the remote sensing image Img is generated through a text generator VLM;

[0023] Combined with CLIP text encoder, the descriptive text Text is mapped into feature space to generate text features The text features It is expressed as:

[0024]

[0025] The text features The multi-layer perceptron MLP is used to transform the dimension and the text attention feature is obtained through the self-attention module The text attention feature It is expressed as:

[0026]

[0027] Text Attention Features Through three consecutive cross-attention modules, cross-attention calculations are performed with the image attention features of different layers in turn to obtain the fused attention feature CAO T ; The fusion attention feature CAO T It is expressed as:

[0028]

[0029] The fusion attention feature CAO T With the 12th layer image attention feature The splicing process is performed to obtain a text-enhanced multimodal fusion feature MMF; the text-enhanced multimodal fusion feature MMF is expressed as:

[0030]

[0031] Among them, Concat represents a concatenation operation.

[0032] Furthermore, the self-attention module embeds features The processing process includes:

[0033] Embedding features Perform layer normalization to obtain layer normalized vector X E ; The vector X E It is expressed as:

[0034]

[0035] Among them, LayerNorm(·) represents the layer normalization function;

[0036] The vector X that normalizes the layer E Input the multi-head attention structure to obtain the fused information MultiHead(X E ); In the multi-head attention structure, each attention head performs attention calculation independently, and multiple attention heads are processed in parallel;

[0037] The fused information MultiHead(X E ) and embedded features Perform residual connection processing and layer normalization processing to obtain the result Output after layer normalization processing; the result Output after layer normalization processing is expressed as:

[0038]

[0039] The result Output after layer normalization is input into the multi-layer perceptron MLP to obtain the multi-layer perceptron output; the multi-layer perceptron output is subjected to residual connection processing with the result Output, and layer normalization processing is performed to obtain the self-attention module output The self-attention module outputs It is expressed as:

[0040]

[0041] Among them, j represents the number of self-attention modules passed.

[0042] Furthermore, in the multi-head attention structure, each attention head transforms the layer-normalized vector X through three independent fully connected layers. E They are mapped to query vector Q, key vector K and value vector V respectively; the query vector Q, key vector K and value vector V in the i-th attention head are expressed as:

[0043]

[0044] in, Represent the independent learnable weight matrices in different fully connected layers respectively;

[0045] Calculate the query vector Q i and the key vector K i The dot product of and scaling is performed to obtain the attention score;

[0046] The attention scores are normalized and summed with the value vector V i Multiply to get the attention calculation result head i ; The attention calculation result head i It is expressed as:

[0047]

[0048] Among them, softmax(·) represents the normalization function, T represents the transposition of the matrix, and d k represents the dimension of the key vector;

[0049] The information extracted by all attention heads is concatenated and transformed through a linear layer to obtain the fused information MultiHead(X E ); the fused information MultiHead(X E ) is expressed as:

[0050] MultiHead(X E )=Concat(head1,head2,…,head h )W O

[0051] Among them, W O Represents a learnable weight matrix, and Concat represents concatenation.

[0052] Furthermore, the self-attention module The processing process includes:

[0053] Text features Perform layer normalization to obtain the layer normalized vector T E ; The vector T E It is expressed as:

[0054]

[0055] Among them, LayerNorm(·) represents the layer normalization function;

[0056] The vector T that normalizes the layerE Input the multi-head attention structure to obtain the fused information MultiHead(T E ); In the multi-head attention structure, each attention head performs attention calculation independently, and multiple attention heads are processed in parallel;

[0057] The fused information MultiHead(T E ) and the embedded feature T E Perform residual connection processing and layer normalization processing to obtain the result Output after layer normalization processing; the result Output after layer normalization processing is expressed as:

[0058] Output = LayerNorm(T E +MultiHead9T E ))

[0059] The result Output after layer normalization is input into the multi-layer perceptron MLP to obtain the multi-layer perceptron output; the multi-layer perceptron output is subjected to residual connection processing with the result Output, and layer normalization processing is performed to obtain the self-attention module output The self-attention module outputs It is expressed as:

[0060]

[0061] Among them, j represents the number of self-attention modules passed.

[0062] Furthermore, in the multi-head attention structure, each attention head transforms the layer-normalized vector T through three independent fully connected layers. E They are mapped to query vector Q, key vector K and value vector V respectively; the query vector Q, key vector K and value vector V in the i-th attention head are expressed as:

[0063]

[0064] in, Represent the independent learnable weight matrices in different fully connected layers respectively;

[0065] Calculate the query vector Q i and the key vector K i The dot product of and scaling is performed to obtain the attention score;

[0066] The attention scores are normalized and summed with the value vector V i Multiply to get the attention calculation result head i ; The attention calculation result head i It is expressed as:

[0067]

[0068] Among them, softmax(·) represents the normalization function, T represents the transposition of the matrix, and d k represents the dimension of the key vector;

[0069] The information extracted by all attention heads is concatenated and transformed through a linear layer to obtain the fused information MultiHead(T E ); the fused information MultiHead(T E ) is expressed as:

[0070] MultiHead(T E )=Concat(head1,head2,…,head h )W O

[0071] Among them, W O Represents a learnable weight matrix, and Concat represents concatenation.

[0072] Furthermore, the cross attention module includes: calculating the branch CAM Img and CAM Text ; The cross attention module is used for image data and text data The processing process includes:

[0073] Compute Branch CAM Img and CAM Text The image data is respectively transformed into and text data Mapped into query vector Q, key vector K and value vector V; where in the calculation branch CAM Img In the ith attention head, the query vector Q, key vector K, and value vector V are represented as Q i Img , K i Img and V i Img ; In the calculation branch CAM Text In the ith attention head, the query vector Q, key vector K, and value vector V are represented as Q i Text , K i Text and V i Text ;

[0074] In the calculation branch CAM ImgIn the example above, we calculate the query vector Q i Text and the key vector K i Img The dot product of and scaling is performed to obtain an attention score; the attention score is normalized and combined with the value vector V i Img Multiply to get the attention calculation result The attention calculation result It is expressed as:

[0075]

[0076] In the calculation branch CAM Text In the example above, we calculate the query vector Q i Img and the key vector K i Text The dot product of and scaling is performed to obtain an attention score; the attention score is normalized and combined with the value vector V i Text Multiply to get the attention calculation result The attention calculation result It is expressed as:

[0077]

[0078] Based on the attention calculation result and Get the calculation branch CAM Img and compute branch CAM Text Output of CAO Img and CAO Text ;

[0079] The output CAO Img and CAO Text Perform connection processing to obtain the output CAO of the cross attention module.

[0080] Furthermore, the S22 includes:

[0081] Based on the multimodal fusion features of the text enhancement, a ReLU nonlinear activation function is used to introduce nonlinear characteristics;

[0082] Combined with the 1×1 convolutional layer, the multimodal fusion features of text enhancement with nonlinear characteristics are introduced to adjust the feature depth and fuse the feature channels;

[0083] The predicted probability of each LCZ category is output through the softmax function; wherein, S22 is expressed as:

[0084] y=softmax(Conv 1×1 (ReLU(MMF)))

[0085] Among them, y represents the LCZ classification result, softmax represents the softmax function, Conv 1×1 represents a 1×1 convolution operation, ReLU represents the ReLU nonlinear activation function, and MMF represents the multimodal fusion feature for text enhancement.

[0086] It can be seen from the above technical solution that compared with the prior art, the technical solution of the present invention has the following advantages:

[0087] Beneficial effects:

[0088] 1. Existing LCZ classification mainly relies on remote sensing image data, and uses deep learning models for feature extraction and analysis, but it is limited in capturing complex urban microclimate characteristics and deep semantics. This method enhances the understanding of scene features in remote sensing images by fusing text and image data and using the geographical and environmental background knowledge in the text, thus improving the accuracy and reliability of classification. Especially when dealing with LCZ categories that are ambiguous or have overlapping features, multimodal data fusion effectively reduces the risk of misclassification.

[0089] 2. By building a self-attention module (SAM) and a cross-attention module (CAM), deep feature extraction and fusion of image and text data are achieved. The self-attention module can capture the global features and long-distance dependencies of the data, while the cross-attention module calculates the cross-modal fusion features and the feature dependencies between each other, thereby generating a richer and more comprehensive feature representation. This mechanism helps the model better understand the complex urban environment and improve the robustness and generalization ability of the classification model.

[0090] 3. This method provides a multimodal LCZ classification model based on text fusion, which can realize the full process automation of LCZ classification tasks in remote sensing images without human intervention. This not only improves the efficiency of classification tasks, but also reduces the risk of human error. In addition, this method can provide a more scientific and reliable basis for urban planning when dealing with urban heat island effects and other environmental problems, and has important practical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0092] Figure 1 A flow chart of a multimodal LCZ classification method based on text fusion provided by an embodiment of the present invention;

[0093] Figure 2 A framework diagram of the LCZ classification model based on text fusion provided by an embodiment of the present invention;

[0094] Figure 3 A feature extraction network framework diagram provided for an embodiment of the present invention;

[0095] Figure 4 A framework diagram of a self-attention module provided in an embodiment of the present invention;

[0096] Figure 5 A multi-head attention structure framework diagram provided for an embodiment of the present invention;

[0097] Figure 6 A framework diagram of a cross-attention module provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0098] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0099] like Figure 1 As shown, this embodiment provides a multimodal LCZ classification method based on text fusion, including:

[0100] S1. Build and train a LCZ classification model based on text fusion to obtain an optimized LCZ classification model;

[0101] S2, inputting the LCZ remote sensing image to be processed into the optimized LCZ classification model to obtain the corresponding LCZ classification result;

[0102] like Figure 2 As shown, the S2 includes:

[0103] S21, processing the LCZ remote sensing image to be processed through a feature extraction network to obtain multimodal fusion features for text enhancement;

[0104] S22. Based on the multimodal fusion features of the text enhancement, an LCZ classification result is obtained through a classification network.

[0105] This multimodal LCZ classification method builds and trains an LCZ classification model that combines image and text data; then the LCZ remote sensing image to be processed is input into the model, and the text-enhanced multimodal fusion features are generated through the feature extraction network, and the final LCZ classification results are output through the classification network. Not only the spatial and spectral information of the image data is utilized, but also the semantic context provided by the text description is incorporated, thereby improving the understanding of complex urban environmental characteristics and classification accuracy, providing a more scientific and reliable tool for urban planning and environmental research.

[0106] The above steps and related technical features are further described in detail below:

[0107] In this embodiment S1, a LCZ classification model based on text fusion is constructed and trained to obtain an optimized LCZ classification model;

[0108] The construction of the LCZ classification model based on text fusion mainly includes: first, the model designs a self-attention module (SAM) and a cross-attention module (CAM), where SAM is used to capture the global features and long-distance dependencies of a single data modality, and CAM is used to calculate cross-modal fusion features and mutual feature dependencies to achieve effective fusion of text and image data. Secondly, the model also includes a feature extraction module based on text fusion, which uses a pre-trained text generator and CLIP text encoder to convert the text description corresponding to the remote sensing image into text features, and deeply fuses it with image features through multiple cross-attention modules to generate feature representations rich in multimodal information. Finally, the text-enhanced multimodal fusion features are processed through the classification network to output the LCZ classification results.

[0109] During the training process, the cross entropy loss function is used to optimize the model. A large number of LCZ remote sensing images and their corresponding text descriptions are used for training, so that the model can learn the feature representations of different LCZ categories and gradually improve the accuracy and robustness of classification.

[0110] The cross entropy loss function L CE It is expressed as:

[0111]

[0112] Where M represents the total number of LCZ categories, y c represents the true value label, represents the probability that the model predicts that it belongs to category c.

[0113] After the training is completed, an optimized LCZ classification model is obtained, which can automatically perform the LCZ classification task and realize the accurate division of urban microclimate zones.

[0114] In this embodiment S2, the LCZ remote sensing image to be processed is input into the optimized LCZ classification model to obtain the corresponding LCZ classification result; including:

[0115] S21, processing the LCZ remote sensing image to be processed through a feature extraction network to obtain multimodal fusion features for text enhancement;

[0116] S22. Based on the multimodal fusion features of the text enhancement, an LCZ classification result is obtained through a classification network.

[0117] like Figure 3 As shown, in step S21, the LCZ remote sensing image to be processed is processed by a feature extraction network, including:

[0118] Based on the input remote sensing image Img, the image embedding features are obtained through the multi-layer perceptron MLP The image embedding feature It is expressed as:

[0119]

[0120] Embedding images into features Input 12 layers of self-attention modules to obtain the second layer of image attention features Layer 4 image attention features Layer 10 image attention features and the 12th layer image attention features

[0121] Furthermore, it also includes:

[0122] Based on the input remote sensing image Img, a descriptive text Text corresponding to the remote sensing image Img is generated through a text generator VLM;

[0123] Combined with CLIP text encoder, the descriptive text Text is mapped into feature space to generate text features The text features It is expressed as:

[0124]

[0125] The text features The multi-layer perceptron MLP is used to transform the dimension and the text attention feature is obtained through the self-attention module The text attention feature It is expressed as:

[0126]

[0127] Text Attention Features Through three consecutive cross-attention modules, cross-attention calculations are performed with the image attention features of different layers in turn to obtain the fused attention feature CAO T ; The fusion attention feature CAO T It is expressed as:

[0128]

[0129] The fusion attention feature CAO T With the 12th layer image attention feature The splicing process is performed to obtain a text-enhanced multimodal fusion feature MMF; the text-enhanced multimodal fusion feature MMF is expressed as:

[0130]

[0131] Among them, Concat represents a concatenation operation.

[0132] like Figure 4 As shown, the self-attention module embeds the features The processing process includes:

[0133] Embedded features Perform layer normalization to obtain layer normalized vector X E ; The vector X E It is expressed as:

[0134]

[0135] Among them, LayerNorm(·) represents the layer normalization function;

[0136] The vector X that normalizes the layer E Input the multi-head attention structure to obtain the fused information MultiHead(X E ); In the multi-head attention structure, each attention head performs attention calculation independently, and multiple attention heads are processed in parallel;

[0137] The fused information MultiHead(X E ) and embedded features Perform residual connection processing and layer normalization processing to obtain the result Output after layer normalization processing; the result Output after layer normalization processing is expressed as:

[0138]

[0139] The result Output after layer normalization is input into the multi-layer perceptron MLP to obtain the multi-layer perceptron output; the multi-layer perceptron output is subjected to residual connection processing with the result Output, and layer normalization processing is performed to obtain the self-attention module output The self-attention module outputs It is expressed as:

[0140]

[0141] Among them, j represents the number of self-attention modules passed.

[0142] like Figure 5 As shown, in the multi-head attention structure, each attention head normalizes the layer vector X through three independent fully connected layers. E They are mapped to query vector Q, key vector K and value vector V respectively; the query vector Q, key vector K and value vector V in the i-th attention head are expressed as:

[0143]

[0144] in, Represent the independent learnable weight matrices in different fully connected layers respectively;

[0145] Calculate the query vector Q i and the key vector K i The dot product of and scaling is performed to obtain the attention score;

[0146] The attention scores are normalized and summed with the value vector V i Multiply to get the attention calculation result head i ; The attention calculation result head i It is expressed as:

[0147]

[0148] Among them, softmax(·) represents the normalization function, T represents the transposition of the matrix, and d k Indicates the dimension of the key vector; d k Used to assist in the scaling operation, which helps avoid producing too large values ​​before performing the softmax operation.

[0149] The information extracted by all attention heads is concatenated and transformed through a linear layer to obtain the fused information MultiHead(X E ); the fused information MultiHead(X E ) is expressed as:

[0150] MultiHead(XE )=Concat(head1,head2,…,head h )W O

[0151] Among them, W O Represents a learnable weight matrix, and Concat represents concatenation.

[0152] Furthermore, the self-attention module The processing process includes:

[0153] Text features Perform layer normalization to obtain the layer normalized vector T E ; The vector T E It is expressed as:

[0154]

[0155] Among them, LayerNorm(·) represents the layer normalization function;

[0156] The vector T that normalizes the layer E Input the multi-head attention structure to obtain the fused information MultiHead(T E ); In the multi-head attention structure, each attention head performs attention calculation independently, and multiple attention heads are processed in parallel;

[0157] The fused information MultiHead(T E ) and the embedded feature T E Perform residual connection processing and layer normalization processing to obtain the result Output after layer normalization processing; the result Output after layer normalization processing is expressed as:

[0158] Output = LayerNorm(T E +MultiHead(T E ))

[0159] The result Output after layer normalization is input into the multi-layer perceptron MLP to obtain the multi-layer perceptron output; the multi-layer perceptron output is subjected to residual connection processing with the result Output, and layer normalization processing is performed to obtain the self-attention module output The self-attention module outputs It is expressed as:

[0160]

[0161] Among them, j represents the number of self-attention modules passed.

[0162] Furthermore, in the multi-head attention structure, each attention head transforms the layer-normalized vector T through three independent fully connected layers. E They are mapped to query vector Q, key vector K and value vector V respectively; the query vector Q, key vector K and value vector V in the i-th attention head are expressed as:

[0163]

[0164] in, Represent the independent learnable weight matrices in different fully connected layers respectively;

[0165] Calculate the query vector Q i and the key vector K i The dot product of and scaling is performed to obtain the attention score;

[0166] The attention scores are normalized and summed with the value vector V i Multiply to get the attention calculation result head i ; The attention calculation result head i It is expressed as:

[0167]

[0168] Among them, softmax(·) represents the normalization function, T represents the transposition of the matrix, and d k represents the dimension of the key vector;

[0169] The information extracted by all attention heads is concatenated and transformed through a linear layer to obtain the fused information MultiHead(T E ); the fused information MultiHead(T E ) is expressed as:

[0170] MultiHead(T E )=Concat(head1,head2,…,head h )W O

[0171] Among them, W O Represents a learnable weight matrix, and Concat represents concatenation.

[0172] In the above step S21, the processing process of the LCZ remote sensing image is implemented in detail. First, the input remote sensing image is converted into image embedding features through a multi-layer perceptron, and then these features are sent to a 12-layer self-attention module to obtain image attention features at different levels. At the same time, based on the remote sensing image content, a text generator is used to generate descriptive text, and these texts are mapped to the feature space to form text features with the help of a CLIP text encoder. Then, the text features are further processed by a multi-layer perceptron and a self-attention module to obtain text attention features. Thereafter, these text attention features will pass through three consecutive cross-attention modules and be fused with image attention features at different levels to generate fused attention features. Finally, the fused attention features are spliced ​​with the 12th-layer image attention features to form text-enhanced multimodal fusion features, providing rich feature representation for subsequent LCZ classification.

[0173] In addition, the self-attention module plays a key role in the feature extraction process. It first performs deep processing on the image and text embedding features through layer normalization and multi-head attention structure to capture the global features and long-distance dependencies of the data. In the multi-head attention structure, each attention head performs attention calculation independently, and multiple heads are processed in parallel to extract feature information from multiple angles. The processed information is then processed by residual connection and layer normalization, as well as further converted by multi-layer perceptron to obtain the output of the self-attention module. This process ensures the richness and diversity of features, laying a solid foundation for subsequent cross-attention calculation and multimodal fusion.

[0174] like Figure 6 As shown, the cross attention module includes: computing branch CAM Img and CAM Text ; The cross attention module is used for image data and text data The processing process includes:

[0175] Compute Branch CAM Img and CAM Text The image data is respectively transformed into and text data Mapped into query vector Q, key vector K and value vector V; where in the calculation branch CAM Img In the ith attention head, the query vector Q, key vector K, and value vector V are represented as Q i Img , K i Img and V i Img ; In the calculation branch CAM TextIn the ith attention head, the query vector Q, key vector K, and value vector V are represented as Q i Text , K i Text and V i Text ;

[0176] In the calculation branch CAM Img In the example above, we calculate the query vector Q i Text and the key vector K i Img The dot product of and scaling is performed to obtain an attention score; the attention score is normalized and combined with the value vector V i Img Multiply to get the attention calculation result The attention calculation result It is expressed as:

[0177]

[0178] In the calculation branch CAM Text In the example above, we calculate the query vector Q i Img and the key vector K i Text The dot product of and scaling is performed to obtain an attention score; the attention score is normalized and combined with the value vector V i Text Multiply to get the attention calculation result The attention calculation result It is expressed as:

[0179]

[0180] Based on the attention calculation result and Get the calculation branch CAM Img and compute branch CAM Text Output of CAO Img and CAO Text ;

[0181] The output CAO Img and CAO Text Perform connection processing to obtain the output CAO of the cross attention module.

[0182] The cross-attention module processes image data and text data separately through computational branches. Each branch uses three independent fully connected layers to map the input into a query vector Q, a key vector K, and a value vector V. In each attention head, the computational branch uses image data as the query vector and text data as the key and value vectors; conversely, the computational branch uses text data as the query vector and image data as the key and value vectors. Then, the dot product of the query vector and the key vector is calculated and scaled in the two branches to obtain the attention scores. These scores are normalized and multiplied with the corresponding value vectors to generate their respective attention calculation results. Finally, the outputs of the two branches are connected to form the comprehensive output of the cross-attention module, which realizes the effective fusion of information between image and text modalities and enhances the model's understanding and classification capabilities of complex LCZ features.

[0183] In step S22 of this embodiment, based on the multimodal fusion features of the text enhancement, an LCZ classification result is obtained through a classification network. Specifically, it includes:

[0184] Based on the multimodal fusion features of the text enhancement, a ReLU nonlinear activation function is used to introduce nonlinear characteristics;

[0185] Combined with the 1×1 convolutional layer, the multimodal fusion features of text enhancement with nonlinear characteristics are introduced to adjust the feature depth and fuse the feature channels;

[0186] The predicted probability of each LCZ category is output through the softmax function; wherein, S22 is expressed as:

[0187] y=softmax(Conv 1×1 (ReLU(MMF)))

[0188] Among them, y represents the LCZ classification result, softmax represents the softmax function, Conv 1×1 represents a 1×1 convolution operation, ReLU represents the ReLU nonlinear activation function, and MMF represents the multimodal fusion feature for text enhancement.

[0189] In this step, by introducing nonlinear characteristics and performing feature adjustment, the classification network's ability to recognize complex LCZ features is significantly improved, thereby improving the accuracy and reliability of classification.

[0190] This embodiment provides a multimodal LCZ classification method based on text fusion. By constructing and training an LCZ classification model that combines image and text data, the accurate division of urban microclimate zones is achieved. In the model construction stage, a self-attention module (SAM) and a cross-attention module (CAM) are designed to capture the global features of a single modality and the fusion features of cross-modality. The pre-trained text generator and CLIP text encoder are used to convert the text description corresponding to the remote sensing image into text features, and deep fusion is performed to generate feature representations rich in multimodal information. The cross entropy loss function is used to optimize the model parameters during the training process to ensure that it learns the feature representations of different LCZ categories.

[0191] In the classification stage, the LCZ remote sensing image to be processed is first processed by the feature extraction network to obtain the text-enhanced multimodal fusion features; then the final LCZ classification results are output through the classification network. Specifically, the feature extraction network not only extracts embedded features from the image and processes them through multi-layer self-attention modules, but also generates text features through the text generator and CLIP text encoder, and deeply fuses them with image features through multiple cross-attention modules to form text-enhanced multimodal fusion features. Finally, the classification network introduces nonlinear activation functions and 1×1 convolutional layers to adjust the fusion features, and outputs the predicted probability of each LCZ category through the softmax function, which significantly improves the accuracy and reliability of classification and provides scientific and reliable tool support for urban planning and environmental research.

[0192] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0193] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multimodal LCZ classification method based on text fusion, characterized in that: include: S1. Build and train a LCZ classification model based on text fusion to obtain an optimized LCZ classification model; S2, inputting the LCZ remote sensing image to be processed into the optimized LCZ classification model to obtain the corresponding LCZ classification result; S2 includes: S21, processing the LCZ remote sensing image to be processed through a feature extraction network to obtain multimodal fusion features for text enhancement; S22. Based on the multimodal fusion features of the text enhancement, an LCZ classification result is obtained through a classification network.

2. The multimodal LCZ classification method based on text fusion according to claim 1, characterized in that: In S1, the cross entropy loss function L is used CE Perform model training; the cross entropy loss function L CE It is expressed as: Where M represents the total number of LCZ categories, y c represents the true value label, represents the probability that the model predicts that it belongs to category c.

3. The multimodal LCZ classification method based on text fusion according to claim 1, characterized in that: In S21, the LCZ remote sensing image to be processed is processed by a feature extraction network, including: Based on the input remote sensing image Img, the image embedding features are obtained through the multi-layer perceptron MLP The image embedding feature It is expressed as: Embedding images into features Input 12 layers of self-attention modules to obtain the second layer of image attention features Layer 4 image attention features Layer 10 image attention features and the 12th layer image attention features 4. The multimodal LCZ classification method based on text fusion according to claim 3 is characterized in that: In S21, the LCZ remote sensing image to be processed is processed by a feature extraction network, and further includes: Based on the input remote sensing image Img, a descriptive text Text corresponding to the remote sensing image Img is generated through a text generator VLM; Combined with CLIP text encoder, the descriptive text Text is mapped into feature space to generate text features The text features It is expressed as: The text features The multi-layer perceptron MLP is used to transform the dimension and the text attention feature is obtained through the self-attention module The text attention feature It is expressed as: Text Attention Features Through three consecutive cross-attention modules, cross-attention calculations are performed with the image attention features of different layers in turn to obtain the fused attention feature CAO T ; The fusion attention feature CAO T It is expressed as: The fusion attention feature CAO T With the 12th layer image attention feature The splicing process is performed to obtain a text-enhanced multimodal fusion feature MMF; the text-enhanced multimodal fusion feature MMF is expressed as: Among them, Concat represents a concatenation operation.

5. The multimodal LCZ classification method based on text fusion according to claim 4 is characterized in that: The self-attention module embeds the features The processing process includes: Embedded features Perform layer normalization to obtain layer normalized vector X E ; The vector X E It is expressed as: Among them, LayerNorm(·) represents the layer normalization function; The vector X that normalizes the layer E Input the multi-head attention structure to obtain the fused information MultiHead(X E ); In the multi-head attention structure, each attention head performs attention calculation independently, and multiple attention heads are processed in parallel; The fused information MultiHead(X E ) and embedded features Perform residual connection processing and layer normalization processing to obtain the result Output after layer normalization processing; the result Output after layer normalization processing is expressed as: The result Output after layer normalization is input into the multi-layer perceptron MLP to obtain the multi-layer perceptron output; the multi-layer perceptron output is subjected to residual connection processing with the result Output, and layer normalization processing is performed to obtain the self-attention module output The self-attention module outputs It is expressed as: Among them, j represents the number of self-attention modules passed.

6. The multimodal LCZ classification method based on text fusion according to claim 5, characterized in that: In the multi-head attention structure, each attention head transforms the layer-normalized vector X through three independent fully connected layers. E They are mapped to query vector Q, key vector K and value vector V respectively; the query vector Q, key vector K and value vector V in the i-th attention head are expressed as: in, Represent the independent learnable weight matrices in different fully connected layers respectively; Calculate the query vector Q i and the key vector K i The dot product of and scaling is performed to obtain the attention score; The attention scores are normalized and summed with the value vector V i Multiply to get the attention calculation result head i ; The attention calculation result head i It is expressed as: Among them, softmax(·) represents the normalization function, T represents the transposition of the matrix, and d k represents the dimension of the key vector; The information extracted by all attention heads is concatenated and transformed through a linear layer to obtain the fused information MultiHead(X E ); the fused information MultiHead(X E ) is expressed as: MultiHead(X E )=Concat(head1,head2,…,head h )W O Among them, W O Represents a learnable weight matrix, and Concat represents concatenation.

7. The multimodal LCZ classification method based on text fusion according to claim 4, characterized in that: The self-attention module is used to analyze the text features The processing process includes: Text features Perform layer normalization to obtain the layer normalized vector T E ; The vector T E It is expressed as: Among them, LayerNorm(·) represents the layer normalization function; The layer normalized vector T E Input the multi-head attention structure to obtain the fused information MultiHead(T E ); In the multi-head attention structure, each attention head performs attention calculation independently, and multiple attention heads are processed in parallel; The fused information MultiHead(T E ) and the embedded feature T E Perform residual connection processing and layer normalization processing to obtain the result Output after layer normalization processing; the result Output after layer normalization processing is expressed as: Output=LayerNorm(T E +MultiHead(T E )) The result Output after layer normalization is input into the multi-layer perceptron MLP to obtain the multi-layer perceptron output; the multi-layer perceptron output is subjected to residual connection processing with the result Output, and layer normalization processing is performed to obtain the self-attention module output The self-attention module outputs It is expressed as: Among them, j represents the number of self-attention modules passed.

8. The multimodal LCZ classification method based on text fusion according to claim 7 is characterized in that: In the multi-head attention structure, each attention head passes through three independent fully connected layers to normalize the vector T E They are mapped to query vector Q, key vector K and value vector V respectively; the query vector Q, key vector K and value vector V in the i-th attention head are expressed as: in, Represent the independent learnable weight matrices in different fully connected layers respectively; Calculate the query vector Q i and the key vector K i The dot product of and scaling is performed to obtain the attention score; The attention scores are normalized and summed with the value vector V i Multiply to get the attention calculation result head i ; The attention calculation result head i It is expressed as: Among them, softmax(·) represents the normalization function, T represents the transposition of the matrix, and d k represents the dimension of the key vector; The information extracted by all attention heads is concatenated and transformed through a linear layer to obtain the fused information MultiHead(T E ); the fused information MultiHead(T E ) is expressed as: MultiHead(T E )=Concat(head1,head2,…,head h )W O Among them, W O Represents a learnable weight matrix, and Concat represents concatenation.

9. The multimodal LCZ classification method based on text fusion according to claim 4, characterized in that: The cross attention module includes: calculating the branch CAM Img and CAM Text ; The cross attention module is used for image data and text data The processing process includes: Compute Branch CAM Img and CAM Text The image data is respectively transformed into and text data Mapped into query vector Q, key vector K and value vector V; where in the calculation branch CAM Img In the ith attention head, the query vector Q, key vector K, and value vector V are represented as Q i Img , K i Img and V i Img ; In the calculation branch CAM Text In the ith attention head, the query vector Q, key vector K, and value vector V are represented as Q i Text , K i Text and V i Text ; In the calculation branch CAM Img In the example above, we calculate the query vector Q i Text and the key vector K i Img The dot product of and scaling is performed to obtain an attention score; the attention score is normalized and combined with the value vector V i Img Multiply to get the attention calculation result The attention calculation result It is expressed as: In the calculation branch CAM Text In the example above, we calculate the query vector Q i Img and the key vector K i Text The dot product of and scaling is performed to obtain an attention score; the attention score is normalized and combined with the value vector V i Text Multiply to get the attention calculation result The attention calculation result It is expressed as: Based on the attention calculation result and Get the calculation branch CAM Img and compute branch CAM Text Output of CAO Img and CAO Text ; The output CAO Img and CAO Text Perform connection processing to obtain the output CAO of the cross attention module.

10. The multimodal LCZ classification method based on text fusion according to claim 1, characterized in that: The S22 comprises: Based on the multimodal fusion features of the text enhancement, a ReLU nonlinear activation function is used to introduce nonlinear characteristics; Combined with the 1×1 convolutional layer, the multimodal fusion features of text enhancement with nonlinear characteristics are introduced to adjust the feature depth and fuse the feature channels; The predicted probability of each LCZ category is output through the softmax function; wherein, S22 is expressed as: y=softmax(Conv 1×1 (ReLU(MMF))) Among them, y represents the LCZ classification result, softmax represents the softmax function, Conv 1×1 represents a 1×1 convolution operation, ReLU represents the ReLU nonlinear activation function, and MMF represents the multimodal fusion feature for text enhancement.

Images