Image analysis method and system for identifying post-earthquake collapsed buildings

Abstract

The application discloses an image analysis method and system for identifying post-earthquake collapsed buildings, and belongs to the technical field of image analysis. First, a recognition network is constructed according to a UNet++ network, a global context enhancement module GCEM and a dynamic feature interaction module DFIM. Second, post-earthquake optical images are simultaneously input into the global context enhancement module GCEM and the UNet++ network. The global context enhancement module GCEM extracts a feature map through a convolution layer, generates a semantic label set through a multi-scale semantic marker MST, and then performs global modeling through a Transformer encoder and a decoder to output a global feature map. The UNet++ network outputs a local feature map through an encoder downsampling and decoder step-by-step fusion operation. Finally, the global feature map and the local feature map are input into the dynamic feature interaction module DFIM, multi-scale feature fusion is realized through adaptive weight adjustment, and a collapsed building recognition result is generated.

Description

Technical Field

[0001] This invention relates to the field of image analysis technology, specifically to an image analysis method and system for identifying buildings that have collapsed after an earthquake. Background Technology

[0002] Post-earthquake collapse building identification is a critical task in disaster emergency response. Traditional methods rely on the joint analysis of pre-earthquake optical imagery and post-earthquake SAR imagery: shape, spectral, and texture features are extracted from optical imagery, combined with polarization and scattering features from SAR imagery, and a classifier is used for identification. However, SAR imagery is limited by satellite revisit cycles and post-earthquake environmental interference, resulting in poor data acquisition timeliness and high interpretation barriers. In contrast, optical imagery has advantages in sensor distribution, ease of acquisition, and interpretation difficulty, but it lacks elevation information and requires the extraction of more abstract and discriminative features. Therefore, the introduction of deep learning technology has become an inevitable choice.

[0003] Currently, deep learning methods based on optical images still have significant shortcomings. For example: 1. Weak global modeling ability: The mainstream model UNet++, due to the limitation of the local receptive field of the convolutional kernel, struggles to capture long-distance spatial dependencies, leading to blurred boundary recognition. 2. Rigid feature fusion mechanism: Fixed-weight fusion strategies cannot dynamically perceive the importance of multi-scale features, resulting in increased false positive / false negative rates. 3. Weak resistance to interference from complex backgrounds: Scenes such as rubble accumulation and vegetation obstruction after an earthquake easily cause feature confusion, significantly reducing recognition accuracy. Although existing improved models attempt optimization, they still struggle to overcome performance bottlenecks due to strong dependence on boundary information or insufficient ability to distinguish environmental interference. Therefore, there is an urgent need for an image analysis technology solution for recognizing collapsed buildings after an earthquake to address the above technical problems. Summary of the Invention

[0004] The purpose of this invention is to provide an image analysis method and system for identifying post-earthquake collapsed buildings. Using UNet++ as the main network framework, a Global Context Enhancement Module (GCEM) is designed. By constructing a Multi-scale Semantic Tokenizer (MST) and introducing the encoding and decoding modules of the Transformer network, long-distance dependencies and global context are captured, enhancing the model's understanding of the structure of the collapsed area, while suppressing background noise.

[0005] Building upon this, a Dynamic Feature Interaction Module (DFIM) was designed to adaptively fuse local and global features, thereby fully leveraging the advantages of features at various scales and improving the accuracy and robustness of the model in recognizing collapsed buildings in complex scenarios. This addresses the problems mentioned in the background section.

[0006] To address the aforementioned technical problems, this invention provides an image analysis method for identifying post-earthquake collapsed buildings, comprising:

[0007] S1. Construct the recognition network, which includes the UNet++ network, the Global Context Enhancement Module (GCEM), and the Dynamic Feature Interaction Module (DFIM).

[0008] When identifying collapsed buildings using high-resolution remote sensing images, UNet++ struggles to balance global information capture with local feature correlation. Its global modeling mechanism is weak when faced with images full of interfering elements after an earthquake, affecting the accuracy and completeness of detection.

[0009] The Global Context Enhancement Module (GCEM) is used to achieve effective association between features and the global scene. The GCEM consists of three parts: Multi-Scale Semantic Tag (MST), Transformer encoder, and Transformer decoder.

[0010] The multi-scale semantic tagger (MST) incorporates a multi-scale attention module, generates a semantic tag set through Softmax, then the Transformer encoder models the context of the semantic tag set, and the Transformer decoder maps it back to the pixel space, thereby effectively associating features with the global scene.

[0011] S2. Simultaneously input the post-earthquake optical images into the Global Context Enhancement Module (GCEM) and the UNet++ network.

[0012] The Global Context Enhancement (GCEM) module extracts feature maps through convolutional layers, generates a semantic tag set through the Multi-Scale Semantic Tag Set (MST), and then performs global modeling through the Transformer encoder and decoder to output a global feature map.

[0013] The Global Context Enhancement Module (GCEM) includes a multi-scale semantic tagger (MST), a Transformer encoder, and a decoder.

[0014] Transformers are suitable for processing sequential data, and their standard input is a two-dimensional matrix, which cannot be directly applied to the three-dimensional structure of an image (height H × width W × number of channels C). Therefore, it is necessary to convert image features into a sequence of semantic units, with each unit corresponding to an embedding vector, treating pixel blocks as independent tags similar to words in text for computation.

[0015] The Multi-Scale Semantic Tag Set (MST) is used to perform multi-scale attention calculations on the feature maps extracted by the convolutional layers, generating a multi-scale semantic tag set; specifically, it consists of two parts.

[0016] The upper part reshapes the input feature map into vector form. The convolutional layer extracts features X∈R. H×W×C The input tensor X is reshaped from (H, W, C) to ((H, W), C); where H, W, and C represent the image height, width, and number of channels, respectively.

[0017] The second half generates a multi-scale spatial attention map in the initial stage through a multi-scale attention module.

[0018] First, multi-scale features are extracted using 1×1, 3×3, and 5×5 convolutional kernels, and average pooling and max pooling are used to enhance feature representation: P i =AP(C i (X))+MP(C i (X)).

[0019] Among them, P i Let C represent the i-th enhancement feature, and let AP and MP distributions represent average pooling and max pooling, respectively. i This represents a convolution operation with a kernel size of i×i (i = 1, 3, 5).

[0020] The attention weight matrix is ​​generated using two fully connected layers with ReLU and sigmoid functions. i = 1, 3, 5 This is a fully connected layer. The multi-scale features are multiplied by the corresponding weight map to obtain the multi-scale attention features A = {A...} i |i=1,3,5}.

[0021] Secondly, by integrating attention features at different scales, we obtain the feature Y = C3(∑(A1,A3,A5))×X. Then, we supplement the features using a spatial attention module to obtain a multi-scale attention map. Using the multi-scale attention map, we calculate the weighted average sum of each pixel in X, and then use pointwise convolution to obtain the multi-scale semantic map Z∈R. H×W×L .

[0022] Finally, the result is multiplied by the tag set obtained in the first part through the Softmax operation to obtain the multi-scale semantic tag set T∈R.L×R The specific formula is as follows:

[0023] T=(σ(Z)) T W=(σ(φ(SA(Y)×Y))) T W.

[0024] Where W represents the feature vector, and SA() represents the spatial attention module. For pointwise convolutional layers with autonomous learning capabilities, σ() represents the Softmax operation.

[0025] The Transformer encoder adds positional encoding to the semantic tag set and extracts contextual features through a multi-head self-attention mechanism; the Transformer decoder maps semantic features back to pixel space through a multi-head cross-attention layer and outputs a global feature map.

[0026] Transformer encoder operation includes:

[0027] First, encode the position U pe ∈R L×R Integrate into the semantic tag set T, and then combine the semantic tag set T∈R L×R The input is fed into the Transformer encoder to obtain a new set of semantic tags T. new After inputting the semantic tag set T into the Transformer encoder, we can obtain q,k,v∈R. L×R The formula is as follows:

[0028] q=TM q k = TM k v = TM v .

[0029] Where q, k, and v are the query vector, key vector, and value vector, respectively, and M... q M k M v ∈R C×d Representing a linear matrix capable of autonomous learning, the formula for a single self-attention layer, Attention(q,k,v), is as follows:

[0030]

[0031] Where d represents the number of channel dimensions for q, k, and v.

[0032] The multi-head self-attention mechanism in the Transformer encoder allows input data to be fed simultaneously into multiple independent attention computation units. Each unit generates its own query vector q, key vector k, and value vector v. Since each head uses different parameters during initialization, these vectors generate differentiated attention weights during computation, enabling the integrated processing of multivariate information from different locations and representation subspaces. The outputs of each head are concatenated, linearly transformed, and then projected to obtain the final value.

[0033] The formula for representing the multi-head self-attention layer MSA(T) is as follows:

[0034] MSA(T)=Concat(head1,...,head n M o .

[0035]

[0036] Among them, head j For the j-th attention head, M o ∈R hd×C represents the linear projection matrix, and h represents the number of self-attention layers.

[0037] After processing by the Transformer encoder, semantic features rich in contextual relationships can be extracted. These high-density semantic features are of great value for identifying target regions. To achieve the mapping and reconstruction of semantic features to pixel space, a Transformer decoder is needed to further optimize and adjust the image features.

[0038] The Transformer decoder operation includes:

[0039] The image features are further optimized and adjusted using a Transformer decoder. The Transformer decoder consists of N... Td The layer is composed of units, each unit including a multi-head cross-attention layer and a multi-layer perceptron layer. The formula for representing the multi-head cross-attention layer is as follows:

[0040] MCA(T)=Concat(head1,...,head n M o .

[0041]

[0042] The perceptron layer also contains two linear projection layers, one of which uses the Gaussian error linear unit (GELU), as shown in the following formula:

[0043] MLP(T) = GELU(TM1)M2.

[0044] Where M1, M2∈R C×C This represents a linear transformation matrix that can learn autonomously.

[0045] The Transformer encoder enhances the structural understanding of collapsed areas by modeling long-distance dependencies through multi-head self-attention. The Transformer decoder maps semantic features back to pixel space through cross-attention, suppressing background interference.

[0046] The UNet++ network outputs local feature maps through encoder downsampling and decoder fusion operations.

[0047] S3. Input the global feature map and local feature map into the dynamic feature interaction module DFIM, and achieve multi-scale feature fusion through weight adjustment to generate the collapse building recognition result.

[0048] While UNet++ networks can achieve progressive fusion and refined reconstruction of multi-scale features through dense skip connections, they struggle to dynamically perceive the differences in importance between features at different scales during cross-level feature fusion. UNet++'s fixed fusion strategy cannot adaptively adjust weights based on the content of the input features, leading to problems such as insufficient response to key features and increased interference from redundant information.

[0049] Therefore, a dynamic feature interaction module DIFM is designed, which can capture the complex relationships between local and global features and selectively focus on the most informative features to learn the joint representation of local and global features.

[0050] The Dynamic Feature Interaction Module (DFIM) performs the following operations:

[0051] First, the global and local feature maps are input into the Dynamic Feature Interaction Module (DFIM). A 3×3 convolutional layer transforms the global and local feature maps, employing residual connections and the ReLU activation function. Specifically, this is represented as follows:

[0052] O g =C3(ReLU(C3(T) g )))+T g .

[0053] O l =C3(ReLU(C3(T) l )))+T l .

[0054] Among them, O g Represents global features, O lC3 represents a local feature, and C3 indicates a convolution operation with a kernel size of 3×3.

[0055] Secondly, the transformed global and local features are concatenated and then subjected to a 1×1 convolution to generate an attention score. The attention weights of the global and local features in different dimensions are calculated using the Softmax function.

[0056] Then, the global and local features are multiplied element-wise with their corresponding attention weights to obtain the weighted WSum of the global features. g Weighted WSum of local features l .

[0057] Finally, the global weighted average and the local weighted average are added element by element to obtain the output result.

[0058] The weighted calculation formula is as follows:

[0059] WSum g =Softmax(C1(Concat(O g O l )))T g .

[0060] WSum l =Softmax(C1(Concat(O g O l )))T l .

[0061] Residual connections are used to enhance feature representation and avoid gradient vanishing. Softmax is used to dynamically allocate global / local feature weights to highlight key scale features.

[0062] An electronic device includes a processor and a memory, the memory storing a computer program, and the processor executing the program to implement an image analysis method for identifying buildings that have collapsed after an earthquake.

[0063] An image analysis system for identifying collapsed buildings after an earthquake includes an image input module, a global feature extraction module, a local feature extraction module, a feature fusion module, and an output module.

[0064] The image input module is used to receive post-earthquake optical images.

[0065] The global feature extraction module is implemented by the global context enhancement module GCEM, which includes convolutional layers, multi-scale semantic tagger MST, Transformer encoder and decoder.

[0066] Convolutional layers are used to extract feature maps from the input image. The Multi-Scale Semantic Tag (MST) generates multi-scale attention features using 1×1, 3×3, and 5×5 convolutional kernels, and outputs a semantic tag set in conjunction with a spatial attention module. The Transformer encoder and decoder add positional encoding to the semantic tag set and output a global feature map through multi-head self-attention and cross-attention mechanisms.

[0067] The local feature extraction module is implemented by the UNet++ network, which outputs local feature maps through encoder downsampling and decoder fusion operations.

[0068] The feature fusion module is implemented by the dynamic feature interaction module DFIM, and performs the following operations:

[0069] The global and local feature maps are transformed by 3×3 convolutional layers, and residual connections are used.

[0070] The concatenated and transformed features are then used to generate attention scores via 1×1 convolution.

[0071] The weights of global and local features are calculated using the Softmax function.

[0072] The weighted global features are added to the local features to generate the collapsed building identification result.

[0073] The GCEM branch extracts global semantic features, the UNet++ branch preserves local detailed features, and DFIM fuses the two to achieve refined boundary recognition.

[0074] The output module is used to output the recognition results to a display device or storage medium.

[0075] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0076] Enhanced global modeling capabilities: By introducing the Transformer codec to build the Global Context Enhancement Module (GCEM), the long-distance spatial dependencies of collapsed buildings (such as the continuity of fractured structures) are effectively captured, overcoming the global semantic loss problem caused by the local convolutional receptive field in UNet++, and significantly improving the overall understanding of large-scale collapse areas.

[0077] Dynamic Feature Fusion Optimization: The Dynamic Feature Interaction Module (DFIM) is designed, which adopts an adaptive weight allocation mechanism (Softmax dynamically calculates global / local feature weights) to replace the fixed fusion strategy of UNet++. This enables the collaborative enhancement of multi-scale features (such as texture details and structural contours) and avoids the suppression of key information or interference from redundant features.

[0078] Improved robustness in complex scenarios: The GCEM module integrates spatial attention and multi-scale convolutional features (1×1 / 3×3 / 5×5 kernels) through the multi-scale semantic tagger MST, accurately distinguishing collapsed targets from background interference (such as rubble piles and vegetation), and significantly reducing false detections and false negatives in severely damaged and mixed ground features.

[0079] Refined Boundary Recognition: The DFIM module combines residual connectivity and feature weighting mechanisms to enhance the expressive power of the edge features of collapsed buildings, solving the problem of blurred contours caused by the strong dependence on boundary information or insufficient cross-level feature coordination in methods such as OB-UNet and CSC-Unet.

[0080] Cross-scene generalization advantage: The collaborative design of the dual-branch architecture (GCEM global modeling + UNet++ local details) enables the model to show stronger adaptability on both distribution-intensive (HIT) and severely damaged (YSU) datasets, especially maintaining recognition integrity in highly mixed scenes.

[0081] Based on the UNet++ network, a Global Context Enhancement (GCEM) module was designed. GCEM constructs a multi-scale semantic tagger (MST) to accurately capture the detailed features and spatial location of collapsed buildings. It also incorporates the encoding and decoding modules of a Transformer network to enhance the global semantic understanding of the collapsed area and effectively model long-distance dependencies. A Dynamic Feature Interaction (DFIM) module was proposed. This module dynamically adjusts the weights of features at different scales based on an adaptive mechanism, fully leveraging the advantages of features at each scale to achieve efficient fusion. Attached Figure Description

[0082] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0083] Figure 1 This is a schematic diagram of the overall structure of the GD-UNet++ network used in the image analysis method for identifying post-earthquake collapsed buildings according to the present invention;

[0084] Figure 2 This is a schematic diagram of the MST structure of the image analysis method for identifying post-earthquake collapsed buildings according to the present invention;

[0085] Figure 3 This is a schematic diagram of the Transformer encoder and decoder of the image analysis method for identifying post-earthquake collapsed buildings according to the present invention;

[0086] Figure 4 This is a schematic diagram of the DIFM structure of the image analysis method for identifying post-earthquake collapsed buildings according to the present invention;

[0087] Figure 5 These are the original images of the dataset and their corresponding ground truth maps of the image analysis method used in this invention to identify post-earthquake collapsed buildings;

[0088] Figure 6 This is a schematic diagram of the HIT dataset detection performance of the image analysis method of this invention for identifying post-earthquake collapsed buildings;

[0089] Figure 7 This is a schematic diagram of the YSU dataset detection performance of the image analysis method of this invention for identifying post-earthquake collapsed buildings. Detailed Implementation

[0090] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0091] This invention provides an image analysis method for identifying buildings collapsed after an earthquake. To improve the model's ability to extract global features, the following method is employed. Figure 1 The GD-UNet++ architecture is used. The GCEM module mainly consists of convolutional layers, MST, and a Transformer encoder / decoder. The original image (Img) is input into the GCEM module through the first branch, where convolutional layers extract feature maps, and the global feature map is generated through global modeling using MST and the Transformer encoder / decoder. Simultaneously, Img is input into UNet++ through the second branch, where it undergoes downsampling by the encoder and progressive fusion by the decoder to obtain a feature layer output containing rich local information. Based on this, the global feature map generated by the GCEM module and the local features of UNet++ are fused using DFIM, enabling multi-scale fusion of global and local features at different levels to obtain the prediction result.

[0092] The specific steps include:

[0093] S1. Construct the recognition network, which includes the UNet++ network, the Global Context Enhancement Module (GCEM), and the Dynamic Feature Interaction Module (DFIM).

[0094] When identifying collapsed buildings using high-resolution remote sensing images, UNet++ struggles to balance global information capture with local feature correlation. Its global modeling mechanism is weak when faced with images full of interfering elements after an earthquake, affecting the accuracy and completeness of detection.

[0095] The Global Context Enhancement Module (GCEM) is used to achieve effective association between features and the global scene. The GCEM consists of three parts: Multi-Scale Semantic Tag (MST), Transformer encoder, and Transformer decoder.

[0096] The multi-scale semantic tagger (MST) incorporates a multi-scale attention module, generates a semantic tag set through Softmax, then the Transformer encoder models the context of the semantic tag set, and the Transformer decoder maps it back to the pixel space, thereby effectively associating features with the global scene.

[0097] S2. Simultaneously input the post-earthquake optical images into the Global Context Enhancement Module (GCEM) and the UNet++ network.

[0098] The Global Context Enhancement (GCEM) module extracts feature maps through convolutional layers, generates a semantic tag set through the Multi-Scale Semantic Tag Set (MST), and then performs global modeling through the Transformer encoder and decoder to output a global feature map.

[0099] The Global Context Enhancement Module (GCEM) includes a multi-scale semantic tagger (MST), a Transformer encoder, and a decoder.

[0100] Transformers are suitable for processing sequential data, and their standard input is a two-dimensional matrix, which cannot be directly applied to the three-dimensional structure of an image (height H × width W × number of channels C). Therefore, it is necessary to convert image features into a sequence of semantic units, with each unit corresponding to an embedding vector, treating pixel blocks as independent tags similar to words in text for computation.

[0101] Please see Figure 2 This invention provides a multi-scale semantic tagger (MST) for performing multi-scale attention calculations on feature maps extracted from convolutional layers to generate a multi-scale semantic tag set; specifically, it includes two parts.

[0102] The upper part reshapes the input feature map into vector form. The convolutional layer extracts features X∈R. H×W×C The input tensor X is reshaped from (H, W, C) to ((H, W), C); where H, W, and C represent the image height, width, and number of channels, respectively.

[0103] The second half generates a multi-scale spatial attention map in the initial stage through a multi-scale attention module.

[0104] First, multi-scale features are extracted using 1×1, 3×3, and 5×5 convolutional kernels, and average pooling and max pooling are used to enhance feature representation: P i =AP(C i(X))+MP(C i (X)).

[0105] Among them, P i Let C represent the i-th enhancement feature, and let AP and MP distributions represent average pooling and max pooling, respectively. i This represents a convolution operation with a kernel size of i×i (i = 1, 3, 5).

[0106] The attention weight matrix is ​​generated using two fully connected layers with ReLU and sigmoid functions. i = 1, 3, 5 This is a fully connected layer. The multi-scale features are multiplied by the corresponding weight map to obtain the multi-scale attention features A = {A...} i |i=1,3,5}.

[0107] Secondly, by integrating attention features at different scales, we obtain the feature Y = C3(∑(A1,A3,A5))×X. Then, we supplement the features using a spatial attention module to obtain a multi-scale attention map. Using the multi-scale attention map, we calculate the weighted average sum of each pixel in X, and then use pointwise convolution to obtain the multi-scale semantic map Z∈R. H×W×L .

[0108] Finally, the result is multiplied by the tag set obtained in the first part through the Softmax operation to obtain the multi-scale semantic tag set T∈R. L×R The specific formula is as follows:

[0109] T=(σ(Z)) T W=(σ(φ(SA(Y)×Y))) T W.

[0110] Where W represents the feature vector, and SA() represents the spatial attention module. For pointwise convolutional layers with autonomous learning capabilities, σ() represents the Softmax operation.

[0111] Please see Figure 3 This invention provides a Transformer encoder and decoder.

[0112] The Transformer encoder adds positional encoding to the semantic tag set and extracts contextual features through a multi-head self-attention mechanism; the Transformer decoder maps semantic features back to pixel space through a multi-head cross-attention layer and outputs a global feature map.

[0113] Transformer encoder operation includes:

[0114] First, encode the position U pe ∈R L×RIntegrate into the semantic tag set T, and then combine the semantic tag set T∈R L×R The input is fed into the Transformer encoder to obtain a new set of semantic tags T. new After inputting the semantic tag set T into the Transformer encoder, we can obtain q,k,v∈R. L×R The formula is as follows:

[0115] q=TM q k = TM k v = TM v .

[0116] Where q, k, and v are the query vector, key vector, and value vector, respectively, and M... q M k M v ∈R C×d Representing a linear matrix capable of autonomous learning, the formula for a single self-attention layer, Attention(q,k,v), is as follows:

[0117]

[0118] Where d represents the number of channel dimensions for q, k, and v.

[0119] The multi-head self-attention mechanism in the Transformer encoder simultaneously feeds input data into multiple independent attention computation units (called "heads"). Each unit generates its own query vector q, key vector k, and value vector v. Since each head uses different parameters during initialization, these vectors generate differentiated attention weights during computation, enabling them to comprehensively process multivariate information from different locations and representation subspaces. The outputs of each head are concatenated, linearly transformed, and then projected to obtain the final value.

[0120] The formula for representing the multi-head self-attention layer MSA(T) is as follows:

[0121] MSA(T)=Concat(head1,...,head n M o .

[0122]

[0123] Among them, head j For the j-th attention head, M o ∈R hd×C represents the linear projection matrix, and h represents the number of self-attention layers.

[0124] After processing by the Transformer encoder, semantic features rich in contextual relationships can be extracted. These high-density semantic features are of great value for identifying target regions. To achieve the mapping and reconstruction of semantic features to pixel space, a Transformer decoder is needed to further optimize and adjust the image features.

[0125] The Transformer decoder operation includes:

[0126] The image features are further optimized and adjusted using a Transformer decoder. The Transformer decoder consists of N... Td The layer is composed of units, each unit including a multi-head cross-attention layer and a multi-layer perceptron layer. The formula for representing the multi-head cross-attention layer is as follows:

[0127] MCA(T)=Concat(head1,...,head n M o .

[0128]

[0129] The perceptron layer also contains two linear projection layers, one of which uses the Gaussian error linear unit (GELU), as shown in the following formula:

[0130] MLP(T) = GELU(TM1)M2.

[0131] Where M1, M2∈R C×C This represents a linear transformation matrix that can learn autonomously.

[0132] The Transformer encoder enhances the structural understanding of collapsed areas by modeling long-distance dependencies through multi-head self-attention. The Transformer decoder maps semantic features back to pixel space through cross-attention, suppressing background interference.

[0133] The UNet++ network outputs local feature maps through encoder downsampling and decoder fusion operations.

[0134] S3. Input the global feature map and local feature map into the dynamic feature interaction module DFIM, and achieve multi-scale feature fusion through weight adjustment to generate the collapse building recognition result.

[0135] While UNet++ networks can achieve progressive fusion and refined reconstruction of multi-scale features through dense skip connections, they struggle to dynamically perceive the differences in importance between features at different scales during cross-level feature fusion. UNet++'s fixed fusion strategy cannot adaptively adjust weights based on the content of the input features, leading to problems such as insufficient response to key features and increased interference from redundant information.

[0136] Therefore, a dynamic feature interaction module DIFM is designed, which can capture the complex relationships between local and global features and selectively focus on the most informative features to learn the joint representation of local and global features.

[0137] Please see Figure 4 The Dynamic Feature Interaction Module (DFIM) performs the following operations:

[0138] First, the global and local feature maps are input into the Dynamic Feature Interaction Module (DFIM). A 3×3 convolutional layer transforms the global and local feature maps, employing residual connections and the ReLU activation function. Specifically, this is represented as follows:

[0139] O g =C3(ReLU(C3(T) g )))+T g .

[0140] O l =C3(ReLU(C3(T) l )))+T l .

[0141] Among them, O g Represents global features, O l C3 represents a local feature, and C3 indicates a convolution operation with a kernel size of 3×3.

[0142] Secondly, the transformed global and local features are concatenated and then subjected to a 1×1 convolution to generate an attention score. The attention weights of the global and local features in different dimensions are calculated using the Softmax function.

[0143] Then, the global and local features are multiplied element-wise with their corresponding attention weights to obtain the weighted WSum of the global features. g Weighted WSum of local features l .

[0144] Finally, the global weighted average and the local weighted average are added element by element to obtain the output result.

[0145] The weighted calculation formula is as follows:

[0146] WSum g=Softmax(C1(Concat(O g O l )))T g .

[0147] WSum l =Softmax(C1(Concat(O g O l )))T l .

[0148] Residual connections are used to enhance feature representation and avoid gradient vanishing. Softmax is used to dynamically allocate global / local feature weights to highlight key scale features (such as collapsed boundary textures).

[0149] An electronic device includes a processor and a memory, the memory storing a computer program, and the processor executing the program to implement an image analysis method for identifying buildings that have collapsed after an earthquake.

[0150] An image analysis system for identifying collapsed buildings after an earthquake includes an image input module, a global feature extraction module, a local feature extraction module, a feature fusion module, and an output module.

[0151] The image input module is used to receive post-earthquake optical images.

[0152] The global feature extraction module is implemented by the global context enhancement module GCEM, which includes convolutional layers, multi-scale semantic tagger MST, Transformer encoder and decoder.

[0153] Convolutional layers are used to extract feature maps from the input image. The Multi-Scale Semantic Tag (MST) generates multi-scale attention features using 1×1, 3×3, and 5×5 convolutional kernels, and outputs a semantic tag set in conjunction with a spatial attention module. The Transformer encoder and decoder add positional encoding to the semantic tag set and output a global feature map through multi-head self-attention and cross-attention mechanisms.

[0154] The local feature extraction module is implemented by the UNet++ network, which outputs local feature maps through encoder downsampling and decoder fusion operations.

[0155] The feature fusion module is implemented by the dynamic feature interaction module DFIM, and performs the following operations:

[0156] The global and local feature maps are transformed by 3×3 convolutional layers, and residual connections are used.

[0157] The concatenated and transformed features are then used to generate attention scores via 1×1 convolution.

[0158] The weights of global and local features are calculated using the Softmax function.

[0159] The weighted global features are added to the local features to generate the collapsed building identification result.

[0160] The GCEM branch extracts global semantic features (suppressing noise such as rubble piles and vegetation); the UNet++ branch preserves local detailed features (such as the broken edges of buildings); and DFIM merges the two to achieve refined boundary recognition.

[0161] The output module is used to output the recognition results to a display device or storage medium.

[0162] Example 1:

[0163] Collapsed buildings are a core focus of post-earthquake emergency response during rescue operations. The semantic segmentation dataset is divided into two categories: collapsed buildings and others. To verify the accuracy and robustness of the proposed network, experiments are conducted using remote sensing imagery from two different regions: HIT and YSU.

[0164] Please see Figure 5 This section displays the original images and their corresponding ground truth maps for two datasets, with red areas representing collapsed buildings. The HIT earthquake in the region reached magnitude 7.0, and the spatial resolution of the satellite imagery is 0.45m. The YSU earthquake in the region reached magnitude 7.1, and the spatial resolution of the satellite imagery is 0.41m in the panchromatic band and 1.65m in the multispectral band.

[0165] The HIT dataset consists of 3228 samples, and the YSU dataset consists of 1942 samples. The HIT and YSU datasets are further divided as shown in Table 1:

[0166] Table 1. Experimental sample allocation for the two datasets.

[0167] Dataset training set Validation set test set HIT 2422 322 484 YSU 1456 194 292

[0168] The experimental environment was built on Ubuntu 16.04 operating system, using the TensorFlow-1.14.0 framework as the development foundation. For hardware, it was equipped with an Nvidia GeForce RTX 2080ti GPU with 11GB of VRAM. In the experimental parameter settings, the initial learning rate was set to 1e-5, and the network was optimized using the Adam optimizer with weight decay of 1e-5, employing a dynamically adjusted learning rate strategy with exponential decay.

[0169] In addition to the base network UNet++, this chapter employs various semantic segmentation methods for comparative experiments based on the HIT and YSU datasets to fully verify the performance of GD-UNet++, as detailed below:

[0170] OB-UNet is a collapsed building recognition method based on object context and boundary enhancement loss mechanism. It embeds an OCR module into the network architecture to improve feature representation capability, and designs a new loss function BE Loss based on the distance between pixels and boundaries, so that the network can focus more on learning boundary pixels.

[0171] ConvNeXt V2 employs a fully convolutional mask encoder and a global response normalization layer, thereby enhancing feature competition between channels;

[0172] CSC-Unet significantly improves the model's performance in recovering spatial detail information by introducing a multi-layer convolutional sparse coding module to replace traditional convolution operations and integrating the ML-CSC module into the UNet network architecture.

[0173] All experiments were conducted using the HIT and YSU datasets. Figure 6 and Figure 7 The results of collapsed building inspections obtained using different methods are presented, and the detailed analysis is as follows:

[0174] Figure 6 This section showcases the detection performance of different methods on the HIT dataset. The first two rows address scenarios with fewer collapsed buildings, where the GD-UNet++ method proposed in this chapter demonstrates significantly fewer false negatives and false positives compared to other methods. Figure 6 In the first row, due to differences in the color and texture of the collapsed buildings, other methods resulted in serious missed detections and false detections, while GD-UNet++ achieved ideal detection results. Figure 6 In the second row, when faced with a complex-shaped collapsed building, although all five methods can outline the general shape, GD-UNet++ has a significant advantage in boundary details. Figure 6 The last two lines show a large number of collapsed buildings, and only GD-UNet++ can detect the collapsed buildings relatively completely.

[0175] Because OB-UNet requires boundary information to calculate BE Loss and removes single-class pixel images from the training set before model training and prediction, its accuracy is not high despite being able to extract some collapsed buildings. ConvNeXt V2's fully convolutional mask encoder struggles to distinguish collapsed buildings from surrounding environmental interference, resulting in a high probability of missed and false detections of collapsed buildings. CSC-Unet's multi-layer convolutional sparse coding module fails to effectively balance the correlation and differences of sparse features across different convolutional layers, leading to false and missed detections. In contrast, GD-UNet++ effectively reduces the number of missed and false detections of collapsed buildings, yielding more accurate results.

[0176] Figure 7 This section showcases detection results obtained using different methods on the YSU dataset. In scenarios with extremely complex backgrounds and completely destroyed buildings, most of the four methods compared were incomplete in extracting the buildings. This is because, compared to the HTI dataset, the YSU dataset shows collapsed buildings with higher degrees of damage and mixed distribution with other ground features, increasing the difficulty of collapse classification and boundary prediction. In contrast, GD-UNet++ demonstrates a significant advantage in collapsed building extraction, with visual results closest to the ground truth map. The first two rows show scenarios with fewer collapsed buildings; other methods misclassified many collapsed building pixels, while GD-UNet++ effectively delineated the outlines and had the fewest missed detections. Figure 7 The last two lines present post-earthquake scenes with many collapsed buildings. In complex scenarios with dense buildings and a mix of collapsed and uncollapsed buildings, other methods are less accurate, with frequent missed detections and false detections. GD-UNet++ has better detection area integrity, effectively reducing missed detections and false detections.

[0177] OB-UNet introduces spatial location information of pixels within objects, while CSC-UNet replaces conventional convolution operations with multi-layer convolutional sparse coding blocks. However, neither method considers semantic differences when fusing at different levels, making it difficult to accurately reconstruct the precise outline of collapsed buildings. Furthermore, although ConvNeXt V2 employs self-supervised learning techniques, it struggles when extracting a large number of collapsed buildings. Figure 7 There are still a large number of false positives and false negatives in the third and fourth rows. Regardless of whether the number of collapsed buildings is large or small, GD-UNet++'s detection results are more complete.

[0178] In summary, GD-UNet++ demonstrates significantly superior detection results and boundary detail recovery capabilities compared to other methods. GD-UNet++ leverages the Global Context Enhancement (GCEM) module to construct a multi-scale semantic tagger (MST), accurately capturing the detailed features of collapsed buildings. Furthermore, by incorporating the encoding and decoding modules of a Transformer network, it enhances the global semantic understanding of the collapsed area and effectively models long-distance dependencies.

[0179] On the other hand, GD-UNet++ uses the Dynamic Feature Interaction Module (DFIM) and an adaptive mechanism to dynamically adjust the weights of features at different scales based on the feature situation, so that the advantages of each scale feature can be fully utilized, making the semantic information of the fused features richer and more accurate.

[0180] Example 2:

[0181] Multiple evaluation indicators were used to evaluate the quantitative accuracy, as shown in Tables 2 and 3:

[0182] Table 2 Quantitative Evaluation of the HIT Dataset

[0183]

[0184] Table 3 Quantitative Evaluation of the YSU Dataset

[0185]

[0186] Table 2 presents the quantitative evaluation data of different methods on the HIT dataset, with bold indicating the best results.

[0187] Compared to UNet++, GD-UNet++ has significant advantages. In terms of mIoU accuracy, GD-UNet++ improves by 7.04% compared to UNet++; for the collapsed building category, both IoU and F1 score have achieved increases of 12.92% or more.

[0188] In comparison with OB-UNet, GD-UNet++ also performed well, with an improvement of 4.78% in mIoU and 13.77% in IoU accuracy for collapsed building classes, and an increase of 12.92% in F1 score.

[0189] For ConvNeXt V2 and CSC-Unet, GD-UNet++ improves mIoU accuracy by more than 0.81%, and for the collapsed building class, it improves IoU accuracy by 1.58%, and the F1 score also increases by 1.02%.

[0190] Table 3 presents the quantitative evaluation data of different methods on the YSU dataset, with bold indicating the best results.

[0191] Compared with other methods, GD-UNet++ shows a significant improvement in accuracy when detecting collapsed building categories.

[0192] In the category of collapsed buildings, the IoU index was improved by 1.19% or more; in terms of F1 score, there was also an improvement of 0.7% or more; the overall mIoU was improved by 0.74% to 11.79%, fully demonstrating the comprehensive advantages of GD-UNet++ in detection of various categories.

[0193] Due to the different levels of complexity between the HIT and YSU datasets, GD-UNet++ performs better on the YSU dataset.

[0194] In the HIT dataset, collapsed buildings are small in scale and extremely densely distributed, greatly increasing the difficulty of detection and boundary prediction. However, in the relatively simple YSU dataset, GD-UNet++, with its unique network architecture design, is able to better capture the features of collapsed buildings, thus achieving more accurate detection and demonstrating superior performance compared to other methods across various metrics.

[0195] To verify the effectiveness of each module, IoU and F1 score were used as evaluation criteria, and the model was analyzed using the HIT and YSU datasets, as shown in Table 4:

[0196] Table 4 shows the accuracy results of the module effectiveness analysis (√ indicates that the component was used, – indicates that the component was not used, and bold indicates the best results).

[0197]

[0198] Analysis of Table 4 shows that on the HIT and YSU datasets, the accuracy of the collapsed building category and other categories both increased when the GCEM or DIFM module was added to the network.

[0199] With the addition of GCEM, the mIoU of the HIT dataset improved by 3.27%, and that of the YSU dataset improved by 5.54%. However, with the addition of DIFM, for the collapsed building category, the IoU of the HIT and YSU datasets increased by 7.36% and 8.11%, respectively, while the F1 scores increased by 7.37% and 7.4%, respectively.

[0200] The experimental results show that embedding GCEM or DIFM modules into the UNet++ base network improves both the overall mIoU and the IoU and F1 score of a single class to varying degrees, thus proving the effectiveness of GCEM and DIFM.

[0201] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0202] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An image analysis method for identifying post-earthquake collapsed buildings, characterized in that: The method includes: S1. Construct the recognition network, which includes the UNet++ network, the Global Context Enhancement Module (GCEM), and the Dynamic Feature Interaction Module (DFIM). The Global Context Enhancement Module (GCEM) includes a multi-scale semantic tagger (MST), a Transformer encoder, and a decoder. The Multi-Scale Semantic Tag Set (MST) is used to perform multi-scale attention calculations on the feature maps extracted by the convolutional layers, generating a multi-scale semantic tag set; it specifically consists of two parts. The upper part reshapes the input feature map into vector form; the convolutional layer extracts features. The input tensor X is reshaped from (H, W, C) to ((H, W), C); where H, W, and C represent the image height, width, and number of channels, respectively. The lower part generates a multi-scale spatial attention map in the initial stage through a multi-scale attention module; First, multi-scale features are extracted using 1×1, 3×3, and 5×5 convolutional kernels, and average pooling and max pooling are used to enhance feature representation. ; in, Let AP and MP represent the i-th enhancement feature, and let AP and MP represent average pooling and max pooling, respectively. Indicates the kernel size as The convolution operation; where, ; The attention weight matrix is ​​generated using two fully connected layers with ReLU and sigmoid functions. , This is a fully connected layer; multi-scale features are multiplied by the corresponding weight map to obtain multi-scale attention features. ; Secondly, features are obtained by integrating attention features at different scales. Then, features are supplemented through a spatial attention module to obtain a multi-scale attention map; using the multi-scale attention map, the weighted average sum of each pixel in X is calculated, and pointwise convolution is used to obtain a multi-scale semantic map. ; Finally, the result is multiplied by the tag set obtained in the first half through the Softmax operation to obtain the multi-scale semantic tag set. The specific formula is as follows: ； in, Represents the eigenvector. Represents the spatial attention module. For pointwise convolutional layers with autonomous learning capabilities, This indicates the Softmax operation; S2. Simultaneously input the post-earthquake optical images into the Global Context Enhancement Module (GCEM) and the UNet++ network; The Global Context Enhancement (GCEM) module extracts feature maps through convolutional layers, generates a semantic tag set through the Multi-Scale Semantic Tag Set (MST), and then performs global modeling through the Transformer encoder and decoder to output a global feature map. The UNet++ network outputs local feature maps through encoder downsampling and decoder fusion operations. S3. Input the global feature map and the local feature map into the dynamic feature interaction module DFIM, and achieve multi-scale feature fusion through weight adjustment to generate the collapsed building recognition result. The Dynamic Feature Interaction Module (DFIM) performs the following operations: First, the global and local feature maps are input into the Dynamic Feature Interaction Module (DFIM). A 3×3 convolutional layer transforms the global and local feature maps, employing residual connections and the ReLU activation function; specifically: ； ； in, Represents global characteristics. Representing local features, This indicates a convolution operation with a kernel size of 3×3; Secondly, the global and local features after concatenation and transformation are combined and then convolved with a 1×1 convolution to generate an attention score; the attention weights of the global and local features in different dimensions are calculated using the Softmax function. Then, the global and local features are multiplied element-wise with their corresponding attention weights to obtain the weighted sum of the global features. Weighted sum of local features ; Finally, the global weighted average and the local weighted average are added element by element to obtain the output result.

2. The image analysis method for identifying post-earthquake collapsed buildings according to claim 1, characterized in that: The Transformer encoder is used to add positional encoding to a set of semantic tags and extract contextual features through a multi-head self-attention mechanism. The Transformer decoder is used to map semantic features back to pixel space through a multi-head cross-attention layer, outputting a global feature map.

3. The image analysis method for identifying post-earthquake collapsed buildings according to claim 2, characterized in that: Transformer encoder operation includes: First encode the position Integrate into the semantic tag set T, and then combine the semantic tag set... The input is fed into the Transformer encoder to obtain a new set of semantic tags. After the semantic tag set T is input into the Transformer encoder, the following can be obtained: The formula is as follows: ； ； ； Where q, k, and v are the query vector, key vector, and value vector, respectively. Represents a linear matrix capable of autonomous learning, a single self-attention layer. The formula for representing is as follows: ； Where d represents the number of channel dimensions for q, k, and v; Multi-head self-attention layer The formula for representing is as follows: ； ； in, For the j-th attention head, , represents the linear projection matrix, and h represents the number of self-attention layers.

4. The image analysis method for identifying post-earthquake collapsed buildings according to claim 3, characterized in that: The image features are further optimized and adjusted using a Transformer decoder. The Transformer decoder consists of... The unit is composed of layers, each of which includes a multi-head cross-attention layer and a multi-layer perceptron layer; The perceptron layer also contains two linear projection layers, one of which uses the Gaussian error linear unit (GELU), as shown in the following formula: ； in This represents a linear transformation matrix that can learn autonomously.

5. The image analysis method for identifying post-earthquake collapsed buildings according to claim 1, characterized in that: The weighted calculation formula is as follows: ； 。 6. An electronic device, characterized in that: It includes a processor and a memory, the memory storing a computer program, and the processor executing the program to implement the method as described in any one of claims 1-5.

7. An image analysis system for identifying post-earthquake collapsed buildings, applied to the image analysis method for identifying post-earthquake collapsed buildings as described in claim 1, characterized in that: The system includes an image input module, a global feature extraction module, a local feature extraction module, a feature fusion module, and an output module; The image input module is used to receive post-earthquake optical images; The global feature extraction module is implemented by the global context enhancement module GCEM, which includes convolutional layers, multi-scale semantic tagger MST, Transformer encoder and decoder; Convolutional layers are used to extract feature maps from the input image; the multi-scale semantic tagger (MST) generates multi-scale attention features through 1×1, 3×3, and 5×5 convolutional kernels, and outputs a set of semantic tags by combining with the spatial attention module; the Transformer encoder and decoder add positional encoding to the set of semantic tags and output a global feature map through multi-head self-attention and cross-attention mechanisms. The local feature extraction module is implemented by the UNet++ network, which outputs local feature maps through encoder downsampling and decoder step-by-step fusion operations; The feature fusion module is implemented by the dynamic feature interaction module DFIM, and performs the following operations: The global and local feature maps are transformed by 3×3 convolutional layers and residual connections are used. The transformed features are concatenated and then convolved with 1×1 to generate attention scores. The weights of the global and local features are calculated using the Softmax function. The weighted global and local features are added together to generate the collapsed building recognition result. The output module is used to output the recognition results to a display device or storage medium.

Images