A method for identifying damage to a building target

Abstract

The application relates to a building target damage identification method, which comprises the following steps: constructing a building target damage identification model; the building target damage identification model comprises a feature extraction module, a semantic segmentation and change detection module; the building target damage identification model is trained based on a training data set to obtain a trained building target damage identification model; a building target pre-attack image and a building target post-attack image to be identified are input into the trained building target damage identification model to obtain a post-attack image damage semantic prediction segmentation graph and a binary classification damage prediction change graph; and the binary classification damage prediction change graph is used to mask the post-attack image damage semantic prediction segmentation graph to obtain a damage semantic change detection graph. The application constructs a light-weight binary classification change detection branch, and combines semantic segmentation to realize high-precision extraction of position and damage type information of multiple types of damage change regions such as broken regions, collapsed regions and hole regions.

Description

Technical Field

[0001] This application relates to the field of multi-type damage identification of building targets, and more specifically, to a method for identifying damage to building targets. Background Technology

[0002] After being attacked, building targets often exhibit various types of damage, including fractured areas, collapsed areas, and voided areas. Identifying these multiple types of damage to building targets is crucial for achieving efficient and accurate damage performance assessment. However, existing methods for identifying these various types of damage to building targets mostly rely on binary qualitative information extraction. This only provides a visual representation of the damage changes after an attack and does not reveal the specific type of damage. Summary of the Invention

[0003] In order to overcome at least one deficiency in the prior art, this application provides a method for identifying damage to building targets.

[0004] Firstly, a method for identifying damage to building targets is provided, including: A building target damage recognition model is constructed. The building target damage recognition model includes a feature extraction module and a semantic segmentation and change detection module. The semantic segmentation and change detection module includes a damage semantic segmentation head and a change detection module based on cross-learning time correlation modeling. The building target damage recognition model is trained based on the training dataset to obtain the trained building target damage recognition model; the samples in the training dataset include dual-temporal image pairs of building targets before and after being hit, namely, the image of the building target before being hit and the image of the building target after being hit. During training, the dual-temporal images of the building target before and after the impact are input to the feature extraction module to obtain semantic features of the image before the impact and semantic features of the image after the impact. The semantic features of the image before the impact and semantic features of the image after the impact are input to the semantic segmentation and change detection module. The damage semantic segmentation head obtains the damage semantic prediction segmentation map of the building target before the impact and the damage semantic prediction segmentation map of the image after the impact based on the semantic features of the image before the impact and the image after the impact. The change detection module based on cross-learning temporal correlation modeling realizes the temporal correlation modeling of the dual-temporal damage semantic features based on the semantic features of the image before the impact and the image after the impact, and completes the binary classification damage change detection to obtain the binary classification damage prediction change map. The images of the building target before and after the impact are input into the trained building target damage recognition model to obtain a damage semantic prediction segmentation map and a binary classification damage prediction change map of the post-impact image. The binary classification damage prediction change map is used to mask the damage semantic prediction segmentation map of the post-impact image to obtain a damage semantic change detection map.

[0005] In one embodiment, the change detection module based on cross-learning time correlation modeling includes a cross-learning time correlation modeling module and a binary damage change detection head; The cross-learning time-related modeling module includes an attention module; the semantic features of the pre-attack image are input into the attention branch to obtain the query. ,value ,key The semantic features of the image after the attack are input into the attention module to obtain the query. ,value ,key ; query s and keys Multiply the results and apply the softmax function to obtain the similarity matrix. ; query s and keys Multiply the results and apply the softmax function to obtain the similarity matrix. Similarity matrix Sum Multiplying, after convolution, and adding to the semantic features of the image before the attack, yields the cross-learned semantic features of the image before the attack; similarity matrix. Sum Multiply, and after convolution, add to the semantic features of the post-hit image to obtain the semantic features of the post-hit image after cross-learning; The semantic features of the pre-attack image after cross-learning and the semantic features of the post-attack image after cross-learning are input into the binary damage change detection head to obtain a binary damage prediction change map.

[0006] In one embodiment, the attention module includes three parallel convolutional layers, two parallel pooling layers, a reshape layer, and a transpose layer; The semantic features of the image before the attack are input into three convolutional layers respectively to obtain the first convolution result, the second convolution result, and the third convolution result; The first and third convolution results are each passed through a pooling layer to obtain the first pooling result and the second pooling result. The first pooling result, the second convolution result, and the second pooling result are all input into the Reshape layer to obtain the first reshaping result, the second reshaping result, and the third reshaping result; The first reshaping result is input into the Transpose layer to obtain the query. The second reshaping result is used as a value. The third reshaping result serves as the key. .

[0007] In one embodiment, the total loss used during training is:

[0008] in, For the total loss, For dual-temporal damage semantic segmentation loss, For the detection of damage changes in two categories, For semantic interaction contrast loss, As an additional constraint loss, and Representing semantic interaction contrast loss respectively and additional constraint losses The weight.

[0009] In one embodiment, the bi-temporal destructive semantic segmentation loss for:

[0010]

[0011] in, Indicates phase Destructive semantic segmentation loss, Indicates phase Destructive semantic segmentation loss, Indicates phase Destructive semantic segmentation loss, Indicates phase The first in the damage semantic truth segmentation graph The pixel belongs to the first The probability of damage of this type. Indicates phase The damage semantic prediction segmentation graph The pixel belongs to the first The probability of damage of this type. Indicates the number of pixels. Indicates the number of damage types.

[0012] In one embodiment, binary damage change detection loss for:

[0013] in, The graph showing the true value changes of binary damage is shown in the figure. The pixel belongs to the first The probability of a class 0 indicates an unchanged type, and 1 indicates a changed type. The figure showing the change in damage prediction for binary classification is shown below. The pixel belongs to the first The probability of a class This represents a graph showing the true value changes in binary damage classification. This represents a graph showing the changes in damage prediction for two categories. This indicates the number of pixels.

[0014] In one embodiment, additional constraint loss for:

[0015] in, The first in the binary classification prediction change graph The pixel value of each pixel. The first of the two-phase semantic difference features The pixel value of each pixel. This is the upper bound for the semantic differences in two-phase damage. Indicates the number of pixels. Representing the semantic difference features of two phases The number of channels.

[0016] In one embodiment, semantic interaction contrastive loss for:

[0017]

[0018]

[0019]

[0020]

[0021]

[0022] in, This indicates local contrast loss. Indicates global comparison loss; Indicates the damage type of phase t The query vector, This indicates a counting operation. and These represent the number of damage types present in the pre-strike and post-strike phases of the current training batch, respectively. Indicates a positive sample. Represents the positive sample set. This represents the set of damage types for phase t; Indicates InfoNCE loss, Indicates the scaling factor. Indicates a negative sample. Represents the negative sample set; This represents the contrast loss between positive sample pairs. This represents the contrast loss between negative sample pairs. This represents the memory queue of the current training batch. This represents the cosine similarity.

[0023] In one embodiment, the method further includes: By mapping the bi-temporal semantic features to a high-dimensional dense feature space, a set of damage feature vectors is obtained; For each time phase and each damage type, multiple damage feature vectors are randomly extracted, and the average feature vector is calculated as the query vector; The set of damage feature vectors that have the same damage type as the query vector constitutes the positive sample set; the set of damage feature vectors that have different damage types from the query vector constitutes the negative sample set.

[0024] Secondly, a building target damage identification device is provided to implement the above-mentioned building target damage identification method.

[0025] Compared with the prior art, this application has the following beneficial effects: 1. The shared parameter semantic feature encoder-decoder extracts the dual-temporal damage semantic features and inputs them into the semantic interactive contrastive learning module. Under the guidance of the damage semantic truth segmentation map, positive and negative sample pairs for contrastive learning are constructed. By constructing a memory queue of damage feature vectors divided by damage type, the interactive contrast of dual-temporal damage semantic features at the local and global levels is completed, which enhances the intra-class similarity and inter-class separability of damage semantic features.

[0026] 2. A damage semantic segmentation branch based on a shared parameter segmentation head is used to complete damage semantic segmentation maps of images before and after a building target impact. Simultaneously, a change detection branch is constructed in parallel with the damage semantic segmentation branch. A cross-learning strategy is used to model the temporal correlation of dual-temporal damage semantic features, enabling the extraction of change information in the images before and after the building target impact. Furthermore, binary change information generated by the change detection branch is used to impose additional constraints on the dual-temporal damage semantic information, capturing the correlation between semantic and binary change information. Finally, the binary change map is used as a mask for the damage semantic segmentation map after the building target impact to obtain the final multi-type damage recognition results for the building target, achieving high-precision extraction of the location and damage type information of various damage change areas such as fractured areas, collapsed areas, and hole areas. Attached Figure Description

[0027] This application can be better understood by referring to the description given below in conjunction with the accompanying drawings, which, together with the detailed description below, are incorporated in and form part of this specification. In the drawings: Figure 1 A flowchart of a method for identifying damage to building targets is shown. Figure 2 A schematic diagram of a building target damage identification model is shown; Figure 3 A schematic diagram of a change detection module based on cross-learning temporal correlation modeling is shown. Figure 4 A flowchart illustrating the implementation of semantic interaction contrastive learning is provided. Figure 5 An example diagram showing the frontal impact position of a building target is shown; Figure 6 An example image of cropping and expanding a building target before and after impact is shown; Figure 7 The results of this application's method and other semantic change detection methods are shown in the identification of multiple types of damage to building targets in image data before and after actual impact. Detailed Implementation

[0028] Exemplary embodiments of the present application will be described below with reference to the accompanying drawings. For clarity and brevity, not all features of the actual embodiments are described in the specification. However, it should be understood that many embodiment-specific decisions can be made in the development of any such actual embodiment to achieve the developer’s specific objectives, and these decisions may vary as the embodiments differ.

[0029] It should also be noted that, in order to avoid obscuring this application with unnecessary details, only the device structure closely related to the solution of this application is shown in the accompanying drawings, while other details that are not closely related to this application are omitted.

[0030] It should be understood that this application is not limited to the described embodiments by virtue of the following description with reference to the accompanying drawings. In this document, embodiments may be combined with each other, features may be substituted or borrowed between different embodiments, and one or more features may be omitted in one embodiment, where feasible.

[0031] This application provides a method for identifying damage to building targets. Figure 1 A flowchart of a method for identifying damage to building targets is shown. See [link / reference]. Figure 1 The method mainly includes the following steps: Step S1: Construct a building target damage recognition model. The building target damage recognition model includes a feature extraction module and a semantic segmentation and change detection module. The semantic segmentation and change detection module includes a damage semantic segmentation head and a change detection module based on cross-learning time correlation modeling.

[0032] Figure 2 A schematic diagram of a building target damage identification model is shown.

[0033] Here, the feature extraction module includes an encoder and a decoder. The encoder uses EfficientNet-B0, and the decoder uses the Feature Pyramid Network (FPN). EfficientNet-B0 (the lightest variant in the EfficientNet family) is used as the encoder to extract dual-temporal deep semantic features, and the Feature Pyramid Network (FPN) is used as the decoder to obtain dual-temporal explicit damage semantic features, providing explicit damage semantic information for subsequent damage semantic segmentation and binary change detection.

[0034] Step S2: Train the building target damage recognition model based on the training dataset to obtain the trained building target damage recognition model; the samples in the training dataset include dual-temporal image pairs of building targets before and after being hit, namely, the image of the building target before being hit and the image of the building target after being hit.

[0035] The experimental data consisted of two-phase images of building targets before and after impact, obtained through real-world experiments, online searches, and impact effect simulations. Then, data annotations were performed on various types of damage, including fractured zones, collapsed zones, and voided zones. Preprocessing operations such as image registration and image enhancement were then applied to this data before it was incorporated into a multi-type damage recognition model for building targets.

[0036] During training, the dual-temporal images of the building target before and after the impact are input to the feature extraction module to obtain semantic features of the image before and after the impact. These semantic features are then input to the semantic segmentation and change detection module. The damage semantic segmentation head uses these features to generate damage semantic prediction segmentation maps for the building target before and after the impact. The change detection module, based on cross-learning temporal correlation modeling, uses these features to model the temporal correlation of the dual-temporal damage semantic features and performs binary damage change detection to obtain a binary damage prediction change map.

[0037] Define the images of the building target before and after the attack as follows: and , where 3 represents the number of channels in the input image (RGB three channels). The height of the input image. The width of the input image. and These are the semantic truth segmentation images of damage to building targets before and after the attack. This is a graph showing the true value changes of damage in two categories. It also includes two temporal images of a building target before and after being hit. and The damage semantic features extracted by the shared parameter semantic feature encoder-decoder are defined as follows: and The goal is to build a damage recognition model for basic building targets. The goal is to learn the semantic information of damage to building targets from images before and after an attack, enabling multi-type damage recognition of building targets. The final optimization objective is:

[0038] in and These are semantic segmentation images of damage to building targets before and after the attack. This represents the total number of all damage types; This is a graph showing the predicted changes in damage based on two categories. This is the total parameter set for the building target damage identification model.

[0039] Step S3: Input the images of the building target before and after the impact into the trained building target damage recognition model to obtain the damage semantic prediction segmentation map and the binary classification damage prediction change map of the post-impact image. Use the binary classification damage prediction change map to mask the damage semantic prediction segmentation map of the post-impact image to obtain the damage semantic change detection map.

[0040] To achieve multi-type damage identification of building targets, including fractured areas, collapsed areas, and void areas, a binary damage prediction change map is used as a change mask (0 for unchanged areas, 1 for changed areas) to segment the semantic damage prediction map of the post-impact image. Masking is applied. The resulting semantic change detection map for multiple types of damage is shown after masking. The damage semantic information of the changed areas is retained, while the damage semantic information of the unchanged areas is discarded, which serves as the result of multi-type damage identification for building targets.

[0041] In one embodiment, existing methods for detecting changes in building target damage only achieve binary classification of changes in minor and irregular damage. Due to limitations in the number of model parameters and the fact that they only consider damage change information (changed and unchanged) while ignoring damage semantic information (fractured areas, collapsed areas, and voided areas), they cannot be directly applied to multi-type damage recognition tasks for building targets. Therefore, a change detection module based on cross-learning temporal correlation modeling is designed, including a cross-learning temporal correlation modeling module and a binary damage change detection head, generating a binary damage change detection map to extract image change information before and after the building target is hit. The cross-learning temporal correlation modeling module uses a cross-learning strategy to model the temporal correlation of damage semantic features in both time phases, enhancing the similarity between the semantic features of damage in both time phases in unchanged areas, aiming to improve the accuracy of binary damage change detection.

[0042] Figure 3 A schematic diagram of a change detection module based on cross-learning temporal correlation modeling is shown. The change detection module based on cross-learning temporal correlation modeling includes a cross-learning temporal correlation modeling module (CLTC) and a binary damage change detection head. The cross-learning time-related modeling module includes an attention module; the semantic features of the pre-attack image are input into the attention branch to obtain the query. ,value ,key The semantic features of the image after the attack are input into the attention module to obtain the query. ,value ,key ; query s and keys Multiply the results and apply the softmax function to obtain the similarity matrix. ; query s and keys Multiply the results and apply the softmax function to obtain the similarity matrix. Similarity matrix Sum Multiplying, after convolution, and adding to the semantic features of the image before the attack, yields the cross-learned semantic features of the image before the attack; similarity matrix. Sum Multiply, and after convolution, add to the semantic features of the post-hit image to obtain the semantic features of the post-hit image after cross-learning; The semantic features of the pre-attack image after cross-learning and the semantic features of the post-attack image after cross-learning are input into the binary damage change detection head to obtain a binary damage prediction change map.

[0043] See Figure 3 The attention module includes three parallel convolutional layers, two parallel pooling layers, a reshape layer, and a transpose layer. The semantic features of the image before the attack are input into three convolutional layers respectively to obtain the first convolution result, the second convolution result, and the third convolution result; The first and third convolution results are each passed through a pooling layer to obtain the first pooling result and the second pooling result. The first pooling result, the second convolution result, and the second pooling result are all input into the Reshape layer to obtain the first reshaping result, the second reshaping result, and the third reshaping result; The first reshaping result is input into the Transpose layer to obtain the query. The second reshaping result is used as a value. The third reshaping result serves as the key. .

[0044] Specifically, firstly, the semantic features of dual-temporal damage. and The input is fed into three different convolutional layers, and the dual-temporal query matrix is ​​extracted through convolution, pooling, reshaping, and transpose operations, respectively. Key matrix Sum matrix :

[0045] in, The number of channels for the input feature; The height of the input feature; The width of the input feature; ; Indicates the use of phase Convolutional layers that destroy semantic features; This represents an adaptive average pooling layer that performs feature downsampling to extract global damage semantic information. To reduce computational complexity, a channel downsampling factor is introduced. It is usually set to 2; Indicates a reshaping operation; This indicates a transpose operation. Subsequently, the query matrices are swapped to obtain two sets. and A cross-global query of dual-temporal features is performed to obtain the similarity matrix of dual-temporal damage semantic features. and :

[0046] By performing a weighted summation of the similarity matrix and the value matrix, the bi-temporal damage semantic features after cross-learning are obtained. and Through the above operations, the similarity of the semantic features of the two-phase damage in the unchanged region is further improved, completing the temporal correlation modeling of the semantic features of the two-phase damage. Finally, binary classification damage change detection is completed using a change detection head.

[0047] in, This represents the convolution operation. This indicates a two-category damage change detection head. This represents a graph showing the changes in damage prediction for two categories. Self-attention mechanisms are crucial for capturing long-term dependencies by exploring the non-local operations of images in the spatiotemporal dimension. However, traditional CNNs, limited by their receptive field, cannot capture global dependencies between features. To overcome this limitation, this paper combines self-attention mechanisms with cross-learning strategies to model temporal correlations at the global level, improving the representation of semantic features of bi-temporal damage in unchanged regions and enhancing the correlation between semantic features of bi-temporal damage in unchanged regions.

[0048] In one embodiment, by learning semantic segmentation of building target damage, the model acquires the ability to recognize object-level damage features. Furthermore, by learning dual-temporal image change detection before and after a building target impact, the model also acquires the ability to recognize damage change features, thus achieving accurate identification of multiple types of damage to building targets, such as fractured areas, collapsed areas, and porous areas. The loss of the multi-task branch based on semantic segmentation and change detection includes three parts: dual-temporal damage semantic segmentation loss, binary classification damage change detection loss, and additional constraint loss from the dual-temporal semantic additional constraint module. Therefore, the total loss for identifying multiple types of damage to building targets is... It consists of four parts: dual-temporal damage semantic segmentation loss, binary classification damage change detection loss, additional constraint loss, and semantic interaction comparison loss, namely:

[0049] in, For the total loss, For dual-temporal damage semantic segmentation loss, For the detection of damage changes in two categories, For semantic interaction contrast loss, As an additional constraint loss, and Representing semantic interaction contrast loss respectively and additional constraint losses The weight, its function is as... and The scaling factor is adjusted by the order of magnitude of other losses.

[0050] Dual-phase damage semantic segmentation loss and two-class damage change detection loss Essentially, both are cross-entropy loss and have equal importance in the model training process, so and They have the same order of magnitude and do not require additional loss weights for adjustment.

[0051] The model's learning objective can be computed by integrating a shared-parameter semantic feature encoder-decoder and a multi-task branch of semantic segmentation and change detection. Definition Let be the total network parameter set for the building target multi-type damage recognition task. The learning objective for the building target multi-type damage recognition task is:

[0052] The Adam optimizer and an end-to-end strategy are employed, utilizing mini-batch gradient descent to minimize the total model loss. The model network parameters are then updated. Through multiple rounds of training iterations, the overall model loss function is... The parameter set exhibits a stable decreasing trend, and synchronously performs autonomous adaptive updates, thereby causing the total loss to eventually converge to the minimum value. Therefore, the update process of the model network parameter set is as follows:

[0053] in It is the learning rate.

[0054] Specifically, cross-entropy loss is used to calculate the difference between the predicted and ground truth segmentation maps of damage semantics before and after a building target is hit, and a dual-temporal damage semantic segmentation loss is employed. for:

[0055]

[0056] in, Indicates phase Destructive semantic segmentation loss, Indicates phase Destructive semantic segmentation loss, Indicates phase Destructive semantic segmentation loss, Indicates phase The first in the damage semantic truth segmentation graph The pixel belongs to the first The probability of damage of this type. Indicates phase The damage semantic prediction segmentation graph The pixel belongs to the first The probability of damage of this type. Indicates the number of pixels. Indicates the number of damage types.

[0057] Specifically, for the binary damage change detection task, a hybrid loss combining cross-entropy loss and dice loss is used as the loss function to mitigate the impact of sample imbalance. for:

[0058] in, The graph showing the true value changes of binary damage is shown in the figure. The pixel belongs to the first The probability of a class 0 indicates an unchanged type, and 1 indicates a changed type. The figure showing the change in damage prediction for binary classification is shown below. The pixel belongs to the first The probability of a class This represents a graph showing the true value changes in binary damage classification. This represents a graph showing the changes in damage prediction for two categories. This indicates the number of pixels.

[0059] Specifically, the multi-task branch based on semantic segmentation and change detection achieves multi-type damage recognition of building targets by jointly performing change detection and semantic segmentation. However, there are important differences between semantic segmentation and change detection tasks. Specifically, semantic segmentation has semantic information for each pixel of the image, while change detection only has information about a small portion of the changed region in the entire image. Therefore, mining the potential constraints of binary change information on damage semantic information is crucial for improving the performance of multi-type damage recognition. Therefore, this embodiment designs a Bi-temporal Semantic Additional Constraint (BSAC) module, which uses the binary change information generated by the change detection branch to impose additional constraints on the bi-temporal damage semantic information. Its purpose is to capture the correlation between semantics and binary change information. The key insight of this embodiment is recognizing that the binary predicted change map can serve as a bridge to incorporate correlation into the bi-temporal semantics. The theoretical basis behind the bi-temporal semantic additional constraint module is that the bi-temporal semantic labels on pixels in the changed region should be the same, while the semantic labels on pixels in the unchanged region should be different. Therefore, in the unchanged region of the binary predicted change map, the difference in bi-temporal damage semantic features is... The absolute value should be close to zero, while the difference should be larger in the region of variation, with additional constraint loss. for:

[0060] in, The first in the binary classification prediction change graph The pixel value of each pixel. The first of the two-phase semantic difference features The pixel value of each pixel. This is the upper bound for the semantic differences in two-phase damage. Indicates the number of pixels. Representing the semantic difference features of two phases The number of channels.

[0061] Here, the dual-temporal semantic difference feature refers to the difference between the semantic features of the image before the attack and the semantic features of the image after the attack.

[0062] The optimization objective is to minimize this loss. Therefore, in the unchanged region ( In (equal to 0), It should be as small as possible to minimize the semantic difference between the two-phase damage. Approaching zero. In the region of change ( In case 1), it is desirable that the semantic difference between the two temporal phases of damage be as large as possible. An upper bound m for the semantic difference between the two temporal phases of damage is defined for the region of change. When it is greater than m, The value is 0, and m is set to 2. The dual-temporal semantic additional constraint module captures the correlation between dual-temporal damage semantic features by constraining binary change information, so that unchanged regions have the same semantic label, indirectly improving the semantic segmentation branch and enhancing feature discrimination ability.

[0063] Specifically, contrastive learning is an effective method that enables models to acquire discriminative feature representations of various types in a dataset. Contrastive learning typically utilizes various augmentation techniques to construct positive and negative sample pairs and compares their distances in the feature space, aiming to minimize the distance between pixels of the same category while maximizing the distance between pixels of different categories. For the problem of identifying multiple types of damage to building targets, such as fractured areas, collapsed areas, and hole areas, this embodiment introduces a Semantic Interactive Contrastive Learning (SICL) module into the multi-type damage identification model for building targets. It utilizes the multi-type damage characteristics of dual-temporal images before and after a building target impact to construct positive and negative samples. Specifically, guided by the damage semantic ground truth segmentation map, training samples belonging to the same damage type but from the same or different temporal phases can be used as positive samples, and training samples belonging to different damage types can be used as negative samples to construct positive and negative sample pairs for contrastive learning. Furthermore, the SICL module implements semantic interactive comparison at both local and global levels, ensuring the balance and diversity of damage types during the contrastive learning process. At the local level, the features involved in each comparison come from a small subset of the training dataset (training samples in the current training batch), which may not cover all damage types, resulting in insufficient balance and diversity of damage types in the feature space. To address this issue, the features involved in the comparison process are extended to multiple training batches for global semantic interactive comparison to obtain a more comprehensive perspective. Through interactive comparison of bi-temporal damage semantic features, the distance between features of the same damage type is narrowed, while the distance between features of different damage types is increased. This addresses the challenges of feature confusion caused by damage category imbalance, confusion, and the complexity of the damage itself, improving the recognition performance of multiple damage types such as broken areas, collapsed areas, and void areas in building targets. Furthermore, the SICL module is only used in the model training phase and does not increase the computational load in the inference phase. Figure 4 The flowchart illustrating the implementation of semantic interaction contrastive learning is shown.

[0064] In contrastive learning, feature maps often need to be mapped to a high-dimensional, dense feature space through a representation head. For local levels, such as... Figure 4 As shown, the bi-temporal images of the current training batch are used to obtain bi-temporal damage semantic features through forward propagation of a shared parameter semantic feature encoder-decoder. and This is mapped to a high-dimensional dense feature space to obtain a set of damage feature vectors. Guided by the damage semantic ground truth segmentation map, the damage type label information of each damage feature vector is known. For each time phase and each damage type, multiple damage feature vectors are randomly extracted. To reduce computational cost, the average feature vector of each damage type in each time phase is used as the query vector. Damage feature vector set and query vector All damage feature vectors with the same damage type are positive samples. This forms a positive sample set to reduce the distance between feature vectors with the same damage type label in both temporal phases. Simultaneously, the damage feature vector set and the query vector... All damage feature vectors with different damage types are negative samples. This forms a negative sample set to increase the distance between feature vectors of different damage types. Finally, pixel-level information noise contrastive estimation (InfoNCE) is used to calculate the local contrastive loss for each damage type in both temporal phases. Then, the loss of all damage types in both temporal phases in the current training batch is averaged, as shown in the following formula:

[0065]

[0066] Local-level SICL performs bi-temporal damage semantic feature enhancement in the current training batch. However, not all damage types may always exist in this small subset, failing to meet the requirement of contrastive learning for diversity of positive and negative samples and reducing the effectiveness of contrastive learning. To address the limitations of local-level SICL, it is introduced to the global level, such as... Figure 4 As shown in the diagram. Specifically, this embodiment creates a memory queue categorized by damage type to store damage feature vectors from historical training batches. During training, damage feature vectors mapped from bi-temporal semantic features are randomly selected and stored in the memory queue. When the memory queue reaches its maximum capacity, it is updated according to a first-in, first-out (FIFO) rule. Subsequently, damage feature vectors from the memory dictionary are introduced into the contrastive learning process of the current training batch. The rules for constructing positive and negative samples remain consistent with those of the local hierarchy. Based on the damage feature vectors in the memory queue, positive and negative samples are obtained.

[0067] By using a memory queue, more positive and negative samples can be extracted from historical training batches for the contrastive learning process, thus ensuring the balance and diversity of damage types. To achieve similarity measurements at different levels, cosine similarity is used to calculate the global-level contrastive loss, as shown in the following formula:

[0068]

[0069]

[0070]

[0071] in Represents the L2 norm; and They are two vectors with the same dimension.

[0072] Then, semantic interaction contrast loss for:

[0073] in, This indicates local contrast loss. Indicates global comparison loss; Indicates the damage type of phase t The query vector, This indicates a counting operation. and These represent the number of damage types present in the pre-strike and post-strike phases of the current training batch, respectively. Indicates a positive sample. Represents the positive sample set. This represents the set of damage types for phase t; Indicates InfoNCE loss, Indicates the scaling factor. Indicates a negative sample. Represents the negative sample set; This represents the contrast loss between positive sample pairs. This represents the contrast loss between negative sample pairs. This represents the memory queue of the current training batch. This represents the cosine similarity.

[0074] It is important to note that during the initial training phase (epoch=1), the memory queue is empty and contains no damage feature vectors, making it impossible to calculate the global hierarchical semantic interaction contrastive loss. Therefore, the global hierarchical semantic interaction contrastive loss should be set to 0 during the initial training phase.

[0075] This application also provides a building target damage identification device for implementing the building target damage identification method of the foregoing embodiments.

[0076] Experimental verification and analysis (1) Data description and setting To verify the effectiveness of the method in this application, an example is provided using a dataset of images of buildings before and after being struck. As mentioned above, images of buildings before and after being struck exhibit four types of damage. This results in the building damage change detection model and the multi-type damage recognition model using the same dataset containing the two-phase images before and after the strike, although the data labels differ. The labels for the building damage change detection image dataset are binary classification ground truth change maps, while the labels for the multi-type damage recognition image dataset for buildings include not only binary classification ground truth change maps but also semantic ground truth segmentation maps of damage before and after the strike. Therefore, after data annotation, binary change detection labels (binary damage ground truth change maps, i.e., changed and unchanged categories) and multi-type damage recognition labels (damage semantic ground truth segmentation maps, i.e., undamaged, broken, collapsed, and punctured categories) are obtained. These are then combined with the images of buildings before and after being struck to form a complete multi-type damage recognition image dataset for buildings. Finally, after dividing and expanding the datasets in an 8:2 ratio, training / test datasets with 2040 and 520 sets of images before and after impact were obtained, respectively. The composition of the building target multi-type damage recognition image dataset is shown in Table 1.

[0077] Table 1. Image dataset for multi-type damage recognition of building targets.

[0078] (2) Specific network structure of the building target damage identification model (hereinafter referred to as SCIMTL) The detailed network structure parameters of the SCIMTL model are shown in Table 2. Specifically, the shared parameter semantic feature encoder contains seven pre-trained, shift-and-flip bottleneck convolutional modules with initial weights, which is the backbone network of EfficientNet-B0; the shared parameter semantic feature decoder consists of a feature pyramid network, including the concatenation and fusion of damage semantic features at four scales; the SICL module contains a two-dimensional convolutional layer as a damage feature vector mapping head, used to map bi-temporal semantic features to a high-dimensional dense feature space; the shared parameter damage semantic segmentation head contains two-dimensional convolutional layers at two scales and an upsampling layer to output the bi-temporal damage semantic prediction segmentation map. Furthermore, to obtain the best performance in recognizing multiple types of damage to building targets, this embodiment uses a Bayesian optimization algorithm to search for hyperparameter combinations. Compared with the traditional grid search method, the Bayesian optimization algorithm has higher computational efficiency and stability. Preliminary experiments determined the search boundary conditions for the hyperparameters to be: the weight parameters of the semantic interaction contrast loss and the additional constraint loss. and The search boundary conditions are all [0.01, 1], and the learning rate is... The search boundary conditions are [0.0001, 0.1], and the batch size is... The search boundary condition is [4, 64], and the number of training iterations is [4, 64]. The search boundary condition is [20, 200], and the number of randomly sampled feature vectors in the semantic interaction contrastive learning module is... and the size of the memory queue The search boundary conditions are all [128, 2048]. Finally, the optimal parameter combination for the SCIMTL model is shown in Table 3.

[0079] Table 2 Network structure parameters of the SCIMTL model

[0080] Table 3 Hyperparameter settings for SCIMTL model training process

[0081] To verify the effectiveness of the SCIMTL model in identifying multiple types of damage to building targets, for the binary change detection branch, F1 score (F1), overall accuracy (OA), and intersection over union (IoU) are introduced as evaluation metrics for the binary change detection results. For the damage semantic segmentation branch, mean intersection over union (mIoU) and mean pixel accuracy (MPA) are used as evaluation metrics for the damage semantic segmentation results. The calculation process is as follows:

[0082] in It refers to the number of damage types; , and It is the first The number of true positive, false positive, and false negative pixels in the class of damage.

[0083] The problem of identifying multiple types of damage to building targets is essentially a task of detecting semantic changes in damage. It obtains a semantic change detection map by combining a binary classification prediction change map and a semantic change prediction segmentation map, which serves as the final result for identifying multiple types of damage to building targets. Therefore, the results of identifying multiple types of damage to building targets exhibit various types of semantic consistency. A semantic consistency metric is introduced as an evaluation indicator for the results of identifying multiple types of damage to building targets. Semantic consistency metrics include binary classification consistency (...). Consistency of change ) and unchanged consistency ( Three evaluation metrics are used. Binary classification consistency considers the consistency between the binary classification change map obtained from the two-phase damage semantic prediction segmentation map before and after a building target impact and the binary classification true value change map. Change-to-no-change consistency considers the consistency between the damage types (different) in the changed areas and the damage types (same) in the unchanged areas in the two-phase damage semantic prediction segmentation map. All semantic consistency metrics are based on IoU calculations, and the calculation process is shown in the formula:

[0084] in and These represent the number of damage types in the changed area and the unchanged area, respectively. and In They represent the first IoU between changed and unchanged types and These represent the IoU in the changed region and the unchanged region, respectively.

[0085] Experimental verification and analysis All the aforementioned experiments were conducted on a test dataset augmented with a mixture of real and simulated image data. To further verify the performance of the proposed SCIMTL model in real-world building target multi-type damage recognition tasks, this application used the SCIMTL model trained in the above experiments, along with other comparative models, to perform real-world data verification on building target images before and after actual impacts. Through the design and data acquisition process of a real-world building target impact experiment, a new set of real-world impact experiments was designed on the front of a six-story reinforced concrete building target to verify the effectiveness of the proposed SCIMTL model in recognizing multi-type damage to building targets on real-world data. The collected images before and after impact were processed using image registration and processing methods to complete the creation of the image dataset. Figure 5 An example diagram showing the frontal impact position of a building target is provided. Figure 6 An example image of cropped and expanded building target before and after being hit is shown, where the image rotation expansion method only shows the expanded image rotated by 45°.

[0086] Figure 7 The table shows the results of multi-type damage recognition of image data before and after actual impact on building targets, comparing the method of this application with other semantic change detection methods. Table 4 presents the results of the evaluation index of multi-type damage recognition of building targets in image data before and after actual impact on building targets, comparing the building target damage recognition model of this application with other semantic change detection methods. Figure 7It can be seen that the SCIMTL model proposed in this paper outperforms other comparative methods in terms of the similarity between the multi-type damage identification results and the ground truth map of the real impact image data of building targets. The SCIMTL model can identify more accurate and smoother multi-type damage identification results, and the false positive and false negative regions are fewer compared with other comparative methods. As shown in Table 4, the SCIMTL model performs better in the evaluation indicators F1, mIoU, and binary change detection, damage semantic segmentation, and damage semantic consistency. and In all aspects, it outperforms other comparative methods. Therefore, it can be concluded that the SCIMTL model has superior multi-type damage recognition performance in image data before and after actual impact on building targets.

[0087] Table 4. Comparison of evaluation metrics between the SCIMTL model and the comparison method for actual data.

[0088] This application addresses the problem of identifying and quantitatively analyzing various types of damage to building targets, including fractured areas, collapsed areas, and hole areas, for missile equipment performance evaluation. First, a multi-task learning model for identifying various types of damage to building targets based on semantic comparison images is designed. Semantic segmentation and binary change detection are integrated into an end-to-end multi-branch framework. The intra-class similarity and inter-class separability of damage semantic features are enhanced through a semantic interaction comparison module. Furthermore, a multi-task learning model is constructed with three branches: semantic segmentation of damage before and after a building target impact, binary classification damage change detection based on cross-learning time correlation modeling, and qualitative information on the location and type of damage in various change areas such as fractured areas, collapsed areas, and hole areas.

[0089] The above descriptions are merely various embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for identifying damage to building targets, characterized in that, include: A building target damage recognition model is constructed; the building target damage recognition model includes a feature extraction module and a semantic segmentation and change detection module, the semantic segmentation and change detection module includes a damage semantic segmentation head and a change detection module based on cross-learning time correlation modeling; The building target damage recognition model is trained based on the training dataset to obtain the trained building target damage recognition model; the samples in the training dataset include dual-temporal image pairs of building targets before and after being hit, namely, the image of the building target before being hit and the image of the building target after being hit. During training, the dual-temporal images of the building target before and after the impact are input into the feature extraction module to obtain semantic features of the image before the impact and semantic features of the image after the impact. The semantic features of the image before the impact and the semantic features of the image after the impact are input into the semantic segmentation and change detection module. The damage semantic segmentation head obtains the damage semantic prediction segmentation map of the building target before the impact and the damage semantic prediction segmentation map of the image after the impact based on the semantic features of the image before the impact and the semantic features of the image after the impact. The change detection module based on cross-learning temporal correlation modeling realizes the temporal correlation modeling of the dual-temporal damage semantic features based on the semantic features of the image before the impact and the semantic features of the image after the impact, and completes the binary classification damage change detection to obtain the binary classification damage prediction change map. The images of the building target before and after the impact are input into the trained building target damage recognition model to obtain a damage semantic prediction segmentation map and a binary classification damage prediction change map of the post-impact image. The binary classification damage prediction change map is used to mask the damage semantic prediction segmentation map of the post-impact image to obtain a damage semantic change detection map.

2. The method as described in claim 1, characterized in that, The change detection module based on cross-learning time correlation modeling includes a cross-learning time correlation modeling module and a binary damage change detection head; The cross-learning time correlation modeling module includes an attention module; The semantic features of the image before the attack are input into the attention branch to obtain the query. ,value ,key The semantic features of the post-attack image are input into the attention module to obtain the query. ,value ,key ; query s and keys Multiply the results and apply the softmax function to obtain the similarity matrix. ; query s and keys Multiply the results and apply the softmax function to obtain the similarity matrix. Similarity matrix Sum Multiplying, after convolution, and adding to the semantic features of the image before the attack, yields the cross-learned semantic features of the image before the attack; similarity matrix. Sum Multiply, and after convolution, add to the semantic features of the post-hit image to obtain the semantic features of the post-hit image after cross-learning; The semantic features of the pre-attack image after cross-learning and the semantic features of the post-attack image after cross-learning are input into the binary damage change detection head to obtain a binary damage prediction change map.

3. The method as described in claim 2, characterized in that, The attention module includes three parallel convolutional layers, two parallel pooling layers, a reshape layer, and a transpose layer; The semantic features of the image before the attack are input into three convolutional layers respectively to obtain the first convolution result, the second convolution result and the third convolution result; The first convolution result and the third convolution result are each passed through a pooling layer to obtain the first pooling result and the second pooling result; The first pooling result, the second convolution result, and the second pooling result are all input into the Reshape layer to obtain the first reshaping result, the second reshaping result, and the third reshaping result; The first reshaping result is input into the Transpose layer to obtain the query. The second reshaping result is used as a value. The third reshaping result serves as a key. .

4. The method as described in claim 1, characterized in that, The total loss used during training is: in, For the total loss, For dual-temporal damage semantic segmentation loss, For the detection of damage changes in two categories, For semantic interaction contrast loss, As an additional constraint loss, and Representing semantic interaction contrast loss respectively and additional constraint losses The weight.

5. The method as described in claim 4, characterized in that, Dual-phase damage semantic segmentation loss for: in, Indicates phase Destructive semantic segmentation loss, Indicates phase Destructive semantic segmentation loss, Indicates phase Destructive semantic segmentation loss, Indicates phase The first in the damage semantic truth segmentation graph The pixel belongs to the first The probability of damage of this type. Indicates phase The damage semantic prediction segmentation graph The pixel belongs to the first The probability of damage of this type. Indicates the number of pixels. Indicates the number of damage types.

6. The method as described in claim 4, characterized in that, Two-category damage change detection loss for: in, The graph showing the true value changes of binary damage is shown in the figure. The pixel belongs to the first The probability of a class 0 indicates an unchanged type, and 1 indicates a changed type. The figure showing the change in damage prediction for binary classification is shown below. The pixel belongs to the first The probability of a class This represents a graph showing the true value changes in binary damage classification. This represents a graph showing the changes in damage prediction for two categories. This indicates the number of pixels.

7. The method as described in claim 4, characterized in that, Additional constraint loss for: in, The first in the binary classification prediction change graph The pixel value of each pixel. The first of the two-phase semantic difference features The pixel value of each pixel. This is the upper bound for the semantic differences in two-phase damage. Indicates the number of pixels. Representing the semantic difference features of two phases The number of channels.

8. The method as described in claim 4, characterized in that, Semantic interaction contrast loss for: in, This indicates local contrast loss. Indicates global comparison loss; Indicates the damage type of phase t The query vector, This indicates a counting operation. and These represent the number of damage types present in the pre-strike and post-strike phases of the current training batch, respectively. Indicates a positive sample. Represents the positive sample set. This represents the set of damage types for phase t; Indicates InfoNCE loss, Indicates the scaling factor. Indicates a negative sample. Represents the negative sample set; This represents the contrast loss between positive sample pairs. This represents the contrast loss between negative sample pairs. This represents the memory queue of the current training batch. This represents the cosine similarity.

9. The method as described in claim 8, characterized in that, The method further includes: By mapping the bi-temporal semantic features to a high-dimensional dense feature space, a set of damage feature vectors is obtained; For each time phase and each damage type, multiple damage feature vectors are randomly extracted, and the average feature vector is calculated as the query vector; The set of damage feature vectors that have the same damage type as the query vector constitutes the positive sample set; the set of damage feature vectors that have different damage types from the query vector constitutes the negative sample set.

10. A building target damage identification device, characterized in that, Used to implement the building target damage identification method according to any one of claims 1-9.

Images