Building target damage assessment method based on multi-modal remote sensing data

Abstract

The invention discloses a multi-modal remote sensing data-based building target damage assessment method, which comprises the following steps of: constructing a text-guided two-stage building target classification network which comprises a CLIP text encoder, a Unet image encoder and a text image fusion decoder, inputting image text description information and pre-disaster / post-disaster dual-temporal remote sensing images, and carrying out multi-modal remote sensing data-based building target damage assessment on the pre-disaster / post-disaster dual-temporal remote sensing images; a CLIP text encoder module extracts text description information features, a Unet image encoder extracts image features of pre-disaster / post-disaster dual-phase remote sensing images, a text image fusion decoder fuses the text description information features and the image features, and building target damage is classified according to the fused features; a building target damage assessment module fusing building vector data and post-disaster remote sensing image data is constructed, the building vector data is converted into a bitmap of a pre-disaster remote sensing image by using the building vector data constructed by an official geographic information base or an OSM base, the bitmap is used as a pre-disaster image in a text-guided two-stage building target classification network BCL-Unet, and the pre-disaster remote sensing image is used as a pre-disaster remote sensing image. And carrying out building target damage grade classification by using a text-guided two-stage building target classification network BCL-Unet. According to the invention, a feasible technical scheme is provided for efficient detection of post-disaster building target damage.

Description

Technical Field

[0001] The invention relates to remote sensing interpretation technology, and in particular to a building target damage assessment method based on multi-modal remote sensing data. Background Art

[0002] Force majeure events such as natural disasters can cause serious damage to building structures. Timely assessment of building damage can effectively reduce the human and economic losses caused by force majeure events. With the development of aerospace technology, post-disaster remote sensing data can be obtained in a timely manner through satellites or drones to quickly assess the damage to building targets in the disaster area. However, the building damage assessment method based on post-disaster remote sensing data is mainly based on the subjective experience of experts, and the automated assessment method has problems such as low accuracy of damage classification level and time-consuming. In order to meet the needs of precise rescue in disaster-stricken areas, pre-disaster data is used as the basis for damage level classification, and a building damage assessment method based on multi-temporal remote sensing data is used to obtain a more accurate building target damage level and achieve accurate assessment of building target damage. However, the following problems exist in the current building target damage assessment based on multi-temporal remote sensing data:

[0003] (1) At present, the assessment of building damage levels based on dual-temporal remote sensing images mainly divides post-disaster building targets into two categories: collapsed and non-collapsed. In order to implement precise rescue in disaster areas, it is necessary to classify the damage levels of building targets more accurately. However, since adjacent damage levels have relatively similar destruction characteristics, it is difficult to accurately classify the damage levels of building targets using only dual-temporal remote sensing images.

[0004] (2) In the task of building damage level assessment based on dual-temporal remote sensing images, it is sometimes not easy to obtain remote sensing images before a disaster. However, the official geographic information database or OSM database contains a large amount of building vector data, which can serve as an excellent supplement to pre-disaster images. Existing technologies usually use building vector data as auxiliary data. When conducting building damage assessment, remote sensing images before and after the change are still required as input, which makes it difficult to solve the problem of missing remote sensing images before the disaster. Summary of the invention

[0005] The purpose of the present invention is to propose a building target damage assessment method based on multimodal remote sensing data, so as to provide a feasible technical solution for efficient detection of building target damage after disasters.

[0006] The technical solution to achieve the purpose of the present invention is: a building target damage assessment method based on multimodal remote sensing data, including two parts:

[0007] Construct a text-guided two-stage building target classification network, namely BCL-Unet. BCL-Unet includes CLIP text encoder, Unet image encoder and text-image fusion decoder. The input image text description information and pre-disaster / post-disaster dual-temporal remote sensing images are used as input. The CLIP text encoder module extracts the text description information features, the Unet image encoder extracts the image features of pre-disaster / post-disaster dual-temporal remote sensing images, and the text-image fusion decoder fuses the text description information features and image features. The damage of building targets is classified according to the fused features.

[0008] A building target damage assessment module is constructed by integrating building vector data and post-disaster remote sensing image data. The building vector data that has been constructed by the official geographic information database or OSM database is used to convert the building vector data into bitmaps of pre-disaster remote sensing images, which are used as pre-disaster images in the text-guided two-stage building target classification network BCL-Unet. The text-guided two-stage building target classification network BCL-Unet is used to classify the building target damage level.

[0009] Furthermore, a text-guided two-stage building object classification network consists of

[0010] (a) Image feature extraction module

[0011] The image feature extraction module is a five-layer network structure based on seresnext50. The first layer consists of a convolution layer, a batch normalization layer and a ReLU activation function. The image data is input into the first layer to obtain the edge and texture information features F1 of the image; F1 passes through the second layer composed of a pooling layer and a residual layer to obtain the shape and local features F2 of the image. The three convolution layers with the same structure are composed of 1*1 convolution, 3*3 convolution and 1*1 convolution. As the third, fourth and fifth layers of the feature extraction network, F2 passes through the third layer to obtain the image features F3, F3 passes through the fourth layer to obtain the image features F4, and F4 passes through the fifth layer to obtain the high-dimensional features F5 of the image.

[0012] (b) Text feature extraction module

[0013] The text feature extraction module adopts the text feature encoder model of the CLIP model. The text description information is input into the text feature encoder of the CLIP model to obtain the text feature vector T1.

[0014] (c) Text-image feature fusion decoding module

[0015] The text image feature fusion decoding module includes five upsampling layers and a feature fusion branch. The structure of each upsampling layer is mainly two 3*3 convolutions. In the first upsampling layer, the high-dimensional feature F5 of the image is input into the first upsampling layer to obtain the feature Fp before image fusion. The image feature Fp and the text feature T1 are input into the feature fusion branch composed of a feature normalization block, a cosine similarity calculation block, and a feature addition block to generate a fusion feature Fc. The feature Fc and the feature F4 are cascaded into the second upsampling layer to obtain the feature F7. The features F7 and F3 are cascaded into the third upsampling layer to obtain the image feature F8. The features F8 and F2 are cascaded into the fourth upsampling layer to obtain the image feature F9. The features F9 and F1 are cascaded into the fifth upsampling layer to obtain the image feature location feature Floc and the classification feature Fcls.

[0016] (d) Damage level classification module

[0017] The image classification feature Fcls is input into the argmax function to generate a preliminary pixel-level damage level map Mcls. The positioning feature Floc is converted into a building positioning map Mloc. Mloc and Mcls are multiplied to obtain the damage level map M of the building target. The damage level of the building target is divided into four categories: basically intact, slightly damaged, severely damaged and destroyed.

[0018] Further, the specific steps are:

[0019] Step 1, obtain remote sensing images of building targets before and after the disaster. If the remote sensing image data before the disaster exists, proceed to step 2, otherwise proceed to step 3;

[0020] Step 2: Generate image text description information based on the remote sensing images of building targets before and after the disaster, and combine the text-guided two-stage building target classification network BCL-Unet to obtain the building target damage classification results;

[0021] Step 3: Obtain the pre-disaster building target vector information through the official geographic information database or OSM database, convert the pre-disaster building vector information into a rasterized bitmap, make a model input dataset based on the generated bitmap and the obtained post-disaster image, and generate image text description information. Combined with the text-guided two-stage building target classification network BCL-Unet, the building target damage classification results are obtained.

[0022] Further, step 1, obtaining remote sensing images of building targets before and after the disaster, the specific method is as follows;

[0023] Use drones or satellites to obtain remote sensing images of post-disaster building targets;

[0024] Retrieve pre-disaster images of the disaster-affected area from the remote sensing image library.

[0025] Furthermore, in step 2 and step 3, image text description information is generated, and the text description information of the pre-disaster image is set to "A photo of + building", and the text description information of the post-disaster image is set to "A photo of + damaged building".

[0026] Furthermore, in step 3, the pre-disaster building target vector information is obtained through the official geographic information database or OSM database, and the pre-disaster building vector information is converted into a rasterized bitmap. The specific method is:

[0027] Call the OSM data interface to obtain the pre-disaster building target vector data, crop the vector data of the area corresponding to the post-disaster image from the obtained vector data, make it have the same geographical location, scale and shape as the post-disaster remote sensing image, and call the vector-to-raster algorithm module in ArcGis to complete the conversion of the building target vector data into a raster bitmap.

[0028] Furthermore, the XBD dataset was selected as the training dataset to generate text description information of pre-disaster and post-disaster images. The text description information of pre-disaster images was set as “A photo of + building”, and the text description information of post-disaster images was set as “A photo of + damaged building”. Combined with pre-disaster and post-disaster images, the text-guided two-stage building object classification network BCL-Unet was trained.

[0029] The model training parameters are set as follows: initial learning rate is 0.01, batch size is set to 8, loss function uses CE and Dice, with weights of 4 and 1 respectively, and the number of network training times is set to 200.

[0030] A building target damage assessment system based on multimodal remote sensing data implements the building target damage assessment method based on multimodal remote sensing data to achieve building target damage assessment based on multimodal remote sensing data.

[0031] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for building target damage assessment based on multimodal remote sensing data is implemented to achieve building target damage assessment based on multimodal remote sensing data.

[0032] A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the method for building target damage assessment based on multimodal remote sensing data is implemented to achieve building target damage assessment based on multimodal remote sensing data.

[0033] Compared with the prior art, the present invention has the following significant advantages: 1) It solves the problem of low accuracy in classification of building damage levels using multi-temporal remote sensing data by using multimodal data fusion methods such as remote sensing images, building vectors, and text guidance information; 2) It improves the classification accuracy of multimodal data for building damage levels by fusing text features obtained by CLIP with image features obtained by CNN networks; 3) It performs feature fusion of building vector data and remote sensing image data for building damage level classification, thus making up for the problem of insufficient accuracy in building damage level classification due to the lack of remote sensing images before the disaster. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a flowchart of a building target damage assessment method based on multi-modal remote sensing data;

[0035] Figure 2 It is the architecture diagram of the text-guided two-stage building object classification network (BCL-Unet);

[0036] Figure 3 This is a diagram of the network architecture that uses vector data to replace pre-disaster images as input. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0038] Combination Figure 1-Figure 3 The present invention proposes a building target damage assessment method based on multimodal remote sensing data, comprising the following steps:

[0039] Step 1: Obtain remote sensing images of building targets before and after the disaster and perform image preprocessing;

[0040] Step 1.1, using remote sensing methods such as drones or satellites to obtain remote sensing images of post-disaster building targets;

[0041] Step 1.2, preprocess the post-disaster remote sensing images, including resampling, projection conversion, radiation calibration, atmospheric correction, and geo-registration processing, and normalize them to a preset size to meet the requirements of image retrieval.

[0042] Step 1.3, retrieve pre-disaster images of the disaster-stricken area from the historical remote sensing image library, and use the retrieval result images as the pre-disaster image set I pre , if I pre If it is not empty, then after preprocessing (the same preprocessing method is used for pre-disaster and post-disaster images), go to step 2, otherwise go to step 3.

[0043] Step 2: Use the open source XBD dataset as the training dataset and the remote sensing images I obtained in steps 1.3 and 1.2. pre As the test data set. Construct a text-guided two-stage building target classification network (BCL-Unet), use the XBD data set to train the model and obtain the optimal parameters; generate image text description information according to step 2.1 as the text input of the model, input the obtained image and text information into the BCL-Unet network, and obtain the building target damage classification result of the input image.

[0044] In step 2.1, for pre-disaster and post-disaster images, text description information is generated, mainly for the buildings in the images, so the text description information of the pre-disaster image is set to "A photo of + building", and the text description information of the post-disaster image is set to "A photo of + damaged building".

[0045] Step 2.2, build the BCL-Unet network. The overall architecture adopts a two-stage design, such as Figure 2 As shown in the figure, the parameters of the two-stage model are the same. In the first stage, the model inputs pre-disaster images and text description information to achieve building positioning; in the second stage, the pre-disaster and post-disaster images and text description information are input to achieve building damage classification. The BCL-Unet network mainly includes four modules, namely image feature extraction module, text feature extraction module, text image feature fusion decoding module, and damage level classification module.

[0046] (a) Image feature extraction module

[0047] The image feature extraction module is based on the five-layer network structure of seresnext50. The first layer consists of a convolutional layer, a batch normalization layer and a ReLU activation function. The image data is input into the first layer to obtain the edge and texture information features F1 of the image; F1 passes through the second layer composed of a pooling layer and a residual layer to obtain the shape and local features F2 of the image. The three convolutional layers with the same structure are composed of 1*1 convolution, 3*3 convolution and 1*1 convolution, which are the third, fourth and fifth layers of the feature extraction network. F2 passes through the third layer to obtain the image features F3, F3 passes through the fourth layer to obtain the image features F4, and F4 passes through the fifth layer to obtain the high-dimensional features F5 of the image.

[0048] (b) Text feature extraction module design

[0049] The text feature extraction module adopts the text feature encoder model of the CLIP model and freezes the text feature encoder parameters of the CLIP model. The text description information generated in step 2.1 is input into the text feature encoder of the CLIP model to obtain the text feature vector T1.

[0050] (c) Design of text image feature fusion decoding module

[0051] The text image feature fusion decoding module includes five upsampling layers and a feature fusion branch. The structure of each upsampling layer is mainly two 3*3 convolutions. In the first upsampling layer, the high-dimensional feature F5 of the image is input into the first upsampling layer to obtain the feature Fp before image fusion. The image feature Fp and the text feature T1 are input into the feature fusion branch consisting of a feature normalization block, a cosine similarity calculation block, and a feature addition block to generate a fusion feature Fc. The feature Fc and the feature F4 are cascaded into the second upsampling layer to obtain the feature F7. The feature F7 and F3 are cascaded into the third upsampling layer to obtain the image feature F8. The feature F8 and F2 are cascaded into the fourth upsampling layer to obtain the image feature F9. The feature F9 and F1 are cascaded into the fifth upsampling layer to obtain the image feature location feature Floc and the classification feature Fcls.

[0052] (d) Damage level classification module

[0053] The image classification feature Fcls is input into the argmax function to generate a preliminary pixel-level damage level map Mcls. The positioning feature Floc is converted into a building positioning map Mloc. Mloc and Mcls are multiplied to obtain the damage level map M of the building target. The damage level of the building target is divided into four categories: basically intact, slightly damaged, severely damaged and destroyed.

[0054] Step 2.2, train the BCL-Unet network model to obtain the optimal parameters. Select the XBD dataset as the training dataset, generate the text description information of the pre-disaster and post-disaster images according to step 2.1, and complete the production of the training dataset. Model training parameter settings: the initial learning rate is 0.01, the batch size is set to 8, the loss function uses CE and Dice, and different weights are set for them respectively, specifically CE+4DICE, and the number of network training is set to 200.

[0055] In step 2.3, the model parameters obtained in step 2.2 are loaded into the BCL-Unet network model, and image text description information is generated according to the test set. The pre-disaster and post-disaster images obtained in step 1 and the generated text description information are input into the BCL-Unet network to obtain the damage level prediction results of the building targets, which are divided into four categories: basically intact, slightly damaged, severely damaged, and destroyed.

[0056] Step 3, obtain the pre-disaster building target vector information through the OSM library, convert the pre-disaster building vector information into a rasterized bitmap as the pre-disaster image, use the pre-disaster image and the post-disaster image obtained in step 1.2, generate text description information using the scheme in step 2.1, make a model input data set, input the prepared data set into the text-guided two-stage building target classification network (BCL-Unet) trained in step 2, and obtain the building target damage classification result.

[0057] Step 3.1, call the OSM data interface to obtain the pre-disaster building target vector data, crop the vector data of the area corresponding to the post-disaster image from the obtained vector data, make it have the same geographical location, scale and shape as the post-disaster remote sensing image, call the vector-to-raster algorithm module in ArcGis to complete the conversion of the building target vector data into a raster bitmap, generate text description information based on the raster bitmap and the post-disaster image obtained in step 1.2, and complete the input data set production.

[0058] Step 3.2, call the BCL-Unet network trained in step 2, such as Figure 3 As shown in the figure, the data obtained in step 3.1 is input into the BCL-Unet network to obtain the damage level prediction results of the building target, which are divided into four categories: basically intact, slightly damaged, seriously damaged and destroyed.

[0059] The present invention also proposes a building target damage assessment system based on multimodal remote sensing data, implements the building target damage assessment method based on multimodal remote sensing data, and realizes building target damage assessment based on multimodal remote sensing data.

[0060] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for building target damage assessment based on multimodal remote sensing data is implemented to achieve building target damage assessment based on multimodal remote sensing data.

[0061] A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the method for building target damage assessment based on multimodal remote sensing data is implemented to achieve building target damage assessment based on multimodal remote sensing data.

[0062] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0063] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A building target damage assessment method based on multimodal remote sensing data, characterized in that: It consists of two parts: Construct a text-guided two-stage building target classification network, namely BCL-Unet. BCL-Unet includes CLIP text encoder, Unet image encoder and text-image fusion decoder. The input image text description information and pre-disaster / post-disaster dual-temporal remote sensing images are used as input. The CLIP text encoder module extracts the text description information features, the Unet image encoder extracts the image features of pre-disaster / post-disaster dual-temporal remote sensing images, and the text-image fusion decoder fuses the text description information features and image features. The damage of building targets is classified according to the fused features. A building target damage assessment module is constructed by integrating building vector data and post-disaster remote sensing image data. The building vector data that has been constructed by the official geographic information database or OSM database is used to convert the building vector data into bitmaps of pre-disaster remote sensing images, which are used as pre-disaster images in the text-guided two-stage building target classification network BCL-Unet. The text-guided two-stage building target classification network BCL-Unet is used to classify the building target damage level.

2. The building target damage assessment method based on multimodal remote sensing data according to claim 1 is characterized in that: A text-guided two-stage architectural object classification network, including (a) Image feature extraction module The image feature extraction module is a five-layer network structure based on seresnext50. The first layer consists of a convolution layer, a batch normalization layer and a ReLU activation function. The image data is input into the first layer to obtain the edge and texture information features F1 of the image; F1 passes through the second layer composed of a pooling layer and a residual layer to obtain the shape and local features F2 of the image. The three convolution layers with the same structure are composed of 1*1 convolution, 3*3 convolution and 1*1 convolution. As the third, fourth and fifth layers of the feature extraction network, F2 passes through the third layer to obtain the image features F3, F3 passes through the fourth layer to obtain the image features F4, and F4 passes through the fifth layer to obtain the high-dimensional features F5 of the image. (b) Text feature extraction module The text feature extraction module adopts the text feature encoder model of the CLIP model. The text description information is input into the text feature encoder of the CLIP model to obtain the text feature vector T1. (c) Text-image feature fusion decoding module The text image feature fusion decoding module includes five upsampling layers and a feature fusion branch. The structure of each upsampling layer is mainly two 3*3 convolutions. In the first upsampling layer, the high-dimensional feature F5 of the image is input into the first upsampling layer to obtain the feature Fp before image fusion. The image feature Fp and the text feature T1 are input into the feature fusion branch composed of a feature normalization block, a cosine similarity calculation block, and a feature addition block to generate a fusion feature Fc. The feature Fc and the feature F4 are cascaded into the second upsampling layer to obtain the feature F7. The features F7 and F3 are cascaded into the third upsampling layer to obtain the image feature F8. The features F8 and F2 are cascaded into the fourth upsampling layer to obtain the image feature F9. The features F9 and F1 are cascaded into the fifth upsampling layer to obtain the image feature location feature Floc and the classification feature Fcls. (d) Damage level classification module The image classification feature Fcls is input into the argmax function to generate a preliminary pixel-level damage level map Mcls. The positioning feature Floc is converted into a building positioning map Mloc. Mloc and Mcls are multiplied to obtain the damage level map M of the building target. The damage level of the building target is divided into four categories: basically intact, slightly damaged, severely damaged and destroyed.

3. The building target damage assessment method based on multimodal remote sensing data according to any one of claims 1-2, characterized in that: The specific steps are: Step 1, obtain remote sensing images of building targets before and after the disaster. If the remote sensing image data before the disaster exists, proceed to step 2, otherwise proceed to step 3; Step 2: Generate image text description information based on the remote sensing images of building targets before and after the disaster, and combine the text-guided two-stage building target classification network BCL-Unet to obtain the building target damage classification results; Step 3: Obtain the pre-disaster building target vector information through the official geographic information database or OSM database, convert the pre-disaster building vector information into a rasterized bitmap, make a model input dataset based on the generated bitmap and the obtained post-disaster image, and generate image text description information. Combined with the text-guided two-stage building target classification network BCL-Unet, the building target damage classification results are obtained.

4. The building target damage assessment method based on multimodal remote sensing data according to claim 3 is characterized in that: Step 1, obtaining remote sensing images of building targets before and after the disaster, the specific method is as follows; Use drones or satellites to obtain remote sensing images of post-disaster building targets; Retrieve pre-disaster images of the disaster-affected area from the remote sensing image library.

5. The building target damage assessment method based on multimodal remote sensing data according to claim 3 is characterized in that: In step 2 and step 3, the image text description information is generated, and the text description information of the pre-disaster image is set to "A photo of + building", and the text description information of the post-disaster image is set to "A photo of + damaged building".

6. The building target damage assessment method based on multimodal remote sensing data according to claim 3 is characterized in that: In step 3, the pre-disaster building target vector information is obtained through the official geographic information database or OSM database, and the pre-disaster building vector information is converted into a rasterized bitmap. The specific method is as follows: Call the OSM data interface to obtain the pre-disaster building target vector data, crop the vector data of the area corresponding to the post-disaster image from the obtained vector data, make it have the same geographical location, scale and shape as the post-disaster remote sensing image, and call the vector-to-raster algorithm module in ArcGis to complete the conversion of the building target vector data into a raster bitmap.

7. The building target damage assessment method based on multimodal remote sensing data according to claim 5 is characterized in that: Select the XBD dataset as the training dataset, generate text description information for pre-disaster and post-disaster images, set the text description information for pre-disaster images to "Aphoto of+building", and set the text description information for post-disaster images to "Aphoto of+damagebuilding". Combine pre-disaster and post-disaster images to train the text-guided two-stage building object classification network BCL-Une. The model training parameters are set as follows: initial learning rate is 0.01, batch size is set to 8, loss function uses CE and Dice, with weights of 4 and 1 respectively, and the number of network training times is set to 200.

8. A building target damage assessment system based on multimodal remote sensing data, characterized in that: Implement the building target damage assessment method based on multimodal remote sensing data as described in any one of claims 1 to 7 to achieve building target damage assessment based on multimodal remote sensing data.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for building target damage assessment based on multimodal remote sensing data as described in any one of claims 1 to 7 is implemented to realize building target damage assessment based on multimodal remote sensing data.

Images