A damage assessment method and system based on remote sensing images

Abstract

The application discloses a kind of based on remote sensing image's damage assessment method and system, using building damage assessment network carries out building damage assessment to remote sensing image;The steps for training building damage assessment network are as follows: obtain the data of building damage assessment;Building positioning is carried out to image before damage;Convolution pooling operation is carried out to image before and after damage, and spatial feature map is obtained;The spatial feature maps of the same level of image before and after damage are fused and deep difference feature extraction is carried out, and disaster category characteristic value is fused feature map;Building positioning result is spliced with the feature map of each layer, and then convolution fusion is carried out to obtain evaluation classification map;Based on the evaluation classification map and corresponding evaluation label output, building damage assessment network is trained.The disaster category information is added to the image of the region to be classified;It can improve the detection ability of the network, has the advantages of high evaluation classification precision, small error and the like.Introduce disaster category information in network, improve the evaluation precision of building damage.

Description

TECHNICAL FIELD

[0001] The present application relates to remote sensing image analysis, in particular to a damage assessment method based on remote sensing images. BACKGROUND

[0002] In the past, damage assessment work mostly needs to be organized by personnel to investigate the site, but because many disasters often have secondary disasters, this work is very time-consuming and labor-intensive and has great danger. Thanks to the rapid development of satellite remote sensing and computer vision technology, people now have the ability to establish some method models to conduct pixel-level damage assessment of buildings in disaster areas using remote sensing satellite images after disasters occur, to obtain the damage of each building in the disaster area, and to further guide the disaster relief work and improve the efficiency of rescue.

[0003] In the early days, people used visual interpretation to assess the damage of buildings in disaster areas, which was subjective and required experts with rich experience to complete the assessment work. At the same time, the image quality of the remote sensing images at that time was limited, which also limited the accuracy of the assessment work. After the rapid development of remote sensing technology, people began to use computer vision to gradually realize automatic interpretation and assessment, greatly improving the efficiency. However, due to the limited ability to mine building damage features, there is still a lot of room for improvement in accuracy and other indicators.

[0004] The most popular method at present is deep learning, which uses deep neural networks to learn and extract features, and building damage assessment has reached a new level. Researchers have widely tried multi-band, multi-modal, and fused features, and designed a large number of network structures such as convolutional neural networks, attention mechanisms, and encoder-decoder structures. However, the current deep learning-based building damage assessment task has to some extent ignored the role of disaster categories in guiding the network to combine disaster damage patterns. SUMMARY

[0005] The present application provides a damage assessment method based on remote sensing images to improve the accuracy of the assessment.

[0006] The present application also provides a damage assessment system based on remote sensing images.

[0007] Technical solution: To solve the above problems, the present application adopts a damage assessment method based on remote sensing images, comprising the following steps:

[0008] (1) Obtain the remote sensing image of the area to be assessed and its disaster category, and obtain the disaster category feature value according to the disaster category;

[0009] Obtain remote sensing images of the area to be assessed and its disaster category, and obtain disaster category characteristic values ​​based on the disaster category;

[0010] (2) Using the trained building damage assessment network to conduct building damage assessment on the remote sensing images of the assessment area, and output the building damage assessment results;

[0011] The trained building damage assessment network is obtained through the following steps:

[0012] (2.1) Obtaining a public dataset for building damage assessment and preprocessing the public dataset to obtain corresponding training and test sets, each of which includes pre-damage images, post-damage images, and corresponding assessment labels;

[0013] (2.2) Using the positioning neural network to locate buildings in the pre-damage images of the training set, the building positioning results of the area to be evaluated are obtained;

[0014] (2.3) Using a classification neural network, convolution and pooling operations are performed on the pre-damage and post-damage images of the training set to obtain a layer-by-layer spatial feature map;

[0015] (2.4) Using the difference rate calculation module to fuse the spatial feature maps of the pre-damage image and the post-damage image at the same level, obtain shallow difference feature maps of each layer. Using the deep difference feature extraction module to extract deep difference features from the shallow difference feature maps of some layers, obtain deep difference feature maps of some layers. Deep difference feature extraction is not performed on the shallow difference feature map of the last layer, and the disaster category feature value is fused into the shallow difference feature map of the last layer.

[0016] (2.5) The building positioning results are combined with the feature maps of each layer, and then convolution fusion is performed to obtain the evaluation classification map;

[0017] (2.6) Based on the output evaluation classification map and the corresponding evaluation labels, the loss is calculated, and the network parameters of the building damage assessment network are optimized through continuous iteration; the optimized building damage assessment network is trained and tested using the training set and test set to obtain the trained building damage assessment network.

[0018] Furthermore, the pre-processing of the public dataset in step (2.1) includes modifying the transparency channel values ​​of the pre-damage image and the corresponding post-damage image to corresponding disaster category characteristic values.

[0019] Furthermore, the learning process of the positioning neural network in step (2.2) is as follows: inputting the pre-damage image into the positioning neural network, outputting the building positioning results corresponding to each building on the pre-damage image; calculating the loss of the building positioning results and the building positioning labels, and continuously iterating and optimizing the positioning neural network.

[0020] Further, the classification neural network in the step (2.3) includes two branches for processing of the pre-damage image and the post-damage image respectively, the two branches have the same structure and the same number of layers; the two branches perform progressive multi-scale feature calculation on the pre-damage image and the post-damage image respectively, and the two branches share parameters between the feature extraction modules at the same level.

[0021] Further, the spatial feature maps of the pre-damage image and the post-damage image at the same level are fused by using a difference rate calculation module to obtain a shallow difference feature map indicating the shallow difference rate before and after damage.

[0022] The difference rate calculation formula is:

[0023]

[0024] Further, in the step (2.4), the deep difference feature extraction module is used to extract deep difference features from the shallow difference feature maps except the first layer and the last layer, and the specific steps of using the deep difference feature extraction module to extract deep difference features from the shallow difference feature maps of each layer are as follows: the shallow difference feature map is divided into several blocks, the divided blocks are spliced in the channel dimension, then the feature extraction is performed through the window attention unit to obtain a feature map, and finally the size of the feature map is restored to the same size as the input shallow difference feature map to obtain a deep difference feature map.

[0025] Further, the fusion of the disaster category feature value into the shallow difference feature map of the last layer is specifically: different disaster categories are set as different numerical values to obtain a disaster category feature value, a disaster category feature map with numerical values corresponding to the disaster category feature value is generated, and the disaster category feature map is spliced with the shallow difference feature map of the last layer.

[0026] The application also adopts a damage evaluation system based on remote sensing images, which comprises:

[0027] The data acquisition module is used for acquiring remote sensing images of the region to be evaluated and disaster categories thereof, and obtaining disaster category feature values according to the disaster categories.

[0028] The model training module is used for obtaining a public data set of building damage assessment, obtaining a corresponding training set and test set by preprocessing the public data set, the training set and the test set both comprising a pre-damage image, a post-damage image and a corresponding evaluation label; a positioning neural network is used to perform building positioning on the pre-damage image of the training set to obtain a building positioning result of an evaluation area; a classification neural network is used to perform convolution and pooling operations on the pre-damage image and the post-damage image of the training set respectively to obtain spatial feature maps of layers in succession; a difference rate calculation module is used to fuse the spatial feature maps of the pre-damage image and the post-damage image of the same level to obtain shallow difference feature maps of layers, a deep difference feature extraction module is used to perform deep difference feature extraction on the shallow difference feature maps of part of the layers to obtain deep difference feature maps of part of the layers, and the shallow difference feature map of the last layer is not subjected to deep difference feature extraction, and a disaster category feature value is fused into the shallow difference feature map of the last layer; the building positioning result and the feature maps of the layers are spliced and then subjected to convolution fusion to obtain an evaluation classification map; based on the output evaluation classification map and the corresponding evaluation label, a loss is calculated, and the network parameters of the building damage assessment network are continuously iteratively optimized; the training and test set are used to train and test the optimized building damage assessment network to obtain a trained building damage assessment network.

[0029] The evaluation module is used to perform building damage assessment on a remote sensing image of an evaluation area by using the trained building damage assessment network in the model training module, and output a building damage assessment result.

[0030] The application also adopts a computer device, which comprises a memory, a processor and a computer program stored on the memory and executable on the processor, and the processor implements the steps of the above method when executing the computer program.

[0031] The application also adopts a computer readable storage medium, which stores a computer program, and the computer program implements the steps of the above method when executed by a processor.

[0032] Advantages: Compared with the prior art, the application has the significant advantages that disaster category information is added to the image of the region to be classified, the trained building damage assessment network is used to process the satellite remote sensing image to be evaluated to perform building damage assessment and output an evaluation result, the detection capability of the network is improved, the evaluation classification precision is high, the error is small and the like. The difference rate calculation module is used to replace the traditional simple addition and subtraction calculation, the spatial features of the pre-damage image and the post-damage image are fused to obtain more effective shallow feature differences, the disaster category information is introduced into the network by using the disaster category feature value, which is beneficial to the network learning of different features of different disaster categories in the damage mode, and the evaluation precision of building damage is further improved. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 Flowchart of the damage assessment method of the present application.

[0034] Figure 2 Network structure diagram of the damage assessment method of the present application.

[0035] Figure 3 Deep difference feature extraction module structure diagram of the damage assessment method of the present application.

[0036] Figure 4 Structure diagram of the damage assessment system of the present application. DETAILED DESCRIPTION

[0037] Embodiment 1

[0038] As shown in the figure, the damage assessment method based on remote sensing image in this embodiment includes the following steps: Figure 1

[0039] (1) Obtain the remote sensing satellite image of the building area to be evaluated; and its disaster category, as shown in Table 1, obtain the disaster category feature value according to the disaster category;

[0040] Table 1 Disaster category feature value

[0041] Disaster category Characteristic value Volcanic eruption 255 Hurricane 254 Earthquake 253 Flood 252 Tsunami 251 Fire / wildfire 250

[0042] (2) Use the trained building damage assessment network to perform building damage assessment on the remote sensing image to be evaluated, and output the building damage assessment result;

[0043] The building damage assessment network is trained on a high-performance computer. The computer configuration is: AMD Ryzen 95900X 12-core 24-thread processor, main frequency 4.2 GHz, Nvidia GeForce RTX 3080ti graphics processor, 12G memory, memory 32GB, operating system Windows 11; The implementation of the building damage assessment network based on remote sensing image considering the disaster category is based on the Pytorch deep learning framework toolkit.

[0044] The trained building damage assessment network is obtained by the following steps:

[0045] (2.1) Download the building damage assessment public data set xDB to obtain the corresponding training set and test set through preprocessing;

[0046] The preprocessing of the obtained image includes:

[0047] ​According to the correspondence between the disaster category in the image file name and the disaster category feature value, the transparency channel value of the pre-damage image and the post-damage image is modified to the disaster category feature value; 5598 pairs of pre-damage images and post-damage images are randomly divided into a training set and a test set according to a ratio of 4:1; the training set and the test set both contain image and label folders for storing paired image files and paired label files respectively.

[0048] (2.2) The pre-built positioning neural network is used for building positioning on the pre-damage image, As shown in stage 1 in Figure 2 ;

[0049] (2.3) The pre-built classification neural network is used for a total of 6 levels of convolution and pooling operations on the pre-damage image and the post-damage image, to obtain two branches and 6 layers of multi-scale spatial feature maps;

[0050] The two branches have the same structure, including multi-scale feature extraction modules that progress layer by layer, and the two branches share parameters between the same level of feature extraction modules, and each module performs convolution, activation function and pooling operation on the image. As shown in the E module of stage 2 in Figure 2 and the branch covered thereby;

[0051] (2.4) The pre-designed difference rate calculation module is used to fuse the pre-damage and post-damage spatial feature maps at the same level to obtain shallow difference feature maps; the pre-designed deep difference feature extraction module is used to extract deep difference features from the shallow difference feature maps; the difference rate calculation module is deployed at level 1 to level 6, and the deep difference feature extraction module is deployed at level 2 to level 5;

[0052] The difference rate calculation module is calculated by the formula, which is described as follows:

[0053]

[0054] As shown in Figure 3 , the deep feature extraction module includes image blocking, sliding window attention unit and post-processing components such as convolution, upsampling and batch normalization; After blocking and attention unit, the feature map is By taking the square root of (H' x W') and upsampling operation, and performing convolution operation on the embedding dimension, the output of this level is restored to the size of the input, and the output is

[0055] (2.5) By fusing the pre-designed disaster category feature value into the shallow difference feature map of the last layer, the model considers different damage modes of different disaster categories.

[0056] As Figure 2 As shown in level 6 of phase 2, the shallow difference feature map of this layer is According to the value of the transparency channel when the network is input, a disaster category feature map with the same size is generated Splicing the two can obtain the final output of level 6

[0057] (2.6) The output convolution of the classification network is constrained by the building positioning result to fuse the features of each layer to obtain an evaluation classification map; as Figure 2 The output of phase 1 And the feature maps output by each level of phase 2 Splicing operation, and the corresponding calculation formula can be expressed as:

[0058] Evaluation result = output convolution (final feature map)

[0059]

[0060] Among them, the preset damage level number is 5, and the number of levels of the classification neural network is 6.

[0061] (2.7) Based on the output evaluation classification map and the corresponding label map, the loss is calculated, and the network parameters of the preset building damage evaluation network are optimized by continuous iteration; the optimized building damage evaluation network is trained and tested by using the training set and the test set, and the trained building damage evaluation network is obtained; including the following specific steps: using the training data in step (2.1), training the building damage evaluation network, the initialization of all convolution kernels in the network adopts Gaussian distribution with mean 0 and variance 0.01, the initialization of bias adopts fixed value 0.0, using Adam optimization algorithm, batch size is 8, model accuracy verification is performed once every 10 generations, and the model basically converges after training 200 generations.

[0062] According to the above steps, an effective building damage evaluation network is established based on the convolution network, aiming to evaluate the damage of buildings by using the difference features before and after damage and disaster category features, and good evaluation classification accuracy is achieved.

[0063] The above-mentioned application can also be extended to other same type multi-source data building damage assessment tasks, only need to set the disaster category value parameter. According to the task condition, it is judged whether the building damage assessment network needs to be retrained. If it is needed, the corresponding relationship between the disaster category and the category feature value is established according to step (2.1), and the network is retrained, so that the building damage assessment network suitable for multi-source remote sensing images can be obtained. The two-stage building damage assessment network considering the progressive extraction of the disaster category and the shallow and deep difference features fully utilizes the pre-damage image, the disaster category and the difference features before and after the damage, and improves the accuracy of the assessment classification.

[0064] Embodiment 2

[0065] As shown in Figure 4 , the embodiment of the application provides a building damage assessment system based on remote sensing images, comprising:

[0066] A data acquisition module is configured to acquire remote sensing images of a region to be assessed and disaster categories thereof, and obtain disaster category feature values according to the disaster categories;

[0067] An assessment module is configured to use the building damage assessment network trained in the model training module to perform building damage assessment on the remote sensing images of the region to be assessed, and output a building damage assessment result.

[0068] A model training module comprises:

[0069] A positioning module is configured to use a positioning neural network to perform building positioning on pre-damage images of a training set, and obtain building positioning results of the region to be assessed; the public data set is preprocessed to obtain corresponding training and test sets, and the training and test sets each comprise pre-damage images, post-damage images and corresponding assessment labels;

[0070] A sampling module is configured to use a classification neural network to perform convolution and pooling operations on the pre-damage images and the post-damage images of the training set respectively, and obtain spatial feature maps of different scales in layers;

[0071] A difference rate calculation module is configured to fuse the spatial feature maps of the pre-damage images and the post-damage images at the same level to obtain shallow difference feature maps of layers;

[0072] A deep difference feature extraction module is configured to extract deep difference features from the shallow difference feature maps of the layers to obtain deep difference feature maps of the layers;

[0073] A disaster category comprehensive module is configured to splice the disaster category feature map and the shallow difference feature map of the bottom layer of the classification neural network;

[0074] The positioning constraint module is configured to splice the building positioning result with the feature maps of the layers, and then perform convolution fusion to obtain an evaluation classification map.

[0075] The optimization module is configured to calculate a loss based on the output evaluation classification map and the corresponding evaluation label, and iteratively optimize the network parameters of the building damage evaluation network.

[0076] The training and testing module is configured to train and test the optimized building damage evaluation network using a training set and a test set to obtain a trained building damage evaluation network.

Claims

1. A damage assessment method based on remote sensing images, characterized in that: The method comprises the following steps: (1) obtaining remote sensing images of an area to be evaluated and a disaster category thereof, and obtaining a disaster category feature value according to the disaster category; (2) using a trained building damage evaluation network to perform building damage evaluation on the remote sensing images of the area to be evaluated, and outputting a building damage evaluation result; The trained building damage evaluation network is obtained through the following steps: (2.1) obtaining a public data set for building damage evaluation, and obtaining a training set and a test set by preprocessing the public data set, wherein the training set and the test set each comprise a pre-damage image, a post-damage image, and a corresponding evaluation label; (2.2) using a positioning neural network to perform building positioning on the pre-damage image in the training set, and obtaining a building positioning result of the area to be evaluated; (2.3) using a classification neural network to perform convolution and pooling operations on the pre-damage image and the post-damage image in the training set respectively, and obtaining spatial feature maps that are progressively obtained layer by layer; (2.4) fusing the spatial feature maps of the pre-damage image and the post-damage image at the same level using a difference rate calculation module, obtaining shallow difference feature maps at each level, using a deep difference feature extraction module to perform deep difference feature extraction on the shallow difference feature maps at part of the levels, obtaining deep difference feature maps at part of the levels, and not performing deep difference feature extraction on the shallow difference feature map at the last level, and fusing the disaster category feature value into the shallow difference feature map at the last level; using the deep difference feature extraction module to perform deep difference feature extraction on the shallow difference feature maps at the first layer and the last layer; using the deep difference feature extraction module to perform deep difference feature extraction on the shallow difference feature maps at each level, the specific steps of which are as follows: the shallow difference feature map is cut into a plurality of blocks, the cut blocks are spliced in the channel dimension, then feature extraction is performed through a window attention unit to obtain a feature map, and finally the size of the feature map is restored to the same size as the input shallow difference feature map to obtain a deep difference feature map; (2.5) splicing the building positioning result and the feature maps at each level, and then performing convolution fusion to obtain an evaluation classification map; (2.6) based on the output evaluation classification map and the corresponding evaluation label, calculating a loss, and continuously iterating and optimizing the network parameters of the building damage evaluation network; using the training set and the test set to train and test the optimized building damage evaluation network, and obtaining the trained building damage evaluation network.

2. The damage assessment method of claim 1, wherein The preprocessing of the public data set in step (2.1) comprises modifying the transparency channel value of the pre-damage image and the corresponding post-damage image to the corresponding disaster category feature value.

3. The damage assessment method of claim 1, wherein, The learning process of the positioning neural network in step (2.2) is as follows: inputting the pre-damage image into the positioning neural network to output building positioning results of each building on the corresponding pre-damage image; calculating a loss of the building positioning result and a building positioning label, and continuously iterating and optimizing the positioning neural network.

4. The damage assessment method of claim 1, wherein The classification neural network in the step (2.3) includes two branches for processing of the pre-damage image and the post-damage image respectively, the two branches have the same structure and the same number of layers; the two branches respectively perform multi-scale feature calculation on the pre-damage image and the post-damage image layer by layer, and the two branches share parameters between the feature extraction modules at the same level.

5. The damage assessment method of claim 1, wherein, The spatial feature maps of the pre-damage image and the post-damage image at the same level are fused by the difference rate calculation module to obtain a shallow difference feature map indicating the difference rate of the pre-damage and post-damage shallow layers. The difference rate calculation formula is: 。 6. A damage assessment system based on remote sensing imagery, characterized in that, It comprises: The data acquisition module is configured to acquire remote sensing images of a to-be-evaluated area and disaster categories thereof, and obtain disaster category feature values according to the disaster categories; The model training module is configured to acquire a public data set for building damage evaluation, obtain a corresponding training set and a test set by preprocessing the public data set, wherein the training set and the test set each include a pre-damage image, a post-damage image, and a corresponding evaluation label; the pre-damage image of the training set is subjected to building positioning by using a positioning neural network to obtain a building positioning result of the to-be-evaluated area; the pre-damage image and the post-damage image of the training set are subjected to convolution and pooling operations by using a classification neural network to obtain spatial feature maps layer by layer; the spatial feature maps of the pre-damage image and the post-damage image at the same level are fused by using a difference rate calculation module to obtain shallow difference feature maps of each layer, and deep difference feature extraction is performed on the shallow difference feature maps of part of the layers by using a deep difference feature extraction module to obtain deep difference feature maps of part of the layers, and the shallow difference feature map of the last layer is not subjected to deep difference feature extraction, and the disaster category feature values are fused into the shallow difference feature map of the last layer; The building positioning result and the feature maps of each layer are spliced, and convolution fusion is performed to obtain an evaluation classification map; based on the output evaluation classification map and the corresponding evaluation label, a loss is calculated, and the network parameters of the building damage evaluation network are continuously iteratively optimized; the training set and the test set are used to train and test the optimized building damage evaluation network to obtain a trained building damage evaluation network; the deep difference feature extraction module is used to perform deep difference feature extraction on the shallow difference feature maps of the layers except the first layer and the last layer, and the deep difference feature extraction module is used to perform deep difference feature extraction on the shallow difference feature maps of each layer, and the specific steps of the deep difference feature extraction module are as follows: the shallow difference feature map is divided into a plurality of blocks, the divided blocks are spliced in the channel dimension, then feature extraction is performed by a window attention unit to obtain a feature map, and finally the size of the feature map is restored to the same size as the input shallow difference feature map to obtain a deep difference feature map; The evaluation module is configured to use the trained building damage evaluation network in the model training module to perform building damage evaluation on the remote sensing images of the to-be-evaluated area, and output a building damage evaluation result.

7. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the method of any one of claims 1 to 6.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 6.

Images