A method for fine-grained building damage change detection based on the SAM model

Abstract

This paper discloses a method for fine-grained building damage change detection using an adaptive SAM model. The method first constructs a fine-grained building change detection dataset, then injects lightweight parameters to fine-tune the SAM model. Building change hint masks are then predicted, followed by decoding the fine-grained change type of building targets. Finally, the target change type is mapped to damage degree, with disappeared buildings assessed as completely damaged and rebuilt buildings assessed as partially damaged. This method utilizes a mask hinting strategy to convert change region prediction into building extraction, thereby increasing the accuracy of building target change detection results.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing images, and in particular relates to a method for fine-grained building damage change detection. Background Art

[0002] As a crucial vehicle for human activity, buildings reflect urban expansion, social development, and land use. Dynamic monitoring of building changes is crucial for urban planning, land resource surveys, illegal building monitoring, and disaster assessment. In scenarios involving building damage and changes caused by human intervention or natural disasters, it is often necessary to capture the fine-grained state of the changed building targets, including complete disappearance or partial destruction. These can be abstracted into two types: disappearance and reconstruction. In scenarios involving farmland and ecological environmental protection, increased attention is paid to newly added buildings. Furthermore, analysis of newly added or rebuilt buildings can also be applied to post-disaster recovery monitoring. In summary, building change types can be categorized into three fine-grained types based on practical applications: disappearance, new construction, and reconstruction. However, current mainstream building change detection methods typically focus solely on whether buildings in the area of interest have changed, failing to capture the fine-grained state of the changed building targets. In the paper "Y.Sun, X.Zhang, J.Huang, H.Wang and Q.Xin, Fine-Grained Building Change Detection From Very High-Spatial-Resolution Remote Sensing Images Based on Deep Multitask Learning, in IEEE Geoscience and Remote Sensing Letters, vol. 19, pp. 1-5, 2022," Sun et al. use a multi-task learning strategy to simultaneously perform change detection on dual-temporal images and extract buildings, and then determine the fine-grained change category to which the changed building belongs based on the temporal phase in which it occurs. This method can effectively determine whether a building has disappeared or been newly built, while reconstructed buildings require complex post-processing to accurately identify them due to the integrity of the change detection results. In addition, this method requires that all building targets in the dual-temporal images be labeled in order to train the building extraction branch, which increases the burden of data labeling and requires high labor costs. Summary of the Invention

[0003] To overcome the shortcomings of existing technologies, this paper provides a method for fine-grained building damage change detection using an adaptive SAM model. This method first constructs a fine-grained building change detection dataset, then injects lightweight parameters to fine-tune the SAM model. Building change hint masks are then predicted, followed by decoding the fine-grained change type of building targets. Finally, the target change type is mapped to damage degree, with disappeared buildings assessed as completely damaged and rebuilt buildings as partially damaged. This method utilizes a mask hinting strategy to convert change region prediction into building extraction, thereby increasing the accuracy of building target change detection results.

[0004] The technical solution adopted by the present invention to solve the technical problem includes the following steps:

[0005] Step 1: Construct a fine-grained building change detection dataset;

[0006] The time phase images where the changed building targets are located are marked. If only the building instances in the previous time phase are marked as disappeared, if only the building instances in the next time phase are marked as newly built, and if the building instances in the previous and next time phases have changed, they are marked as rebuilt.

[0007] Step 2: Inject lightweight parameters to fine-tune the SAM model image encoder;

[0008] Step 3: Predict building change hint mask;

[0009] By aggregating and fine-tuning the SAM model image encoder at different stages, the dual-phase differential features are used to predict the change area and the change confidence is calculated. Figure 2 The value is obtained by hint mask;

[0010] Step 4: Decode the fine-grained change type of the building target;

[0011] Combining the hint mask and the fine-tuned SAM model image encoder output representation, the changing building targets are extracted in the dual-temporal image, and the change type of the changing building targets is determined according to the temporal phase in which they are located.

[0012] Step 5: Map the target change type to the damage degree;

[0013] Disappeared buildings are assessed as completely destroyed, while rebuilt buildings are assessed as partially destroyed.

[0014] Furthermore, the step 1 is specifically as follows:

[0015] Collect dual-phase remote sensing images and mark the building instances that have changed in the previous and next phase images. 0 represents non-changed buildings and non-building areas, and 1 represents changed buildings. Assume that the previous and next phase images are marked as I pre and I post , the corresponding labels are L pre and L post, the change area label and fine-grained building change type are obtained according to the following formula:

[0016] L f (i,j)=L pre (i,j)+2*L post (i,j)

[0017]

[0018] Among them L f (i, j)∈{0,1,2,3}, “1”, “2” and “3” represent the labels corresponding to the three types of buildings: disappearance, new construction and reconstruction, respectively. l (i,j)∈{0,1}, where “1” indicates the area where the building changes, and “0” indicates other areas.

[0019] Furthermore, the step 2 is specifically as follows:

[0020] A small number of learnable parameters are embedded in different layers of the SAM model image encoder; assuming that the transformer depth of the SAM model image encoder is n, the i-th transformer T i The output representation is F i ∈R C×H×W , the learnable embedding is denoted as E i ∈R m×C , the output representation F i With learnable embedding E i Fusion and input to the i+1th transformer T i+1 That is, we get the fine-tuned representation F i+1 , the above process is expressed by the following formula:

[0021] F i+1 =T i+1 (Cat(F i ,E i )), i=0,1,…,n-1

[0022] Where F0 is the shallow representation of the SAM model image encoder without fine-tuning, and Cat(x,y) represents the concatenation operation of the spatial dimension.

[0023] Furthermore, the step 3 is specifically as follows:

[0024] According to the representation of different levels of the fine-tuned SAM model image encoder, the dual-phase differential representation is obtained, and the differential representation corresponding to the i-th transformer is recorded as D i ,but:

[0025]

[0026] where f i1(x) represents the dimensionality reduction function, which consists of a convolution with a kernel of 1, a batch normalization layer, and a linear rectifier unit activation function, which is used to compress the number of channels of the original representation; f i2 The composition of (x) is the same as f i1 (x) is consistent, where the convolution weights are different; abs(x) is the absolute value function; on this basis, the differential representations of different levels are aggregated to obtain the change representation, which is described by the following formula:

[0027] Ch i+1 =g i2 (D i+1 +g i1 (Ch i )), Ch1=D1, i=1,2,3,…,n-1

[0028] Among them Ch i represents the change representation of different levels obtained through progressive aggregation, g i1 (x) and g i2 The composition of (x) is the same as f i1 (x) consistency; the progressive aggregation strategy can effectively reduce the redundancy of representations and pass the significant representations to the last level to obtain Ch n ; Finally, use Ch n Predicted Building Change Tips Mask BCM:

[0029] BCM=Binarized(z(Ch n )) ↑16

[0030] where BCM∈R H×W , Binarized(x) represents the binarization function; z(x) represents a convolution operation with a kernel size of 3, and the output is the building change confidence map; the building change hint mask prediction branch network is based on the change area label C l It is used as a supervisory signal and trained with binary cross entropy loss.

[0031] Furthermore, the step 4 is specifically as follows:

[0032] The change hint mask is first downsampled by a factor of 4 and then fed into the hint encoder:

[0033] E m =PromptEncoder(BCM ↓4 ),

[0034] Among them E mRepresents the encoded change prompt embedding; PromptEncoder(x) is composed of a convolution kernel with a stride of 2 and a kernel size of 2, a layer normalization operation, and a Gaussian error linear unit activation function, which obtains a prompt embedding that is consistent with the shape of the fine-tuned SAM model image encoder output representation. The change prompt embedding is then fused with the dual-temporal features to decode the corresponding change building target.

[0035] Ch bi =Decoder(E m +F ni ),i∈{pre,post}

[0036] The decoder Decoder(x) consists of a multi-layer perceptron that outputs Ch bi is the confidence distribution of the building target changes in the current phase image; the confidence distribution of the building changes in the two phases is binarized and the formula of step 1 is used to calculate the fine-grained building change type; the building target fine-grained change type prediction branch network is trained using binary cross entropy loss, L pre and L post as a supervisory signal.

[0037] The beneficial effects of the present invention are as follows:

[0038] 1. More specific detection results: The present invention classifies changed building targets into three types: disappeared, newly built, and rebuilt, making the detection results more specific.

[0039] 2. Higher detection accuracy: The present invention converts the change area prediction into building extraction through a mask prompting strategy, thereby increasing the accuracy of building target change detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a specific flow chart for the implementation of the present invention.

[0041] Figure 2 This is an example of the results of the embodiment of the present invention. DETAILED DESCRIPTION

[0042] The present invention will be further described below with reference to the accompanying drawings and examples.

[0043] This invention addresses the shortcomings of existing methods by effectively integrating the Segment Anything Model (SAM) proposed by Kirillov et al. (A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. Berg, W. Lo, P. Dollár, and Ross Girshick, Segment anything. arXiv preprint arXiv:2304.02643, 2023). This method proposes a method for fine-grained building damage change detection that adapts the SAM model. This method efficiently fine-tunes the SAM model and combines it with a predicted hint mask of building change regions. This allows the SAM model to output the changed building targets in the dual-temporal image, directly obtaining the fine-grained state of the changed building targets without the need for complex post-processing or labeling of all buildings. Finally, the damage severity can be assessed based on the type of building change.

[0044] 1. Construct a fine-grained building change detection dataset. Specifically, the time-phase images where the changed building targets are located are annotated. The images corresponding only to the previous time phase are labeled as disappeared, the images corresponding only to the next time phase are labeled as newly built, and the images where the building instances have changed between the previous and next time phases are labeled as rebuilt.

[0045] 2. Inject lightweight parameters to fine-tune the SAM model. A parameter-efficient fine-tuning strategy is used to adapt the SAM model encoder output features for mask-cue-based building extraction.

[0046] 3. Predict building change hint mask. Predict the change area by aggregating the bi-temporal difference features of the fine-tuned SAM model at different stages and calculate the change confidence. Figure 2 The hint mask is obtained by valuating it.

[0047] 4. Decode the fine-grained change type of building objects. Combining the change hint mask and the fine-tuned encoder output representation, we extract the changing building objects in the bi-temporal image and determine their change type based on the phase in which they are located.

[0048] 5. The mapping target change type is the degree of damage. Disappearing buildings are assessed as completely destroyed, while rebuilt buildings are assessed as partially destroyed. Newly constructed buildings typically do not exist in damage change analysis scenarios.

[0049] Example:

[0050] Reference Figure 1 The implementation steps of the fine-grained building damage change detection method of the present invention are as follows:

[0051] Step 1: Construct a fine-grained building change detection dataset: collect dual-temporal remote sensing images and annotate the building instances that have changed in the previous and next phase images. 0 represents a non-changed building (including unchanged buildings and non-building areas), and 1 represents a changed building. Assume that the previous and next phase images are denoted as I pre and I post , the corresponding labels are L pre and L post , then the change area label and fine-grained building change type can be obtained according to the following formula:

[0052] L f (i,j)=L pre (i,j)+2*L post (i,j),

[0053]

[0054] Among them L f (i, j)∈{0,1,2,3}, “1”, “2” and “3” represent the labels corresponding to the three types of buildings: disappearance, new construction and reconstruction, respectively. l (i,j) indicates whether a pixel has undergone architectural changes.

[0055] Step 2: Inject lightweight parameters to fine-tune the SAM model: A small amount of learnable parameters are embedded into different layers of the SAM model image encoder through a parameter-efficient fine-tuning strategy. Assume that the transformer depth of the SAM model image encoder is n, and the i-th transformer T i The output representation is F i ∈R C×H×W , the learnable embedding is denoted as E i ∈R m×C , the output representation F i With learnable embedding E i Fusion and input to the i+1th transformer T i+1 The fine-tuned representation F can be obtained i+1 , the above process can be expressed by the following formula:

[0056] F i+1 =T i+1 (Cat(F i ,E i )), (i=0,1,…,n-1),

[0057] Where F0 is the shallow representation of the SAM model image encoder without fine-tuning, and Cat(x,y) represents the concatenation operation of the spatial dimension.

[0058] Step 3: Predict the building change hint mask: First, obtain the dual-phase differential representation based on the representation of different levels of the fine-tuned SAM model image encoder. The differential representation corresponding to the i-th transformer is denoted as D i ,but

[0059]

[0060] where f i1 (x) represents the dimensionality reduction function, which consists of a convolution with a kernel of 1, a batch normalization layer, and a linear rectifier unit activation function, which is used to compress the number of channels of the original representation. i2 The composition of (x) is the same as f i1 (x) is consistent, where the convolution weights are different. abs(x) is the absolute value function. On this basis, the differential representations of different levels are aggregated to obtain the change representation, which can be described by the following formula:

[0061] Ch i+1 =g i2 (D i+1 +g i1 (Ch i )),Ch1=D1, (i=1,2,3,…,n-1),

[0062] Among them Ch i represents the change representation of different levels obtained through progressive aggregation, g i1 (x) and g i2 The composition of (x) is the same as f i1 (x) is consistent. The strategy of progressive aggregation can effectively reduce the redundancy of representation and pass the significant representation to the last level to obtain Ch n Finally, using Ch n Predicted building changes prompt mask BCM,

[0063] BCM=Binarized(z(Ch n )) ↑16 ,

[0064] where BCM∈R H×W , Binarized(x) represents the binarization function. z(x) represents a convolution operation with a kernel size of 3, and the output is the building change confidence map. The building change hint mask prediction sub-network is based on the change area label C l It is used as a supervisory signal and trained with binary cross entropy loss.

[0065] Step 4: Decode the fine-grained change type of the building target: Convert the binary change mask into the change region hint embedding. Specifically, the change hint mask is first downsampled by 4 times and then input into the hint encoder.

[0066] Em =PromptEncoder(BCM ↓4 ),

[0067] Among them E m Represents the encoded change prompt embedding. PromptEncoder(x) consists of a convolution kernel with a stride of 2 and a kernel size of 2, a layer normalization operation, and a Gaussian error linear unit activation function. It can obtain a prompt embedding that is consistent with the shape of the fine-tuned SAM model image encoder output representation. The change prompt embedding is then fused with the dual-temporal features to decode the corresponding change building target.

[0068] Ch bi =Decoder(E m +F ni ),(i∈{pre,post})

[0069] The decoder Decoder(x) consists of a multi-layer perceptron that outputs Ch bi is the confidence distribution of the building target changes in the current phase image. Binarize the confidence distribution of the building changes in the two phases and calculate it using the formula in step 1 to obtain the fine-grained building change type. The building target fine-grained change type prediction network is trained using binary cross entropy loss, L pre and L post as a supervisory signal.

[0070] Step 5: Map the target change type to the damage degree: the disappeared building target is assessed as completely damaged, and the rebuilt building target is assessed as partially damaged. In the damage change analysis scenario, there is usually no new building type.

[0071] The effects of the present invention can be further illustrated by the following simulation experiments.

[0072] 1. Simulation conditions

[0073] The present invention is a central processing unit The simulation experiments were conducted using Python and the PyTorch deep learning framework on an i7-6800K 3.40GHz CPU, an NVIDIA GeForce GTX3090 GPU, and an Ubuntu operating system. The data used in the experiments was re-annotated from the LEVIR-CD+ dataset, detailing the fine-grained classification of each changed building object. The LEVIR-CD+ dataset, proposed by Shen et al. in "Looking: A satellite side-looking dataset for building change detection. Remote Sensing, vol. 13, no. 24, 2021," is a building change detection dataset with 637 image pairs in the training set and 318 pairs in the test set. The re-annotated changed regions are classified as disappearance, new construction, and reconstruction.

[0074] 2. Simulation content

[0075] To compare the effectiveness of this invention, we used a transformer-based change detection method (ChangeFormer) proposed by Bandara et al. in the paper "W. Bandara and V. Patel, A Transformer-Based Siamese Network for Change Detection, in Proc. IEEE International Geoscience and Remote Sensing Symposium, 2022, pp. 207-210." We calculated the accuracy, recall, and F-measure. The comparison results are shown in Table 1.

[0076] Table 1 Comparison of experimental results

[0077]

[0078] As can be seen from Table 1, when detecting different types of building change areas, the F value of the present invention is better than ChangeFormer; when detecting disappeared buildings, the effect of the present invention is significantly better than ChangeFormer; when detecting reconstructed buildings, the recall rate is higher, reflecting the effectiveness of the present invention in detecting reconstructed buildings. Overall, the present invention can achieve better results when detecting different types of building change areas. In addition, it can be seen from Figure 2The result examples show that the present invention can effectively judge three different types of building changes: disappearance, new construction and reconstruction, where green represents disappearance, red represents new construction and blue represents reconstruction.

Claims

1. A method for fine-grained building damage change detection using an adapted SAM model, characterized in that: The steps include: Step 1: Construct a fine-grained building change detection dataset; The time phase images where the changed building targets are located are marked. If only the building instances in the previous time phase are marked as disappeared, if only the building instances in the next time phase are marked as newly built, and if the building instances in the previous and next time phases have changed, they are marked as rebuilt. Step 2: Inject lightweight parameters to fine-tune the SAM model image encoder; The step 2 is specifically as follows: A small number of learnable parameters are embedded in different layers of the SAM model image encoder; assuming that the transformer depth of the SAM model image encoder is , No. converters The output representation is , the learnable embedding is denoted as , the output representation With learnable embeddings Merge and enter converters That is, the fine-tuned representation , the above process is expressed by the following formula: ， =0, 1, …, in is the shallow representation of the SAM model image encoder without fine-tuning, Represents the concatenation operation of spatial dimensions; Step 3: Predict building change hint mask; The change region is predicted by aggregating and fine-tuning the bi-temporal differential features of the SAM model image encoder at different stages, and the change confidence map is binarized to obtain the hint mask. The step 3 is specifically as follows: According to the representation of different levels of the fine-tuned SAM model image encoder, the dual-phase difference representation is obtained, corresponding to the first The differential representation of the converter is denoted as ,but: ， =1, 2, 3, …, in represents the dimensionality reduction function, which consists of a convolution with a kernel of 1, a batch normalization layer, and a linear rectifier unit activation function, used to compress the number of channels of the original representation; The composition of Consistent, where the weights of the convolution are different; is the absolute value function; on this basis, the differential representations of different levels are aggregated to obtain the change representation, which is described by the following formula: ， =1, 2, 3, …, in Represents the change representation of different levels obtained through progressive aggregation, and The composition is the same as Consistent; the progressive aggregation strategy can effectively reduce the redundancy of representations and pass significant representations to the last level to obtain ;Finally use Predicted building change hint mask : in , represents the binarization function; Represents a convolution operation with a kernel size of 3, and the output is a building change confidence map; the building change hint mask prediction sub-network is used to change the area label As a supervisory signal and trained with binary cross entropy loss; Step 4: Decode the fine-grained change type of the building target; Combining the hint mask and the fine-tuned SAM model image encoder output representation, the changing building targets are extracted in the dual-temporal image, and the change type of the changing building targets is determined according to the temporal phase in which they are located. Step 5: Map the target change type to the damage degree; Disappeared buildings are assessed as completely destroyed, while rebuilt buildings are assessed as partially destroyed.

2. The method for fine-grained building damage change detection using an adapted SAM model according to claim 1, characterized in that: The step 1 is specifically as follows: Collect dual-temporal remote sensing images and mark the building instances that have changed in the previous and next phase images. 0 represents non-changed buildings and non-building areas, and 1 represents changed buildings. Assume that the previous and next phase images are respectively marked as and , and the corresponding labels are and , the change area label and fine-grained building change type are obtained according to the following formula: in , "1", "2" and "3" represent the labels corresponding to the three types of buildings: disappearance, new construction and reconstruction. , where "1" represents the building change area and "0" represents other areas.

3. The method for fine-grained building damage change detection using an adapted SAM model according to claim 1, characterized in that: The step 4 is specifically as follows: The change hint mask is first downsampled by a factor of 4 and then fed into the hint encoder: ， in Changes in representational encoding suggest embedding; It consists of a convolution kernel with a stride of 2 and a kernel size of 2, a layer normalization operation, and a Gaussian error linear unit activation function. It obtains a hint embedding that is consistent with the shape of the output representation of the fine-tuned SAM model image encoder. Then, the change hint embedding is fused with the dual-temporal features to decode the corresponding change building targets. { pre, post} Decoder It consists of a multi-layer perceptron, output is the confidence distribution of the building target changes in the current phase image; the confidence distribution of the building changes in the two phases is binarized and calculated using the formula in step 1 to obtain the fine-grained building change type; the building target fine-grained change type prediction network is trained using binary cross entropy loss. and as a supervisory signal.

Images