Remote sensing image building damage assessment method combining global and local clues

Abstract

The invention relates to the technical field of building damage assessment, in particular to a global and local clue-combined remote sensing image building damage assessment method, which comprises the following steps of: acquiring pre-disaster and post-disaster initial double-time-phase images of a building in a target area, pre-disaster and post-disaster dual-time-phase images meeting certain pairing conditions are obtained based on the initial dual-time-phase images; performing weight sharing coding on the dual-time-phase image to obtain a multi-scale dual-time-phase feature; obtaining a multi-scale difference enhancement fusion feature based on the multi-scale dual-time-phase feature; based on global and local clues, multi-scale difference enhancement fusion features are connected and fused, and output features are obtained; and predicting a damage assessment result and an error prediction result of the building based on the output features to obtain a final damage assessment result of the building. Therefore, the problems of poor modeling effect, difficulty in application migration, insufficient global-local feature utilization, lack of difficult sample sensing ability and the like caused by small feature difference in related technologies are solved.

Description

Technical Field

[0001] The present application relates to the technical field of building damage assessment, and in particular to a remote sensing image building damage assessment method combining global and local clues. Background Art

[0002] After a disaster occurs, it is crucial to quickly and accurately assess the scope and severity of the disaster area for subsequent rescue and reconstruction. The traditional ground survey and assessment process can obtain relatively detailed damage information, but it takes a long time, has high manpower and material costs, and has many limitations, which cannot meet the needs of disaster emergency response and other tasks. With the development of remote sensing technology, high-resolution remote sensing satellites can obtain large-scale image data of disaster areas in a short period of time, and have gradually been applied to the determination of disaster-stricken areas and disaster emergency response, providing a new means for efficient large-scale building damage assessment.

[0003] In the related technologies, damage assessment can be carried out by focusing on collapsed and non-collapsed buildings based on features such as color and texture and by considering the changes in the buildings themselves. It can also be based on deep learning technology and, according to the data used, be divided into a method that only uses post-disaster images and a method that uses both pre-disaster and post-disaster dual-phase images. According to the evaluation technology used, it can be divided into a method based on object-level scene classification and a method based on pixel-level change detection.

[0004] However, in related technologies, the differences in extracted features are small, resulting in poor modeling effects and difficulty in migration and application. In addition, global-local features are not fully utilized and the ability to perceive difficult samples is lacking, which urgently needs to be improved. Summary of the invention

[0005] The present application provides a remote sensing image building damage assessment method that combines global and local clues to solve the problems in related technologies, such as small differences in extracted features, resulting in poor modeling effect, difficulty in migration and application, insufficient utilization of global-local features, and lack of difficult sample perception ability.

[0006] The first aspect of the present application provides a remote sensing image building damage assessment method combining global and local clues, comprising the following steps: acquiring a pre-disaster initial dual-phase image and a post-disaster initial dual-phase image of the building in the target area, and obtaining a pre-disaster dual-phase image and a post-disaster dual-phase image that meet a preset pairing condition based on the pre-disaster initial dual-phase image and the post-disaster initial dual-phase image; performing weight-sharing encoding on the pre-disaster dual-phase image and the post-disaster dual-phase image to obtain a pre-disaster multi-scale dual-phase feature of the pre-disaster dual-phase image and a post-disaster multi-scale dual-phase feature of the post-disaster dual-phase image; based on the The pre-disaster multi-scale bi-phase features and the post-disaster multi-scale bi-phase features obtain the multi-scale difference enhancement fusion features between the pre-disaster bi-phase images and the post-disaster bi-phase images; based on the global and local clues between the pre-disaster bi-phase images and the post-disaster bi-phase images, the multi-scale difference enhancement fusion features are connected and fused to obtain the output features between the pre-disaster bi-phase images and the post-disaster bi-phase images; based on the output features, the damage assessment results and the error prediction results of the building are predicted to obtain the final damage assessment result of the building according to the damage assessment results and the error prediction results.

[0007] Optionally, in one embodiment of the present application, the multi-scale difference enhanced fusion features between the pre-disaster dual-phase image and the post-disaster dual-phase image based on the pre-disaster multi-scale dual-phase features and the post-disaster multi-scale dual-phase features include: obtaining the differential features between the pre-disaster multi-scale dual-phase features and the post-disaster multi-scale dual-phase features based on the pre-disaster multi-scale dual-phase features and the post-disaster multi-scale dual-phase features; obtaining the differential channel attention features of the differential features, the pre-disaster channel attention features of the pre-disaster multi-scale dual-phase features, and the post-disaster channel attention features of the post-disaster multi-scale dual-phase features based on channel attention; obtaining the initial pre-disaster differential enhanced channel attention features and the initial post-disaster differential enhanced channel attention features based on the differential channel attention features, the pre-disaster channel attention features, and the post-disaster channel attention features; obtaining the differential channel attention features based on the initial disaster The pre-disaster differential enhancement channel attention features and the initial post-disaster differential enhancement channel attention features are used to obtain the pre-disaster channel differential enhancement features and the post-disaster channel differential enhancement features; based on the pre-disaster channel differential enhancement features and the post-disaster channel differential enhancement features, the channel differential features between the pre-disaster channel differential enhancement features and the post-disaster channel differential enhancement features are obtained; using spatial attention to obtain the differential spatial attention features of the channel differential features, the pre-disaster spatial attention features of the pre-disaster multi-scale dual-temporal features, and the post-disaster spatial attention features of the post-disaster multi-scale dual-temporal features; based on the differential spatial attention features, the pre-disaster spatial attention features and the post-disaster spatial attention features, the pre-disaster spatial differential enhancement features and the post-disaster spatial differential enhancement features are obtained; based on the pre-disaster spatial differential enhancement features and the post-disaster spatial differential enhancement features, the multi-scale difference enhancement fusion features are obtained.

[0008] Optionally, in one embodiment of the present application, the multi-scale difference enhancement fusion features are connected and fused based on the global and local clues between the pre-disaster dual-phase images and the post-disaster dual-phase images to obtain the output features between the pre-disaster dual-phase images and the post-disaster dual-phase images, including: based on the global and local clues between the pre-disaster dual-phase images and the post-disaster dual-phase images, the difference enhancement fusion features of the first scale in the multi-scale difference enhancement fusion features are connected and fused to obtain the scale intermediate features of the first scale; based on the scale intermediate features, the scale output features of the first scale are obtained; based on the scale output features, the difference enhancement fusion features of the first scale, and the difference enhancement fusion features of the second scale, the output features are obtained.

[0009] Optionally, in one embodiment of the present application, predicting the damage assessment result and the error prediction result of the building based on the output features to obtain the final damage assessment result of the building according to the damage assessment result and the error prediction result includes: based on the output features, connecting and fusing the difference enhancement fusion features of the third scale, the difference enhancement fusion features of the fourth scale, and the difference enhancement fusion features of the fifth scale of the multi-scale difference enhancement fusion features through a jump connection to obtain the damage assessment result and the error prediction result; generating the final loss assessment result based on the damage assessment result and the error prediction result.

[0010] Optionally, in one embodiment of the present application, based on the output features, the difference enhancement fusion features of the third scale, the difference enhancement fusion features of the fourth scale, and the difference enhancement fusion features of the fifth scale of the multi-scale difference enhancement fusion features are connected and fused by jump connection to obtain the damage assessment result and the error prediction result, including: performing middle-layer inter-layer fusion decoding based on the difference enhancement fusion features of the fourth scale and the difference enhancement fusion features of the fifth scale to obtain the middle-layer fusion decoding features of the fourth scale and the middle-layer fusion decoding features of the fifth scale; performing shallow inter-layer fusion decoding based on the difference enhancement fusion features of the third scale, the middle-layer fusion decoding features of the fourth scale, and the middle-layer fusion decoding features of the fifth scale to obtain the damage assessment result and the error prediction result.

[0011] The second aspect of the present application provides a remote sensing image building damage assessment device that combines global and local clues, including: an acquisition module, used to acquire a pre-disaster initial dual-phase image and a post-disaster initial dual-phase image of the building in the target area, and obtain a pre-disaster dual-phase image and a post-disaster dual-phase image that meet a preset pairing condition based on the pre-disaster initial dual-phase image and the post-disaster initial dual-phase image; an encoding module, used to perform weight-sharing encoding on the pre-disaster dual-phase image and the post-disaster dual-phase image to obtain the pre-disaster multi-scale dual-phase features of the pre-disaster dual-phase image and the post-disaster multi-scale dual-phase features of the post-disaster dual-phase image; a difference enhancement fusion module, used to perform weight-sharing encoding on the pre-disaster dual-phase image and the post-disaster dual-phase image based on the pre-disaster initial dual-phase image and the post-disaster initial dual-phase image; The pre-disaster multi-scale dual-phase features and the post-disaster multi-scale dual-phase features obtain the multi-scale difference enhancement fusion features between the pre-disaster dual-phase images and the post-disaster dual-phase images; a global and local clue fusion module is used to connect and fuse the multi-scale difference enhancement fusion features based on the global and local clues between the pre-disaster dual-phase images and the post-disaster dual-phase images to obtain the output features between the pre-disaster dual-phase images and the post-disaster dual-phase images; an evaluation module is used to predict the damage assessment results and error prediction results of the building based on the output features to obtain the final damage assessment result of the building according to the damage assessment results and the error prediction results.

[0012] Optionally, in one embodiment of the present application, the difference enhancement fusion module includes: a difference unit, used to obtain the difference features between the pre-disaster multi-scale bi-phase features and the post-disaster multi-scale bi-phase features based on the pre-disaster multi-scale bi-phase features and the post-disaster multi-scale bi-phase features; a channel attention unit, used to obtain the differential channel attention features of the differential features, the pre-disaster channel attention features of the pre-disaster multi-scale bi-phase features, and the post-disaster channel attention features of the post-disaster multi-scale bi-phase features based on channel attention; a differential enhancement channel attention unit, used to obtain the initial pre-disaster differential enhancement channel attention features and the initial post-disaster differential enhancement channel attention features based on the differential channel attention features, the pre-disaster channel attention features, and the post-disaster channel attention features; a channel differential enhancement unit, used to obtain the initial pre-disaster differential enhancement channel attention features and the initial post-disaster differential enhancement channel attention features based on the initial pre-disaster differential enhancement channel attention features and the initial post-disaster differential enhancement channel attention features. The intention feature obtains the pre-disaster channel differential enhancement feature and the post-disaster channel differential enhancement feature; the channel difference unit is used to obtain the channel differential feature between the pre-disaster channel differential enhancement feature and the post-disaster channel differential enhancement feature based on the pre-disaster channel differential enhancement feature and the post-disaster channel differential enhancement feature; the spatial attention unit is used to use spatial attention to obtain the differential spatial attention feature of the channel differential feature, the pre-disaster spatial attention feature of the pre-disaster multi-scale dual-temporal feature, and the post-disaster spatial attention feature of the post-disaster multi-scale dual-temporal feature; the spatial differential enhancement unit is used to obtain the pre-disaster spatial differential enhancement feature and the post-disaster spatial differential enhancement feature based on the differential spatial attention feature, the pre-disaster spatial attention feature and the post-disaster spatial attention feature; the difference enhancement fusion unit is used to obtain the multi-scale difference enhancement fusion feature based on the pre-disaster spatial differential enhancement feature and the post-disaster spatial differential enhancement feature.

[0013] Optionally, in one embodiment of the present application, the global and local clue fusion module includes: a fusion unit, used to connect and fuse the difference enhancement fusion features of the first scale in the multi-scale difference enhancement fusion features based on the global and local clues between the pre-disaster dual-phase images and the post-disaster dual-phase images to obtain the scale intermediate features of the first scale; a first generation unit, used to obtain the scale output features of the first scale based on the scale intermediate features; and a second generation unit, used to obtain the output features based on the scale output features, the difference enhancement fusion features of the first scale, and the difference enhancement fusion features of the second scale.

[0014] Optionally, in one embodiment of the present application, the evaluation module includes: a first evaluation unit, used to connect and fuse the third scale difference enhancement fusion features, the fourth scale difference enhancement fusion features and the fifth scale difference enhancement fusion features of the multi-scale difference enhancement fusion features through a jump connection based on the output features to obtain the damage assessment result and the error prediction result; a second evaluation unit, used to generate the final loss assessment result based on the damage assessment result and the error prediction result.

[0015] Optionally, in one embodiment of the present application, the first evaluation unit includes: a first fusion decoding subunit, used to perform middle-layer inter-layer fusion decoding based on the difference enhancement fusion features of the fourth scale and the difference enhancement fusion features of the fifth scale to obtain the middle-layer fusion decoding features of the fourth scale and the middle-layer fusion decoding features of the fifth scale; a second fusion decoding subunit, used to perform shallow inter-layer fusion decoding based on the difference enhancement fusion features of the third scale, the middle-layer fusion decoding features of the fourth scale, and the middle-layer fusion decoding features of the fifth scale to obtain the damage assessment result and the error prediction result.

[0016] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the remote sensing image building damage assessment method combining global and local clues as described in the above embodiment.

[0017] A fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-mentioned remote sensing image building damage assessment method combining global and local cues.

[0018] The fifth aspect of the present application provides a computer program product, including a computer program, which, when executed, implements the above-mentioned remote sensing image building damage assessment method combining global and local cues.

[0019] The embodiment of the present application first obtains multi-scale dual-phase features by weight-sharing encoding of pre-disaster and post-disaster dual-phase images; secondly, the multi-scale dual-phase features are differentially enhanced and fused to obtain multi-scale differentially enhanced fused features; again, based on global and local clues, connection fusion is performed to obtain output features; finally, the output features are used to output auxiliary damage assessment results and error prediction results, and then the final damage assessment results are output. The embodiment of the present application enhances the difference between dual-phase features, has a good effect in suppressing change noise, performs well in improving the semantic information extraction ability of the network, and can improve the judgment ability of difficult samples. Therefore, the problems in the related technology that the extracted features are small in difference, resulting in poor modeling effect, difficulty in migration application, insufficient utilization of global-local features, and lack of perception ability of difficult samples are solved.

[0020] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0022] Figure 1 A flowchart of a remote sensing image building damage assessment method combining global and local clues provided according to an embodiment of the present application;

[0023] Figure 2 It is a schematic diagram of the structure of a DEFM (Defeature Enhancement and Fusion Module) provided according to an embodiment of the present application;

[0024] Figure 3 A schematic diagram of a calculation process for obtaining multi-scale difference enhancement fusion features using DEFM according to an embodiment of the present application;

[0025] Figure 4 A schematic diagram of the structure of a GLCFM (Global Local Clue Fusion Module) provided according to an embodiment of the present application;

[0026] Figure 5 A flowchart of obtaining output features according to an embodiment of the present application;

[0027] Figure 6 A schematic diagram of the structure of a SSB (Selective Scan Block) provided according to an embodiment of the present application;

[0028] Figure 7 A schematic diagram of the structure of a HMCB (Hierarchical Multi-scale Convolutional Block) provided according to an embodiment of the present application;

[0029] Figure 8 A schematic diagram of the structure of a DEAD (Dynamic Error Awareness Decoder) provided according to an embodiment of the present application;

[0030] Fig. 9 A flowchart of obtaining a final damage assessment result of a building according to an embodiment of the present application;

[0031] Fig.10 A quantitative result diagram of a comparative experiment provided according to an embodiment of the present application;

[0032] Fig.11 This is a qualitative result diagram of a comparative experiment provided according to an embodiment of the present application;

[0033] Fig.12 A block diagram of the working principle of a remote sensing image building damage assessment method combining global and local clues provided according to an embodiment of the present application;

[0034] Fig.13 A schematic block diagram of a remote sensing image building damage assessment device combining global and local clues provided according to an embodiment of the present application;

[0035] Fig.14 It is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0036] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0037] The following describes the method for assessing building damage in remote sensing images by combining global and local clues in the embodiment of the present application with reference to the accompanying drawings. In view of the problem that the extracted feature differences mentioned in the above background technology are small, resulting in poor modeling effect, difficulty in migration and application, insufficient utilization of global-local features, and lack of perception ability of difficult samples, the present application provides a method for assessing building damage in remote sensing images by combining global and local clues. In the method, firstly, multi-scale dual-phase features are obtained by weight sharing encoding of pre-disaster and post-disaster dual-phase images; secondly, multi-scale dual-phase features are differentially enhanced and fused to obtain multi-scale differential enhanced fusion features; again, based on global and local clues, connection fusion is performed to obtain output features; finally, auxiliary damage assessment results and error prediction results are output using the output features, and then the final damage assessment results are output. The embodiment of the present application enhances the difference between dual-phase features, has a good effect on suppressing change noise, performs well in improving the semantic information extraction ability of the network, and can improve the judgment ability of difficult samples. Thus, the problem that the extracted feature differences are small, resulting in poor modeling effect, difficulty in migration and application, insufficient utilization of global-local features, and lack of perception ability of difficult samples in the related technology is solved.

[0038] Specifically, Figure 1 The present invention provides a flowchart of a method for assessing building damage in remote sensing images by combining global and local clues according to an embodiment of the present application.

[0039] like Figure 1 As shown, the remote sensing image building damage assessment method combining global and local clues includes the following steps:

[0040] In step S101, the pre-disaster initial dual-phase images and the post-disaster initial dual-phase images of the buildings in the target area are acquired, and the pre-disaster dual-phase images and the post-disaster dual-phase images that meet the preset pairing conditions are obtained based on the pre-disaster initial dual-phase images and the post-disaster initial dual-phase images.

[0041] In some embodiments, the embodiments of the present application can obtain pre-disaster initial dual-temporal images of the target area including buildings.

[0042] In some embodiments, the embodiments of the present application can obtain initial dual-temporal images of the target area including buildings after a disaster.

[0043] In some embodiments, the embodiments of the present application pair and crop the acquired pre-disaster initial dual-phase images and post-disaster initial dual-phase images, thereby obtaining pre-disaster dual-phase images and post-disaster dual-phase images that meet certain pairing conditions. The certain pairing conditions can be set by those skilled in the art according to actual conditions, and this application does not impose specific restrictions.

[0044] In step S102, weight-sharing encoding is performed on the pre-disaster bi-temporal image and the post-disaster bi-temporal image to obtain pre-disaster multi-scale bi-temporal features of the pre-disaster bi-temporal image and post-disaster multi-scale bi-temporal features of the post-disaster bi-temporal image.

[0045] As a possible implementation method, the embodiment of the present application can send the pre-disaster dual-phase images and the post-disaster dual-phase images that meet certain pairing conditions into a weight sharing encoder for weight sharing encoding, thereby obtaining multi-scale dual-phase features before and after the disaster, which can be expressed as but not limited to: F i pre , Among them, H i =H / 2 i , W i =W / 2 i , H and W are the length and width of the input image, respectively, C i is the number of channels, C 1 =64, C i =C 1 2 i-2 , {i=2,3,4,5}.

[0046] In step S103, a multi-scale difference enhancement fusion feature between the pre-disaster bi-temporal image and the post-disaster bi-temporal image is obtained based on the pre-disaster multi-scale bi-temporal feature and the post-disaster multi-scale bi-temporal feature.

[0047] In the actual implementation process, the embodiment of the present application can send the pre-disaster multi-scale bi-temporal features and the post-disaster multi-scale bi-temporal features into DEFM to obtain the multi-scale difference enhancement fusion features.

[0048] In addition, in the embodiment of the present application, the structural diagram of DEFM is as follows Figure 2 shown.

[0049] Optionally, in one embodiment of the present application, a multi-scale difference enhancement fusion feature between a pre-disaster dual-phase image and a post-disaster dual-phase image is obtained based on the pre-disaster multi-scale dual-phase features and the post-disaster multi-scale dual-phase features, including: obtaining differential features between the pre-disaster multi-scale dual-phase features and the post-disaster multi-scale dual-phase features based on the pre-disaster multi-scale dual-phase features and the post-disaster multi-scale dual-phase features; obtaining differential channel attention features of differential features based on channel attention, pre-disaster channel attention features of pre-disaster multi-scale dual-phase features, and post-disaster channel attention features of post-disaster multi-scale dual-phase features; obtaining initial pre-disaster differential enhancement channel attention features and initial post-disaster differential enhancement channel attention features based on differential channel attention features, pre-disaster channel attention features, and post-disaster channel attention features; obtaining differential channel attention features based on initial pre-disaster differential enhancement channel attention features, pre-disaster channel attention features, and post-disaster channel attention features; obtaining differential channel attention features based on initial pre-disaster differential enhancement channel attention features. The differential enhanced channel attention features and the initial post-disaster differential enhanced channel attention features are used to obtain the pre-disaster channel differential enhanced features and the post-disaster channel differential enhanced features; based on the pre-disaster channel differential enhanced features and the post-disaster channel differential enhanced features, the channel differential features between the pre-disaster channel differential enhanced features and the post-disaster channel differential enhanced features are obtained; using spatial attention, the differential spatial attention features of the channel differential features, the pre-disaster spatial attention features of the pre-disaster multi-scale dual-temporal features, and the post-disaster spatial attention features of the post-disaster multi-scale dual-temporal features are obtained; based on the differential spatial attention features, the pre-disaster spatial attention features and the post-disaster spatial attention features, the pre-disaster spatial differential enhanced features and the post-disaster spatial differential enhanced features are obtained; based on the pre-disaster spatial differential enhanced features and the post-disaster spatial differential enhanced features, the multi-scale difference enhanced fusion features are obtained.

[0050] Those skilled in the art can understand that the calculation process of the embodiment of the present application using DEFM to obtain the multi-scale difference enhancement fusion feature is as follows: Figure 3 As shown, the specific content can be:

[0051] Step S301: Obtain differential channel attention features, pre-disaster channel attention features, and post-disaster channel attention features.

[0052] For example, the embodiment of the present application can input the multi-scale bi-phase feature F i pre , {i=1,2,3,4,5}, subtract the two and take the absolute value to obtain the differential feature F i sub , obtain differential channel attention features through channel attention F i pre and F i post The same process is also used to obtain the pre-disaster channel attention features and post-disaster channel attention features Among them, the calculation formula of the channel attention feature in the embodiment of the present application can be but is not limited to:

[0053] Ac=MLP(GAP(F))+MLP(GMP(F)),

[0054] Among them, F represents the input feature, MLP(·) represents the multi-layer perceptron with shared parameters, GAP(·) represents the global average pooling, GMP(·) represents the global maximum pooling, and Ac is the obtained channel attention feature.

[0055] Step S302: Obtain pre-disaster differential enhancement channel attention features and post-disaster differential enhancement channel attention features.

[0056] For example, the embodiments of the present application may and By 1×1×C i Transpose to 1×C i ×1, superimposed along the channel dimension, and adaptively fused through 1×7 convolution to obtain the initial pre-disaster differential enhanced channel attention features and initial post-disaster differentially enhanced channel attention features The calculation formula can be expressed as but not limited to:

[0057]

[0058] Among them, Conv 1×7 (·) represents 1×7 convolution, [·,·] represents superposition along the channel dimension, and σ(·) represents the Sigmoid function.

[0059] Furthermore, the embodiment of the present application obtains the initial pre-disaster and post-disaster differential enhanced channel attention features and Transpose back to 1×1×C i , and multiply it with the original feature, and use the residual connection method to generate the pre-disaster channel differential enhancement feature and post-disaster channel differential enhancement characteristics The calculation formula can be expressed as but not limited to:

[0060]

[0061] in, is element-wise multiplication.

[0062] Step S303: Obtain spatial attention features.

[0063] Exemplarily, the embodiment of the present application enhances the characteristics of pre-disaster and post-disaster channel differentials. and Subtract and take the absolute value to obtain the channel differential feature Obtaining pre-disaster multi-scale bi-temporal features via spatial attention Multi-scale and dual-temporal characteristics after disasters and channel differential characteristics Spatial attention features and The calculation formula of the spatial attention feature of the embodiment of the present application can be, but is not limited to, expressed as:

[0064] As=Conv 7×7 ([GAP(F),GMP(F)]),

[0065] Among them, As is the obtained spatial attention feature.

[0066] Step S304: Acquire spatial difference enhancement features.

[0067] Exemplarily, the embodiment of the present application uses the differential spatial attention feature and pre-disaster spatial attention characteristics and post-disaster spatial attention characteristics Superposition along the channel dimension, generating differential enhanced spatial attention features through 3×3 convolution and Sigmoid function and And generate the pre-disaster spatial difference enhancement features through residual connection and post-disaster spatial differential enhancement characteristics The calculation formula can be expressed as but not limited to:

[0068]

[0069]

[0070] Step S305: Output multi-scale difference enhanced fusion features.

[0071] Exemplarily, the present application embodiment enhances the pre-disaster spatial difference feature and post-disaster spatial differential enhancement characteristics Perform channel superposition and obtain the final multi-scale difference enhancement fusion features through two consecutive convolutional layers The calculation formula can be expressed as but not limited to:

[0072]

[0073] f i fus =ReLU(BN(Conv 3×3 (f i inter ))),

[0074] Among them, BN(·) represents batch normalization, and ReLU(·) represents ReLU activation. Specifically, the embodiment of the present application fully interacts the bi-phase features through 1×1 convolution to obtain the intermediate features And dimension compression is performed through 3×3 convolution to generate multi-scale difference enhanced fusion features.

[0075] In step S104, based on the global and local clues between the pre-disaster dual-temporal images and the post-disaster dual-temporal images, the multi-scale difference enhanced fusion features are connected and fused to obtain the output features between the pre-disaster dual-temporal images and the post-disaster dual-temporal images.

[0076] As a possible implementation method, the embodiment of the present application can send the multi-scale difference enhancement fusion feature to GLCFM based on the global and local clues between the pre-disaster dual-phase image and the post-disaster dual-phase image, and connect the fusion feature and the decoded output feature through a jump connection to obtain the output feature f between the pre-disaster dual-phase image and the post-disaster dual-phase image. 3 out .

[0077] In addition, in the embodiment of the present application, the structural diagram of GLCFM is as follows Figure 4 shown.

[0078] Optionally, in one embodiment of the present application, based on global and local clues between the pre-disaster dual-phase images and the post-disaster dual-phase images, multi-scale difference enhancement fusion features are connected and fused to obtain output features between the pre-disaster dual-phase images and the post-disaster dual-phase images, including: based on the global and local clues between the pre-disaster dual-phase images and the post-disaster dual-phase images, the difference enhancement fusion features of the first scale in the multi-scale difference enhancement fusion features are connected and fused to obtain scale intermediate features of the first scale; based on the scale intermediate features, scale output features of the first scale are obtained; based on the scale output features, the difference enhancement fusion features of the first scale, and the difference enhancement fusion features of the second scale are obtained.

[0079] In some embodiments, the process of obtaining output features in the embodiment of the present application is as follows: Figure 5 As shown, the main steps are:

[0080] Step S501: Obtain scale output features.

[0081] Based on the global and local clues between the pre-disaster dual-phase images and the post-disaster dual-phase images, the first-scale intermediate features are obtained by connecting and fusing the multi-scale difference enhancement fusion features.

[0082] Exemplarily, the embodiment of the present application enhances the fusion feature of the difference of the first scale The scale output features are generated based on the scale intermediate features after inputting into GLCFM-V (Global Local Clue Fusion Module-series connection) to obtain the scale intermediate features.

[0083] Specifically, the embodiments of the present application will Send it to SSB to get the intermediate scale features The calculation formula can be but is not limited to:

[0084] f i mid =Lin(SSB(LN(f i fus )))+f i fus ,

[0085] Among them, Lin(·) is the linear layer and LN(·) is the layer normalization.

[0086] In addition, it should be noted that the structural diagram of the SSB in the embodiment of the present application can be as follows: Figure 6 As shown, the specific calculation process can be:

[0087] It is understandable that the core of the SSB embodiment of the present application is SS2D (2D Selective Scan Module, 2D selective scanning module), which inputs the feature Input two branches, one branch obtains the gated feature Z through a linear layer i , and the other branch obtains the global feature Y through SS2D i , the two are multiplied element by element to obtain the output And through channel attention adjustment, the main process can be expressed as but not limited to:

[0088]

[0089] Here, SiLU(·) represents SiLU activation, CA(·) and SA(·) represent complete channel and spatial attention, respectively.

[0090] Among them, the embodiment SS2D of the present application first expands the two-dimensional features in the vertical and horizontal directions into four one-dimensional sequences in different orders, and applies one-dimensional selective scanning to the four one-dimensional sequences respectively to collect global context in different directions, and finally folds the scanned one-dimensional sequences back to two dimensions and adds them to obtain global features.

[0091] Furthermore, the embodiment of the present application inputs the scale intermediate features output by SSB into HMCB, and outputs the scale output features The calculation formula can be but is not limited to:

[0092] f i out =DWConv(HMCB(LN(f i mid )))+f i mid ,

[0093] Where i=5, DWConv(·) is the DW convolution.

[0094] In addition, it should be noted that the structural diagram of the HMCB in the embodiment of the present application can be as follows: Figure 7 As shown, the specific calculation process can be:

[0095] It can be understood that the HMCB of the present application embodiment includes two branches, a multi-scale convolution branch and a 1×1 convolution branch, which converts the input feature The channel dimension is divided into four parts Convolution is performed on each feature with different expansion rates, and cross-channel connections are added to enhance the cross-channel and scale interaction capabilities of the module. The features obtained by layered convolution The channels are stacked along the channel dimension, and then the channels are integrated through 1×1 convolution to obtain the output features. The calculation formula can be expressed as but not limited to:

[0096]

[0097] in, Represents a 3×3 convolution with a dilation rate of j. The 1×1 convolution branch is responsible for collecting local features, focusing on the channel dimension. Features of the 1×1 convolution branch Added to the output of the multi-scale convolution branch, the final output feature is obtained It can be expressed as, but not limited to:

[0098]

[0099] Step S502: Obtain output features.

[0100] For example, the embodiment of the present application outputs the scale output feature of GLCFM-V Input into GLCFM-H (GlobalLocal Clue Fusion Module-parallel fusion, global and local clue parallel fusion module) and f i fus , i∈{4,5}, the final output feature is f 3 out .

[0101] Specifically, the embodiment of the present application first performs feature fusion and compression through a linear layer to obtain a compressed feature f i sq Normalized and sent to SSB and HCMB branches respectively to obtain the global feature f i G and local features f i L The two types of features are adjusted and added through the linear layer and DW (Depthwise) convolution layer, and the residual connection structure is introduced. The added features are deeply fused through the normalization layer and the linear layer, and the output features are obtained by upsampling and dimension transformation through pixel shuffling and linear layer. The calculation process of GLCFM-H in the embodiment of the present application can be expressed as, but not limited to:

[0102] f i sq =Lin([f i fus ,f i out ]),

[0103] f i G =Lin(SSB(LN(f i sq ))),

[0104] f i L =DWConv(HMCB(LN(f i sq ))),

[0105]

[0106] Among them, PS(·) is the pixel shuffle operation.

[0107] In step S105, the damage assessment result and the error prediction result of the building are predicted based on the output features, so as to obtain the final damage assessment result of the building according to the damage assessment result and the error prediction result.

[0108] In the actual implementation process, the embodiment of the present application can output the output feature f of GLCFM 3 out Input into DEAD to predict the damage assessment result P of the building dam1 and the wrong prediction result P err , and then get the final damage assessment result P of the building dam2 .

[0109] Among them, the structural diagram of DEAD in the embodiment of the present application is as follows Figure 8 As shown, it can be understood that: the basic unit of DEAD in the embodiment of the present application is CFDM (Convolutional Feature Fusion and Decoding Module, inter-layer convolutional feature fusion and decoding module). According to the position of the module, CDFM can be divided into CFDM-M (Convolutional Feature Fusion and Decoding Module at Middle Layers, middle layer inter-layer fusion decoding module) and CFDM-S (Convolutional Feature Fusion Decoder Module at Shallow Layers, shallow layer inter-layer fusion decoding module).

[0110] Among them, the embodiment CFDM-M of the present application includes two consecutive convolution blocks (3×3 convolution + batch normalization + ReLU activation). The first block fuses the inter-layer features of channel superposition, and the second block performs semantic extraction on the fused features; CFDM-S changes the second block to 3×3 convolution to output the result.

[0111] Optionally, in one embodiment of the present application, damage assessment results and error prediction results of a building are predicted based on the output features to obtain a final damage assessment result of the building based on the damage assessment results and the error prediction results, including: based on the output features, connecting and fusing the difference enhancement fusion features of the third scale, the difference enhancement fusion features of the fourth scale, and the difference enhancement fusion features of the fifth scale of the multi-scale difference enhancement fusion features through a jump connection to obtain the damage assessment results and the error prediction results; generating a final loss assessment result based on the damage assessment results and the error prediction results.

[0112] The difference enhanced fusion features f are connected by skip connection 1 out , and f 3 out , and output a damage assessment result, an error prediction result, and a final damage assessment result;

[0113] As a possible implementation method, the flowchart of the embodiment of the present application for obtaining the final damage assessment result of the building is as follows: Fig. 9 As shown, the main steps are:

[0114] Step S901: Obtain damage assessment results and error prediction results.

[0115] Exemplarily, in the embodiment DEAD of the present application, the output f of GLCFM is 3out Input into two independent branches respectively, connect and fuse the difference enhancement fusion feature f through skip connection 1 fus , f 3 fus , and then output a damage assessment result P at the end of the branch dam1 and an error prediction result P err The damage assessment truth value GT dam and P dam1 Make the difference and take the absolute value, normalize it and use it as the true value GT of the wrong prediction err , perform gradient truncation in the difference process to prevent the gradient propagation of the wrong prediction branch from interfering with the training of the damage evaluation branch. Its calculation formula can be expressed as but not limited to:

[0116] GT err =λ·SUM(|GT dam1 -δ(P dam1 )|),

[0117] Among them, P dam1 , GT err ∈R H×W×1 , C cls is the total number of output categories, SUM(·) represents the addition along the channel dimension, δ(·) represents the Softmax function along the channel dimension, and λ is the normalization coefficient, which is set to 0.5 here.

[0118] Step S902: Generate final loss assessment results.

[0119] Exemplarily, the features output by the CFDM-S modules at the end of the two branches of the embodiment of the present application are superimposed along the channel and sent to an additional CFDM-S module, which is further refined in combination with the error prediction features, and the final damage assessment result P is output. dam2 The final network is jointly optimized by two damage assessment losses and one error prediction loss, which can be expressed as but not limited to:

[0120]

[0121] in, and Assess the loss for both injuries, For the error assessment loss, L mix is the class imbalance loss function, which uses the cross entropy loss L ce As the main loss, L is added to alleviate the problem of category imbalance. Lovasz The loss is used as a correction, and its calculation formula can be expressed as but not limited to:

[0122] L mix =L ce +L Lovasz ,

[0123] Optionally, in one embodiment of the present application, based on the output features, the difference enhancement fusion features of the third scale, the difference enhancement fusion features of the fourth scale, and the difference enhancement fusion features of the fifth scale of the multi-scale difference enhancement fusion features are connected and fused by jump connection to obtain damage assessment results and error prediction results, including: performing middle-layer inter-layer fusion decoding based on the difference enhancement fusion features of the fourth scale and the difference enhancement fusion features of the fifth scale to obtain middle-layer fusion decoding features of the fourth scale and middle-layer fusion decoding features of the fifth scale; performing shallow inter-layer fusion decoding based on the difference enhancement fusion features of the third scale, the middle-layer fusion decoding features of the fourth scale, and the middle-layer fusion decoding features of the fifth scale to obtain damage assessment results and error prediction results.

[0124] In some embodiments, the embodiments of the present application can send the difference enhancement fusion features of the fourth scale and the difference enhancement fusion features of the fifth scale into CFDM-M to obtain the middle-layer fusion decoding features, and send the difference enhancement fusion features of the third scale, the middle-layer fusion decoding features of the fourth scale, and the middle-layer fusion decoding features of the fifth scale into CFDM-S to obtain damage assessment results and error prediction results.

[0125] Furthermore, in order to verify the effectiveness of this method, the present application embodiment conducted a comparative experiment with multiple latest building damage assessment methods on the largest public satellite image building damage assessment dataset xBD. The quantitative and qualitative results of the experiment are as follows: Fig.10 and Fig.11 As shown. Quantitative results show that the embodiment of the present application has achieved excellent results in the F1 scores of building extraction and damage classification, especially the F1 score of damage classification is significantly better than the comparison method. Qualitative results show that the damage assessment results obtained in the embodiment of the present application have a more detailed building extraction contour, and can more accurately identify the damage category of large-area buildings and small buildings. The above experiments prove the effectiveness of this method in the task of building damage assessment.

[0126] like Fig.12 As shown, a specific embodiment explains the working content of the remote sensing image building damage assessment method combining global and local clues proposed in the embodiment of the present application.

[0127] Embodiment 1:

[0128] First, the embodiment of the present application obtains a pre-disaster initial dual-phase image and a post-disaster initial dual-phase image of the target area including buildings, and then obtains a pre-disaster dual-phase image and a post-disaster dual-phase image that meet certain pairing conditions.

[0129] Secondly, the embodiment of the present application sends the pre-disaster dual-phase images and the post-disaster dual-phase images that meet certain pairing conditions into a weight-sharing encoder to obtain multi-scale dual-phase features before and after the disaster.

[0130] Again, the embodiment of the present application sends the multi-scale dual-phase features to Figure 2 In the DEFM shown in the figure, multi-scale difference enhancement fusion features are obtained. The specific process is as follows Figure 3 shown.

[0131] Next, the embodiment of the present application sends the multi-scale difference enhancement fusion feature to Figure 4 In the GLCFM shown in Figure 2, the output features are obtained. The specific process is as follows: Figure 5 shown.

[0132] Finally, the embodiment of the present application inputs the output characteristics of the GLCFM output into the following Figure 8 In the DEAD shown in Figure 1, the final damage assessment result of the building is obtained. The specific process is as follows: Fig. 9 shown.

[0133] According to the method for building damage assessment of remote sensing images combining global and local clues proposed in the embodiment of the present application, firstly, weight-sharing encoding is performed on the pre-disaster and post-disaster dual-phase images to obtain multi-scale dual-phase features; secondly, the multi-scale dual-phase features are subjected to difference enhancement fusion to obtain multi-scale difference enhancement fusion features; again, connection fusion is performed based on global and local clues to obtain output features; finally, the output features are used to output auxiliary damage assessment results and error prediction results, and then the final damage assessment results are output. The embodiment of the present application enhances the difference between dual-phase features, has a good effect in suppressing change noise, performs well in improving the semantic information extraction ability of the network, and can improve the judgment ability of difficult samples. Therefore, the problems in the related technology that the extracted features are small in difference, resulting in poor modeling effect, difficulty in migration application, insufficient utilization of global-local features, and lack of perception ability of difficult samples are solved.

[0134] Next, a remote sensing image building damage assessment device combining global and local clues proposed in an embodiment of the present application is described with reference to the accompanying drawings.

[0135] Fig.13 It is a block diagram of a remote sensing image building damage assessment device combining global and local cues provided according to an embodiment of the present application.

[0136] like Fig.13As shown, the remote sensing image building damage assessment device 10 combining global and local clues includes: an acquisition module 100, an encoding module 200, a difference enhancement fusion module 300, a global and local clue fusion module 400 and an assessment module 500.

[0137] Among them, the acquisition module 100 is used to obtain the pre-disaster initial dual-phase images and the post-disaster initial dual-phase images of the buildings in the target area, and obtain the pre-disaster dual-phase images and the post-disaster dual-phase images that meet the preset pairing conditions based on the pre-disaster initial dual-phase images and the post-disaster initial dual-phase images.

[0138] The encoding module 200 is used to perform weight-sharing encoding on the pre-disaster dual-phase images and the post-disaster dual-phase images to obtain pre-disaster multi-scale dual-phase features of the pre-disaster dual-phase images and post-disaster multi-scale dual-phase features of the post-disaster dual-phase images.

[0139] The difference enhancement fusion module 300 is used to obtain multi-scale difference enhancement fusion features between the pre-disaster dual-phase image and the post-disaster dual-phase image based on the pre-disaster multi-scale dual-phase features and the post-disaster multi-scale dual-phase features.

[0140] The global and local clue fusion module 400 is used to connect and fuse the multi-scale difference enhancement fusion features based on the global and local clues between the pre-disaster dual-phase images and the post-disaster dual-phase images to obtain the output features between the pre-disaster dual-phase images and the post-disaster dual-phase images.

[0141] The evaluation module 500 is used to predict the damage assessment result and the error prediction result of the building based on the output characteristics, so as to obtain the final damage assessment result of the building according to the damage assessment result and the error prediction result.

[0142] Optionally, in one embodiment of the present application, the difference enhancement fusion module 300 includes: a difference unit, a channel attention unit, a difference enhancement channel attention unit, a channel difference enhancement unit, a channel difference unit, a spatial attention unit, a spatial difference enhancement unit and a difference enhancement fusion unit.

[0143] Among them, the difference unit is used to obtain the difference characteristics between the pre-disaster multi-scale bi-temporal characteristics and the post-disaster multi-scale bi-temporal characteristics based on the pre-disaster multi-scale bi-temporal characteristics and the post-disaster multi-scale bi-temporal characteristics.

[0144] The channel attention unit is used to obtain differential channel attention features of differential features based on channel attention, pre-disaster channel attention features of pre-disaster multi-scale dual-temporal features, and post-disaster channel attention features of post-disaster multi-scale dual-temporal features.

[0145] The differential enhanced channel attention unit is used to obtain initial pre-disaster differential enhanced channel attention features and initial post-disaster differential enhanced channel attention features based on the differential channel attention features, the pre-disaster channel attention features and the post-disaster channel attention features.

[0146] The channel differential enhancement unit is used to obtain pre-disaster channel differential enhancement features and post-disaster channel differential enhancement features based on the initial pre-disaster differential enhancement channel attention features and the initial post-disaster differential enhancement channel attention features.

[0147] The channel difference unit is used to obtain the channel difference feature between the pre-disaster channel difference enhancement feature and the post-disaster channel difference enhancement feature based on the pre-disaster channel difference enhancement feature and the post-disaster channel difference enhancement feature.

[0148] The spatial attention unit is used to use spatial attention to obtain differential spatial attention features of channel differential features, pre-disaster spatial attention features of pre-disaster multi-scale dual-temporal features, and post-disaster spatial attention features of post-disaster multi-scale dual-temporal features.

[0149] The spatial difference enhancement unit is used to obtain the pre-disaster spatial difference enhancement feature and the post-disaster spatial difference enhancement feature based on the differential spatial attention feature, the pre-disaster spatial attention feature and the post-disaster spatial attention feature.

[0150] The difference enhancement fusion unit is used to obtain multi-scale difference enhancement fusion features based on the pre-disaster spatial difference enhancement features and the post-disaster spatial difference enhancement features.

[0151] Optionally, in one embodiment of the present application, the global and local clue fusion module 400 includes: a fusion unit, a first generation unit and a second generation unit.

[0152] Among them, the fusion unit is used to connect and fuse the difference enhancement fusion features of the first scale in the multi-scale difference enhancement fusion features based on the global and local clues between the pre-disaster dual-phase images and the post-disaster dual-phase images to obtain the scale intermediate features of the first scale.

[0153] The first generating unit is used to obtain a scale output feature of a first scale based on the scale intermediate feature.

[0154] The second generating unit is used to obtain the output feature based on the scale output feature, the difference enhancement fusion feature of the first scale and the difference enhancement fusion feature of the second scale.

[0155] Optionally, in one embodiment of the present application, the evaluation module 500 includes: a first evaluation unit and a second evaluation unit.

[0156] Among them, the first evaluation unit is used to connect and fuse the third scale difference enhancement fusion feature, the fourth scale difference enhancement fusion feature and the fifth scale difference enhancement fusion feature of the multi-scale difference enhancement fusion feature through a jump connection based on the output features to obtain a damage assessment result and an error prediction result.

[0157] The second evaluation unit is used to generate a final loss assessment result based on the damage assessment result and the error prediction result.

[0158] Optionally, in one embodiment of the present application, the first evaluation unit includes: a first fusion decoding subunit and a second fusion decoding subunit.

[0159] Among them, the first fusion decoding subunit is used to perform middle-layer inter-layer fusion decoding based on the difference enhancement fusion features of the fourth scale and the difference enhancement fusion features of the fifth scale to obtain the middle-layer fusion decoding features of the fourth scale and the middle-layer fusion decoding features of the fifth scale.

[0160] The second fusion decoding subunit is used to perform shallow inter-layer fusion decoding based on the difference enhancement fusion features of the third scale, the middle-level fusion decoding features of the fourth scale, and the middle-level fusion decoding features of the fifth scale to obtain damage assessment results and error prediction results.

[0161] It should be noted that the aforementioned explanation of the embodiment of the method for assessing building damage using remote sensing images combining global and local clues is also applicable to the device for assessing building damage using remote sensing images combining global and local clues of this embodiment, and will not be repeated here.

[0162] According to the remote sensing image building damage assessment device that combines global and local clues proposed in the embodiment of the present application, firstly, weight-sharing encoding is performed on the pre-disaster and post-disaster dual-phase images to obtain multi-scale dual-phase features; secondly, the multi-scale dual-phase features are differentially enhanced and fused to obtain multi-scale differentially enhanced fused features; again, connection fusion is performed based on global and local clues to obtain output features; finally, the output features are used to output auxiliary damage assessment results and error prediction results, and then the final damage assessment results are output. The embodiment of the present application enhances the difference between dual-phase features, has a good effect in suppressing change noise, performs well in improving the semantic information extraction ability of the network, and can improve the judgment ability of difficult samples. Therefore, the problems in the related technology that the extracted features are small in difference, resulting in poor modeling effect, difficulty in migration application, insufficient utilization of global-local features, and lack of perception ability of difficult samples are solved.

[0163] Fig.14 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. The electronic device may include:

[0164] A memory 1401 , a processor 1402 , and a computer program stored in the memory 1401 and executable on the processor 1402 .

[0165] When the processor 1402 executes the program, the remote sensing image building damage assessment method combining global and local clues provided in the above embodiment is implemented.

[0166] Furthermore, the electronic device further comprises:

[0167] The communication interface 1403 is used for communication between the memory 1401 and the processor 1402 .

[0168] The memory 1401 is used to store computer programs that can be executed on the processor 1402 .

[0169] The memory 1401 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0170] If the memory 1401, the processor 1402 and the communication interface 1403 are implemented independently, the communication interface 1403, the memory 1401 and the processor 1402 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.14 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0171] Optionally, in a specific implementation, if the memory 1401, the processor 1402 and the communication interface 1403 are integrated on a chip, the memory 1401, the processor 1402 and the communication interface 1403 can communicate with each other through an internal interface.

[0172] The processor 1402 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0173] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned remote sensing image building damage assessment method combining global and local cues.

[0174] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed, implements the above remote sensing image building damage assessment method combining global and local clues.

[0175] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0176] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0177] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0178] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.

[0179] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented by hardware, as in another embodiment, it can be implemented by any one or a combination of multiple of the following technologies known in the art: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0180] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0181] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0182] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A method for building damage assessment from remote sensing images combining global and local cues, characterized in that: The following steps are involved: Acquire a pre-disaster initial dual-phase image and a post-disaster initial dual-phase image of the building in the target area, and obtain a pre-disaster dual-phase image and a post-disaster dual-phase image that meet a preset pairing condition based on the pre-disaster initial dual-phase image and the post-disaster initial dual-phase image; Performing weight-sharing encoding on the pre-disaster dual-phase image and the post-disaster dual-phase image to obtain pre-disaster multi-scale dual-phase features of the pre-disaster dual-phase image and post-disaster multi-scale dual-phase features of the post-disaster dual-phase image; Based on the pre-disaster multi-scale dual-phase features and the post-disaster multi-scale dual-phase features, a multi-scale difference enhancement fusion feature between the pre-disaster dual-phase image and the post-disaster dual-phase image is obtained; Based on the global and local clues between the pre-disaster dual-phase image and the post-disaster dual-phase image, connecting and fusing the multi-scale difference enhancement fusion features to obtain output features between the pre-disaster dual-phase image and the post-disaster dual-phase image; A damage assessment result and an error prediction result of the building are predicted based on the output characteristics, so as to obtain a final damage assessment result of the building according to the damage assessment result and the error prediction result.

2. The method according to claim 1, characterized in that The method of obtaining the multi-scale difference enhancement fusion features between the pre-disaster dual-phase image and the post-disaster dual-phase image based on the pre-disaster multi-scale dual-phase features and the post-disaster multi-scale dual-phase features includes: Obtaining differential features between the pre-disaster multi-scale bi-temporal features and the post-disaster multi-scale bi-temporal features based on the pre-disaster multi-scale bi-temporal features and the post-disaster multi-scale bi-temporal features; Based on the channel attention, a differential channel attention feature of the differential feature, a pre-disaster channel attention feature of the pre-disaster multi-scale dual-temporal feature, and a post-disaster channel attention feature of the post-disaster multi-scale dual-temporal feature are obtained; Obtaining an initial pre-disaster differential enhanced channel attention feature and an initial post-disaster differential enhanced channel attention feature based on the differential channel attention feature, the pre-disaster channel attention feature and the post-disaster channel attention feature; Obtaining a pre-disaster channel differential enhancement feature and a post-disaster channel differential enhancement feature based on the initial pre-disaster differential enhancement channel attention feature and the initial post-disaster differential enhancement channel attention feature; Obtaining a channel differential feature between the pre-disaster channel differential enhancement feature and the post-disaster channel differential enhancement feature based on the pre-disaster channel differential enhancement feature and the post-disaster channel differential enhancement feature; Using spatial attention, obtain the differential spatial attention features of the channel differential features, the pre-disaster spatial attention features of the pre-disaster multi-scale dual-temporal features, and the post-disaster spatial attention features of the post-disaster multi-scale dual-temporal features; Obtaining a pre-disaster spatial differential enhancement feature and a post-disaster spatial differential enhancement feature based on the differential spatial attention feature, the pre-disaster spatial attention feature, and the post-disaster spatial attention feature; The multi-scale difference enhancement fusion feature is obtained based on the pre-disaster spatial difference enhancement feature and the post-disaster spatial difference enhancement feature.

3. The method according to claim 1, characterized in that: The connecting and fusing the multi-scale difference enhancement fusion features based on the global and local clues between the pre-disaster dual-phase image and the post-disaster dual-phase image to obtain the output features between the pre-disaster dual-phase image and the post-disaster dual-phase image, includes: Based on the global and local clues between the pre-disaster dual-temporal image and the post-disaster dual-temporal image, connecting and fusing the difference enhancement fusion features of the first scale in the multi-scale difference enhancement fusion features to obtain the scale intermediate features of the first scale; Obtaining a scale output feature of the first scale based on the scale intermediate feature; The output feature is obtained based on the scale output feature, the difference enhancement fusion feature of the first scale and the difference enhancement fusion feature of the second scale.

4. The method according to claim 1, characterized in that: The predicting the damage assessment result and the error prediction result of the building based on the output feature to obtain the final damage assessment result of the building according to the damage assessment result and the error prediction result includes: Based on the output features, connecting and fusing the difference enhancement fusion features of the third scale, the difference enhancement fusion features of the fourth scale, and the difference enhancement fusion features of the fifth scale of the multi-scale difference enhancement fusion features through a jump connection method to obtain the damage assessment result and the error prediction result; The final loss assessment result is generated based on the damage assessment result and the error prediction result.

5. The method according to claim 4, characterized in that The method of connecting and fusing the third scale difference enhancement fusion feature, the fourth scale difference enhancement fusion feature and the fifth scale difference enhancement fusion feature of the multi-scale difference enhancement fusion feature by a jump connection based on the output feature to obtain the damage assessment result and the error prediction result includes: Performing middle-layer inter-layer fusion decoding based on the difference enhancement fusion feature of the fourth scale and the difference enhancement fusion feature of the fifth scale to obtain the middle-layer fusion decoding feature of the fourth scale and the middle-layer fusion decoding feature of the fifth scale; Shallow inter-layer fusion decoding is performed based on the difference enhancement fusion features of the third scale, the middle-layer fusion decoding features of the fourth scale, and the middle-layer fusion decoding features of the fifth scale to obtain the damage assessment result and the error prediction result.

6. A remote sensing image building damage assessment device combining global and local clues, characterized in that: include: An acquisition module is used to acquire a pre-disaster initial dual-phase image and a post-disaster initial dual-phase image of the buildings in the target area, and obtain a pre-disaster dual-phase image and a post-disaster dual-phase image that meet a preset pairing condition based on the pre-disaster initial dual-phase image and the post-disaster initial dual-phase image; An encoding module, used for performing weight-sharing encoding on the pre-disaster dual-phase image and the post-disaster dual-phase image to obtain pre-disaster multi-scale dual-phase features of the pre-disaster dual-phase image and post-disaster multi-scale dual-phase features of the post-disaster dual-phase image; A difference enhancement fusion module, used for obtaining multi-scale difference enhancement fusion features between the pre-disaster dual-phase image and the post-disaster dual-phase image based on the pre-disaster multi-scale dual-phase features and the post-disaster multi-scale dual-phase features; A global and local clue fusion module, used for connecting and fusing the multi-scale difference enhancement fusion features based on the global and local clues between the pre-disaster dual-phase image and the post-disaster dual-phase image, so as to obtain the output features between the pre-disaster dual-phase image and the post-disaster dual-phase image; An evaluation module is used to predict the damage assessment result and the error prediction result of the building based on the output characteristics, so as to obtain the final damage assessment result of the building according to the damage assessment result and the error prediction result.

7. The device according to claim 6, characterized in that The difference enhancement fusion module comprises: A difference unit, configured to obtain a difference feature between the pre-disaster multi-scale bi-temporal feature and the post-disaster multi-scale bi-temporal feature based on the pre-disaster multi-scale bi-temporal feature and the post-disaster multi-scale bi-temporal feature; A channel attention unit, used for obtaining, based on channel attention, a differential channel attention feature of the differential feature, a pre-disaster channel attention feature of the pre-disaster multi-scale dual-temporal feature, and a post-disaster channel attention feature of the post-disaster multi-scale dual-temporal feature; A differentially enhanced channel attention unit, used to obtain an initial pre-disaster differentially enhanced channel attention feature and an initial post-disaster differentially enhanced channel attention feature based on the differential channel attention feature, the pre-disaster channel attention feature and the post-disaster channel attention feature; A channel differential enhancement unit, configured to obtain a pre-disaster channel differential enhancement feature and a post-disaster channel differential enhancement feature based on the initial pre-disaster differential enhancement channel attention feature and the initial post-disaster differential enhancement channel attention feature; A channel difference unit, configured to obtain a channel difference feature between the pre-disaster channel difference enhancement feature and the post-disaster channel difference enhancement feature based on the pre-disaster channel difference enhancement feature and the post-disaster channel difference enhancement feature; A spatial attention unit, configured to obtain, by using spatial attention, a differential spatial attention feature of the channel differential feature, a pre-disaster spatial attention feature of the pre-disaster multi-scale dual-temporal feature, and a post-disaster spatial attention feature of the post-disaster multi-scale dual-temporal feature; A spatial difference enhancement unit, configured to obtain a pre-disaster spatial difference enhancement feature and a post-disaster spatial difference enhancement feature based on the differential spatial attention feature, the pre-disaster spatial attention feature, and the post-disaster spatial attention feature; The difference enhancement fusion unit is used to obtain the multi-scale difference enhancement fusion feature based on the pre-disaster spatial difference enhancement feature and the post-disaster spatial difference enhancement feature.

8. The device according to claim 6, characterized in that The global and local clue fusion module includes: A fusion unit, configured to connect and fuse the difference enhancement fusion features of the first scale in the multi-scale difference enhancement fusion features based on global and local clues between the pre-disaster dual-temporal image and the post-disaster dual-temporal image to obtain the scale intermediate features of the first scale; A first generating unit, configured to obtain a scale output feature of the first scale based on the scale intermediate feature; The second generating unit is used to obtain the output feature based on the scale output feature, the difference enhancement fusion feature of the first scale and the difference enhancement fusion feature of the second scale.

9. The device according to claim 6, characterized in that The evaluation module comprises: A first evaluation unit is configured to connect and fuse the difference enhancement fusion feature of the third scale, the difference enhancement fusion feature of the fourth scale, and the difference enhancement fusion feature of the fifth scale of the multi-scale difference enhancement fusion feature through a jump connection based on the output feature, so as to obtain the damage assessment result and the error prediction result; The second evaluation unit is used to generate the final loss assessment result based on the damage assessment result and the error prediction result.

10. The device according to claim 9, characterized in that The first evaluation unit comprises: A first fusion decoding subunit is used to perform middle-layer inter-layer fusion decoding based on the difference enhancement fusion feature of the fourth scale and the difference enhancement fusion feature of the fifth scale to obtain the middle-layer fusion decoding feature of the fourth scale and the middle-layer fusion decoding feature of the fifth scale; The second fusion decoding subunit is used to perform shallow inter-layer fusion decoding based on the difference enhancement fusion feature of the third scale, the middle-level fusion decoding feature of the fourth scale, and the middle-level fusion decoding feature of the fifth scale to obtain the damage assessment result and the error prediction result.

11. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the remote sensing image building damage assessment method combining global and local clues as described in any one of claims 1 to 5.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the remote sensing image building damage assessment method combining global and local clues as described in any one of claims 1 to 5.

13. A computer program product, characterized in that It comprises a computer program, which, when executed, is used to implement the remote sensing image building damage assessment method combining global and local clues as described in any one of claims 1-5.

Images