Flooded infrastructure change detection method based on feature fusion deep supervision network

Abstract

The invention provides a flooded infrastructure change detection method based on a feature fusion deep supervision network, and the method comprises the steps: obtaining a pre-disaster image and a post-disaster image; inputting the pre-disaster image into a pre-disaster encoder to obtain a plurality of pre-disaster image features of different scales; inputting the post-disaster image into a post-disaster encoder to obtain a plurality of post-disaster image features of different scales; inputting the plurality of pre-disaster image features of different scales and the plurality of post-disaster image features of different scales into a multi-attention constrained deep supervision multi-scale feature fusion module to obtain a plurality of pre-disaster channel space correction feature maps of different scales and a plurality of post-disaster channel space correction feature maps of different scales; inputting the pre-disaster channel space correction feature map and the post-disaster channel space correction feature map into a decoder to obtain a change detection map; according to the invention, the change monitoring capability of the flooded building road in the flood complex scene can be improved.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a method for detecting changes in flooded infrastructure based on a feature fusion deep supervision network. Background Art

[0002] To mitigate the immediate impact of flooding, first responders need to respond quickly and assess damage when or soon after a flood occurs. The most important of these is assessing damage to buildings and roads to determine which buildings have been damaged by flooding and which roads have been blocked by flooding.

[0003] At present, how to identify flooded buildings and flooded roads, in actual business applications, the solution is to directly overlay land cover data using post-disaster flood vector maps. On the one hand, the failure to update land cover data in a timely manner leads to misjudgment of disaster losses; on the other hand, the above method cannot detect flooded buildings because floods block buildings.

[0004] This shows that the flood disaster assessment method in the relevant technology has the technical problem of low accuracy in monitoring the changes of flooded buildings and flooded roads. Summary of the invention

[0005] The present invention provides a method for detecting changes in flooded infrastructure based on a feature fusion deep supervision network, which is used to solve the defect of low accuracy in monitoring changes in flooded buildings and roads in the prior art flood disaster assessment method, and improve the ability to monitor changes in flooded buildings and roads in complex flood scenes.

[0006] The present invention provides a method for detecting changes in flooded infrastructure based on a feature fusion deep supervision network, comprising the following steps. Obtaining pre-disaster images and post-disaster images, wherein the pre-disaster images are used to represent the images of the target object before it suffers from the flood disaster, and the post-disaster images are used to represent the images of the target object after it suffers from the flood disaster; inputting the pre-disaster images into a pre-disaster encoder to obtain pre-disaster image features of multiple different scales output by the pre-disaster encoder; inputting the post-disaster images into a post-disaster encoder to obtain post-disaster image features of multiple different scales output by the post-disaster encoder; inputting the pre-disaster image features of multiple different scales into a multi-attention constrained deep supervision multi-scale feature fusion module to obtain pre-disaster channel spaces of multiple different scales output by the multi-attention constrained deep supervision multi-scale feature fusion module. Correction feature map; input the multiple post-disaster image features of different scales into the deep supervision multi-scale feature fusion module of the multi-attention constraints, and obtain multiple post-disaster channel space correction feature maps of different scales output by the deep supervision multi-scale feature fusion module of the multi-attention constraints; wherein the multi-attention constraints include a channel attention mechanism and a spatial attention mechanism; input the pre-disaster channel space correction feature map and the post-disaster channel space correction feature map into a decoder, and obtain a change detection map output by the decoder, wherein the pre-disaster channel space correction feature map and the pre-disaster channel space correction feature map have the same scale, and the change detection map is used to represent the change monitoring comparison result of the target object.

[0007] According to a method for detecting changes in flooded infrastructure based on a feature fusion deep supervision network provided by the present invention, the pre-disaster encoder comprises: a first pre-disaster encoder, a second pre-disaster encoder, a third pre-disaster encoder and a fourth pre-disaster encoder, and the pre-disaster image is input into the pre-disaster encoder to obtain a plurality of pre-disaster image features of different scales output by the pre-disaster encoder, comprising: inputting the pre-disaster image into the first pre-disaster encoder to obtain a first pre-disaster image feature output by the first pre-disaster encoder, the scale of the first pre-disaster image feature being one quarter of the scale of the pre-disaster image, and the number of channels of the first pre-disaster image feature being 1; inputting the first pre-disaster image feature into the second pre-disaster encoder to obtain the second pre-disaster image feature. The second pre-disaster image feature output by the first encoder has a scale of one eighth of the pre-disaster image, and the number of channels of the second pre-disaster image feature is 2; the second pre-disaster image feature is input into the third pre-disaster encoder to obtain the third pre-disaster image feature output by the third pre-disaster encoder, the scale of the third pre-disaster image feature is one sixteenth of the pre-disaster image, and the number of channels of the third pre-disaster image feature is 4; the third pre-disaster image feature is input into the fourth pre-disaster encoder to obtain the fourth pre-disaster image feature output by the fourth pre-disaster encoder, the scale of the fourth pre-disaster image feature is one thirty-second of the pre-disaster image, and the number of channels of the fourth pre-disaster image feature is 8.

[0008] According to a method for detecting changes in flooded infrastructure based on a feature fusion deep supervision network provided by the present invention, the number of channels of the pre-disaster encoder is 192, 384, 768, and 1536 respectively; the number of modules of the pre-disaster encoder is 3, 3, 9, and 3 respectively; the structure of the pre-disaster encoder includes: deep convolution, layer normalization, a first point-by-point convolution, a Gaussian error linear unit, and a second point-by-point convolution.

[0009] According to a method for detecting changes in flooded infrastructure based on a feature fusion deep supervision network provided by the present invention, the pre-disaster image features of the plurality of different scales are input into a multi-attention constrained deep supervision multi-scale feature fusion module to obtain a plurality of pre-disaster channel space correction feature maps of different scales output by the multi-attention constrained deep supervision multi-scale feature fusion module, including: taking each of the plurality of different scales as a target scale and performing the following operations to obtain a pre-disaster channel space correction feature map of the target scale: up-sampling the pre-disaster image features of the plurality of different scales to obtain pre-disaster image features of target scales with a plurality of different channel numbers; .... The pre-disaster image features of the target scale are channel stitched to obtain a target scale feature map; the target scale feature map is input into a channel attention module to obtain a channel attention feature map output by the channel attention module; the channel attention feature map is multiplied by the target scale feature map to obtain a channel correction feature map; the target scale feature map is input into a spatial attention module to obtain a spatial attention feature map output by the spatial attention module; the spatial attention feature map is multiplied by the target scale feature map to obtain a spatial correction feature map; the channel correction feature map is multiplied by the spatial correction feature map to obtain a pre-disaster channel spatial correction feature map of the target scale.

[0010] According to a method for detecting changes in flooded infrastructure based on a feature fusion deep supervision network provided by the present invention, the decoder includes a first decoder, a second decoder, a third decoder and a fourth decoder, and the pre-disaster channel space correction feature map and the post-disaster channel space correction feature map are input into the decoder to obtain a change detection map output by the decoder, including: channel splicing of the pre-disaster channel space correction feature map of a fourth scale and the post-disaster channel space correction feature map of a fourth scale to obtain a fourth spliced ​​feature map; inputting the fourth spliced ​​feature map into a fourth decoder to obtain a third-scale feature map output by the fourth decoder; and splicing the pre-disaster channel space correction feature map of the third scale and the post-disaster channel space correction feature map of the third scale and the third-scale feature map of the third scale. The image is channel-stitched to obtain a third spliced ​​feature map; the third spliced ​​feature map is input into a third decoder to obtain a second-scale feature map output by the third decoder; the second-scale pre-disaster channel space correction feature map, the second-scale post-disaster channel space correction feature map and the second-scale feature map are channel-stitched to obtain a second spliced ​​feature map; the second spliced ​​feature map is input into a second decoder to obtain a first-scale feature map output by the second decoder; the first-scale pre-disaster channel space correction feature map, the first-scale post-disaster channel space correction feature map and the first-scale feature map are channel-stitched to obtain a first spliced ​​feature map; the first spliced ​​feature map is input into a first decoder to obtain a change detection map output by the first decoder.

[0011] According to a method for detecting changes in flooded infrastructure based on a feature fusion deep supervision network provided by the present invention, the structure of the decoder includes: convolution, first batch normalization, first rectified linear unit, inverse convolution, second batch normalization and second rectified linear unit. The present invention also provides a device for detecting changes in flooded infrastructure based on a feature fusion deep supervision network, including the following modules: an acquisition module, used to acquire pre-disaster images and post-disaster images, wherein the pre-disaster images are used to represent the images of the target object before suffering from the flood disaster, and the post-disaster images are used to represent the images of the target object after suffering from the flood disaster; a pre-disaster encoding module, used to input the pre-disaster images into a pre-disaster encoder to obtain pre-disaster image features of multiple different scales output by the pre-disaster encoder; a post-disaster encoding module, used to input the post-disaster images into a post-disaster encoder to obtain post-disaster image features of multiple different scales output by the post-disaster encoder; a multi-scale fusion module, used to input the pre-disaster image features of multiple different scales into a multi-attention constrained deep supervision multi-scale feature fusion module to obtain the multi-attention constrained A deep supervised multi-scale feature fusion module outputs multiple pre-disaster channel spatial correction feature maps of different scales; the multi-scale fusion module is also used to input the multiple post-disaster image features of different scales into the deep supervised multi-scale feature fusion module with multiple attention constraints, to obtain multiple post-disaster channel spatial correction feature maps of different scales output by the deep supervised multi-scale feature fusion module with multiple attention constraints; wherein the multi-attention constraints include a channel attention mechanism and a spatial attention mechanism; a decoding module is used to input the pre-disaster channel spatial correction feature map and the post-disaster channel spatial correction feature map into a decoder, to obtain a change detection map output by the decoder, wherein the pre-disaster channel spatial correction feature map and the pre-disaster channel spatial correction feature map have the same scale, and the change detection map is used to represent the change monitoring comparison result of the target object.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for detecting changes in flooded infrastructure based on a feature fusion deep supervision network as described above is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described methods for detecting changes in flooded infrastructure based on a feature fusion deep supervision network.

[0014] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-mentioned methods for detecting changes in flooded infrastructure based on a feature fusion deep supervision network.

[0015] The method for detecting changes in flooded infrastructure based on a feature fusion deep supervision network provided by the present invention can extract pre-disaster and post-disaster image features of multiple scales by acquiring pre-disaster images of the target object before it suffers from a flood disaster and post-disaster images after it suffers from a flood disaster, and inputting the pre-disaster images and post-disaster images into a pre-disaster encoder and a post-disaster encoder respectively. The multi-scale feature extraction can enhance the model's recognition ability for target objects of different sizes and shapes, and improve the accuracy of change detection. A deep-supervised multi-scale feature fusion module with multiple attention constraints is used to fuse pre-disaster image features of different scales and post-disaster image features of different scales respectively, and a deep supervision strategy is used to ensure that features at each level are effectively utilized, which helps to capture change information at different scales. , improve the robustness and generalization ability of the model; introduce channel attention mechanism and spatial attention mechanism, respectively focus on important channels and spatial positions in the image, so as to enhance the ability to extract key information; input the channel space correction feature map before the disaster and the channel space correction feature map after the disaster into the decoder, and obtain the change detection map. Due to the joint effect of multi-scale feature extraction, multi-attention constraint and deep supervision feature fusion in the previous steps, the change detection map can accurately reflect the changes of the target object before and after the flood disaster, and improve the change monitoring ability of flooded buildings and roads in complex flood scenes, thereby solving the technical problem of low accuracy in monitoring the changes of flooded buildings and roads in the flood disaster assessment method in related technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a flow chart of a method for detecting changes in flooded infrastructure based on a feature fusion deep supervision network provided by the present invention.

[0018] Figure 2 It is a simplified change detection structure diagram of the flooded infrastructure change detection method provided by the present invention.

[0019] Figure 3 It is an overall network structure diagram of the flooded infrastructure change detection method based on feature fusion deep supervision network provided by the present invention.

[0020] Figure 4It is a detailed network structure diagram of the flooded infrastructure change detection method based on feature fusion deep supervision network provided by the present invention.

[0021] Figure 5 This is a diagram of a multi-attention constrained deep-supervised high-dimensional-low-dimensional multi-scale feature fusion module provided by the present invention.

[0022] Figure 6 This is a comparison diagram of monitoring results of flooded building roads with different common change detection structures provided by the present invention.

[0023] Figure 7 This is a comparison diagram of ablation experiment results provided by the present invention.

[0024] Figure 8 This is a comparison chart of the monitoring results of flooded building roads using the MSFF provided by the present invention embedded in different deep learning models.

[0025] Fig. 9 It is a structural schematic diagram of a flooded infrastructure change detection device based on a feature fusion deep supervision network provided by the present invention.

[0026] Fig.10 It is a schematic diagram of the physical structure of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] Floods are one of the major natural disasters. With global climate change, flood disasters are characterized by frequent, sudden, and significant chain effects. Floods cause huge losses of life, infrastructure damage, and property losses; as well as a series of other environmental problems such as post-disaster cleanup and reconstruction, and health care. In order to mitigate the direct impact of floods, emergency personnel need to respond quickly and assess the damage when or shortly after the flood. The most important of these is to assess the damage to buildings and roads to determine which places need help and which roads are blocked by floods. Remote sensing images are the main tool for quickly mapping flood-prone areas, but detailed emergency monitoring and damage assessment of the affected areas take a lot of time. Most of the flooded buildings and roads in floods are still in normal condition. Buildings block floods, which makes conventional flood detection methods unable to detect flooded buildings. The slender shape of the roads themselves also affects the decline of model extraction capabilities. Therefore, it is of great significance to use deep learning models to quickly and accurately quantify the affected buildings and roads.

[0029] At present, how to judge flooded buildings and roads, in actual business applications, the solution is to directly superimpose land cover data using post-disaster flood vector maps. On the one hand, the failure to update land cover data in a timely manner leads to misjudgment of disaster losses; on the other hand, the above method cannot detect flooded buildings because buildings block floods. Dual-temporal image change detection technology can accurately determine the damage status of infrastructure before and after floods. By analyzing and comparing geographic information, remote sensing images and other data from different periods, remote sensing image change detection plays an important role in geographic data updating, disaster damage assessment and other aspects. Compared with the change detection of buildings and roads in normal state, the change detection of damaged infrastructure is more challenging. Because there are additional interference features in disaster scenes: such as non-building structures damaged in earthquake disasters, the detection of damaged infrastructure in earthquake disasters has been widely studied; such as the mixture of floods of various landforms in flood disasters. However, the change detection of flooded buildings and roads is a difficult task. Unlike the complete destruction and severe collapse caused by earthquakes, most of the flooded buildings and roads are still in normal condition when viewed from the sky in flood disasters. Therefore, there is an urgent need for feature extractors with stronger context-awareness and change detection structures that can fully exploit the bi-temporal semantic information before and after floods. With the introduction of the ultra-high-resolution flooded building and road dataset xBD, change detection structures have been initially applied to the detection of flooded buildings and roads in flood scenarios. However, different change detection structures have different abilities in describing the changes in flooded buildings and roads, and it is still unclear which change detection structure has the strongest ability.

[0030] Deep learning models are very good at discovering high-dimensional structures in complex data. They use convolutional structures to capture spatial contextual information and play an important role in the field of emergency monitoring and damage assessment of disasters. The convolutional neural network was pioneered to apply to image semantic segmentation, and innovations were made in the encoding-decoding structure to improve the upsampling effect by saving the pooling index. In subsequent studies, multi-scale contextual information was effectively integrated through U-shaped structures, pyramid pooling modules, hollow convolutions, multi-branch parallel convolutions, etc., enhancing the model's ability to understand complex scenes. However, when the above research fully considers the use of scale-related contextual information, there are still some limitations on how to reduce complexity and generate multi-scale features with more powerful capabilities. A very deep network is constructed through a residual structure, and the width, depth and resolution of the network are balanced through NAS search technology. It is introduced from the natural language field to the semantic segmentation field, and the sequence is modeled based on the attention mechanism. The design ideas of Transformer are borrowed to fully implement the former module with a deep learning model (CNN, Convolutional Neural Network). The design ideas of these encoders significantly enhance the feature extraction capabilities of the network structure.

[0031] Multi-scale feature fusion and layer feature fusion strategies are designed to achieve high-dimensional features and low-dimensional features are fused with each other, and a large number of channels are introduced in the fusion strategy. There is significant spatial similarity and channel redundancy in multi-channel convolution kernels. In order to alleviate channel redundancy, channel attention mechanisms (e.g., SE (Squeeze-and-Excitation, ECA (Efficient Channel Attention)) focus on the attention of the channel dimension and assign appropriate weights to feature channels at different levels. Since convolution kernels produce similar responses to similar patterns in the local space, CBAM (Convolutional Block Attention Module) and scSE (Concurrent Spatial and Channel Squeeze & Excitation) consider the attention of both spatial and channel dimensions at the same time, further solving the problem of spatial irrelevance. Some studies embed the attention mechanism in the encoder or decoder in the network structure, ignoring the independence of high-dimensional-low-dimensional multi-scale features, and the use of multi-scale features in change detection tasks is relatively limited. Dynamic multi-scale feature fusion or multi-attention constraint methods are designed to learn fusion weights from features and guide the organic fusion of shallow and deep semantic features. These multi-scale feature fusions based on the attention mechanism rely on specific deep learning models and cannot be directly applied in deep learning models with different structures.

[0032] In summary, in the emergency monitoring and disaster assessment tasks of flood disasters, professional researchers in the current business application field use flood vector maps to superimpose land cover data. Since the land cover data is not updated in time or the buildings block the flood, the monitoring accuracy of the changes in flooded building roads is not high. The present invention designs and uses the dual-phase image change detection structure FloodCD before and after the flood. Different feature extractors and network structures are combined together, and their ability to detect changes in flooded building roads in ultra-high resolution remote sensing images is tested. For the different capabilities of deep learning models in complex flood scenes, the present invention explores the effectiveness of different network structures in detecting changes in flooded building roads. Faced with the problem that the high-dimensional and low-dimensional feature maps of different sizes extracted by the deep learning model at each stage are not effectively fused and utilized, the present invention constructs a plug-and-play multi-attention constrained multi-scale feature extraction module, and uses a deep supervised learning strategy to make full use of the global-local multi-scale features of the flood scene to improve the accuracy of change detection. The flooded infrastructure change detection method based on feature fusion deep supervision network proposed in the present invention improves the change monitoring ability of flooded building roads in complex flood scenes, and confirms its excellence in flood scenes.

[0033] Optionally, the method for detecting changes in flooded infrastructure based on a feature fusion deep supervision network in an embodiment of the present invention may be executed by a server, or by a terminal device, or jointly by a server and a terminal device, taking the example of the method for detecting changes in flooded infrastructure based on a feature fusion deep supervision network in this embodiment being executed by a server.

[0034] Figure 1 is a flow chart of a flooded infrastructure change detection method based on a feature fusion deep supervision network provided by the present invention, such as Figure 1 As shown, the method includes the following steps.

[0035] Step 101, obtaining pre-disaster images and post-disaster images.

[0036] Among them, the pre-disaster image is used to represent the image of the target object before it suffers from the flood disaster, and the post-disaster image is used to represent the image of the target object after it suffers from the flood disaster.

[0037] In the emergency monitoring and disaster assessment tasks of flood disasters, professional researchers in the business application field currently use flood vector maps to overlay land cover data. Since the land cover data is not updated in a timely manner or buildings block floods, the monitoring accuracy of changes in flooded building roads is not high.

[0038] In an embodiment of the present invention, a pre-disaster image of the target object before it suffers from a flood disaster and a post-disaster image after it suffers from a flood disaster are obtained. Here, the pre-disaster image and the post-disaster image can be ultra-high-resolution remote sensing images of the target object; the target object can be a flooded building or a flooded road, etc., a building object affected by the flood disaster.

[0039] Step 102 , inputting the pre-disaster image into a pre-disaster encoder, and obtaining a plurality of pre-disaster image features of different scales output by the pre-disaster encoder.

[0040] In the embodiment of the present invention, a pre-disaster encoder is used to encode the pre-disaster image, and multiple pre-disaster image features of different scales are extracted, so that different levels of information in the image can be captured.

[0041] Here, the pre-disaster encoder may include a first pre-disaster encoder, a second pre-disaster encoder, a third pre-disaster encoder and a fourth pre-disaster encoder, which are used to convert the input pre-disaster image into pre-disaster image features of different scales.

[0042] For example, pre-disaster image features at different scales include: pre-disaster image features at a quarter scale, pre-disaster image features at a one-eighth scale, pre-disaster image features at a one-sixteenth scale, and pre-disaster image features at a one-thirty-second scale.

[0043] It should be noted that the number of channels of pre-disaster image features at different scales is different. For example, the number of channels of pre-disaster image features at a quarter scale is 1, the number of channels of pre-disaster image features at a one-eighth scale is 2, the number of channels of pre-disaster image features at a one-sixteenth scale is 4, and the number of channels of pre-disaster image features at a one-thirty-second scale is 8.

[0044] Through the embodiments of the present invention, the pre-disaster encoder can extract pre-disaster image features of multiple different scales, covering different levels of information from global to local, which helps to understand the pre-disaster image content more comprehensively.

[0045] Step 103 , inputting the post-disaster image into a post-disaster encoder to obtain a plurality of post-disaster image features of different scales output by the post-disaster encoder.

[0046] In the embodiment of the present invention, a post-disaster encoder is used to encode the post-disaster image, and multiple post-disaster image features of different scales are extracted, so that different levels of information in the image can be captured.

[0047] Here, the post-disaster encoder may include a first post-disaster encoder, a second post-disaster encoder, a third post-disaster encoder and a fourth post-disaster encoder, which are used to convert the input post-disaster image into post-disaster image features of different scales.

[0048] For example, post-disaster image features of different scales include: post-disaster image features of a quarter scale, post-disaster image features of a one-eighth scale, post-disaster image features of a one-sixteenth scale, and post-disaster image features of a one-thirty-second scale.

[0049] It should be noted that the number of channels of post-disaster image features at different scales is different. For example, the number of channels of post-disaster image features at a quarter scale is 1, the number of channels of post-disaster image features at a one-eighth scale is 2, the number of channels of post-disaster image features at a one-sixteenth scale is 4, and the number of channels of post-disaster image features at a one-thirty-second scale is 8.

[0050] Through the embodiments of the present invention, the post-disaster encoder can extract post-disaster image features of multiple different scales, covering different levels of information from global to local, which is helpful for a more comprehensive understanding of the post-disaster image content.

[0051] Step 104 , inputting the pre-disaster image features of multiple different scales into a multi-attention constrained deep-supervised multi-scale feature fusion module, and obtaining the pre-disaster channel space correction feature maps of multiple different scales output by the multi-attention constrained deep-supervised multi-scale feature fusion module.

[0052] In an embodiment of the present invention, a multi-attention mechanism is introduced in the feature fusion process, including a channel attention mechanism and a spatial attention mechanism; deep supervision refers to exerting supervision at different scales of feature fusion to ensure that features at each scale can be fully utilized.

[0053] Here, Deep Supervision is used to add an auxiliary classifier in the target middle hidden layer of the deep neural network as a network branch to supervise the backbone network, so as to solve the problems of gradient disappearance and slow convergence in deep neural network training.

[0054] Under multi-attention constraints and deep supervision, multi-scale feature fusion is performed on pre-disaster image features of multiple different scales according to the target scale to obtain the pre-disaster channel space correction feature map of the target scale. The target scale is changed and multi-scale feature fusion is repeated until all different scales are traversed to obtain the pre-disaster channel space correction feature map of each scale in multiple different scales.

[0055] Through the embodiments of the present invention, after being processed by the multi-attention constrained deep-supervised multi-scale feature fusion module, a plurality of pre-disaster channel space correction feature maps of different scales are finally output, which not only contain the information in the original image (i.e., the pre-disaster image), but also have been enhanced by multi-scale, multi-attention mechanism and deep supervision, and have higher information density and accuracy.

[0056] Step 105, inputting the post-disaster image features of multiple different scales into the multi-attention constrained deep-supervised multi-scale feature fusion module, and obtaining the post-disaster channel space correction feature maps of multiple different scales output by the multi-attention constrained deep-supervised multi-scale feature fusion module.

[0057] Among them, the multi-attention constraints include channel attention mechanism and spatial attention mechanism; Under multi-attention constraints and deep supervision, according to the target scale, multi-scale feature fusion is performed on post-disaster image features of multiple different scales to obtain the post-disaster channel space correction feature map of the target scale. The target scale is changed and multi-scale feature fusion is repeated until all different scales are traversed to obtain the post-disaster channel space correction feature map of each scale in multiple different scales.

[0058] Through the embodiments of the present invention, after being processed by the multi-attention constrained deep-supervised multi-scale feature fusion module, a plurality of post-disaster channel space correction feature maps of different scales are finally output, which not only contain the information in the original image (i.e., the post-disaster image), but also have been enhanced by multi-scale, multi-attention mechanism and deep supervision, and have higher information density and accuracy.

[0059] Step 106, inputting the pre-disaster channel space correction feature map and the post-disaster channel space correction feature map into a decoder to obtain a change detection map output by the decoder.

[0060] Among them, the scale of the pre-disaster channel space correction feature map is the same as that of the pre-disaster channel space correction feature map, and the change detection map is used to represent the change monitoring comparison results of the target object.

[0061] In an embodiment of the present invention, the decoder includes a first decoder, a second decoder, a third decoder and a fourth decoder, each of which is used to decode the pre-disaster channel space correction feature map and the post-disaster channel space correction feature map of different scales.

[0062] In an embodiment of the present invention, the pre-disaster channel spatial correction feature map and the post-disaster channel spatial correction feature map are input into a decoder to generate a change detection map for displaying the comparison results of the changes of the target object before and after the flood disaster, providing an important basis for disaster assessment, emergency response and post-disaster reconstruction.

[0063] refer to Figure 2 , Figure 2 It is a simplified change detection structure diagram of the flooded infrastructure change detection method provided by the present invention.

[0064] like Figure 2As shown in the figure, the pre-disaster images and the post-disaster images are respectively input into the encoder to obtain pre-disaster image features of multiple scales and post-disaster image features of multiple scales; the pre-disaster image features of multiple scales and the post-disaster image features of multiple scales are input into the multi-attention constrained deep supervision multi-scale feature fusion module to obtain pre-disaster channel spatial correction feature maps of multiple scales and post-disaster channel spatial correction feature maps of multiple scales; the pre-disaster channel spatial correction feature maps of multiple scales and the post-disaster channel spatial correction feature maps of multiple scales are channel-joined and input into the decoder to obtain a change detection map.

[0065] Through the above steps of the embodiment of the present invention, pre-disaster images and post-disaster images are obtained, wherein the pre-disaster images are used to represent the images of the target object before suffering from the flood disaster, and the post-disaster images are used to represent the images of the target object after suffering from the flood disaster; the pre-disaster images are input into the pre-disaster encoder to obtain the pre-disaster image features of multiple different scales output by the pre-disaster encoder; the post-disaster images are input into the post-disaster encoder to obtain the post-disaster image features of multiple different scales output by the post-disaster encoder; the pre-disaster image features of multiple different scales are input into the multi-attention constrained deep supervision multi-scale feature fusion module to obtain the multiple Pre-disaster channel space correction feature maps of different scales; input multiple post-disaster image features of different scales into the multi-attention-constrained deep-supervised multi-scale feature fusion module, and obtain multiple post-disaster channel space correction feature maps of different scales output by the multi-attention-constrained deep-supervised multi-scale feature fusion module; wherein the multi-attention constraint includes a channel attention mechanism and a spatial attention mechanism; input the pre-disaster channel space correction feature map and the post-disaster channel space correction feature map into the decoder, and obtain the change detection map output by the decoder, wherein the pre-disaster channel space correction feature map and the pre-disaster channel space correction feature map have the same scale, and the change detection map is used to represent the change monitoring comparison results of the target object. The change monitoring capability of flooded buildings and roads in complex flood scenes is improved, thereby solving the technical problem that the flood disaster assessment method in the related technology has low accuracy in monitoring the changes of flooded buildings and flooded roads.

[0066] According to the flooded infrastructure change detection method based on feature fusion deep supervision network provided by the present invention, the pre-disaster encoder includes: a first pre-disaster encoder, a second pre-disaster encoder, a third pre-disaster encoder and a fourth pre-disaster encoder; the pre-disaster image is input to the pre-disaster encoder to obtain a plurality of pre-disaster image features of different scales output by the pre-disaster encoder, including: Input the pre-disaster image into the first pre-disaster encoder to obtain the first pre-disaster image feature output by the first pre-disaster encoder, the scale of the first pre-disaster image feature is one quarter of the pre-disaster image, and the number of channels of the first pre-disaster image feature is 1; Input the first pre-disaster image feature into the second pre-disaster encoder to obtain the second pre-disaster image feature output by the second pre-disaster encoder, the scale of the second pre-disaster image feature is one eighth of the pre-disaster image, and the number of channels of the second pre-disaster image feature is 2; Inputting the second pre-disaster image feature into the third pre-disaster encoder to obtain the third pre-disaster image feature output by the third pre-disaster encoder, wherein the scale of the third pre-disaster image feature is one sixteenth of the pre-disaster image, and the number of channels of the third pre-disaster image feature is 4; The third pre-disaster image feature is input into the fourth pre-disaster encoder to obtain the fourth pre-disaster image feature output by the fourth pre-disaster encoder. The scale of the fourth pre-disaster image feature is one thirty-second of the pre-disaster image, and the number of channels of the fourth pre-disaster image feature is 8.

[0067] Here, the scale of the feature is the height (H) and width (W) of the feature map.

[0068] refer to Figure 3 , Figure 3 It is an overall network structure diagram of the flooded infrastructure change detection method based on feature fusion deep supervision network provided by the present invention.

[0069] Input the pre-disaster image into the first encoder (encoder_1) to obtain the first pre-disaster image feature ( ,in, Indicates characteristics, represents the encoder, Before the disaster, represents the set of real numbers, and Respectively represent the height and width of the original image (pre-disaster image), The first pre-disaster image feature is input into the second encoder (encoder_2) to obtain the second pre-disaster image feature ( ); Input the second pre-disaster image feature into the third encoder (encoder_3) to obtain the third post-disaster image feature ( ); Input the third post-disaster image feature to the fourth encoder (encoder_4) to obtain the fourth post-disaster image feature ( ).

[0070] The encoding process of post-disaster images is the same as that of pre-disaster images (where Indicates after the disaster), the present invention will not be repeated here.

[0071] In some embodiments, the post-disaster encoder includes: a first post-disaster encoder, a second post-disaster encoder, a third post-disaster encoder, and a fourth post-disaster encoder.

[0072] Input the post-disaster image into the first post-disaster encoder to obtain the first post-disaster image feature output by the first post-disaster encoder, the scale of the first post-disaster image feature is one quarter of the post-disaster image, and the number of channels of the first post-disaster image feature is 1; Input the first post-disaster image feature into the second post-disaster encoder to obtain the second post-disaster image feature output by the second post-disaster encoder, the scale of the second post-disaster image feature is one eighth of the post-disaster image, and the number of channels of the second post-disaster image feature is 2; Inputting the second post-disaster image feature into the third post-disaster encoder to obtain the third post-disaster image feature output by the third post-disaster encoder, the scale of the third post-disaster image feature is one sixteenth of the post-disaster image, and the number of channels of the second post-disaster image feature is 4; The third post-disaster image feature is input into the fourth post-disaster encoder to obtain the fourth post-disaster image feature output by the fourth post-disaster encoder. The scale of the fourth post-disaster image feature is one thirty-second of the post-disaster image, and the number of channels of the second post-disaster image feature is 8.

[0073] Through the embodiments of the present invention, pre-disaster images and post-disaster images can be encoded into features of different scales respectively, thereby obtaining different levels of information from the global to the local level in the images.

[0074] According to the flooded infrastructure change detection method based on feature fusion deep supervision network provided by the present invention, the number of channels of the pre-disaster encoder is 192, 384, 768, and 1536 respectively; the number of modules of the pre-disaster encoder is 3, 3, 9, and 3 respectively; the structure of the pre-disaster encoder includes: deep convolution, layer normalization, first point-by-point convolution, Gaussian error linear unit, and second point-by-point convolution.

[0075] refer to Figure 4 , Figure 4 It is a detailed network structure diagram of the flooded infrastructure change detection method based on the feature fusion deep supervision network provided by the present invention, which includes multiple encoders, a multi-attention constrained deep supervision multi-scale feature fusion module, multiple channel splicing and multiple decoders.

[0076] It should be noted that the encoder includes a pre-disaster encoder and a post-disaster encoder, and the pre-disaster encoder and the post-disaster encoder have the same structure and characteristics.

[0077] In the embodiment of the present invention, in order to balance the number of model parameters and the performance of feature extraction, ConvNeXt-Tiny is introduced as the backbone of the encoder, and pre-trained weights are added on its basis to construct the following Figure 4The encoder, decoder, and high-dimensional-low-dimensional multi-scale feature connection fusion module (i.e., multi-attention constrained deep supervision multi-scale feature fusion module) are shown. The four-stage channels of the encoder are (192, 384, 768, 1536), and the number of modules is (3, 3, 9, 3). The encoder structure includes deep convolution, layer normalization (LayerNorm), point-by-point convolution, Gaussian error linear unit (GELU) and point-by-point convolution. The above encoder structure can have stronger context perception and extract more accurate semantic information.

[0078] Deep convolution can capture local features of input data, and as the size of the convolution kernel increases, the range of features captured will also expand accordingly. In the encoder structure, the deep convolution layer is usually located at the beginning of the module to preliminarily extract the features of the input data.

[0079] LayerNorm is a normalization technique used to normalize each sample of input data. It can help speed up the model training process and improve the stability of the model. In the encoder structure, LayerNorm is usually located after the convolution layer and is used to normalize the output of the convolution layer.

[0080] Point-wise convolution (also called 1x1 convolution) is a special convolution operation with a convolution kernel size of 1x1. It is mainly used to change the number of channels of the input data while keeping the spatial resolution of the data unchanged. In the encoder structure, point-wise convolution is often used to connect modules at different stages and to achieve dimensionality reduction or dimensionality increase of features.

[0081] GELU (Gaussian Error Linear Unit) is a nonlinear activation function that combines the advantages of the ReLU activation function and introduces the randomness of the Gaussian distribution. In the encoder structure, the GELU activation function is used to increase the nonlinear expression ability of the model, thereby capturing more complex features.

[0082] Through the embodiments of the present application, the deep convolution layer and point-by-point convolution layer in the encoder structure gradually extract deeper features by continuously convolving and transforming the input data. These features not only contain local information of the input data, but also contextual information. Therefore, the encoder structure has strong context perception; at the same time, components such as LayerNorm and GELU also help improve the model's ability to extract semantic information.

[0083] According to the flooded infrastructure change detection method based on feature fusion deep supervision network provided by the present invention, multiple pre-disaster image features of different scales are input into a multi-attention constrained deep supervision multi-scale feature fusion module, and multiple pre-disaster channel space correction feature maps of different scales output by the multi-attention constrained deep supervision multi-scale feature fusion module are obtained, including: Each of the multiple different scales is used as the target scale to perform the following operations to obtain the pre-disaster channel space correction feature map of the target scale: Up-sample the pre-disaster image features of multiple different scales to obtain pre-disaster image features of multiple target scales with different numbers of channels; Channel splicing is performed on the pre-disaster image features of multiple target scales with different numbers of channels to obtain a target scale feature map; Input the target scale feature map into the channel attention module to obtain the channel attention feature map output by the channel attention module; Multiply the channel attention feature map with the target scale feature map to obtain the channel correction feature map; Input the target scale feature map into the spatial attention module to obtain the spatial attention feature map output by the spatial attention module; Multiply the spatial attention feature map with the target scale feature map to obtain the spatial correction feature map; The channel correction feature map is multiplied by the spatial correction feature map to obtain the pre-disaster channel spatial correction feature map of the target scale.

[0084] refer to Figure 5 , Figure 5 This is a diagram of a multi-attention constrained deep-supervised high-dimensional-low-dimensional multi-scale feature fusion module provided by the present invention.

[0085] The quarter-scale of the pre-disaster image is determined as the target scale, and the pre-disaster image features with a quarter-scale and a channel number of 1 ( ) is upsampled to obtain the pre-disaster image features with a quarter scale and a channel number of 1 ( ).

[0086] The pre-disaster image features with a scale of one-eighth and a channel number of 2 ( ) is upsampled to obtain the pre-disaster image features with a quarter scale and 2 channels ( ).

[0087] The pre-disaster image features with a scale of one-sixteenth and a channel number of 4 ( ) is upsampled to obtain the pre-disaster image features with a quarter scale and 4 channels ( ).

[0088] The pre-disaster image features with a scale of 1 / 32 and 8 channels ( ) is upsampled to obtain the pre-disaster image features with a quarter scale and 8 channels ( ).

[0089] The pre-disaster image features obtained by upsampling at a quarter of the target scale are channel-joined to obtain a target scale feature map.

[0090] The target scale feature map is input into the spatial-channel compression activation module.

[0091] Perform adaptive average pooling on the target scale feature map to obtain the first spatial feature ( ); Perform convolution and corrected linear transformation on the first spatial feature to obtain the second spatial feature ( ); Convolve and activate the second spatial feature to obtain the third spatial feature ( ), that is, the spatial attention feature map; multiply the third spatial feature with the target scale feature map to obtain the spatial compression-activation feature map (that is, the spatial correction feature map).

[0092] Among them, the rectified linear unit (ReLU) is a commonly used activation function, which is usually used to increase the nonlinear characteristics of the network. The ReLU layer usually follows the convolutional layer and performs a nonlinear transformation on the output of the convolutional layer, that is, setting all negative values ​​to 0 and keeping the positive values ​​unchanged.

[0093] For example, first, the convolution layer uses a set of convolution filters to perform convolution operations on the input image to generate a series of feature maps. Then, the ReLU layer (rectified linear unit) performs a nonlinear transformation on the output of the convolution layer. Specifically, each element in the feature map is processed by the ReLU function, all negative values ​​are set to 0, and positive values ​​remain unchanged.

[0094] Convolve the target scale feature map to obtain the first channel feature ( ), that is, the channel attention feature map; the channel attention feature map is multiplied by the target scale feature map to obtain the channel compression-activation feature map (that is, the channel correction feature map).

[0095] Multiply the spatial compression-activation feature map with the channel compression-activation feature map to obtain the spatial channel compression-activation feature map ( ), which is the channel space correction feature map.

[0096] Here, one-quarter scale, one-eighth scale, one-sixteenth scale and one-thirty-second scale are taken as target scales, and upsampling, channel splicing and space-channel compression activation operations are repeated to obtain the pre-disaster channel space correction feature map of each different scale.

[0097] It can be understood that the specific process of inputting post-disaster image features of multiple different scales into the multi-attention constrained deep supervision multi-scale feature fusion module to obtain the post-disaster channel space correction feature maps of multiple different scales output by the multi-attention constrained deep supervision multi-scale feature fusion module is the same as the above process except that the input is replaced by the post-disaster image, and the present invention will not be repeated here.

[0098] According to the flooded infrastructure change detection method based on feature fusion deep supervision network provided by the present invention, the decoder includes a first decoder, a second decoder, a third decoder and a fourth decoder, and the pre-disaster channel space correction feature map and the post-disaster channel space correction feature map are input into the decoder to obtain a change detection map output by the decoder, including: Channel-joining is performed on the pre-disaster channel space correction feature map of the fourth scale and the post-disaster channel space correction feature map of the fourth scale to obtain a fourth joint feature map; Inputting the fourth concatenated feature map into a fourth decoder to obtain a third scale feature map output by the fourth decoder; Channel splicing is performed on the pre-disaster channel space correction feature map of the third scale, the post-disaster channel space correction feature map of the third scale, and the third scale feature map to obtain a third splicing feature map; Inputting the third concatenated feature map into a third decoder to obtain a second scale feature map output by the third decoder; Channel splicing is performed on the pre-disaster channel space correction feature map of the second scale, the post-disaster channel space correction feature map of the second scale, and the second scale feature map to obtain a second spliced ​​feature map; Inputting the second concatenated feature map into the second decoder to obtain a first scale feature map output by the second decoder; Perform channel splicing on the pre-disaster channel space correction feature map of the first scale, the post-disaster channel space correction feature map of the first scale, and the first scale feature map to obtain a first splicing feature map; The first concatenated feature map is input into the first decoder to obtain a change detection map output by the first decoder.

[0099] like Figure 3 As shown, the multi-attention constrained deep supervision multi-scale feature fusion module outputs a fourth-scale pre-disaster channel spatial correction feature map and a fourth-scale post-disaster channel spatial correction feature map, a third-scale pre-disaster channel spatial correction feature map and a third-scale post-disaster channel spatial correction feature map, a second-scale pre-disaster channel spatial correction feature map and a second-scale post-disaster channel spatial correction feature map, and a first-scale pre-disaster channel spatial correction feature map and a first-scale post-disaster channel spatial correction feature map.

[0100] The fourth scale pre-disaster channel space correction feature map ( ) and the fourth scale post-disaster channel space correction feature map ( ) to perform channel splicing to obtain a fourth splicing feature map; the fourth splicing feature map is input into a fourth decoder (decoder_4) to obtain a third scale feature map.

[0101] The third-scale feature map and the third-scale pre-disaster channel space correction feature map ( ) and the third-scale post-disaster channel spatial correction feature map ( ) to perform channel splicing to obtain a third splicing feature map; the third splicing feature map is input into a third decoder (decoder_3) to obtain a second scale feature map.

[0102] The second-scale feature map and the second-scale pre-disaster channel space correction feature map ( ) and the second-scale post-disaster channel spatial correction feature map ( ) to perform channel splicing to obtain a second splicing feature map; the second splicing feature map is input into the third decoder (decoder_2) to obtain a first scale feature map.

[0103] The first scale feature map and the first scale pre-disaster channel space correction feature map ( ) and the first scale post-disaster channel spatial correction feature map ( ) to perform channel splicing to obtain a first splicing feature map; the second splicing feature map is input into the third decoder (decoder_1) to obtain a change detection map.

[0104] The change detection map is input into the change detection head to obtain the flood label output by the change detection head, where 0 represents background, 1 represents normal buildings, 2 represents flooded buildings, 3 represents normal roads, and 4 represents flooded roads.

[0105] In an embodiment of the present invention, image features of different scales (including: pre-disaster image features and post-disaster image features) are resampled and channel stitching is performed, and convolution, batch normalization, rectified linear unit, space-channel compression activation, convolution, batch normalization are performed to obtain channel space correction feature maps of different scales (including: pre-disaster channel space correction feature map and post-disaster channel space correction feature map).

[0106] According to the flooded infrastructure change detection method based on feature fusion deep supervision network provided by the present invention, the structure of the decoder includes: convolution, first batch normalization, first rectified linear unit, inverse convolution, second batch normalization and second rectified linear unit.

[0107] like Figure 4As shown in the figure, the structure of the decoder consists of convolution, inverse convolution, BN (batch normalization) and ReLU (rectified linear unit), which realizes the spatial and semantic feature fusion and complementation of high-dimensional and low-dimensional multi-scale features.

[0108] The following describes an example of a method for detecting changes in flooded infrastructure based on a feature fusion deep supervision network provided by the present invention in a practical application scenario.

[0109] Deep learning is widely used in change detection due to its excellent feature extraction ability. There are many different change detection structures based on deep learning methods. The change detection framework based on DL can be divided into a single-branch structure and a dual-branch structure according to its feature extraction strategy. The single-branch structure uses early fusion to merge the temporal images into multispectral images. However, due to the lack of independent original input images, it is difficult for the single-branch structure to extract the fine temporal features of each dual-temporal image, which is not conducive to the subsequent change detection of the network. The dual-branch structure is a conjoined network consisting of two sub-networks with shared weights or non-shared weights. It has two dual-temporal fusion modes: mid-term fusion and late fusion. Each branch independently learns the high-level feature representation of a single-phase image through multiple convolutions. This powerful representation ability enables the dual-branch network to effectively capture the change information in complex RS images, thereby improving the accuracy of change detection. In order to make the model have stronger context perception ability for single-temporal images, the present invention adopts a mid-term fusion strategy, and embeds a multi-scale feature fusion (MSFF) module to improve the mid-term fusion strategy and design a change detection structure FloodCD, such as Figure 4 As shown, a simplified change detection structure (i.e., a multi-attention constrained deep-supervised multi-scale feature fusion module), encoder-decoder and fusion method are shown.

[0110] In order to balance the number of model parameters and the performance of feature extraction, the present invention introduces ConvNeXt-Tiny as the backbone of the encoder, and adds pre-trained weights on its basis to construct Figure 4 The encoder, decoder, and high-dimensional-low-dimensional multi-scale feature connection fusion module (i.e., multi-attention constrained deep supervision multi-scale feature fusion module) are shown in the figure. The four-stage channels of the encoder are (192, 384, 768, 1536), and the number of modules is (3, 3, 9, 3). The encoder structure includes deep convolution, layernorm, point-by-point convolution, GELU, and point-by-point convolution. The above encoder structure can have stronger context perception and extract more accurate semantic information. The decoder structure consists of convolution, inverse convolution, BN, and ReLU, which realizes the spatial and semantic feature fusion supplement of high-dimensional-low-dimensional multi-scale features.

[0111] PSPNet first proposed pyramid-structured feature fusion. In subsequent research work, feature fusion was widely used in image segmentation, and the self-attention feature fusion network (SA-FFNet) was introduced to improve the semantic segmentation performance. For example, the multi-scale cross-attention hierarchical network (MSCCA-Net) feature fusion focuses on the mutual fusion between multiple scales, but there is still the problem that high-dimensional-low-dimensional multi-scale features cannot be fully utilized. Even if all fine-grained spatial information and coarse-grained semantic information are merged into the decoder through full-scale jump connections, it cannot be plug-and-play into various promising deep learning models, and it is heavily dependent on the specific structure of the module. In order to overcome the problem of multi-scale feature fusion module migration and make full use of multi-scale features, the embodiment of the present invention constructs a plug-and-play multi-attention constrained deep supervision high-dimensional-low-dimensional multi-scale feature fusion module (i.e., multi-attention constrained deep supervision multi-scale feature fusion module), which will significantly improve the detection accuracy of flooded building road changes in flood scenes.

[0112] like Figure 5 As shown in the figure, scSE is a joint attention mechanism that combines the channel attention mechanism and the spatial attention mechanism of the convolution module. The target scale feature map is input into the channel attention module to obtain the channel attention feature map output by the channel attention module; the channel attention feature map is multiplied with the target scale feature map to obtain the channel correction feature map; the target scale feature map is input into the spatial attention module to obtain the spatial attention feature map output by the spatial attention module; the spatial attention feature map is multiplied with the target scale feature map to obtain the spatial correction feature map; the channel correction feature map is multiplied with the spatial correction feature map to obtain the pre-disaster channel spatial correction feature map of the target scale.

[0113] Different from cross entropy loss, dice loss and focal loss as the main loss function of the network structure, cross entropy loss is used as the main loss function and focal is used as the auxiliary loss function to optimize the hidden layer feature extraction capability. The present invention directly optimizes the gradient back propagation process of the hidden layer weights by adding deep supervised learning to the hidden layer of the network, making it biased towards high-discrimination feature maps. This enhances the flood inundation feature extraction capability of the hidden layer in the network. Images before and after floods and water body masks , the second decoder of FloodCDNet is recorded as , and added a deep supervised classifier , thus introducing deep supervision optimization for the network encoder and the first two decoders, which can prompt the model to learn these different levels of features and improve the detection results of flooded building road changes. The deep supervision classifier can refer to the following formula (1): (1)

[0114] in, represents a deep supervised classifier, For Model In the training set The heat map generated by reasoning above, For the second decoder The generated feature map, is the batchsize of samples in the training set, each is the onehot label of the labeled data, Upsampling of feature maps, represents the pre-disaster image mask, Represents the post-disaster image mask.

[0115] refer to Figure 3 , Figure 3 The workflow of FloodCDNet for detecting changes in flooded buildings and roads from ultra-high resolution remote sensing images is proposed. FloodCDNet adopts a weight-sharing dual-temporal semantic feature fusion change detection structure and learns five categories of change information through an encoder-decoder architecture. This method improves the recognition ability of flooded buildings and roads by learning the semantic features of buildings and roads in the normal state of pre-disaster images and the flood information around flooded buildings and flooded roads after the disaster. For the feature extractor, a ConvNeXt-based twin network is adopted. It can extract the semantic features of buildings, roads, and floods from dual-temporal images, and obtain the semantic features of flooded buildings and roads with strong context perception. In order to strengthen the extraction and fusion of semantic features at different levels of dual-temporal images, the present invention constructs a deep-supervised high-dimensional-low-dimensional multi-scale feature fusion module with ready-to-use multi-attention constraints, thereby obtaining a robust global-local semantic representation of infrastructure and floods, and significantly improving the accuracy of rapid extraction of flooded buildings and roads. Subsequently, data enhancement strategies and deep supervision modules for supervising hidden layers are gradually introduced to improve detection accuracy.

[0116] In order to verify the superiority of the change detection structure adopted by the present invention, the present invention re-implemented the early fusion (FC-EF), mid-term fusion conc (Fc-siam-conc) and subtract (Fc-siam-diff) fusions F1, F2, and F3 respectively; and designed the late fusion (Fc-siam-last-conc) strategy F4. The comparison of the miou accuracy results of the change detection results of flooded building roads on the SpaceNet8 dataset shows that Fc-siam-conc 60.56% is better than all other structures, exceeding FC-EF, Fc-siam-diff, and Fc-siam-last-conc by 14.32%, 4.78%, and 1.31% respectively. The mid-term conc fusion strategy performs outstandingly in change detection of flooded building roads.

[0117] refer to Figure 6 , Figure 6 This is a comparison diagram of monitoring results of flooded building roads with different common change detection structures provided by the present invention.

[0118] like Figure 6 As shown, the experimental results of different change detection structures are shown. (Buildings and roads are focused on at the same time) The mid-term fusion method performs better than the early fusion and late fusion methods in terms of missed detection of flooded buildings (red circles) and misdetected pixels (orange circles). Figure 6 (a)); Figure 6 Middle (d) shows that the normal buildings missed less in the mid-term fusion than in the early fusion. Although the normal buildings are not as good as those in the late fusion in terms of indicators, the missed detection of normal buildings is significantly less ( Figure 6 In the area where there are both flooded buildings and normal buildings (roads), the false positive rate (orange circles) of the three fusion methods is high, and the number of pixels missed on normal roads is also high ( Figure 6 (b)).

[0119] Refer to Table 1, which is a comparison table of the monitoring accuracy of flooded building roads with different common change detection structures provided by the present invention.

[0120] Table 1

[0121] In order to evaluate the flooded building road change detection model FloodCDNet used, we used EfficientNet-B5 and ResNeSt50 as the backbone to build models with the same dual-time fusion structure, denoted as M2 and M3. FC-siam-conc, ChangeFormer, and SNUNet were implemented in the study and denoted as M1, M4, and M5. All models use the same parameters to ensure fair comparison. The table lists the IOU accuracy of different models on the SpaceNet8 dataset. FloodCDNet outperforms other methods in MIOU, exceeding FC-siam-conc, EfficientNet-B5, ResNeSt50, ChangeFormer, and SNUNet by 7.32%, 6.73%, 6.16%, 10.02%, and 16.15% respectively. The models M2, M3, and M6 constructed by the present invention are all better than other change detection models. The performance of the changeformer based on tranformer is not as good as that of similar products of CNN.

[0122] refer to Figure 2 , Table 2 is a comparison table of the monitoring accuracy of flooded building roads using different common deep learning models provided by the present invention.

[0123] Table 2

[0124] The present invention conducted ablation experiments on the SpaceNet8 dataset, which included the proposed modules data enhancement, multi-scale feature fusion, and deep supervised learning, and compared their performance with baseline methods that lack these features. The table lists the accuracy of the detection results of changes in flooded building roads, indicating that the integration of these modules improves the accuracy of the evaluation indicator iou. It is worth noting that the miou value obtained by the baseline method is reduced by 6.11%. This means that these modules greatly improve the accuracy of change detection in flooded building roads. In particular, the addition of data enhancement strategies has a significant impact on the accuracy of change detection, specifically an increase of 4.78% in miou. At the same time, after adding MSFF, the accuracy of change detection in flooded building roads is improved by 1.33%.

[0125] refer to Figure 7 , Figure 7 This is a comparison diagram of ablation experiment results provided by the present invention.

[0126] Figure 7 The ablation results are shown, where the boundary rules of the changed areas identified by B4 are the most consistent with the ground truth. In particular, the addition of the MSFF module enhances the recognition of normal buildings and flooded buildings ( Figure 7The orange circles in (a), (b) and (d) Figure 7 (b) and Figure 7 As shown in the orange and red circles in (c), after adding deep supervision, the flooded roads and roads become more regular and show fine boundary details.

[0127] Refer to Table 3, which is a comparison of the monitoring accuracy of different modules on flooded building road changes in the ablation experiment provided by the present invention.

[0128] Table 3

[0129] In order to evaluate the simplicity of MSFF and deep supervised learning plug-and-play and their impact on change detection results, we conducted ablation experiments on M1, M2, and M3. The table lists the evaluation results of Fc-siam-conc, Resnest50, Efficientnet-b5, and Convnext-tiny after adding MSFF and deep supervised learning, which shows that the detection accuracy is improved when using them.

[0130] refer to Figure 8 , Figure 8 This is a comparison chart of the monitoring results of flooded building roads using the MSFF provided by the present invention embedded in different deep learning models.

[0131] Figure 8 The results of embedding the MFLL module are shown. The results obtained using the ConvNeXt feature extractor are more accurate than the other methods in terms of flooded buildings, flooded roads, and normal buildings and normal roads ( Figure 8 Middle (a), Figure 8 (b) and Figure 8 In (d), the false detection and missed detection ratios are low, and the shape contours are clear. This shows that ConvNeXt as a feature extractor is more robust in identifying various objects. In addition, the missed detection problem on normal roads is more prominent ( Figure 8 In (c) and Figure 8 in (d).

[0132] Refer to Table 4, which is a comparison table of the monitoring accuracy of flooded building roads using MSFF embedded in different deep learning models provided by the present invention.

[0133] Table 4

[0134] The following is a description of the flooded infrastructure change detection device based on the feature fusion deep supervision network provided by the present invention. The flooded infrastructure change detection device based on the feature fusion deep supervision network described below and the flooded infrastructure change detection method based on the feature fusion deep supervision network described above can be referenced to each other.

[0135] refer to Fig. 9 , Fig. 9 It is a structural schematic diagram of a flooded infrastructure change detection device based on a feature fusion deep supervision network provided by the present invention, which includes: an acquisition module 901, a pre-disaster encoding module 902, a post-disaster encoding module 903, a multi-scale fusion module 904, and a decoding module 905.

[0136] The acquisition module 901 is used to acquire pre-disaster images and post-disaster images, wherein the pre-disaster images are used to represent images of the target object before it suffers from the flood disaster, and the post-disaster images are used to represent images of the target object after it suffers from the flood disaster; A pre-disaster encoding module 902 is used to input the pre-disaster image into a pre-disaster encoder to obtain pre-disaster image features of multiple different scales output by the pre-disaster encoder; The post-disaster encoding module 903 is used to input the post-disaster image into the post-disaster encoder to obtain a plurality of post-disaster image features of different scales output by the post-disaster encoder; A multi-scale fusion module 904 is used to input the pre-disaster image features of multiple different scales into the multi-attention-constrained deep-supervised multi-scale feature fusion module to obtain the pre-disaster channel space correction feature maps of multiple different scales output by the multi-attention-constrained deep-supervised multi-scale feature fusion module; The multi-scale fusion module 904 is further used to input the post-disaster image features of multiple different scales into the multi-attention-constrained deep-supervised multi-scale feature fusion module to obtain the post-disaster channel space correction feature maps of multiple different scales output by the multi-attention-constrained deep-supervised multi-scale feature fusion module; Among them, the multi-attention constraints include channel attention mechanism and spatial attention mechanism; The decoding module 905 is used to input the pre-disaster channel space correction feature map and the post-disaster channel space correction feature map into the decoder to obtain a change detection map output by the decoder, wherein the pre-disaster channel space correction feature map and the pre-disaster channel space correction feature map have the same scale, and the change detection map is used to represent the change monitoring comparison results of the target object.

[0137] Specifically, the above-mentioned flooded infrastructure change detection device based on feature fusion deep supervision network provided by the present invention can implement all the method steps implemented by the above-mentioned flooded infrastructure change detection method embodiment based on feature fusion deep supervision network, and can achieve the same technical effect. The parts and beneficial effects that are the same as the method embodiment in this embodiment will not be specifically repeated here.

[0138] refer to Fig.10 , Fig.10 is a schematic diagram of the physical structure of the electronic device provided by the present invention, such as Fig.10 As shown, the electronic device may include: a processor 1010, a communication interface 1020, a memory 1030 and a communication bus 1040, wherein the processor 1010, the communication interface 1020 and the memory 1030 communicate with each other through the communication bus 1040. The processor 1010 may call the logic instructions in the memory 1030 to execute a flooded infrastructure change detection method based on a feature fusion deep supervision network, the method comprising: obtaining pre-disaster images and post-disaster images, wherein the pre-disaster images are used to represent the images of the target object before the flood disaster, and the post-disaster images are used to represent the images of the target object after the flood disaster; inputting the pre-disaster images into the pre-disaster encoder to obtain pre-disaster image features of multiple different scales output by the pre-disaster encoder; inputting the post-disaster images into the post-disaster encoder to obtain post-disaster image features of multiple different scales output by the post-disaster encoder; inputting the pre-disaster image features of multiple different scales into a multi-attention constrained deep supervision multi-scale feature fusion module to obtain a multi-attention The method comprises the following steps: inputting the pre-disaster channel space correction feature maps of multiple different scales output by the deep supervision multi-scale feature fusion module under multi-attention constraints; inputting the post-disaster image features of multiple different scales into the deep supervision multi-scale feature fusion module under multi-attention constraints, and obtaining the post-disaster channel space correction feature maps of multiple different scales output by the deep supervision multi-scale feature fusion module under multi-attention constraints; wherein the multi-attention constraints include the channel attention mechanism and the spatial attention mechanism; inputting the pre-disaster channel space correction feature map and the post-disaster channel space correction feature map into the decoder, and obtaining the change detection map output by the decoder, wherein the pre-disaster channel space correction feature map and the pre-disaster channel space correction feature map have the same scale, and the change detection map is used to represent the change monitoring comparison result of the target object.

[0139] In addition, the logic instructions in the above-mentioned memory 1030 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0140] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the flooded infrastructure change detection method based on the feature fusion deep supervision network provided by the above-mentioned methods, and the method includes: obtaining pre-disaster images and post-disaster images, wherein the pre-disaster images are used to represent the images of the target object before suffering from the flood disaster, and the post-disaster images are used to represent the images of the target object after suffering from the flood disaster; inputting the pre-disaster images into a pre-disaster encoder to obtain a plurality of pre-disaster image features of different scales output by the pre-disaster encoder; inputting the post-disaster images into a post-disaster encoder to obtain a plurality of post-disaster image features of different scales output by the post-disaster encoder; and inputting the pre-disaster images into a post-disaster encoder to obtain a plurality of post-disaster image features of different scales output by the post-disaster encoder; and The image features are input into a deep-supervised multi-scale feature fusion module with multiple attention constraints to obtain a plurality of pre-disaster channel space correction feature maps of different scales output by the deep-supervised multi-scale feature fusion module with multiple attention constraints; the post-disaster image features of multiple different scales are input into a deep-supervised multi-scale feature fusion module with multiple attention constraints to obtain a plurality of post-disaster channel space correction feature maps of different scales output by the deep-supervised multi-scale feature fusion module with multiple attention constraints; wherein the multi-attention constraints include a channel attention mechanism and a spatial attention mechanism; the pre-disaster channel space correction feature map and the post-disaster channel space correction feature map are input into a decoder to obtain a change detection map output by the decoder, wherein the pre-disaster channel space correction feature map and the pre-disaster channel space correction feature map have the same scale, and the change detection map is used to represent the change monitoring comparison result of the target object.

[0141] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the flooded infrastructure change detection method based on a feature fusion deep supervision network provided by the above-mentioned methods, the method comprising: obtaining pre-disaster images and post-disaster images, wherein the pre-disaster images are used to represent the images of the target object before the flood disaster, and the post-disaster images are used to represent the images of the target object after the flood disaster; inputting the pre-disaster images into a pre-disaster encoder to obtain a plurality of pre-disaster image features of different scales output by the pre-disaster encoder; inputting the post-disaster images into a post-disaster encoder to obtain a plurality of post-disaster image features of different scales output by the post-disaster encoder; and inputting the plurality of pre-disaster image features of different scales into a deep multi-attention constrained encoder. A multi-scale feature fusion module is supervised to obtain a plurality of pre-disaster channel space correction feature maps of different scales output by the deep supervision multi-scale feature fusion module with multiple attention constraints; the post-disaster image features of multiple different scales are input into the deep supervision multi-scale feature fusion module with multiple attention constraints to obtain a plurality of post-disaster channel space correction feature maps of different scales output by the deep supervision multi-scale feature fusion module with multiple attention constraints; wherein the multi-attention constraints include a channel attention mechanism and a spatial attention mechanism; the pre-disaster channel space correction feature map and the post-disaster channel space correction feature map are input into the decoder to obtain a change detection map output by the decoder, wherein the pre-disaster channel space correction feature map and the pre-disaster channel space correction feature map have the same scale, and the change detection map is used to represent the change monitoring comparison result of the target object.

[0142] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0143] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting changes in flooded infrastructure based on a feature fusion deep supervision network, characterized in that: include: Acquire pre-disaster images and post-disaster images, wherein the pre-disaster images are used to represent images of the target object before it suffers from the flood disaster, and the post-disaster images are used to represent images of the target object after it suffers from the flood disaster; Inputting the pre-disaster image into a pre-disaster encoder to obtain pre-disaster image features of multiple different scales output by the pre-disaster encoder; Inputting the post-disaster image into a post-disaster encoder to obtain a plurality of post-disaster image features of different scales output by the post-disaster encoder; Inputting the pre-disaster image features of the plurality of different scales into a multi-attention-constrained deep-supervised multi-scale feature fusion module, and obtaining a plurality of pre-disaster channel space correction feature maps of different scales output by the multi-attention-constrained deep-supervised multi-scale feature fusion module; Inputting the post-disaster image features of the plurality of different scales into the multi-attention constrained deep-supervised multi-scale feature fusion module, and obtaining the post-disaster channel space correction feature maps of the plurality of different scales output by the multi-attention constrained deep-supervised multi-scale feature fusion module; Wherein, the multi-attention constraints include a channel attention mechanism and a spatial attention mechanism; The pre-disaster channel spatial correction feature map and the post-disaster channel spatial correction feature map are input into a decoder to obtain a change detection map output by the decoder, wherein the pre-disaster channel spatial correction feature map and the pre-disaster channel spatial correction feature map have the same scale, and the change detection map is used to represent the change monitoring comparison result of the target object.

2. The method for detecting changes in flooded infrastructure based on feature fusion deep supervision network according to claim 1 is characterized in that: The pre-disaster encoder comprises: a first pre-disaster encoder, a second pre-disaster encoder, a third pre-disaster encoder and a fourth pre-disaster encoder; the pre-disaster image is input into the pre-disaster encoder to obtain a plurality of pre-disaster image features of different scales output by the pre-disaster encoder, including: Inputting the pre-disaster image into the first pre-disaster encoder to obtain a first pre-disaster image feature output by the first pre-disaster encoder, wherein the scale of the first pre-disaster image feature is one quarter of the pre-disaster image, and the number of channels of the first pre-disaster image feature is 1; Inputting the first pre-disaster image feature into the second pre-disaster encoder to obtain a second pre-disaster image feature output by the second pre-disaster encoder, wherein the scale of the second pre-disaster image feature is one eighth of the pre-disaster image, and the number of channels of the second pre-disaster image feature is 2; Inputting the second pre-disaster image feature into the third pre-disaster encoder to obtain a third pre-disaster image feature output by the third pre-disaster encoder, wherein the scale of the third pre-disaster image feature is one sixteenth of the pre-disaster image, and the number of channels of the third pre-disaster image feature is 4; The third pre-disaster image feature is input into the fourth pre-disaster encoder to obtain a fourth pre-disaster image feature output by the fourth pre-disaster encoder, the scale of the fourth pre-disaster image feature is one thirty-second of the pre-disaster image, and the number of channels of the fourth pre-disaster image feature is 8.

3. The method for detecting changes in flooded infrastructure based on feature fusion deep supervision network according to claim 2 is characterized in that: The number of channels of the pre-disaster encoder is 192, 384, 768, and 1536 respectively; the number of modules of the pre-disaster encoder is 3, 3, 9, and 3 respectively; the structure of the pre-disaster encoder includes: depth convolution, layer normalization, first point-by-point convolution, Gaussian error linear unit, and second point-by-point convolution.

4. The method for detecting changes in flooded infrastructure based on feature fusion deep supervision network according to claim 1 is characterized in that: The inputting of the plurality of pre-disaster image features of different scales into a multi-attention-constrained deep-supervised multi-scale feature fusion module to obtain a plurality of pre-disaster channel space correction feature maps of different scales output by the multi-attention-constrained deep-supervised multi-scale feature fusion module includes: Each of the multiple different scales is used as a target scale to perform the following operations to obtain a pre-disaster channel space correction feature map of the target scale: Upsampling the plurality of pre-disaster image features of different scales respectively to obtain pre-disaster image features of target scales of a plurality of different numbers of channels; Channel stitching is performed on the pre-disaster image features of the target scales with different numbers of channels to obtain a target scale feature map; Inputting the target scale feature map into a channel attention module to obtain a channel attention feature map output by the channel attention module; Multiplying the channel attention feature map with the target scale feature map to obtain a channel correction feature map; Inputting the target scale feature map into a spatial attention module to obtain a spatial attention feature map output by the spatial attention module; Multiplying the spatial attention feature map with the target scale feature map to obtain a spatial correction feature map; The channel correction feature map is multiplied by the space correction feature map to obtain a pre-disaster channel space correction feature map of the target scale.

5. The method for detecting changes in flooded infrastructure based on feature fusion deep supervision network according to claim 1, characterized in that: The decoder includes a first decoder, a second decoder, a third decoder and a fourth decoder, and the pre-disaster channel space correction feature map and the post-disaster channel space correction feature map are input into the decoder to obtain the change detection map output by the decoder, including: Channel-joining is performed on the pre-disaster channel space correction feature map of the fourth scale and the post-disaster channel space correction feature map of the fourth scale to obtain a fourth joint feature map; Inputting the fourth concatenated feature map into a fourth decoder to obtain a third scale feature map output by the fourth decoder; Perform channel splicing on the pre-disaster channel space correction feature map of the third scale, the post-disaster channel space correction feature map of the third scale, and the third scale feature map to obtain a third splicing feature map; Inputting the third concatenated feature map into a third decoder to obtain a second scale feature map output by the third decoder; Perform channel splicing on the pre-disaster channel space correction feature map of the second scale, the post-disaster channel space correction feature map of the second scale, and the second-scale feature map to obtain a second splicing feature map; Inputting the second concatenated feature map into a second decoder to obtain a first scale feature map output by the second decoder; Channel-joining is performed on the pre-disaster channel space correction feature map of the first scale, the post-disaster channel space correction feature map of the first scale, and the first-scale feature map to obtain a first joint feature map; The first concatenated feature map is input into a first decoder to obtain a change detection map output by the first decoder.

6. The method for detecting changes in flooded infrastructure based on feature fusion deep supervision network according to claim 5 is characterized in that: The structure of the decoder includes: convolution, first batch normalization, first rectified linear unit, inverse convolution, second batch normalization and second rectified linear unit.

7. A flooded infrastructure change detection device based on feature fusion deep supervision network, characterized in that: include: An acquisition module, used to acquire pre-disaster images and post-disaster images, wherein the pre-disaster images are used to represent images of the target object before it suffers from the flood disaster, and the post-disaster images are used to represent images of the target object after it suffers from the flood disaster; A pre-disaster encoding module, used for inputting the pre-disaster image into a pre-disaster encoder to obtain pre-disaster image features of multiple different scales output by the pre-disaster encoder; A post-disaster encoding module, used for inputting the post-disaster image into a post-disaster encoder to obtain a plurality of post-disaster image features of different scales output by the post-disaster encoder; A multi-scale fusion module, used for inputting the pre-disaster image features of the plurality of different scales into a multi-attention-constrained deep-supervised multi-scale feature fusion module, and obtaining a plurality of pre-disaster channel space correction feature maps of different scales output by the multi-attention-constrained deep-supervised multi-scale feature fusion module; The multi-scale fusion module is further used to input the post-disaster image features of the multiple different scales into the multi-attention constrained deep-supervised multi-scale feature fusion module to obtain the post-disaster channel space correction feature maps of multiple different scales output by the multi-attention constrained deep-supervised multi-scale feature fusion module; Wherein, the multi-attention constraints include a channel attention mechanism and a spatial attention mechanism; A decoding module is used to input the pre-disaster channel space correction feature map and the post-disaster channel space correction feature map into a decoder to obtain a change detection map output by the decoder, wherein the pre-disaster channel space correction feature map and the pre-disaster channel space correction feature map have the same scale, and the change detection map is used to represent the change monitoring comparison result of the target object.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for detecting changes in flooded infrastructure based on a feature fusion deep supervision network as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for detecting changes in flooded infrastructure based on a feature fusion deep supervision network as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for detecting changes in flooded infrastructure based on a feature fusion deep supervision network as described in any one of claims 1 to 6 is implemented.

Images