Multi-modal flood area identification method based on double-correlation cross attention

Abstract

The invention discloses a multi-modal flood area identification method based on double-correlation cross attention, relates to the technical field of remote sensing image processing and remote sensing image identification, and aims to solve the problem of low flood inundation identification precision in the prior art. Comprising the following steps: respectively carrying out image segmentation processing on an SAR image and an optical image to obtain a first non-overlapping image block and a second non-overlapping image block; respectively coding the first non-overlapping image block and the second non-overlapping image block by adopting a double-branch coder to obtain a multi-layer target SAR feature and a multi-layer target optical feature; determining double-correlation features between the target SAR features and the target optical features of each layer based on a double-correlation cross attention mechanism; and fusing the double correlation features, and determining a flood coverage area in the to-be-identified flood area based on the fused features. The multi-modal flood area identification method based on double-correlation cross attention is used for improving flood inundation identification precision.

Description

Technical Field

[0001] The invention relates to the technical field of remote sensing image processing and remote sensing image recognition, and in particular to a multimodal flood area recognition method based on dual-correlation cross-attention. Background Art

[0002] Flood disasters are one of the most common and destructive natural disasters in the world, and one of the most frequent meteorological disasters in my country, affecting a large number of people. Against the backdrop of climate change, the scale and scope of flood disasters are expanding due to the increase in extreme precipitation, posing serious challenges to social and economic development and the ecological environment. The development of accurate flood forecasting and real-time monitoring technology is an urgent requirement in disaster prevention and mitigation, and has important social significance and economic value. Remote sensing technology can obtain a wide range of surface information in real time, quickly and accurately, and is widely used in the identification and extraction of flood-inundated areas. Using remote sensing technology to obtain real-time and accurate flood-inundated area data has become a basic work for relevant departments to fully grasp the disaster situation, quickly and efficiently carry out emergency response, and conduct flood emergency rescue management.

[0003] In the identification of flood inundation areas based on remote sensing, synthetic aperture radar (SAR) has significant advantages. It can work all day and all weather and is not affected by bad weather conditions. In addition, SAR signals are very sensitive to the identification of water bodies and have strong penetration ability. They can provide information on the flooding status, making them the first choice for flood inundation area identification and monitoring. However, due to the limitations of the imaging mechanism, the spatial resolution of SAR images is usually low, and they are seriously affected by noise, terrain and shadows, which to a certain extent limits the accuracy of SAR images in the identification of flood inundation areas. In contrast, optical images have rich spectral information and relatively higher spatial resolution, and have obvious advantages in ground object identification, but they are easily affected by clouds and have difficulty in providing information on the flooding status. Given the complementary characteristics between SAR and optical data, some related studies have used both optical and SAR images to identify flood-covered areas, but these methods are usually based on traditional machine learning methods, and are insufficient in feature extraction and characterization of optical and SAR images. At the same time, simply superimposing SAR and optical images cannot achieve deep interaction of different modal features, and it is difficult to fully utilize the complementary advantages of SAR and optical data in the task of flood coverage area identification, resulting in low accuracy of flood inundation identification. Summary of the invention

[0004] The purpose of the present invention is to provide a multimodal flood area recognition method based on dual-correlation cross-attention, which is used to fully explore and utilize the complementary advantages of SAR and optical imaging modalities and significantly improve the accuracy of remote sensing flood recognition.

[0005] In order to achieve the above object, the present invention provides the following technical solutions: The present invention provides a multimodal flood area recognition method based on dual-correlation cross attention, comprising: Acquire SAR images and optical images of the flood area to be identified; Performing image segmentation processing on the SAR image and the optical image respectively to obtain a first non-overlapping image block corresponding to the SAR image and a second non-overlapping image block corresponding to the optical image; Using a dual-branch encoder to encode the first non-overlapping image block and the second non-overlapping image block respectively, to obtain multi-layer target SAR features and multi-layer target optical features; Determine the dual correlation features between the target SAR features and the target optical features at each layer based on the dual correlation cross attention mechanism; Fusing the dual correlation features to obtain target fused features; The flood coverage area in the to-be-identified flood area is determined based on the target fusion features.

[0006] Technical effect: Compared with the prior art, the present invention provides a multimodal flood area recognition method based on dual-correlation cross-attention, which performs flood area recognition based on SAR images and optical images, overcomes the limitations of single-modal images in flood area recognition, and achieves more accurate recognition performance. The SAR and optical images are segmented and processed, and then a dual-branch encoder is used to encode the first non-overlapping image block and the second non-overlapping image block respectively, to obtain multi-layer target SAR features and multi-layer target optical features. The dual-correlation features between the SAR features and the optical features of each layer are determined through the dual-correlation cross-attention mechanism, which can not only pay attention to the common information between the modalities, but also pay attention to the unique information between the modalities, and fully retain the complementary information between the multimodal data, thereby improving the ability to distinguish and express multimodal features. The dual-correlation features are fused to further enhance the feature expression ability, thereby improving the accuracy of flood area recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings: Figure 1 A flow chart of a multimodal flood area recognition method based on dual-correlation cross-attention provided by the present invention; Figure 2 The overall framework diagram of a multimodal flood area recognition method based on dual-correlation cross-attention provided by the present invention; Figure 3Schematic diagram of the calculation process of the feature fusion module composed of DCCMF and SA provided by the present invention. DETAILED DESCRIPTION

[0008] In order to clearly describe the technical solutions of the embodiments of the present invention, in the embodiments of the present invention, words such as "first" and "second" are used to distinguish the same items or similar items with basically the same functions and effects. For example, the first threshold and the second threshold are only used to distinguish different thresholds, and their order is not limited. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0009] It should be noted that, in the present invention, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0010] In the present invention, "at least one" means one or more, and "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, the combination of a and b, the combination of a and c, the combination of b and c, or the combination of a, b and c, where a, b, c can be single or multiple.

[0011] Before introducing the embodiments of the present invention, the following definitions are given for the relevant terms involved in the embodiments of the present invention: Swin-Transformer: is a visual Transformer model that has achieved remarkable results in many fields such as image classification, object detection, and semantic segmentation. Swin-Transformer is an improvement based on the idea of ​​Vision Transformer, but it solves the problems of ViT in low-resolution feature mapping and quadratic growth of complexity with image size. Swin-Transformer achieves efficient image processing and multi-scale feature extraction by introducing a sliding window mechanism and a hierarchical structure.

[0012] U-Net-like decoder: The U-Net-like decoder is a network structure designed and improved based on the decoder part in the U-Net network architecture. U-Net is a deep learning architecture designed specifically for image segmentation tasks. Its decoder part plays a vital role in image segmentation and is responsible for gradually restoring the spatial resolution and details of the image. The main function of the U-Net decoder is to gradually restore the spatial resolution and details of the image through the upsampling process. This is usually achieved through a series of deconvolution layers and upsampling layers. After each upsampling step, the decoder merges the feature map with the feature map of the corresponding layer of the encoder to restore the lost spatial information. This skip connection mechanism is a key feature of the U-Net architecture, which helps to recover detail information during the upsampling process and allows the network to learn more accurate outputs.

[0013] In recent years, deep learning has developed rapidly, and a series of new deep learning models have emerged, showing powerful feature learning and representation capabilities. These models can automatically extract multi-level, high-level feature representations from complex data, overcoming the limitations of hand-designed features in traditional machine learning methods. Therefore, based on the significant complementary characteristics between SAR and optical data, and the effective extraction and representation capabilities of deep learning for multimodal features, a multimodal flood coverage area recognition method based on deep learning optical image and SAR image collaboration is studied, which can fully explore and utilize the complementary advantages of the two modalities and significantly improve the effectiveness of remote sensing flood monitoring. The existing remote sensing-based flood inundation area recognition method does not take into account the feature interaction between SAR and optical images, and cannot give full play to the coordinated and complementary advantages between SAR images and optical images, and the accuracy of flood inundation recognition is not high.

[0014] In response to the above problems, this application proposes a multimodal flood area recognition method based on dual-correlation cross-attention. First, an asymmetric dual-branch encoder is used to extract multi-scale features of SAR images and optical images respectively. Then, a feature fusion module containing dual-correlation cross-attention mechanism and self-attention mechanism is constructed. The dual-correlation cross-attention mechanism is used to extract high-correlation features and low-correlation features from optical images to SAR images and SAR images to optical images respectively, and the complement of attention weights is used to capture the unique information between modalities. For high-correlation and low-correlation features, self-attention mechanisms are applied respectively to further enhance the feature expression ability, highlight important features, suppress redundant information, and realize the effective extraction and fusion of optical features and SAR features. The decoder constructed at last adopts a U-Net-like structure, and fuses features at different levels through a multi-level step-by-step feature fusion module. At the same time, jump connections are introduced in the decoder to alleviate the gradient vanishing problem in deep networks and promote information flow and gradient propagation. By combining an asymmetric dual-branch encoder, a cross-modal cross-attention feature fusion module and a U-Net-like decoder, the present invention constructs an efficient and robust multimodal flood area recognition framework, which makes full use of the complementary information of optical images and SAR images and realizes accurate flood area extraction. The present invention can use computer software to realize the automatic operation process. The following is an explanation with reference to the accompanying drawings.

[0015] See also Figure 1 ,This application provides a multimodal flood area recognition method based on dual correlation cross attention, e.g. Figure 1 As shown, the method comprises the following steps: Step 101: Acquire SAR images and optical images in the flood area to be identified; Specifically, obtain SAR images of the same phase after the flood occurs and optical imaging ;in, l is the number of bands of the SAR image, h is the height of the SAR image, w is the width of the SAR image; L is the number of bands of the optical image, H is the height of the optical image, W is the width of the optical image; , , ; R is the size of the feature.

[0016] Perform image preprocessing operations on the SAR image and the optical image to obtain a preprocessed SAR image and a preprocessed optical image. The preprocessing may be operations such as spatial registration and normalization.

[0017] The preprocessed SAR image is upsampled to obtain an upsampled SAR image , the spatial resolution of the upsampled SAR image is the same as that of the preprocessed optical image.

[0018] Step 102: performing image segmentation processing on the SAR image and the optical image respectively to obtain a first non-overlapping image block corresponding to the SAR image and a second non-overlapping image block corresponding to the optical image; The SAR image in this step is the SAR image after upsampling in step 101 The optical image is a pre-processed optical image. Figure 2 As shown, the optical feature extraction network extracts features from the optical image, and the SAR feature extraction network extracts features from the SAR image. Before feature extraction, the image is segmented by the image segmentation module, and then encoded by the encoder. Specifically, step 102 includes: segmenting the SAR image according to a preset block size to obtain a plurality of first non-overlapping image blocks; The optical image is segmented according to a preset block size to obtain a plurality of second non-overlapping image blocks.

[0019] Exemplarily, the preset block size may be 4×4.

[0020] Step 103: using a dual-branch encoder to encode the first non-overlapping image block and the second non-overlapping image block respectively, to obtain multi-layer target SAR features and multi-layer target optical features; like Figure 2 As shown, the dual-branch encoder includes a SAR encoder and an optical encoder. In the optical feature extraction network, the multi-scale features of the optical image are extracted by the optical encoder, and in the SAR feature extraction network, the multi-scale features of the SAR image are extracted by the SAR encoder; wherein, the SAR encoder and the optical encoder both include 4 encoding layers, wherein the first encoding layer is composed of a linear embedding layer and a Swin-Transformer module, and any layer from the second encoding layer to the fourth encoding layer is composed of an image block merging layer and a Swin-Transformer module. The linear embedding layer is used to embed the segmented image blocks into a high-dimensional vector space, and the embedding dimension size is CIt can be 96. The image block merging layer is used to spatially downsample the feature map, reduce the spatial resolution, and increase the number of channels. The downsampling rate can be 2 each time, and the number of channels can be increased by 2. The Swin-Transformer module is used for self-attention calculation and feature transformation. In the optical encoder, the depths of the Swin-Transformer modules used from the first layer to the fourth layer are 2, 2, 6, and 2 layers, respectively. In particular, the depth of the Swin-Transformer module used in the third encoding layer of the SAR encoder is less than the depth of the Swin-Transformer module used in the third encoding layer of the optical encoder. The depths of the Swin-Transformer modules used from the first layer to the fourth layer of the SAR encoder are 2, 2, 2, and 2 layers, respectively.

[0021] Specifically, a dual-branch encoder is used to encode the first non-overlapping image block and the second non-overlapping image block respectively, and the multi-layer target SAR features and the multi-layer target optical features are obtained, including: For a first coding layer of the SAR encoder: mapping the first non-overlapping image block to an initial embedding space to obtain a first image block feature map; Performing self-attention calculation and feature transformation on the feature map of the first image block to obtain the SAR output feature of the current layer; For any coding layer from the second coding layer to the fourth coding layer of the SAR encoder: spatially downsample the SAR output features of the previous layer and increase the number of channels to obtain processed SAR output features; Perform self-attention calculation and feature transformation on the processed SAR output features to obtain the SAR output features of the current layer; Complete the encoding process of the four-layer SAR encoding layer to obtain the four-layer SAR output features; Reshape each layer of SAR output features into a four-dimensional tensor form to obtain four layers of target SAR features; Four-layer target SAR features Indicates. i Indicates the sequence number of the coding layer, , B is the batch size, C i For the i The channel dimension size of the features generated by the layer encoding layer, H i For the i The size of the spatial width dimension of the features generated by the layer encoding layer, W i For the i The layer encodes the size of the high-dimensional space of features generated by the layer.

[0022] For a first encoding layer of the optical encoder: mapping the second non-overlapping image block to an initial embedding space to obtain a second image block feature map; Performing self-attention calculation and feature transformation on the feature map of the second image block to obtain the optical output feature of the current layer; For any coding layer from the second coding layer to the fourth coding layer of the optical encoder: spatially downsampling the optical output features of the previous layer and increasing the number of channels to obtain processed optical output features; Perform self-attention calculation and feature transformation on the processed optical output features to obtain the optical output features of the current layer; Complete the encoding processing of four optical coding layers to obtain four layers of optical output features; The optical output features of each layer are reshaped into a four-dimensional tensor form to obtain four layers of target optical features.

[0023] Four-layer target optical features express.

[0024] like Figure 2 As shown in the figure, the target SAR features and target optical features generated in each coding layer will be input into the cross-modal feature fusion module for fusion. The cross-modal feature fusion module is a feature fusion module that includes dual-correlation cross-attention fusion DCCMF and self-attention SA mechanism. The cross-modal feature fusion module first performs cross-attention calculation on the two input modalities to capture the high-correlation features and low-correlation features between the two modalities of SAR image and optical image. Figure 3 The calculation process of the feature fusion module composed of DCCMF and SA is shown, and then specifically described in conjunction with step 104 and step 105. The calculation process of cross attention is like step 104.

[0025] Step 104: determining dual correlation features between the target SAR features and the target optical features of each layer based on a dual correlation cross attention mechanism; The dual correlation features include high correlation features of target SAR features to target optical features, low correlation features of target SAR features to target optical features, high correlation features of target optical features to target SAR features, and low correlation features of target optical features to target SAR features, where high correlation features refer to features with a high degree of correlation, which can be understood as common information between modalities, and low correlation features refer to features with a low degree of correlation, which can be understood as unique information between modalities.

[0026] The specific steps of step 104 are as follows: The dual correlation features between the target SAR features and the target optical features of each layer determined based on the cross attention mechanism include: According to formula (1): (1) Calculate and obtain the query vector, key vector and value vector of the target optical feature to the target SAR feature; in, For the i Layer target optical characteristics, For the i Layer target SAR characteristics, For the i The size of the features generated by the layer encoding layer, B is the size of the batch, C i For the i The channel dimension size of the features generated by the layer encoding layer, H i For the i The size of the spatial width dimension of the features generated by the layer encoding layer, W i For the i The size of the high-dimensional space of the features generated by the layer encoding layer, is the query vector of the target optical feature to the target SAR feature, h is the number of multi-head attention layers, For the i Effect of layer target optical characteristics on target SAR characteristics h The query vector of the attention head; is the key vector of the target SAR feature, is the value vector of the target SAR feature, For the i SAR characteristics of layered targets h The key vector of the attention head, For the i SAR characteristics of layered targets h The value vector of the attention head, , and For different change matrices; According to formula (2): (2) Calculate and obtain the query vector, key vector and value vector of the target SAR feature to the target optical feature; in, is the query vector of the target SAR feature to the target optical feature, For the i The influence of SAR characteristics of layered targets on target optical characteristics h The query vector of the attention head; is the key vector of the target optical feature, is the value vector of the target optical feature, For the i Optical characteristics of layer targets h The key vector of the attention head, For the i Optical characteristics of layer targets h The value vector of the attention heads; Substitute the query vector of the target SAR feature to the target optical feature and the key vector of the target optical feature into formula (3): (3) Obtain the attention weight of the target SAR feature to the target optical feature; Substitute the query vector of the target optical feature to the target SAR feature and the key vector of the target SAR feature into formula (4): (4) Obtain the attention weight of the target optical feature to the target SAR feature; in, For Flatten and transpose to get, For Flatten and transpose to get, For Flatten and transpose to get, For Flatten and transpose to get, is the attention weight of the target SAR feature to the target optical feature, is the attention weight of the target optical feature to the target SAR feature; Substitute the attention weight of the target optical feature to the target SAR feature into formula (5): (5) Obtain high correlation characteristics of target optical characteristics to target SAR characteristics; Substitute the attention weight of the target SAR feature to the target optical feature into formula (6): (6) Obtain high correlation characteristics of target SAR characteristics to target optical characteristics; in, For Flatten and transpose to get, For Flatten and transpose to get, It is the high correlation feature of the target optical feature to the target SAR feature. It is a feature with high correlation between target SAR feature and target optical feature; Substitute the attention weight of the target optical feature to the target SAR feature into formula (7): (7) Obtaining low correlation characteristics of target optical characteristics to target SAR characteristics; Substitute the attention weight of the target SAR feature to the target optical feature into formula (8): (8) Obtaining low correlation characteristics of target SAR characteristics to target optical characteristics; in, is the low correlation characteristic of the target optical characteristics to the target SAR characteristics, It is a low correlation feature between the target SAR feature and the target optical feature.

[0027] Step 105: Fusing the dual correlation features to obtain target fused features; Specifically, step S1: cascading the high correlation features of the target SAR features to the target optical features and the high correlation features of the target optical features to the target SAR features of each layer to obtain the cascaded high correlation features. , , Indicates a cascade operation.

[0028] Step S2: Cascade the low correlation features of the target SAR features to the target optical features and the low correlation features of the target optical features to the target SAR features in each layer to obtain the cascaded low correlation features , .

[0029] Step S3: using a self-attention mechanism to perform weighted processing on the cascaded high-correlation features and the cascaded low-correlation features to obtain weighted high-correlation features and weighted low-correlation features; According to formula (9): (9) Calculate the weighted high correlation features; It is a weighted high correlation feature. It is the self-attention mechanism; According to formula (10): (10) The weighted low-correlation features are calculated. It is the weighted low correlation feature.

[0030] The calculation process of the self-attention mechanism SA is: First, given the input feature graph X, since the steps of weighting the high-correlation features after cascading are the same as those of weighting the low-correlation features after cascading, X is used here to represent both. The query vector, key vector, and value vector of the feature graph X are calculated according to formula (11): (11) in, Q is the query vector of feature graph X, K is the key vector of feature graph X, V is the value vector of the feature map X; it can be understood that the input feature map X is the feature map corresponding to the high correlation features after cascading and the low correlation features after cascading.

[0031] Then according to formula (12): (12) Calculating attention weights A ; in, For After flattening and transposing, For After flattening and transposing, d is the scaling factor.

[0032] Then according to formula (13): (13) Calculating weighted features .

[0033] in, For Flatten and transpose.

[0034] Finally, according to formula (14), residual connection calculation is performed with the original feature map to obtain the weighted features, as shown in formula (14): (14) in, is the original feature map, is the weighted feature of the output, is a learnable weight. It can be understood that the original feature map is a feature map corresponding to the high correlation feature after cascading and the low correlation feature after cascading.

[0035] Step S4: fusing the weighted high-correlation features and the weighted low-correlation features to obtain fused features; Step S5: Perform convolution operation on the fused features to obtain a convolution feature map; reduce the dimension of the target fused features to the target output channel number through convolution. .

[0036] Step S6: performing a batch normalization operation on the convolution feature map to obtain a normalized feature map; Step S7: Perform nonlinear mapping on the normalized feature map through the ReLU activation function to obtain the target fusion feature.

[0037] Specifically, according to formula (15), batch normalization and ReLU activation function are performed on the convolution feature map for nonlinear mapping, as shown in formula (15): (15) in, For a 1×1 convolutional layer, For the i Features output by the cross-modal feature fusion module.

[0038] Step 106: Determine the flood coverage area in the to-be-identified flood area based on the target fusion features.

[0039] The specific steps include: Step S10: using a multi-layer decoder to gradually upsample and decode the target fusion features to obtain a flood coverage area feature map of the flood area to be identified; The decoder can be a U-Net-like decoder. The main task of the decoder is to gradually upsample and decode the target fusion features obtained by the feature fusion module to generate the final segmentation output. Figure 2 As shown in the figure, the first three layers of the decoder are composed of an image expansion layer and a Swin-Transformer module. The image expansion layer is used to spatially upsample the feature map, improve the spatial resolution, and reduce the number of channels of the feature map. The upsampling rate of each upsampling is 2, and the number of channels is reduced to 1 / 2 of the input each time. The Swin-Transformer module is mainly used for feature decoding. j The layer decoding layer receives two inputs. j =1, receives the features output by the feature fusion module at the corresponding level And the skip connection features from the output of the previous fusion module ,when When , the decoding layer receives the output features of the decoding layer above it and the features output from the feature fusion module of the previous layer The decoding layer first performs skip connection features Perform average pooling and adjust the spatial size to obtain , and then and Cascading in the channel dimension, we get , then adjust the features The dimension in the channel dimension and the linear projection layer Linear is used to reduce the number of channels of the feature map to obtain Finally, the feature map Enter to j The Swin-Transformer module in the layer decoding layer performs feature decoding and extraction to obtain the decoded features .

[0040] like Figure 2 As shown in the figure, the target fusion features output by the feature fusion module are, from top to bottom: the first layer fusion features, the second layer fusion features, the third layer fusion features and the fourth layer fusion features; the decoder is, from bottom to top, the first layer decoding layer, the second layer decoding layer, the third layer decoding layer and the fourth layer decoding layer.

[0041] The specific step S10 includes the following steps: Step S101: For the first decoding layer: use average pooling AvgPool The third layer fusion features The spatial dimension is halved, and the number of channels is doubled using 1×1 convolution to obtain the first increased dimensional feature ; The first dimension feature is increased Fusion features with the fourth layer Cascade to get the first cascade feature ; The cascade splicing is performed on the channel dimension.

[0042] The first cascade feature Upsample and reduce the number of channels to obtain the first reduced dimensional feature ; Upsampling is performed through the image expansion layer, and the number of channels is reduced through the linear projection layer Linear. The upsampling multiple is 2, and the number of channels after reduction is 1 / 2 of the original.

[0043] The Swin-Transformer module in the first decoding layer is used to reduce the first dimension feature Perform feature decoding and extraction to obtain the output features of the first decoding layer ; Step S102: For the second decoding layer, use average pooling to fusion features of the second layer The spatial dimension of the corresponding feature map is halved, and the number of channels of the feature map corresponding to the second layer of fusion features is increased to twice the original number, obtaining the second increased dimension feature. ; The second dimension feature is increased The output features of the first decoding layer Cascade on the channel dimension to obtain the second cascade feature ; Reduce the second cascade feature The number of channels corresponding to the feature map is used to obtain the second reduced dimensional feature ; Specifically, the number of channels of the feature map corresponding to the second cascade feature is reduced to 1 / 2 of the current one through the linear projection layer; The second reduced dimensional feature is input into the Swin-Transformer module in the second decoding layer for feature decoding and extraction, and the output feature of the second decoding layer is obtained. .

[0044] Step S103: For the third decoding layer, use average pooling to fusion features of the first layer The spatial dimension of is halved, and the number of channels of the feature map corresponding to the first layer of fusion features is increased to twice the original number, and the third increased dimension feature is obtained. ; Add the third dimension feature The output features of the second decoding layer Cascade on the channel dimension to obtain the third cascade feature ; Reduce third cascade features The number of channels corresponding to the feature map is used to obtain the third reduced dimensional feature ; Specifically, the number of channels of the feature map corresponding to the third cascade feature is reduced to 1 / 2 of the current one through the linear projection layer; The third reduced dimensional feature is input into the Swin-Transformer module in the third decoding layer for feature decoding and extraction, and the output feature of the third decoding layer is obtained. .

[0045] The number of channels and spatial dimensions of the feature map corresponding to the output features of the third decoding layer are expanded to obtain the flood coverage area feature map of the flood area to be identified. Specifically, the number of channels is first expanded to 16 times the original through the linear mapping layer, and then the spatial dimension is expanded to 4 times the original through the rearrangement operation, and the number of channels is correspondingly reduced to the original size. The final output flood coverage area feature map of the flood area to be identified has a spatial resolution consistent with the input data. It can be understood that the input data is the upsampled SAR image and the preprocessed optical image.

[0046] Step S20: Segment the flood coverage area feature map of the flood area to be identified to obtain the flood coverage area in the flood area to be identified.

[0047] Specifically, a 1×1 convolutional layer is used to reduce the number of channels of the flood coverage area feature map of the flood area to be identified to the number of categories, which is 1 in this task, i.e., the flood coverage area. The feature map with the number of channels reduced to 1 is mapped to the probability space through the Sigmoid nonlinear activation function to obtain the flood coverage area in the flood area to be identified.

[0048] The beneficial effects of the present invention are: 1. The present invention uses SAR images and optical images of the same flood-affected area in the same phase to carry out research on a multimodal flood-inundated area identification method. Compared with the existing mainstream single-modality: flood coverage area identification method using SAR or optical images alone, the present invention can fully tap the complementary advantages between SAR images and optical images, overcome the limitations of single-modality images in flood-inundated area identification, and achieve more accurate identification performance.

[0049] 2. The method of the present invention is a coding and decoding structure, and both the encoder and the decoder use Swin-Transformer as the basic network. In view of the difference in information richness contained in optical images and SAR images, an asymmetric dual-branch encoder is designed. Among them, considering that optical images usually contain richer texture and detail information than SAR images, a deeper network is required to extract high-level features. Therefore, 6 Swin-Transformer modules are used in the third encoding layer of the optical encoder, while the SAR encoder only uses 2 Swin-Transformer modules at all levels. The deeper encoder structure can extract richer texture and semantic features of optical images, while the SAR encoder uses a relatively shallow encoder structure to capture the geometric structure and scattering characteristics of the data structure, ensuring effective feature extraction while reducing the number of parameters of the SAR encoder, improving the generalization ability of the model, and preventing overfitting.

[0050] 3. The present invention proposes a new feature fusion module, which aims to effectively fuse the features of optical images and SAR images to solve the problem of complementary information extraction in multimodal data. In particular, the feature fusion module includes a dual correlation cross attention fusion (DCCAF) module and a self attention (SA) module. DCCAF is used to extract complementary information between optical images and SAR images, which is different from the current cross attention method that can only focus on the high correlation features between multimodal data, while ignoring the unique features of each modality. The present invention constructs a new dual correlation cross attention mechanism (DCCAF), which includes two branches, one branch uses the cross self attention mechanism to extract high correlation features between modalities, and the other branch uses the complement cross attention mechanism to extract unique information with low correlation between modalities. In this way, the model can not only pay attention to the common information between modalities, but also pay attention to the unique information between modalities, and fully retain the complementary information between multimodal data, thereby improving the ability to distinguish and express multimodal features.

[0051] 4. The present invention uses the self-attention mechanism to fuse the high-correlation and low-correlation features between modalities obtained by the dual-correlation cross-attention fusion module. The self-attention mechanism is applied to the high-correlation and low-correlation features respectively to further enhance the feature expression ability. The introduction of the self-attention mechanism enables the model to automatically focus on important features during the fusion process, suppress the redundant and unimportant information between the high-correlation and low-correlation features of multimodal data, and help improve the model's discrimination ability and generalization performance. On the other hand, by performing self-attention enhancement on high-correlation and low-correlation features respectively, the model can better capture the complex dependencies in spatial and channel dimensions, improve the feature expression ability, and thus improve the accuracy of flood coverage area identification.

[0052] 5. The decoder established by the present invention is a U-Net-like structure, which fuses features at different levels through a multi-level step-by-step feature fusion module. Among them, the shallow high-resolution features provide fine spatial detail features, while the deep low-resolution features contain global semantic information. The cross-scale feature fusion enables the decoder to simultaneously learn the precise boundaries and overall structural features of the flood area, thereby generating high-quality recognition results.

[0053] Although the present invention is described herein in conjunction with various embodiments, in the process of implementing the claimed invention, those skilled in the art may understand and implement other variations of the disclosed embodiments by viewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "one" or "an" does not exclude multiple situations. A single processor or other unit may implement several functions listed in a claim. Certain measures are recorded in different dependent claims, but this does not mean that these measures cannot be combined to produce good results.

[0054] Although the present invention has been described in conjunction with specific features and embodiments thereof, it is apparent that various modifications and combinations may be made thereto without departing from the spirit and scope of the present invention. Accordingly, this specification and the accompanying drawings are merely exemplary illustrations of the present invention as defined by the appended claims and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present invention. Obviously, those skilled in the art may make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, the present invention is intended to include such modifications and variations if they fall within the scope of the claims of the present invention and their equivalents.

Claims

1. A multimodal flood area recognition method based on dual-correlation cross-attention, characterized in that: include: Acquire SAR images and optical images of the flood area to be identified; Performing image segmentation processing on the SAR image and the optical image respectively to obtain a first non-overlapping image block corresponding to the SAR image and a second non-overlapping image block corresponding to the optical image; Using a dual-branch encoder to encode the first non-overlapping image block and the second non-overlapping image block respectively, to obtain multi-layer target SAR features and multi-layer target optical features; Determine the dual correlation features between the target SAR features and the target optical features at each layer based on the dual correlation cross attention mechanism; Fusing the dual correlation features to obtain target fused features; The flood coverage area in the to-be-identified flood area is determined based on the target fusion features.

2. The multimodal flood area recognition method based on dual-correlation cross attention according to claim 1 is characterized in that: The dual correlation features include a high correlation feature of the target SAR feature to the target optical feature, a low correlation feature of the target SAR feature to the target optical feature, a high correlation feature of the target optical feature to the target SAR feature, and a low correlation feature of the target optical feature to the target SAR feature; The step of fusing the dual correlation features to obtain a target fusion feature comprises: Cascading high correlation features of target SAR features to target optical features and high correlation features of target optical features to target SAR features in each layer, to obtain cascaded high correlation features; Cascading the low correlation features of the target SAR features to the target optical features and the low correlation features of the target optical features to the target SAR features in each layer to obtain the cascaded low correlation features; A self-attention mechanism is used to perform weighted processing on the cascaded high-correlation features and the cascaded low-correlation features to obtain weighted high-correlation features and weighted low-correlation features; Fusing the weighted high-correlation features and the weighted low-correlation features to obtain fused features; Performing a convolution operation on the fused features to obtain a convolution feature map; Performing a batch normalization operation on the convolutional feature map to obtain a normalized feature map; The normalized feature map is nonlinearly mapped through the ReLU activation function to obtain the target fusion feature.

3. The multimodal flood area recognition method based on dual-correlation cross attention according to claim 1 is characterized in that: The performing image segmentation processing on the SAR image and the optical image respectively to obtain a first non-overlapping image block corresponding to the SAR image and a second non-overlapping image block corresponding to the optical image comprises: Segmenting the SAR image according to a preset block size to obtain a plurality of first non-overlapping image blocks; The optical image is segmented according to a preset block size to obtain a plurality of second non-overlapping image blocks.

4. The multimodal flood area recognition method based on dual-correlation cross attention according to claim 1 is characterized in that: The determining of the flood coverage area in the to-be-identified flood area based on the target fusion feature comprises: A multi-layer decoder is used to gradually upsample and decode the target fusion features to obtain a flood coverage area feature map of the flood area to be identified; The flood coverage area feature map of the flood area to be identified is segmented to obtain the flood coverage area in the flood area to be identified.

5. The multimodal flood area recognition method based on dual-correlation cross attention according to claim 4 is characterized in that: The target fusion features include first layer fusion features, second layer fusion features, third layer fusion features and fourth layer fusion features; The multi-layer decoder is used to gradually upsample and decode the target fusion features to obtain a flood coverage area feature map of the flood area to be identified, including: For the first decoding layer: the spatial dimension of the third layer fusion feature is halved, and the number of channels is increased to obtain a first increased dimension feature; Cascading the first dimension-increased feature and the fourth-layer fusion feature to obtain a first cascade feature; Performing upsampling and channel reduction processing on the first cascade features to obtain first reduced-dimensional features; Performing feature decoding and extraction processing on the first reduced-dimensional features to obtain output features of a first decoding layer; For the second decoding layer: the spatial dimension of the second layer fusion feature is halved and the number of channels is increased to obtain a second increased dimension feature; Cascading the second dimension-increased feature with the output feature of the first decoding layer in the channel dimension to obtain a second cascade feature; Adjusting the dimension of the second cascade feature in the channel dimension and reducing the number of channels to obtain a second reduced-dimensional feature; Decoding and extracting the second reduced-dimensional features to obtain output features of a second decoding layer; For the third decoding layer: the spatial dimension of the first layer fusion feature is halved and the number of channels is increased to obtain the third increased dimensional feature; Cascading the third dimension-added feature with the output feature of the second decoding layer in the channel dimension to obtain a third cascade feature; Adjusting the dimension of the third cascade feature in the channel dimension and reducing the number of channels to obtain a third reduced-dimensional feature; Performing feature decoding and extraction on the third reduced-dimensionality features to obtain output features of a third decoding layer; The number of channels and the spatial dimension of the output features of the third decoding layer are expanded to obtain a flood coverage area feature map of the flood area to be identified.

6. The multimodal flood area recognition method based on dual-correlation cross-attention according to claim 4 is characterized in that: The flood coverage area characteristic map of the flood area to be identified is segmented to obtain the flood coverage area in the flood area to be identified, including: The number of channels of the feature map of the flood coverage area in the flood area to be identified is reduced to 1, and the feature map with the number of channels reduced to 1 is mapped to the probability space through a nonlinear activation function to obtain the flood coverage area in the flood area to be identified.

7. The multimodal flood area recognition method based on dual-correlation cross-attention according to claim 1 is characterized in that: The dual-branch encoder includes a SAR encoder and an optical encoder, and each branch encoder includes 4 coding layers; the dual-branch encoder is used to encode the first non-overlapping image block and the second non-overlapping image block respectively, and the multi-layer target SAR features and multi-layer target optical features are obtained, including: For a first coding layer of the SAR encoder: mapping the first non-overlapping image block to an initial embedding space to obtain a first image block feature map; Performing self-attention calculation and feature transformation on the feature map of the first image block to obtain the SAR output feature of the current layer; For any coding layer from the second coding layer to the fourth coding layer of the SAR encoder: spatially downsample the SAR output features of the previous layer and increase the number of channels to obtain processed SAR output features; Perform self-attention calculation and feature transformation on the processed SAR output features to obtain the SAR output features of the current layer; Complete the encoding process of the four-layer SAR encoding layer to obtain the four-layer SAR output features; Reshape each layer of SAR output features into a four-dimensional tensor form to obtain four layers of target SAR features; For a first encoding layer of the optical encoder: mapping the second non-overlapping image block to an initial embedding space to obtain a second image block feature map; Performing self-attention calculation and feature transformation on the feature map of the second image block to obtain the optical output feature of the current layer; For any coding layer from the second coding layer to the fourth coding layer of the optical encoder: spatially downsampling the optical output features of the previous layer and increasing the number of channels to obtain processed optical output features; Perform self-attention calculation and feature transformation on the processed optical output features to obtain the optical output features of the current layer; Complete the encoding processing of four optical coding layers to obtain four layers of optical output features; The optical output features of each layer are reshaped into a four-dimensional tensor form to obtain four layers of target optical features.

8. The multimodal flood area recognition method based on dual-correlation cross-attention according to claim 1 is characterized in that: The dual correlation features between the target SAR features and the target optical features of each layer determined based on the dual correlation cross attention mechanism include: According to the formula: ； Calculate and obtain the query vector, key vector and value vector of the target optical feature to the target SAR feature; in, is the target optical feature of the i-th layer, is the SAR feature of the i-th layer target, The size of the feature generated for the i-th encoding layer, is the batch size, C i The channel dimension size of the feature generated by the i-th encoding layer, H i The size of the spatial width dimension of the feature generated by the i-th encoding layer, W i The size of the high-dimensional space for generating features for the i-th encoding layer, is the query vector of the target optical feature to the target SAR feature, h is the number of multi-head attention layers, is the query vector of the h attention heads of the target optical features at the i-th layer for the target SAR features; is the key vector of the target SAR feature, is the value vector of the target SAR feature, is the key vector of the h attention heads of the target SAR features at the i-th layer, is the value vector of the h attention heads of the SAR features of the i-th target, , and For different change matrices; According to the formula: ； Calculate and obtain the query vector, key vector and value vector of the target SAR feature to the target optical feature; in, is the query vector of the target SAR feature to the target optical feature, is the query vector of the h attention heads of the target SAR features at the i-th layer for the target optical features; is the key vector of the target optical feature, is the value vector of the target optical feature, is the key vector of the h attention heads of the target optical features at the i-th layer, is the value vector of the h attention heads of the target optical features at the i-th layer; Substitute the query vector of the target SAR feature to the target optical feature and the key vector of the target optical feature into: ； Obtain the attention weight of the target SAR feature to the target optical feature; Substitute the query vector of the target optical feature to the target SAR feature and the key vector of the target SAR feature into: ； Obtain the attention weight of the target optical feature to the target SAR feature; in, For Flatten and transpose to get, For Flatten and transpose to get, For Flatten and transpose to get, For Flatten and transpose to get, is the attention weight of the target SAR feature to the target optical feature, is the attention weight of the target optical feature to the target SAR feature; Substitute the attention weight of the target optical feature to the target SAR feature into: ； Obtain high correlation characteristics of target optical characteristics to target SAR characteristics; Substitute the attention weight of the target SAR feature to the target optical feature: ； Obtain high correlation characteristics of target SAR characteristics to target optical characteristics; in, For Flatten and transpose to get, For Flatten and transpose to get, It is the high correlation feature of the target optical feature to the target SAR feature. It is a feature with high correlation between target SAR feature and target optical feature; Substitute the attention weight of the target optical feature to the target SAR feature into: ； Obtaining low correlation characteristics of target optical characteristics to target SAR characteristics; Substitute the attention weight of the target SAR feature to the target optical feature: ； Obtaining low correlation characteristics of target SAR characteristics to target optical characteristics; in, is the low correlation characteristic of the target optical characteristics to the target SAR characteristics, It is a low correlation feature between the target SAR feature and the target optical feature.

9. The multimodal flood area recognition method based on dual-correlation cross-attention according to claim 7 is characterized in that: The depth of the Swin-Transformer module used in the third coding layer of the SAR encoder is less than the depth of the Swin-Transformer module used in the third coding layer of the optical encoder.

10. The multimodal flood area recognition method based on dual-correlation cross-attention according to claim 1 is characterized in that: The obtaining of SAR images and optical images in the flood area to be identified includes: Obtain SAR images and optical images at the same time phase after the flood occurs; The SAR image is upsampled to obtain an upsampled SAR image, and the spatial resolution of the upsampled SAR image is the same as the spatial resolution of the optical image.

Images