A non-aligned SAR-optical image coupled contrast fusion building segmentation method

Abstract

The present invention relates to a non-aligned SAR-optical image coupled contrast fusion building segmentation method. The present invention comprises the following steps: constructing a multimodal dataset for building remote sensing image segmentation; constructing a non-aligned SAR-optical image coupled contrast fusion building segmentation model; training a non-aligned SAR-optical image coupled contrast fusion building segmentation model; and obtaining the result of the non-aligned SAR-optical image coupled contrast fusion building segmentation model. Compared with the prior art, the present invention solves the problem that the prior method focuses on strict modal alignment and ignores the gap between modalities when dealing with the weak alignment problem of multimodal image pairs. The present invention captures the optimal alignment position by guiding the adaptive feature alignment module through offset, thereby realizing adaptive alignment between modalities. Further, the complementary features between modalities are extracted through the coupled contrast learning framework of multimodal image fusion, thereby reducing the redundant information of the fusion result and ensuring the accuracy and robustness of the segmentation result.

Description

Technical Field

[0001] The invention relates to the technical field of multimodal remote sensing image processing, in particular to a non-alignment resistant SAR-optical image coupled contrast fusion building segmentation method. Background Art

[0002] In recent years, with the continuous advancement of remote sensing technology, the multimodal fusion of synthetic aperture radar (SAR) and optical images has received widespread attention in the application field of building segmentation. Multimodal remote sensing images combine information obtained by different sensors and can provide richer ground features and spatial information, providing strong support for urban planning, environmental monitoring and disaster assessment. However, despite the obvious advantages of multimodal remote sensing images in information acquisition, their processing and analysis still face many challenges.

[0003] Traditional remote sensing image segmentation methods usually rely on single-modal data and fail to fully utilize the complementarity of multimodal information, resulting in poor segmentation results in complex scenes. In addition, the alignment problem between modalities also brings additional difficulties to the segmentation task, affecting the accuracy and reliability of the segmentation results.

[0004] In order to meet these challenges, researchers have gradually explored multimodal image segmentation methods based on deep learning. However, existing multimodal learning frameworks often lack effective feature alignment and information fusion mechanisms, resulting in insufficient knowledge transfer between the segmentation model and the fusion model, affecting the detail retention of the segmentation results and the consistency of cross-modal features. Therefore, a new method is urgently needed that can effectively deal with the weak alignment problem between modalities and enhance the accuracy and robustness of the segmentation results.

[0005] In response to this demand, the present invention proposes a non-aligned SAR-optical image coupled contrast fusion building segmentation method. This innovative method makes full use of the information between modalities and captures the best alignment position by offset-guided adaptive feature alignment module, thereby achieving adaptive alignment between modalities. Further, coupled contrast learning is used to extract complementary features between modalities, thereby reducing the redundant information of the fusion result and obtaining a better fusion result, which promotes the further development of multi-source remote sensing data fusion processing and image segmentation tasks. Summary of the invention

[0006] The purpose of the present invention is to overcome the limitations of existing methods, solve the weak alignment problem of multimodal image fusion and the information redundancy problem of fusion results, and provide a non-aligned SAR-optical image coupled contrast fusion building segmentation method.

[0007] In order to achieve the above object, the technical solution of the present invention is as follows:

[0008] A non-aligned SAR-optical image coupled contrast fusion building segmentation method specifically comprises the following steps:

[0009] 11) Constructing a multimodal dataset for building remote sensing image segmentation: Obtaining paired SAR images and optical imaging and the corresponding building label mask Used to construct the building segmentation task dataset, divide the training set and validation set in a ratio of 7:3; further randomly crop and randomly flip the building segmentation task dataset to expand the dataset;

[0010] 12) Construct a non-aligned SAR-optical image coupled contrast fusion building segmentation model: The model consists of a multimodal dual-branch feature extraction module, an offset-guided adaptive feature alignment module, and a coupled contrast learning framework for multimodal image fusion;

[0011] 13) Training the SAR-optical image coupled contrast fusion building segmentation model that is resistant to non-alignment: Use the divided training set to train the parameters of the building segmentation model constructed in 12) until the training reaches 100 epochs, and finally retain the trained network parameters;

[0012] 14) Obtaining the results of the SAR-optical image coupled contrast fusion building segmentation model that is resistant to non-alignment: Input the acquired real SAR and optical image data to be fused into the trained SAR-optical image coupled contrast fusion building segmentation model to obtain the segmentation results.

[0013] The construction of the non-aligned SAR-optical image coupled contrast fusion building segmentation model specifically includes the following steps:

[0014] 21) Construct a multimodal dual-branch feature extraction module for extracting modality-specific features, which includes a SAR image encoder and an optical image encoder;

[0015] 211) Construct a SAR image encoder for extracting SAR image features, which includes three consecutive downsampling structures. The downsampling structure is specifically: a batch normalization layer, a ReLU layer, a convolution kernel size of A convolutional layer, a batch normalization layer, a Relu layer, and a convolution kernel size of A convolutional layer, a batch normalization layer, a Relu layer, and a convolution kernel size of The convolutional layer;

[0016] 212) An optical image encoder for extracting optical image features is constructed, which includes three consecutive downsampling structures. The downsampling structures are as follows: A convolutional layer, a Relu layer, a batch normalization layer, a A convolutional layer, a Relu layer, and a batch normalization layer;

[0017] 22) Construct an offset-guided adaptive feature alignment module for guiding multimodal feature alignment, which includes a cross-modal spatial offset estimation submodule, an offset-guided deformable alignment and fusion submodule, and an upsampling submodule;

[0018] 221) Construct a cross-modal spatial offset estimation submodule for accurately estimating feature-level offsets, which includes an invariant feature encoder, a specific feature encoder, a spatial attention mechanism, a channel attention mechanism, and a Convolutional layer and a Relu activation function;

[0019] 2211) Construct an invariant feature encoder, which includes a convolutional layer and a SiLU activation function;

[0020] 2212) Construct a specific feature encoder, which includes a Convolutional layer and a C3 module, the C3 module consists of three Convolutional layers and bottleneck layers in Resnet50;

[0021] 222) Construct an offset-guided deformable alignment and fusion submodule, which includes a deformable convolution layer, a bilinear interpolation operation, and a Convolution and two concatenation operations;

[0022] 223) Construct an upsampling submodule, which includes three consecutive upsampling structures. The upsampling structure has three layers, which are: Deconvolution layer, a Relu layer, and a batch normalization layer;

[0023] 23) Construct a coupled contrastive learning framework for multimodal image fusion, which includes a Vgg-19 network with pre-trained weights, a contrast loss function for improving foreground saliency, a contrast loss function for preserving background details of optical images, and a segmentation submodule;

[0024] 231) Construct a contrast loss function for improving foreground saliency, where the foreground features of SAR images are set as positive samples and the foreground features of optical images are set as negative samples;

[0025] 232) Construct a contrast loss function for preserving the background details of optical images. The background features of optical images are set as positive samples, and the background features of SAR images are set as negative samples.

[0026] 233) Construct a segmentation submodule, which includes a fully connected layer and a sigmoid activation function.

[0027] The training of the non-aligned SAR-optical image coupled contrast fusion building segmentation model specifically includes the following steps:

[0028] 31) The training of the SAR-optical image coupled contrast fusion building segmentation model that is resistant to non-alignment includes two stages. The first stage is the training stage of coupled contrast learning. The multimodal dual-branch feature extraction module and the offset-guided adaptive feature alignment module are obtained through pre-training of the remote sensing image segmentation multimodal dataset, which can fully integrate the complementary features in the SAR image and the optical image. The second stage is the fine-tuning stage of the SAR-optical image coupled contrast fusion building segmentation model that is resistant to non-alignment. The entire model is fine-tuned through the remote sensing image segmentation multimodal dataset to make the segmentation result better. The training of the SAR-optical image coupled contrast fusion building segmentation model that is resistant to non-alignment includes the following steps:

[0029] 32) Input the SAR image and optical image in the preprocessed data set into the multimodal dual-branch feature extraction module to obtain SAR image features and optical image features, which specifically includes the following steps:

[0030] 321) After the SAR image is input into the SAR image encoder, three consecutive downsampling structures are executed to obtain the SAR image features. ;

[0031] 322) After the optical image is input into the optical image encoder, three consecutive downsampling structures are performed to obtain the optical image features. ;

[0032] 33) SAR image features and optical imaging features The input offset guided adaptive feature alignment module is used to guide the multimodal feature alignment, which specifically includes the following steps:

[0033] 331) Execute a cross-modal spatial offset estimation submodule for accurately estimating feature-level offsets, specifically comprising the following steps:

[0034] 3311) SAR image characteristics and optical imaging features After the invariant feature encoder with shared parameters, convolution operation and SiLU activation function operation are performed to obtain the shared features of SAR images. , Optical image sharing features ;

[0035] 3312) SAR image characteristics and optical imaging features After passing through the specific feature encoder, the convolution operation and the C3 module are performed to obtain the specific features of the SAR image. , optical image specific features ;

[0036] 3313) SAR image sharing features After the spatial attention module, the SAR images share features at the same time Perform maximum pooling and average pooling, concatenate the maximum pooling output and the average pooling output, and then perform the concatenation Convolution, then execute the sigmoid function, the probability matrix obtained Shares features with SAR images Perform point multiplication to obtain spatially enhanced SAR image features ;

[0037] 3314) Optical image sharing features After the spatial attention module, the optical image shares features at the same time Perform the maximum pooling operation and the average pooling operation, concatenate the two pooling results, and then perform the concatenated result Convolution, and then through the sigmoid function, the probability matrix obtained Shares features with optical images Perform point multiplication to obtain spatially enhanced optical image features ;

[0038] 3315) Characteristics of spatially enhanced SAR images and spatially enhanced optical imaging features Perform the splicing operation to obtain the changed features , the characteristics of the changes After the channel attention module, Perform maximum pooling and average pooling, then pass the output of maximum pooling and the output of average pooling through a multi-layer perceptron network, then add the outputs of the multi-layer perceptron network, and pass the result of the addition through a sigmoid function to obtain an attention map , attention map and spatially enhanced SAR image features Perform matrix multiplication to get , attention map and spatially enhanced optical imaging features Perform matrix multiplication to get , and spatially enhanced SAR image features Adding the spatial and channel enhanced SAR image features , and spatially enhanced optical imaging features Adding the spatial and channel enhanced optical image features , formulated as follows:

[0039] ,

[0040] ,

[0041] in, is a multi-layer perceptron network, is the maximum pooling operation, is the average pooling operation;

[0042] 3316) SAR image features for spatial and channel enhancement , spatial and channel enhanced optical imaging features Perform the concatenation operation, then the convolution operation and the Relu activation function operation to get the offset ;

[0043] 332) executing an offset-guided deformable alignment and fusion submodule for adaptive alignment;

[0044] 3321) Sharing features of optical images Perform deformable convolution to obtain adaptively aligned optical image features , the deformable convolution formula is as follows:

[0045] ,

[0046] in, is the value of the corresponding position of the feature after alignment, is the kernel weight, It is The kernel weights, is the convolution operation, is the location of the central sampling point, It is A fixed offset, It is for Locations compensation, is the offset from learned modulation scalar;

[0047] 3322) Optical image features for adaptive alignment Perform bilinear interpolation to obtain the complete optical image features for adaptive alignment ;

[0048] 3323) Complete optical image features for adaptive alignment Shares features with SAR images Execute the splicing operation, then execute Convolution, get the output ,right , optical image specific features and SAR image specific features Perform the splicing operation to obtain the fused features , formulated as follows:

[0049] ,

[0050] in, yes convolution, It is a splicing operation;

[0051] 333) Execute the upsampling submodule, perform 3 consecutive upsampling structures, the upsampling structure includes a Deconvolution layer, a Relu layer, and a batch normalization layer to get the fused image ;

[0052] 34) A coupled contrastive learning framework for multimodal image fusion;

[0053] 341) Optical imaging , SAR images and fused image Label mask Perform pixel-level multiplication to obtain the optical foreground image , SAR foreground image and the prospect of fused images ;

[0054] Optical imaging , SAR images and fused image With background mask Perform pixel-level multiplication to obtain the optical background image , SAR background image and the background of the fused image ;

[0055] Optical foreground image , SAR foreground image , the prospect of fusion image , optical background image , SAR background image , the background of the fused image Cut Image blocks;

[0056] Import these image blocks into the Vgg-19 network with pre-trained weights to obtain the foreground features of the fused image , SAR image foreground features , optical image foreground features , fusion image background features , Background characteristics of SAR images , background characteristics of optical images , Indicates negative samples;

[0057] 342) Constructing a coupled contrastive learning loss function , Contrastive loss function for improving foreground saliency is included and a contrast loss function for preserving background details of optical images , , and The formula is as follows:

[0058] ,

[0059] ,

[0060] ,

[0061] in, Indicates negative samples, represents the L1 norm, are learnable weights, represents the coupled contrastive learning loss function, represents the contrast loss function used to improve foreground saliency, represents the contrast loss function used to preserve the background details of the optical image;

[0062] During the training process, according to The formula calculates the loss value, calculates the gradient of the loss function, back-propagates the error from the output layer back to each layer of the network, determines the gradient vector by back-propagating the loss value, and updates the model parameters of the coupled contrast learning stage of the anti-non-aligned SAR-optical image coupled contrast fusion building segmentation model;

[0063] Determine whether 100 epochs have been reached. If so, the training of the coupled contrast learning phase of the anti-non-aligned SAR-optical image coupled contrast fusion building segmentation model is completed. Otherwise, continue training.

[0064] 35) Based on the training in the coupled contrastive learning stage, a pre-trained multimodal dual-branch feature extraction module and an offset-guided adaptive feature alignment module are obtained. In the fine-tuning stage, a segmentation submodule in the coupled contrastive learning framework of multimodal image fusion is added after the offset-guided adaptive feature alignment module for supervised training;

[0065] 36) Input the SAR images and optical images in the dataset into the anti-non-aligned SAR-optical image coupled contrast fusion building segmentation model, perform supervised fine-tuning learning, and train a remote sensing image building segmentation model with a multimodal dual-branch feature extraction module, an offset-guided adaptive feature alignment module, and a multimodal image fusion coupled contrast learning framework, specifically including the following steps:

[0066] 361) Input the SAR image and optical image in the preprocessed data set into the multimodal dual-branch feature extraction module, and perform the SAR image encoder and optical image encoder operations respectively to obtain the SAR image feature and optical imaging features ;

[0067] 362) SAR image features and optical imaging features Input to the offset-guided adaptive feature alignment module to guide multimodal feature alignment and fusion to obtain a fused image ;

[0068] 363) Fusion image The segmentation submodule in the coupled contrast learning framework of multimodal image fusion is input to perform full connection operation and softmax activation function operation to perform predictive segmentation, which is formulated as follows:

[0069] ,

[0070] in, represents the fully connected layer operation, Represents the softmax activation function operation, Represents the predicted segmentation result;

[0071] 37) Cross entropy loss is used as the loss function to optimize the network segmentation accuracy. The cross entropy loss function expression is as follows:

[0072] ,

[0073] in, is the number of training samples, The basic truth value samples, The predicted value is samples;

[0074] 38) By calculating the gradient of the loss function, the error is back-propagated from the output layer to each layer of the network, the gradient vector is determined by back-propagating the loss value, and the model parameters of the fine-tuning stage of the anti-non-aligned SAR-optical image coupled contrast fusion building segmentation model are updated;

[0075] It is determined whether the set number of training rounds of 100 is reached. If so, the training of the fine-tuning stage of the anti-non-aligned SAR-optical image coupled contrast fusion building segmentation model is completed, otherwise the training continues. Beneficial Effects

[0076] The SAR-optical image coupled contrast fusion building segmentation method proposed in this invention is highly innovative and superior. First, by constructing an offset-guided adaptive feature alignment module, the problem of over-reliance on modal alignment in traditional methods when processing multimodal images is solved, and the information between modalities is fully utilized to achieve adaptive alignment. This module can adaptively capture the optimal alignment position, thereby improving the alignment quality and the accuracy of the fusion results.

[0077] Secondly, the complementary features between modalities are extracted through the coupled contrast learning framework of multimodal image fusion, which effectively enhances the robustness and accuracy of the segmentation model. This framework can extract complementary features between modalities, thereby reducing the redundant information of the fusion result and obtaining a better fusion result. In addition, the present invention gives full play to the advantages of SAR and optical imaging in multimodal remote sensing data fusion, improves the processing ability of complex scenes, and has broad application prospects, especially in the fields of urban monitoring, environmental protection and disaster response, showing good practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 It is a non-aligned SAR-optical image coupled contrast fusion building segmentation method;

[0079] Figure 2 It is a model framework diagram of the coupled contrast learning stage of the SAR-optical image coupled contrast fusion building segmentation method that resists non-alignment;

[0080] Figure 3 This is a model framework diagram of the fine-tuning stage of the SAR-optical image coupled contrast fusion building segmentation method that is resistant to misalignment;

[0081] Figure 4 It is a framework diagram of the multimodal dual-branch feature extraction module and the offset-guided adaptive feature alignment module;

[0082] Figure 5 It is a coupled contrastive learning framework diagram for multimodal fusion. DETAILED DESCRIPTION

[0083] In order to have a further understanding and recognition of the structural features and the effects achieved by the present invention, a preferred embodiment and accompanying drawings are used for detailed description as follows:

[0084] like Figure 1 As shown, the anti-non-alignment SAR-optical image coupled contrast fusion building segmentation method described in the present invention is characterized by comprising the following steps:

[0085] The first step is to construct a multimodal dataset for building remote sensing image segmentation:

[0086] Acquire paired SAR images and optical imaging and the corresponding building label mask Used to construct the building segmentation task dataset, divide the training set and validation set in a ratio of 7:3; further randomly crop and randomly flip the building segmentation task dataset to expand the dataset;

[0087] The second step is to build a non-aligned SAR-optical image coupled contrast fusion building segmentation model:

[0088] The specific steps are as follows:

[0089] (1) Constructing a multimodal dual-branch feature extraction module for extracting modality-specific features, which includes a SAR image encoder and an optical image encoder;

[0090] (1-1) Construct a SAR image encoder for extracting SAR image features, which includes three consecutive downsampling structures. The downsampling structure is as follows: a batch normalization layer, a ReLU layer, and a convolution kernel size of A convolutional layer, a batch normalization layer, a Relu layer, and a convolution kernel size of A convolutional layer, a batch normalization layer, a Relu layer, and a convolution kernel size of The convolutional layer;

[0091] (1-2) Constructing an optical image encoder for extracting optical image features, which includes three consecutive downsampling structures. The downsampling structures are as follows: A convolutional layer, a Relu layer, a batch normalization layer, a A convolutional layer, a Relu layer, and a batch normalization layer;

[0092] (2) Constructing an offset-guided adaptive feature alignment module for guiding multimodal feature alignment, which includes a cross-modal spatial offset estimation submodule, an offset-guided deformable alignment and fusion submodule, and an upsampling submodule;

[0093] (2-1) Construct a cross-modal spatial offset estimation submodule for accurately estimating feature-level offsets, which includes an invariant feature encoder, a specific feature encoder, a spatial attention mechanism, a channel attention mechanism, and a Convolutional layer and a Relu activation function;

[0094] (2-1-1) Construct an invariant feature encoder, which includes a convolutional layer and a SiLU activation function;

[0095] (2-1-2) Construct a specific feature encoder, which includes a Convolutional layer and a C3 module, the C3 module consists of three Convolutional layers and bottleneck layers in Resnet50;

[0096] (2-2) Construct an offset-guided deformable alignment and fusion submodule, which includes a deformable convolution layer, a bilinear interpolation operation, and a Convolution and two concatenation operations;

[0097] (2-3) Construct an upsampling submodule, which includes three consecutive upsampling structures. The upsampling structure has three layers, which are: Deconvolution layer, a Relu layer, and a batch normalization layer;

[0098] (3) Construct a coupled contrastive learning framework for multimodal image fusion, which consists of a Vgg-19 network with pre-trained weights, a contrastive loss function for improving foreground saliency, a contrastive loss function for preserving background details of optical images, and a segmentation submodule;

[0099] (3-1) Construct a contrast loss function for improving foreground saliency, where the foreground features of SAR images are set as positive samples and the foreground features of optical images are set as negative samples;

[0100] (3-2) Constructing a contrast loss function for preserving the background details of optical images. The background features of optical images are set as positive samples, and the background features of SAR images are set as negative samples.

[0101] (3-3) Construct a segmentation submodule, which includes a fully connected layer and a sigmoid activation function;

[0102] The third step is to train a non-aligned SAR-optical image coupled contrast fusion building segmentation model:

[0103] The specific steps are as follows:

[0104] (1) The training of the SAR-optical image coupled contrast fusion building segmentation model that is resistant to non-alignment includes two stages. The first stage is the training stage of coupled contrast learning. The multimodal dual-branch feature extraction module and the offset-guided adaptive feature alignment module are obtained through pre-training on the multimodal dataset of remote sensing image segmentation, which can fully integrate the complementary features in SAR images and optical images. The second stage is the fine-tuning stage of the SAR-optical image coupled contrast fusion building segmentation model that is resistant to non-alignment. The entire model is fine-tuned through the multimodal dataset of remote sensing image segmentation to make the segmentation result better. The training of the SAR-optical image coupled contrast fusion building segmentation model that is resistant to non-alignment includes the following steps:

[0105] (2) The SAR image and optical image in the preprocessed data set are input into the multimodal dual-branch feature extraction module to obtain SAR image features and optical image features, which specifically includes the following steps:

[0106] (2-1) After the SAR image is input into the SAR image encoder, three consecutive downsampling structures are executed to obtain the SAR image features. ;

[0107] (2-2) After the optical image is input into the optical image encoder, three consecutive downsampling structures are performed to obtain the optical image features. ;

[0108] (3) SAR image features and optical imaging features The input offset guided adaptive feature alignment module is used to guide the multimodal feature alignment, which specifically includes the following steps:

[0109] (3-1) Execute a cross-modal spatial offset estimation submodule to accurately estimate the feature-level offset, which specifically includes the following steps:

[0110] (3-1-1) SAR image characteristics and optical imaging features After the invariant feature encoder with shared parameters, convolution operation and SiLU activation function operation are performed to obtain the shared features of SAR images. , Optical image sharing features ;

[0111] (3-1-2) SAR image characteristics and optical imaging features After passing through the specific feature encoder, the convolution operation and the C3 module are performed to obtain the specific features of the SAR image. , optical image specific features ;

[0112] (3-1-3) SAR image sharing features After the spatial attention module, the SAR images share features at the same time Perform maximum pooling and average pooling, concatenate the maximum pooling output and the average pooling output, and then perform the concatenation Convolution, then execute the sigmoid function, the probability matrix obtained Shares features with SAR images Perform point multiplication to obtain spatially enhanced SAR image features ;

[0113] (3-1-4) Optical image sharing features After the spatial attention module, the optical image shares features at the same time Perform the maximum pooling operation and the average pooling operation, concatenate the two pooling results, and then perform the concatenated result Convolution, and then through the sigmoid function, the probability matrix obtained Shares features with optical images Perform point multiplication to obtain spatially enhanced optical image features ;

[0114] (3-1-5) Characteristics of spatially enhanced SAR images and spatially enhanced optical imaging features Perform the splicing operation to obtain the changed features , the characteristics of the changes After the channel attention module, Perform maximum pooling and average pooling, then pass the output of maximum pooling and the output of average pooling through a multi-layer perceptron network, then add the outputs of the multi-layer perceptron network, and pass the result of the addition through a sigmoid function to obtain an attention map , attention map and spatially enhanced SAR image features Perform matrix multiplication to get , attention map and spatially enhanced optical imaging features Perform matrix multiplication to get , and spatially enhanced SAR image features Adding the spatial and channel enhanced SAR image features , and spatially enhanced optical imaging features Adding the spatial and channel enhanced optical image features , formulated as follows:

[0115] ,

[0116] ,

[0117] in, is a multi-layer perceptron network, is the maximum pooling operation, is the average pooling operation;

[0118] (3-1-6) SAR image features enhanced in space and channels , spatial and channel enhanced optical imaging features Perform the concatenation operation, then the convolution operation and the Relu activation function operation to get the offset ;

[0119] (3-2) executing the offset-guided deformable alignment and fusion submodule for adaptive alignment;

[0120] (3-2-1) Sharing features of optical images Perform deformable convolution to obtain adaptively aligned optical image features , the deformable convolution formula is as follows:

[0121] ,

[0122] in, is the value of the corresponding position of the feature after alignment, is the kernel weight, It is The kernel weights, is the convolution operation, is the location of the central sampling point, It is A fixed offset, It is for Locations compensation, is the offset from learned modulation scalar;

[0123] (3-2-2) Optical image features for adaptive alignment Perform bilinear interpolation to obtain the complete optical image features for adaptive alignment ;

[0124] (3-2-3) Complete optical image features for adaptive alignment Shares features with SAR images Execute the splicing operation, then execute Convolution, get the output ,right , optical image specific features and SAR image specific features Perform the splicing operation to obtain the fused features , formulated as follows:

[0125] ,

[0126] in, yes convolution, It is a splicing operation;

[0127] (3-3) Execute the upsampling submodule and perform three consecutive upsampling structures. The upsampling structure includes a Deconvolution layer, a Relu layer, and a batch normalization layer to get the fused image ;

[0128] (4) A coupled contrastive learning framework for performing multimodal image fusion;

[0129] (4-1) Optical image , SAR images and fused image Label mask Perform pixel-level multiplication to obtain the optical foreground image , SAR foreground image and the prospect of fused images ;

[0130] Optical imaging , SAR images and fused image With background mask Perform pixel-level multiplication to obtain the optical background image , SAR background image and the background of the fused image ;

[0131] Optical foreground image , SAR foreground image , the prospect of fusion image , optical background image , SAR background image , the background of the fused image Cut Image blocks;

[0132] Import these image blocks into the Vgg-19 network with pre-trained weights to obtain the foreground features of the fused image , SAR image foreground features , optical image foreground features , fusion image background features , Background characteristics of SAR images , background characteristics of optical images , Indicates negative samples;

[0133] (4-2) Constructing coupled contrastive learning loss function , Contrastive loss function for improving foreground saliency is included and a contrast loss function for preserving background details of optical images , , and The formula is as follows:

[0134] ,

[0135] ,

[0136] ,

[0137] in, Indicates negative samples, represents the L1 norm, are learnable weights, represents the coupled contrastive learning loss function, represents the contrast loss function used to improve foreground saliency, represents the contrast loss function used to preserve the background details of the optical image;

[0138] During the training process, according to The formula calculates the loss value, calculates the gradient of the loss function, back-propagates the error from the output layer back to each layer of the network, determines the gradient vector by back-propagating the loss value, and updates the model parameters of the coupled contrast learning stage of the anti-non-aligned SAR-optical image coupled contrast fusion building segmentation model;

[0139] Determine whether 100 epochs have been reached. If so, the training of the coupled contrast learning phase of the anti-non-aligned SAR-optical image coupled contrast fusion building segmentation model is completed. Otherwise, continue training.

[0140] (5) Based on the training in the coupled contrastive learning stage, a pre-trained multimodal dual-branch feature extraction module and an offset-guided adaptive feature alignment module are obtained. In the fine-tuning stage, a segmentation submodule in the coupled contrastive learning framework of multimodal image fusion is added after the offset-guided adaptive feature alignment module for supervised training;

[0141] (6) The SAR images and optical images in the dataset are input into the SAR-optical image coupled contrast fusion building segmentation model that is resistant to misalignment, and supervised fine-tuning learning is performed to train a remote sensing image building segmentation model with a multimodal dual-branch feature extraction module, an offset-guided adaptive feature alignment module, and a multimodal image fusion coupled contrast learning framework, which specifically includes the following steps:

[0142] (6-1) Input the SAR image and optical image in the preprocessed data set into the multimodal dual-branch feature extraction module, and execute the SAR image encoder and optical image encoder operations respectively to obtain the SAR image feature and optical imaging features ;

[0143] (6-2) SAR image features and optical imaging features Input to the offset-guided adaptive feature alignment module to guide multimodal feature alignment and fusion to obtain a fused image ;

[0144] (6-3) The fused image The segmentation submodule in the coupled contrast learning framework of multimodal image fusion is input to perform full connection operation and softmax activation function operation to perform predictive segmentation, which is formulated as follows:

[0145] ,

[0146] in, represents the fully connected layer operation, Represents the softmax activation function operation, Represents the predicted segmentation result;

[0147] (7) The cross entropy loss is used as the loss function to optimize the network segmentation accuracy. The cross entropy loss function expression is as follows:

[0148] ,

[0149] in, is the number of training samples, The basic truth value samples, The predicted value is samples;

[0150] (8) By calculating the gradient of the loss function, the error is back-propagated from the output layer to each layer of the network, the gradient vector is determined by back-propagating the loss value, and the model parameters of the fine-tuning stage of the non-aligned SAR-optical image coupled contrast fusion building segmentation model are updated;

[0151] Determine whether the set number of training rounds of 100 is reached. If so, the training of the fine-tuning stage of the anti-non-aligned SAR-optical image coupled contrast fusion building segmentation model is completed, otherwise, continue training;

[0152] Step 4: Obtain the results of the SAR-optical image coupling contrast fusion building segmentation model that is resistant to non-alignment:

[0153] The acquired real SAR and optical image data to be fused are input into the trained anti-non-alignment SAR-optical image coupled contrast fusion building segmentation model to obtain the segmentation result.

[0154] The proposed method of building segmentation by coupled contrast fusion of SAR-optical images that is resistant to misalignment significantly improves the accuracy and robustness of the segmentation results, and effectively solves the challenges brought by weak alignment between modalities. Compared with the existing remote sensing image segmentation technology, this method fully utilizes the information between modalities by constructing an offset-guided adaptive feature alignment module, thereby improving the alignment quality and enhancing the accuracy of the fusion results. In addition, this method gives full play to the advantages of SAR and optical images in multimodal data fusion, extracts complementary features between modalities through coupled contrast learning, reduces redundant information, and further improves the processing capability and stability of the model. This method shows excellent practical value and broad application prospects, especially in the fields of urban monitoring, environmental protection and disaster response.

[0155] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.

Claims

1. A non-aligned SAR-optical image coupled contrast fusion building segmentation method, characterized in that: The specific steps include: 11) Construct a multimodal dataset for building remote sensing image segmentation: Obtain paired SAR images S and optical images O and the corresponding building label masks M to construct a building segmentation task dataset, and divide the training set and validation set into a ratio of 7:3; further randomly crop and randomly flip the building segmentation task dataset to expand the dataset; 12) Construct a non-aligned SAR-optical image coupled contrast fusion building segmentation model: The model consists of a multimodal dual-branch feature extraction module, an offset-guided adaptive feature alignment module, and a coupled contrast learning framework for multimodal image fusion; the specific steps are as follows: 121) constructing a multimodal dual-branch feature extraction module for extracting modality-specific features, which includes a SAR image encoder and an optical image encoder; 1211) constructing a SAR image encoder for extracting SAR image features, which includes three consecutive downsampling structures, and the downsampling structure is specifically: a batch normalization layer, a Relu layer, a convolution layer with a convolution kernel size of 1×1, a batch normalization layer, a Relu layer, a convolution layer with a convolution kernel size of 3×3, a batch normalization layer, a Relu layer, and a convolution layer with a convolution kernel size of 1×1; 1212) constructing an optical image encoder for extracting optical image features, which includes three consecutive downsampling structures, and the downsampling structure is specifically: a 3×3 convolution layer, a Relu layer, a batch normalization layer, a 3×3 convolution layer, a Relu layer, and a batch normalization layer; 122) constructing an offset-guided adaptive feature alignment module for guiding multimodal feature alignment, which includes a cross-modal spatial offset estimation submodule, an offset-guided deformable alignment and fusion submodule, and an upsampling submodule; 1221) constructing a cross-modal spatial offset estimation submodule for accurately estimating feature-level offsets, which includes an invariant feature encoder, a specific feature encoder, a spatial attention mechanism, a channel attention mechanism, a 3×3 convolutional layer, and a Relu activation function; 12211) constructing an invariant feature encoder comprising a convolutional layer and a SiLU activation function; 12212) construct a specific feature encoder, which includes a 3×3 convolutional layer and a C3 module, the C3 module includes three 3×3 convolutional layers and the bottleneck layer in Resnet50; 1222) constructing an offset-guided deformable alignment and fusion submodule, which includes a deformable convolution layer, a bilinear interpolation operation, a 1×1 convolution, and two splicing operations; 1223) constructing an upsampling submodule, which includes three consecutive upsampling structures. The upsampling structure specifically has three layers, namely: a 3×3 deconvolution layer, a ReLU layer, and a batch normalization layer; 123) Construct a coupled contrastive learning framework for multimodal image fusion, which includes a Vgg-19 network with pre-trained weights, a contrastive loss function for improving foreground saliency, a contrastive loss function for preserving background details of optical images, and a segmentation submodule; 1231) constructing a contrast loss function for improving foreground significance, the foreground features of SAR images are set as positive samples, and the foreground features of optical images are set as negative samples; 1232) constructing a contrast loss function for retaining the background details of the optical image, wherein the background features of the optical image are set as positive samples, and the background features of the SAR image are set as negative samples; 1233) construct a segmentation submodule, which includes a fully connected layer and a sigmoid activation function; 13) Training the SAR-optical image coupled contrast fusion building segmentation model that is resistant to non-alignment: Use the divided training set to train the parameters of the building segmentation model constructed in 12) until the training reaches 100 epochs, and finally retain the trained network parameters; 14) Obtaining the results of the SAR-optical image coupled contrast fusion building segmentation model that is resistant to non-alignment: Input the acquired real SAR and optical image data to be fused into the trained SAR-optical image coupled contrast fusion building segmentation model that is resistant to non-alignment to obtain the segmentation results.

2. The method for building segmentation based on SAR-optical image coupling contrast fusion and anti-non-alignment according to claim 1, characterized in that: The training of the non-aligned SAR-optical image coupled contrast fusion building segmentation model specifically includes the following steps: 21) The training of the SAR-optical image coupled contrast fusion building segmentation model that is resistant to non-alignment includes two stages. The first stage is the training stage of coupled contrast learning. The multimodal dual-branch feature extraction module and the offset-guided adaptive feature alignment module are obtained through pre-training of the remote sensing image segmentation multimodal dataset, which can fully integrate the complementary features in the SAR image and the optical image. The second stage is the fine-tuning stage of the SAR-optical image coupled contrast fusion building segmentation model that is resistant to non-alignment. The entire model is fine-tuned through the remote sensing image segmentation multimodal dataset to make the segmentation result better. The training of the SAR-optical image coupled contrast fusion building segmentation model that is resistant to non-alignment specifically includes the following steps: 22) Inputting the SAR image and the optical image in the preprocessed data set into the multimodal dual-branch feature extraction module to obtain the SAR image features and the optical image features, specifically including the following steps: 221) After the SAR image is input into the SAR image encoder, three consecutive downsampling structures are executed to obtain the SAR image feature F SAR ; 222) After the optical image is input into the optical image encoder, three consecutive downsampling structures are executed to obtain the optical image feature F optical ; 23) The SAR image feature F SAR and optical image feature F optical The input offset guided adaptive feature alignment module is used to guide the multimodal feature alignment, which specifically includes the following steps: 231) executing a cross-modal spatial offset estimation submodule for accurately estimating feature-level offsets, specifically comprising the following steps: 2311)SAR image features SAR and optical image feature F optical After the invariant feature encoder with shared parameters, convolution operation and SiLU activation function operation are performed to obtain the shared features of SAR images. Optical image sharing features 2312)SAR image features SAR and optical image feature F optical After passing through the specific feature encoder, the convolution operation and the C3 module are performed to obtain the specific features of the SAR image. Optical image specific features 2313) SAR image sharing features After the spatial attention module, the SAR images share features at the same time Perform maximum pooling and average pooling, concatenate the maximum pooling output and the average pooling output, then perform a 7×7 convolution on the concatenated result, and then execute the sigmoid function to obtain the probability matrix Shares features with SAR images Perform point multiplication to obtain spatially enhanced SAR image features 2314) Optical image sharing features After the spatial attention module, the optical image shares features at the same time Perform the maximum pooling operation and the average pooling operation, concatenate the two pooling results, and then perform a 7×7 convolution on the concatenated result, and then pass it through the sigmoid function to obtain the probability matrix Shares features with optical images Perform point multiplication to obtain spatially enhanced optical image features 2315) for spatially enhanced SAR image features and spatially enhanced optical imaging features Perform the concatenation operation to obtain the changed feature F dif , the changing feature F dif After the channel attention module, F dif Perform maximum pooling and average pooling, then pass the output of maximum pooling and the output of average pooling through a multi-layer perceptron network, then add the outputs of the multi-layer perceptron network, and pass the result of the addition through a sigmoid function to obtain the attention map M att , attention map M att and spatially enhanced SAR image features Perform matrix multiplication to get Attention Map M att and spatially enhanced optical imaging features Perform matrix multiplication to get and spatially enhanced SAR image features Adding the spatial and channel enhanced SAR image features and spatially enhanced optical imaging features Adding the spatial and channel enhanced optical image features The formula is as follows: Among them, f mlp is a multi-layer perceptron network, Max(·) is the maximum pooling operation, and Avg(·) is the average pooling operation; 2316) for spatial and channel enhanced SAR image features Optical imaging features of spatial and channel enhancement Perform the concatenation operation, then perform the convolution operation and the Relu activation function operation to obtain the offset φ c ; 232) executing an offset-guided deformable alignment and fusion submodule for adaptive alignment; 2321) Sharing features of optical images Perform deformable convolution to obtain adaptively aligned optical image features The deformable convolution formula is as follows: Among them, y(p) is the value of the corresponding position of the aligned feature, K is the kernel weight, and w k is the kth kernel weight, x(·) is the convolution operation, p is the position of the center sampling point, and p k is the kth fixed offset, Δp k is the kth position φ c Compensation, Δm k is the offset φ c The learned modulation scalar; 2322) Optical image features for adaptive alignment Perform bilinear interpolation to obtain the complete optical image features for adaptive alignment 2323) for adaptive alignment of complete optical image features Shares features with SAR images Perform the concatenation operation and then perform a 1×1 convolution to get the output F c , for F c , optical image specific features and SAR image specific features Perform the splicing operation to obtain the fused feature F f , formulated as follows: Among them, f 1× 1 is 1×1 convolution, Cat(·) is the concatenation operation; 233) Execute the upsampling submodule, execute three consecutive upsampling structures, the upsampling structure includes a 3×3 deconvolution layer, a ReLU layer, and a batch normalization layer to obtain a fused image Fused; 24) A coupled contrastive learning framework for performing multimodal image fusion; 241) The optical image O, SAR image S and fused image Fused are respectively multiplied by the label mask M at the pixel level to obtain the optical foreground image O fore , SAR foreground image S fore and the foreground of the fused image fore ; The optical image O, SAR image S and fused image Fused are respectively multiplied by the background mask 1-M at the pixel level to obtain the optical background image O back , SAR background image S back and the background of the fused image back ; The optical foreground image O fore , SAR foreground image S fore , Fused image foreground fore , optical background image O back , SAR background image S back , Fused image background back Cut into 8×8 image blocks; Import these image blocks into the Vgg-19 network with pre-trained weights to obtain the foreground features u of the fused image i , SAR image foreground features Optical image foreground features Background features v of fused image i , Background characteristics of SAR images Background characteristics of optical images m represents the mth negative sample; 242) Construct coupled contrastive learning loss function L con , L con Includes a contrast loss function L for improving foreground saliency SAR and the contrast loss function L used to preserve the background details of the optical image optical , L con , L SAR and L optical The formula is as follows: L con =L SAR +L optical , Where m represents the mth negative sample, ||·|| represents the L1 norm, and w i is the learnable weight, L con represents the coupled contrastive learning loss function, L SAR represents the contrast loss function used to improve foreground saliency, L optical represents the contrast loss function used to preserve the background details of the optical image; During the training process, according to L con The formula calculates the loss value, calculates the gradient of the loss function, back-propagates the error from the output layer back to each layer of the network, determines the gradient vector by back-propagating the loss value, and updates the model parameters of the coupled contrast learning stage of the anti-non-aligned SAR-optical image coupled contrast fusion building segmentation model; Determine whether 100 epochs have been reached. If so, the training of the coupled contrast learning phase of the anti-non-aligned SAR-optical image coupled contrast fusion building segmentation model is completed. Otherwise, continue training. 25) Based on the training in the coupled contrast learning stage, a pre-trained multimodal dual-branch feature extraction module and an offset-guided adaptive feature alignment module are obtained. In the fine-tuning stage, a segmentation submodule in the coupled contrast learning framework of multimodal image fusion is added after the offset-guided adaptive feature alignment module for supervised training; 26) Input the SAR images and optical images in the data set into the SAR-optical image coupled contrast fusion building segmentation model that is resistant to misalignment, perform supervised fine-tuning learning, and train a remote sensing image building segmentation model with a multimodal dual-branch feature extraction module, an offset-guided adaptive feature alignment module, and a multimodal image fusion coupled contrast learning framework, specifically including the following steps: 261) The SAR image and optical image in the preprocessed data set are input into the multimodal dual-branch feature extraction module, and the SAR image encoder and optical image encoder operations are performed respectively to obtain the SAR image feature F SAR and optical image feature F optical ; 262) The SAR image feature F SAR and optical image feature F optical The input is sent to the offset-guided adaptive feature alignment module to guide the multimodal feature alignment and fusion to obtain the fused image Fused; 263) The fused image Fused is input into the segmentation submodule in the coupled contrast learning framework of multimodal image fusion, and the full connection operation and softmax activation function operation are performed to perform predictive segmentation, which is formulated as follows: p=soft max(L(Fused)), Where L(·) represents the fully connected layer operation, softmax(·) represents the softmax activation function operation, and p represents the predicted segmentation result; 27) The cross entropy loss is used as the loss function to optimize the network segmentation accuracy. The cross entropy loss function expression is as follows: Where N is the number of training samples, x i is the i-th sample of the ground truth, y i is the i-th sample of the predicted value; 28) By calculating the gradient of the loss function, the error is back-propagated from the output layer to each layer of the network, the gradient vector is determined by back-propagating the loss value, and the model parameters of the fine-tuning stage of the SAR-optical image coupled contrast fusion building segmentation model against non-alignment are updated; It is determined whether the set number of training rounds of 100 is reached. If so, the training of the fine-tuning stage of the anti-non-aligned SAR-optical image coupled contrast fusion building segmentation model is completed, otherwise the training continues.

Images