A multi-modal remote sensing image segmentation method and device based on cross fusion

Abstract

The application provides a multi-modal remote sensing image segmentation method and device based on cross fusion, and relates to the technical field of image processing. The method comprises the following steps: constructing a segmentation model, wherein the segmentation model comprises a plurality of first transformer layers, a first FFN, a first FAN, a plurality of second transformer layers, a second feature fusion network FFN, a second FAN, a plurality of third transformer layers, a third FFN, a third FAN, a plurality of fourth transformer layers, a fourth FFN, a neck network and a decoder; iteratively training the segmentation model by using a plurality of training sets; saving the optimal parameters by using each round of parameters and a verification set to obtain a final segmentation model, so that the final segmentation result of the multi-modal remote sensing image to be processed is obtained through the final segmentation model. The accuracy of multi-modal remote sensing image segmentation is high.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a multimodal remote sensing image segmentation method and apparatus based on cross-fusion. Background Technology

[0002] Multimodal remote sensing image segmentation is a method that combines data from different remote sensing sensors to improve the accuracy and detail of ground feature segmentation. This method leverages the complementarity of information provided by different sensors; by fusing multiple remote sensing data, it can more comprehensively describe the characteristics of ground features, thereby improving the accuracy of ground feature segmentation. The application of multimodal remote sensing image segmentation is based on the principle that different sensors have varying sensitivities to the reflective or emissive properties of ground features, thus providing information that differs spatially, spectrally, and temporally. By combining this differing information, ground features can be identified and segmented more effectively, especially in complex environments such as urban areas or vegetated areas, where multimodal remote sensing image segmentation is particularly important.

[0003] Currently, the operation of multimodal remote sensing image segmentation is as follows: First, image features at different modalities are acquired separately; second, features at each scale are fused using an attention mechanism, and upsampling is used to gradually fuse information from all scales to obtain the segmentation result; finally, the dice loss and focal loss between the segmentation result and the ground truth label are calculated, and end-to-end training is completed using backpropagation. However, existing technologies use independent feature extraction. This independent feature extraction process, lacking interaction between different modalities, results in some information duplication and fails to effectively take into account complementary information between modalities, significantly affecting the information extraction and utilization rate, leading to low accuracy in multimodal remote sensing image segmentation. Summary of the Invention

[0004] The purpose of this invention is to provide a multimodal remote sensing image segmentation method and apparatus based on cross-fusion, thereby solving the problem of low accuracy in multimodal remote sensing image segmentation.

[0005] To address the aforementioned technical problems, the embodiments of the present invention provide the following technical solutions:

[0006] The first aspect of this invention provides a multimodal remote sensing image segmentation method based on cross-fusion, the method comprising:

[0007] Multiple datasets are acquired and preprocessed. The preprocessed datasets are then divided into corresponding training and validation sets. The datasets are collections of labeled multimodal remote sensing images, with labels used to mark the terrain in the multimodal remote sensing images.

[0008] A segmentation model is constructed, which includes multiple first transformer layers, a first feature fusion network, a first feature adaptive network, multiple second transformer layers, a second feature fusion network, a second feature adaptive network, multiple third transformer layers, a third feature fusion network, a third feature adaptive network, multiple fourth transformer layers, a fourth feature fusion network, a neck network, and a decoder. The number of first transformer layers, the number of second transformer layers, the number of third transformer layers, and the number of fourth transformer layers are all the same as the number of categories in the multimodal remote sensing image.

[0009] The segmentation model is iteratively trained using multiple training sets to obtain the parameters of the segmentation model in each round.

[0010] The optimal parameters are saved using the parameters and validation set in each round to obtain the final segmentation model. This final segmentation model is then used to segment the multimodal remote sensing image to be processed, and the final segmentation result is obtained.

[0011] A second aspect of the present invention provides a multimodal remote sensing image segmentation device based on cross-fusion, the device comprising:

[0012] The partitioning module is used to acquire multiple datasets and preprocess them. The preprocessed datasets are divided into corresponding training and validation sets. The multiple datasets are collections of labeled multimodal remote sensing images, and the labels are used to mark the terrain in the multimodal remote sensing images.

[0013] The building module is used to construct the segmentation model, which includes multiple first transformer layers, a first feature fusion network, a first feature adaptive network, multiple second transformer layers, a second feature fusion network, a second feature adaptive network, multiple third transformer layers, a third feature fusion network, a third feature adaptive network, multiple fourth transformer layers, a fourth feature fusion network, a neck network, and a decoder. The number of multiple first transformer layers, multiple second transformer layers, multiple third transformer layers, and multiple fourth transformer layers are all the same as the number of categories in the multimodal remote sensing image.

[0014] The training module is used to iteratively train the segmentation model using multiple training sets to obtain the parameters of the segmentation model in each round.

[0015] The save module is used to save the optimal parameters using the parameters of each round and the validation set to obtain the final segmentation model. This final segmentation model is then used to segment the multimodal remote sensing image to be processed, and the final segmentation result is obtained.

[0016] Compared to existing technologies, the present invention provides a multimodal remote sensing image segmentation method and apparatus based on cross-fusion, which acquires multiple datasets and preprocesses them, dividing the preprocessed datasets into corresponding training and validation sets. The multiple datasets are collections of labeled multimodal remote sensing images, with labels used to mark terrain in the images. A segmentation model is constructed, comprising multiple first transformer layers, a first feature fusion network, a first feature adaptive network, multiple second transformer layers, a second feature fusion network, a second feature adaptive network, multiple third transformer layers, a third feature fusion network, and a third feature... The segmentation model consists of a feature adaptation network, multiple fourth transformer layers, a fourth feature fusion network, a neck network, and a decoder. The number of first, second, third, and fourth transformer layers is the same as the number of categories in the multimodal remote sensing image. The segmentation model is iteratively trained using multiple training sets to obtain parameters for each round. The optimal parameters are saved using the parameters from each round and the validation set to obtain the final segmentation model. This final model is then used to segment the multimodal remote sensing image to obtain the final segmentation result. Thus, through the first feature fusion network, first feature adaptation network, second feature fusion network, second feature adaptation network, third feature fusion network, third feature adaptation network, and fourth feature fusion network in the segmentation model, interactive extraction of information between multiple modalities of remote sensing images is achieved. This effectively takes into account the complementary information between modalities and adaptively projects the fused feature map to match the feature space of each modality, which is beneficial for subsequent better extraction of modal features, resulting in higher accuracy in multimodal remote sensing image segmentation. Attached Figure Description

[0017] The above and other objects, features, and advantages of exemplary embodiments of the present invention will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of the invention are illustrated by way of example and not limitation, with the same or corresponding reference numerals denoteing the same or corresponding parts, wherein:

[0018] Figure 1 A flowchart illustrating a multimodal remote sensing image segmentation method based on cross-fusion is shown schematically.

[0019] Figure 2 A schematic diagram illustrating the execution of the segmentation model is shown.

[0020] Figure 3 A schematic diagram of a multimodal remote sensing image segmentation device based on cross-fusion is shown. Detailed Implementation

[0021] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the invention and to fully convey the scope of the invention to those skilled in the art.

[0022] It should be noted that, unless otherwise stated, the technical or scientific terms used in this invention should have the ordinary meaning as understood by those skilled in the art.

[0023] The methods described in the embodiments of the present invention will be explained in detail below.

[0024] Figure 1 The flowchart illustrating a cross-fusion-based multimodal remote sensing image segmentation method according to an embodiment of the present invention is shown in the illustration. See [link to flowchart illustration]. Figure 1 As shown, the method may include:

[0025] S101. Obtain multiple datasets and preprocess them, dividing the preprocessed datasets into corresponding training and validation sets.

[0026] Among them, multiple datasets are collections of labeled multimodal remote sensing images, with labels used to mark the terrain in the multimodal remote sensing images.

[0027] Multiple datasets include multiple Potsdam datasets and multiple Wuhan University Optical and Synthetic Aperture Radar Dataset (WHU-OPT-SAR) datasets. Each Potsdam dataset and each WHU-OPT-SAR dataset is a dataset containing multimodal remote sensing images.

[0028] Multimodal remote sensing images include visible light remote sensing images, multispectral remote sensing images, infrared remote sensing images, near-infrared remote sensing images, SAR remote sensing images, and digital surface model (DSM) remote sensing images, etc.

[0029] The Potsdam dataset contains 28 6000*6000 remote sensing images, including four-band (RGBIR) remote sensing image files in Tag Image File Format (TIFF) and single-band DSM remote sensing images. The remote sensing images are labeled with six terrain types and stored as TIFF images in the form of image masks. With a stride of 512, each remote sensing image is cropped into 512*512 non-overlapping image patches. These cropped remote sensing images are used as the dataset for the segmentation model.

[0030] The WHU-OPT-SAR dataset contains 100 5556*3704 remote sensing images, composed of four bands: near-infrared, red, green, and SAR. The remote sensing images are labeled with seven terrain types and stored as TIFF images in the form of image masks. With a stride of 512, each remote sensing image is cropped into 512*512 non-overlapping image patches. These cropped images are used as the dataset for the segmentation model.

[0031] Preprocessing of multiple datasets, including:

[0032] Step A1: Perform standardization and normalization on multiple datasets in sequence to obtain normalized datasets.

[0033] Multiple datasets are standardized to scale the multimodal remote sensing images from multiple datasets to a range based on the calculated mean and standard deviation, so that they conform to a normal distribution.

[0034] Normalization is applied to multiple standardized datasets to scale them proportionally into a small, specific range, resulting in normalized datasets. By default, max-min normalization is used to scale them to [0,1].

[0035] Step A2: Perform data augmentation on the normalized datasets to obtain preprocessed datasets.

[0036] Data augmentation includes random horizontal flipping, scaling, and random cropping.

[0037] Specifically, the normalized datasets are sequentially subjected to random horizontal flipping, scaling, and random cropping to obtain preprocessed datasets.

[0038] Random horizontal flipping involves randomly flipping multimodal remote sensing images from multiple normalized datasets along the vertical central axis, essentially mirroring the multimodal remote sensing images from multiple normalized datasets left and right. The default probability is 0.5.

[0039] Scaling involves randomly scaling multimodal remote sensing images from multiple datasets that have been randomly horizontally flipped. The scaled size is not fixed.

[0040] Random cropping involves randomly selecting a sub-region from multimodal remote sensing images across multiple scaled datasets, according to a preset size, as a training sample. The default probability is 0.5. The preset size is 224*224.

[0041] After preprocessing multiple datasets, the preprocessed datasets are randomly partitioned into training and validation sets. The training set comprises 80% and the validation set comprises 20%.

[0042] The multimodal remote sensing images in each dataset can be extracted and saved separately according to their categories, and used as data for subsequent parallel experiments.

[0043] S102. Construct a segmentation model.

[0044] The segmentation model comprises multiple first transformer layers, a first feature fusion network, a first feature adaptation network, multiple second transformer layers, a second feature fusion network, a second feature adaptation network, multiple third transformer layers, a third feature fusion network, a third feature adaptation network, multiple fourth transformer layers, a fourth feature fusion network, a neck network, and a decoder. The number of first transformer layers, second transformer layers, third transformer layers, and fourth transformer layers are all the same as the number of categories in the multimodal remote sensing image.

[0045] Each of the first, second, third, and fourth transformer layers includes multiple blocks. The number of blocks in each of the first, second, third, and fourth transformer layers is different. Each block sequentially includes a third normalization layer, a multi-head attention layer, a fourth normalization layer, and a multilayer perceptron (MLP).

[0046] For example, when the number of categories in a multimodal remote sensing image is 3, the corresponding number of multiple first transformer layers, multiple second transformer layers, multiple third transformer layers, and multiple fourth transformer layers are all 3.

[0047] For example, each first transformer layer includes 2 blocks, each second transformer layer includes 5 blocks, each third transformer layer includes 8 blocks, and each fourth transformer layer includes 11 blocks.

[0048] The neck network consists of a third 1×1 convolutional layer and a 3×3 convolutional layer. The decoder consists of a fourth 1×1 convolutional layer and a fifth 1×1 convolutional layer.

[0049] The global attention transformation (Vision Transformer_base, ViT_base) is used as the backbone network. The backbone network, ViT_base, consists of multiple first transformer layers, a first feature fusion network (FFN), a first feature adaptive network (FAN), multiple second transformer layers, a second FFN, a second FAN, multiple third transformer layers, a third FFN, a third FAN, multiple fourth transformer layers, and a fourth FFN.

[0050] FFN and FAN are referred to as parallel sensing adaptive cross-modal layers.

[0051] S103. Use multiple training sets to iteratively train the segmentation model to obtain the parameters of the segmentation model in each round.

[0052] Specifically, the segmentation model is iteratively trained using multiple training sets to obtain the parameters of the segmentation model for each round, including:

[0053] Step B1: Iteratively train the segmentation model using multiple training sets to obtain the corresponding multi-round preset segmentation results.

[0054] Each training set includes remote sensing images of multiple categories. Each category of remote sensing image corresponds to a first transformer layer, a second transformer layer, a third transformer layer, and a fourth transformer layer.

[0055] Figure 2 A schematic diagram illustrating the execution of the segmentation model is shown below.Figure 2 As shown, the segmentation model is iteratively trained using multiple training sets to obtain corresponding multi-round preset segmentation results, including:

[0056] Step B111: Input the remote sensing images of each category in the training set into the corresponding first transformer layer in sequence, so that each first transformer layer outputs the corresponding first feature map.

[0057] Step B112: Input each first feature map into the first feature fusion network so that the first feature fusion network outputs the first fused feature map.

[0058] The first feature fusion network, the second feature fusion network, the third feature fusion network, and the fourth feature fusion network all include a first 1×1 convolutional layer, a first normalization layer, a splicing layer, and a fusion layer.

[0059] Specifically, each first feature map is input into the first feature fusion network so that the first feature fusion network outputs a first fused feature map, including:

[0060] Step B1121: Input each first feature map into the first 1×1 convolutional layer and the first normalization layer in sequence, so that the first normalization layer outputs each first intermediate feature map.

[0061] In this context, one first feature map corresponds to one first intermediate feature map.

[0062] Step B1122: Input each first intermediate feature map into the concatenation layer so that each first intermediate feature map is concatenated with the four-dimensional vector along the channel dimension of the image to output multiple features.

[0063] The four-dimensional vector has the shape (N,1,H,W), and all elements of (N,1,H,W) are 1.

[0064] Step B1123: Input multiple features into the fusion layer so that the fusion layer outputs the first fused feature map.

[0065] Specifically, multiple features are input into the fusion layer so that the fusion layer outputs a first fused feature map, including:

[0066] Step B11231: Add dimensions to the first feature and the second feature respectively and calculate the Cartesian product to obtain the first calculated feature.

[0067] Step B11232: The first calculated feature is sequentially rearranged in terms of dimensions, flattened, and dimensions are added to obtain the first intermediate feature.

[0068] Step B11233: Rearrange the dimensions and add dimensions to the third feature in sequence to obtain the first updated feature.

[0069] Step B11234: Calculate the Cartesian product of the first updated feature and the first intermediate feature to obtain the second calculated feature, and then rearrange, flatten and add dimensions to the second calculated feature in sequence to obtain the second intermediate feature.

[0070] Step B11235: Accordingly, rearrange the dimensions and add dimensions to the nth feature in sequence to obtain the (n-2)th updated feature; calculate the Cartesian product of the (n-2)th updated feature and the (n-2)th intermediate feature to obtain the (n-1)th calculated feature, and rearrange the dimensions, flatten and add dimensions to the (n-1)th calculated feature in sequence to obtain the (n-1)th intermediate feature.

[0071] Step B11236: Continue until all features have been processed and the first fused feature map is obtained.

[0072] For example, the multimodal remote sensing image has 3 categories, 3 first transformer layers, 3 first intermediate feature maps, and a four-dimensional vector with shape (N,1,H,W), where all elements of (N,1,H,W) are 1. Each first intermediate feature map is concatenated with the four-dimensional vector along the image's channel number (C) dimension, resulting in 3 features, i.e., feature X1∈R. N×(C1+1)×H×W Feature X2∈R N×(C2+1)×H×W and feature X3∈R N×(C3+1)×H×W Where N represents the total number of samples in a batch, H and W represent the height and width of the feature dimensions, respectively, C1, C2, and C3 represent the number of channels for each feature, and R represents the real number field. N×(C1+1)×H×W Let the representation be a real number space with dimensions (N, C1+1, H, W). Dimensions are added to the first feature (feature X1) and the second feature (feature X2) respectively (one feature is added to N, and the other to C). After adding dimensions to features X1 and X2, the Cartesian product is calculated to obtain the first computed feature. The first computed feature is then subjected to a series of rearrangements, flattening, and dimension additions to obtain the first intermediate feature (feature X). 1 ∈R N ×H×W×(C1+1)*(C2+1)×1 Then, the dimensions of the third feature, X3, are rearranged and dimensions are added sequentially to obtain the first updated feature, X. 2 ∈R N×H×W×1×(C3+1) Then update the first feature, i.e., feature X. 2 and the first intermediate feature, i.e., feature X 1 Calculate the Cartesian product to obtain the second calculated feature. Then, sequentially rearrange, flatten, and add dimensions to the second calculated feature to obtain the second intermediate feature, i.e., feature X. 3 ∈R N×(C1+1)*(C2+1)*(C3+1)×H×W .

[0073] Step B113: Input the first fused feature map into the first feature adaptive network so that the first feature adaptive network outputs the second fused feature map.

[0074] The first feature adaptive network, the second feature adaptive network, and the third feature adaptive network all include a second 1×1 convolutional layer and a second normalization layer.

[0075] Specifically, inputting the first fused feature map into the first feature adaptive network so that the first feature adaptive network outputs the second fused feature map includes: sequentially inputting the first fused feature map into the second 1×1 convolutional layer and the second normalization layer so that the second normalization layer outputs the second fused feature map.

[0076] The second fusion feature map is the modality fusion feature.

[0077] Step B114: Add the second fused feature map to each of the first feature maps respectively to obtain the corresponding second feature maps.

[0078] Each second feature map is used as the input to each second transformer layer.

[0079] Step B115: Input each second feature map into the corresponding second transformer layer and the second feature fusion network in sequence, so that the second feature fusion network outputs the third fusion feature map; input the third fusion feature map into the second feature adaptive network, so that the second feature adaptive network outputs the fourth fusion feature map.

[0080] The specific operation of inputting each second feature map into the corresponding second transformer layer and the second feature fusion network in sequence so that the second feature fusion network outputs the third fused feature map is described in steps B111 and B112 above (including steps B1121 to B1123).

[0081] For details on how to input the third fused feature map into the second feature adaptive network so that the second feature adaptive network outputs the fourth fused feature map, please refer to step B113 above.

[0082] Step B116: Add the fourth fused feature map to each of the second feature maps respectively to obtain the corresponding third feature maps.

[0083] Step B117: Input each third feature map into the corresponding third transformer layer and the third feature fusion network in sequence, so that the third feature fusion network outputs the fifth fused feature map; input the fifth fused feature map into the third feature adaptive network, so that the third feature adaptive network outputs the sixth fused feature map.

[0084] The specific operation of inputting each third feature map into the corresponding third transformer layer and the third feature fusion network in sequence so that the third feature fusion network outputs the fifth fused feature map is described in steps B111 and B112 above (including steps B1121 to B1123).

[0085] For details on how to input the fifth fused feature map into the third feature adaptive network so that the third feature adaptive network outputs the sixth fused feature map, please refer to step B113 above.

[0086] Step B118: Add the sixth fused feature map to each of the third feature maps respectively to obtain the corresponding fourth feature maps.

[0087] Step B119: Input each fourth feature map into the corresponding fourth transformer layer and the fourth feature fusion network in sequence, so that the fourth feature fusion network outputs the seventh fused feature map.

[0088] The specific operation of inputting each fourth feature map into the corresponding fourth transformer layer and the fourth feature fusion network in sequence so that the fourth feature fusion network outputs the seventh fused feature map is described in steps B111 and B112 above (including steps B1121 to B1123).

[0089] The feature fusion network enables interactive extraction of information from multiple modal remote sensing images, effectively taking into account the complementary information between modalities. The feature adaptive network adaptively projects the fused feature map to match the feature space of each modality, which is beneficial for subsequent better extraction of modal features.

[0090] Step B120: Input the first fusion feature map, the third fusion feature map, the fifth fusion feature map and the seventh fusion feature map into the neck network and the decoder in sequence, so that the decoder outputs multiple rounds of preset segmentation results.

[0091] Specifically, the fused feature maps from four layers—the first, third, fifth, and seventh fused feature maps—are each processed by a parameter-independent third 1×1 convolutional layer to adjust their channel count to 768. Next, bilinear interpolation is applied to each feature map layer to generate a feature map set with scaling ratios of [4, 2, 1, 0.5]. Finally, each feature map layer is processed by a parameter-independent 3×3 convolutional layer to obtain multi-scale features with 768 channels.

[0092] The multi-scale feature maps output by the neck network are fed into the decoder, which uses an UpperNet network. For each scale of the feature map output by the neck network, UpperNet uses upsampling (such as bilinear interpolation or transposed convolution) to enlarge its size to the same size as the original input image. The upsampled feature map is then passed through a fourth 1×1 convolutional layer to reduce the number of channels and decrease computational complexity. The upsampled feature map is then fused with the corresponding low-level features at the same scale by element-wise addition. Finally, the fused result is passed through a fifth 1×1 convolutional layer to obtain a multi-round preset segmentation result. The number of channels in this fifth 1×1 convolution corresponds to the number of semantic segmentation categories and is used to predict the category of each pixel.

[0093] Step B2: Calculate the cross-entropy loss function value for each round based on the preset segmentation results and corresponding labels.

[0094] The cross-entropy loss function is used to calculate the cross-entropy loss function value for each round. The cross-entropy loss function value for each round reflects the difference between the preset segmentation results and the labels input to the segmentation model.

[0095] Step B3: Update the parameters of the segmentation model for each round using the backpropagation algorithm based on the cross-entropy loss function value for each round.

[0096] The optimizer selected is the Adam (AdamW) optimizer with weight decay. Different learning rates and weight decays are used for models corresponding to different transformer layers, set as needed. A poly strategy is used for the learning rate, and the number of training epochs is set to 50.

[0097] Based on the cross-entropy loss function value in each round, the parameters of the segmentation model are updated using the backpropagation algorithm and optimizer to enable the segmentation model to better predict the target.

[0098] S104. Save the optimal parameters using the parameters of each round and the validation set to obtain the final segmentation model. Then, use the final segmentation model to segment the multimodal remote sensing image to be processed and obtain the final segmentation result.

[0099] Specifically, using the parameters and validation set from each round, the optimal parameters are saved to obtain the final segmentation model, including:

[0100] Step C1: Using the parameters of each round, process the validation set separately to obtain the prediction results for each round corresponding to the validation set.

[0101] Step C2: Calculate the cross-union ratio (CUI) of remote sensing images of multiple categories in each round of prediction results to obtain multiple CUI corresponding to each round of prediction results.

[0102] In addition to calculating the crossover ratio (CROR) of remote sensing images of multiple categories in each round of prediction, the F1 score of remote sensing images of multiple categories in each round of prediction can also be calculated.

[0103] Step C3: Calculate the mean of multiple crossover ratios corresponding to the prediction results of each round, and select the parameter corresponding to the highest mean to save it, so as to obtain the final segmentation model. In order to use the final segmentation model to segment the multimodal remote sensing image to be processed, the final segmentation result can be obtained.

[0104] After saving the optimal parameters using the parameters of each round and the validation set to obtain the final segmentation model, the process also includes: inputting the multimodal remote sensing image to be processed into the final segmentation model so that the final segmentation model outputs the final segmentation result.

[0105] This invention can be used to register multimodal remote sensing images acquired by satellites, and then extract complementary information between different modalities for image segmentation to obtain segmentation maps with land cover category labels.

[0106] Based on the above Figure 1As can be seen from the implementation method, this embodiment of the invention acquires multiple datasets and preprocesses them. The preprocessed datasets are divided into corresponding training and validation sets. The multiple datasets are collections of labeled multimodal remote sensing images, and the labels are used to mark the terrain in the multimodal remote sensing images. A segmentation model is constructed, which includes multiple first transformer layers, a first feature fusion network, a first feature adaptive network, multiple second transformer layers, a second feature fusion network, a second feature adaptive network, multiple third transformer layers, a third feature fusion network, a third feature adaptive network, multiple fourth transformer layers, a fourth feature fusion network, a neck network, and a decoder. The number of multiple first transformer layers, multiple second transformer layers, multiple third transformer layers, and multiple fourth transformer layers are all the same as the number of categories in the multimodal remote sensing images. The segmentation model is iteratively trained using multiple training sets to obtain the parameters of the segmentation model in each round. The optimal parameters are saved using the parameters of each round and the validation set to obtain the final segmentation model. The final segmentation model is then used to segment the multimodal remote sensing images to be processed to obtain the final segmentation result. In this way, by using the first feature fusion network, the first feature adaptive network, the second feature fusion network, the second feature adaptive network, the third feature fusion network, the third feature adaptive network, and the fourth feature fusion network in the segmentation model, the interactive extraction of information between multiple modal remote sensing images is realized. This effectively takes into account the complementary information between modalities and adaptively projects the fused feature map to match the feature space of each modal feature, which is conducive to better extraction of modal features in the subsequent process, resulting in a higher accuracy of multimodal remote sensing image segmentation.

[0107] Based on the same inventive concept, as an implementation of the above-mentioned cross-fusion-based multimodal remote sensing image segmentation method, this embodiment of the invention also provides a cross-fusion-based multimodal remote sensing image segmentation device. Figure 3 This is a structural diagram of the multimodal remote sensing image segmentation device based on cross-fusion in an embodiment of the present invention. See also... Figure 3 As shown, the device may include:

[0108] The partitioning module 301 is used to acquire multiple datasets and preprocess them. The preprocessed datasets are divided into corresponding training and validation sets. The multiple datasets are a collection of labeled multimodal remote sensing images. The labels are used to mark the terrain in the multimodal remote sensing images.

[0109] Module 302 is used to construct a segmentation model. The segmentation model includes multiple first transformer layers, a first feature fusion network, a first feature adaptive network, multiple second transformer layers, a second feature fusion network, a second feature adaptive network, multiple third transformer layers, a third feature fusion network, a third feature adaptive network, multiple fourth transformer layers, a fourth feature fusion network, a neck network, and a decoder. The number of multiple first transformer layers, multiple second transformer layers, multiple third transformer layers, and multiple fourth transformer layers are all the same as the number of categories in the multimodal remote sensing image.

[0110] Training module 303 is used to iteratively train the segmentation model using multiple training sets to obtain the parameters of the segmentation model in each round.

[0111] The storage module 304 is used to store the optimal parameters using the parameters of each round and the validation set to obtain the final segmentation model, so as to perform segmentation processing on the multimodal remote sensing image to be processed through the final segmentation model and obtain the final segmentation result.

[0112] Training module 303 is specifically used to iteratively train the segmentation model using multiple training sets to obtain corresponding multi-round preset segmentation results; calculate the cross-entropy loss function value for each round based on the preset segmentation results and corresponding labels; and update the parameters of the segmentation model for each round using the backpropagation algorithm based on the cross-entropy loss function value for each round.

[0113] Training module 303, each training set includes remote sensing images of multiple categories. Each category of remote sensing image corresponds to a first transformer layer, a second transformer layer, a third transformer layer, and a fourth transformer layer. The segmentation model is iteratively trained using multiple training sets to obtain corresponding multi-round preset segmentation results. This includes: sequentially inputting remote sensing images of each category from each training set into their respective first transformer layers, so that each first transformer layer outputs its corresponding first feature map; inputting each first feature map into a first feature fusion network, so that the first feature fusion network outputs a first fused feature map; inputting the first fused feature map into a first feature adaptive network, so that the first feature adaptive network outputs a second fused feature map; adding the second fused feature map to each of the first feature maps respectively to obtain the corresponding second feature maps; and sequentially inputting each second feature map into its corresponding second transformer layer and second transformer layer. A feature fusion network is used to output a third fused feature map from the second feature fusion network. The third fused feature map is then input into a second feature adaptive network to output a fourth fused feature map. The fourth fused feature map is added to each of the second feature maps to obtain the corresponding third feature maps. Each third feature map is then sequentially input into the corresponding third transformer layer and the third feature fusion network to output a fifth fused feature map. The fifth fused feature map is then input into the third feature adaptive network to output a sixth fused feature map. The sixth fused feature map is then added to each of the third feature maps to obtain the corresponding fourth feature maps. Each fourth feature map is then sequentially input into the corresponding fourth transformer layer and the fourth feature fusion network to output a seventh fused feature map. The first, third, fifth, and seventh fused feature maps are then sequentially input into a neck network and a decoder to output a multi-round preset segmentation result from the decoder.

[0114] The construction module 302, the first feature fusion network, the second feature fusion network, the third feature fusion network and the fourth feature fusion network all include a first 1×1 convolutional layer, a first normalization layer, a splicing layer and a fusion layer.

[0115] The training module 303 sequentially inputs each first feature map into the first 1×1 convolutional layer and the first normalization layer, so that the first normalization layer outputs each first intermediate feature map, with one first feature map corresponding to one first intermediate feature map; inputs each first intermediate feature map into the concatenation layer, so that each first intermediate feature map is concatenated with the four-dimensional vector along the channel dimension of the image, and outputs multiple features; inputs multiple features into the fusion layer, so that the fusion layer outputs the first fused feature map.

[0116] Training module 303 inputs multiple features into the fusion layer to output a first fused feature map. This includes: adding dimensions to the first and second features and calculating their Cartesian products to obtain a first computed feature; sequentially rearranging, flattening, and adding dimensions to the first computed feature to obtain a first intermediate feature; sequentially rearranging and adding dimensions to the third feature to obtain a first updated feature; calculating the Cartesian product of the first updated feature and the first intermediate feature to obtain a second computed feature, and sequentially rearranging, flattening, and adding dimensions to the second computed feature to obtain a second intermediate feature; correspondingly, sequentially rearranging and adding dimensions to the nth feature to obtain the (n-2)th updated feature; calculating the Cartesian product of the (n-2)th updated feature and the (n-2)th intermediate feature to obtain the (n-1)th computed feature, and sequentially rearranging, flattening, and adding dimensions to the (n-1)th computed feature to obtain the (n-1)th intermediate feature; and so on, until all features are processed to obtain the first fused feature map.

[0117] Training module 303, including a first feature adaptive network, a second feature adaptive network, and a third feature adaptive network, each includes a second 1×1 convolutional layer and a second normalization layer; inputting the first fused feature map into the first feature adaptive network so that the first feature adaptive network outputs a second fused feature map includes: sequentially inputting the first fused feature map into the second 1×1 convolutional layer and the second normalization layer so that the second normalization layer outputs a second fused feature map.

[0118] Module 302 is constructed, and each of the first transformer layer, the second transformer layer, the third transformer layer, and the fourth transformer layer includes multiple blocks. The number of multiple blocks corresponding to each first transformer layer, each of the second transformer layer, each of the third transformer layer, and each of the fourth transformer layer are different. Each block includes, in sequence, a third normalization layer, a multi-head attention layer, a fourth normalization layer, and a multilayer perceptron.

[0119] The storage module 304 is specifically used to process the validation set using the parameters of each round to obtain the prediction results of each round corresponding to the validation set; calculate the intersection-union ratio (IU) of multiple categories of remote sensing images in each round prediction results to obtain multiple IU corresponding to each round prediction results; calculate the mean of the multiple IU corresponding to each round prediction results, and select the parameter corresponding to the highest mean to save it, so as to obtain the final segmentation model, so as to segment the multimodal remote sensing image to be processed through the final segmentation model and obtain the final segmentation result.

[0120] It should be noted that the above description of the embodiment of the multimodal remote sensing image segmentation device based on cross-fusion is similar to the description of the embodiment of the multimodal remote sensing image segmentation method based on cross-fusion, and has similar beneficial effects. For any technical details not disclosed in the embodiments of the multimodal remote sensing image segmentation device based on cross-fusion of the present invention, please refer to the description of the embodiment of the multimodal remote sensing image segmentation method based on cross-fusion of the present invention for understanding.

[0121] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multimodal remote sensing image segmentation method based on cross-fusion, characterized in that, The cross-fusion-based multimodal remote sensing image segmentation method includes: Multiple datasets are acquired and preprocessed. The preprocessed datasets are then divided into corresponding training and validation sets. The multiple datasets are collections of labeled multimodal remote sensing images, and the labels are used to mark the terrain in the multimodal remote sensing images. A segmentation model is constructed, comprising multiple first transformer layers, a first feature fusion network, a first feature adaptive network, multiple second transformer layers, a second feature fusion network, a second feature adaptive network, multiple third transformer layers, a third feature fusion network, a third feature adaptive network, multiple fourth transformer layers, a fourth feature fusion network, a neck network, and a decoder. The number of the multiple first transformer layers, the multiple second transformer layers, the multiple third transformer layers, and the multiple fourth transformer layers are all the same as the number of categories in the multimodal remote sensing image. The first feature fusion network, the second feature fusion network, the third feature fusion network, and the fourth feature fusion network each include a first 1×1 convolutional layer, a first normalization layer, a stitching layer, and a fusion layer. The fusion layer is configured to sequentially perform operations on the input features, such as adding dimensions, calculating the Cartesian product, rearranging dimensions, and flattening, to output the corresponding fused feature map. The segmentation model is iteratively trained using multiple training sets to obtain the parameters of the segmentation model for each round. The optimal parameters are saved using the parameters of each round and the validation set to obtain the final segmentation model. The final segmentation model is then used to segment the multimodal remote sensing image to be processed, and the final segmentation result is obtained.

2. The multimodal remote sensing image segmentation method based on cross-fusion according to claim 1, characterized in that, The step of iteratively training the segmentation model using multiple training sets to obtain the parameters of the segmentation model for each round includes: The segmentation model is iteratively trained using the multiple training sets to obtain corresponding multi-round preset segmentation results; Based on the preset segmentation results and the corresponding labels for each round, calculate the cross-entropy loss function value for each round; Based on the cross-entropy loss function value for each round, the parameters of the segmentation model are updated using the backpropagation algorithm.

3. The multimodal remote sensing image segmentation method based on cross-fusion according to claim 2, characterized in that, Each training set includes remote sensing images of multiple categories. A remote sensing image of one category corresponds to a first transformer layer, a second transformer layer, a third transformer layer, and a fourth transformer layer. The step of iteratively training the segmentation model using the multiple training sets to obtain corresponding multi-round preset segmentation results includes: The remote sensing images of each category in the training set are sequentially input into the corresponding first transformer layer so that each first transformer layer outputs the corresponding first feature map. Each of the first feature maps is input into the first feature fusion network, so that the first feature fusion network outputs a first fused feature map; The first fused feature map is input into the first feature adaptive network so that the first feature adaptive network outputs the second fused feature map. The second fused feature map is added to each of the first feature maps respectively to obtain the corresponding second feature maps; Each of the second feature maps is sequentially input into the corresponding second transformer layer and the second feature fusion network, so that the second feature fusion network outputs a third fused feature map; the third fused feature map is input into the second feature adaptive network, so that the second feature adaptive network outputs a fourth fused feature map; The fourth fused feature map is added to each of the second feature maps respectively to obtain the corresponding third feature maps; Each of the third feature maps is sequentially input into the corresponding third transformer layer and the third feature fusion network, so that the third feature fusion network outputs a fifth fused feature map; the fifth fused feature map is input into the third feature adaptive network, so that the third feature adaptive network outputs a sixth fused feature map; The sixth fused feature map is added to each of the third feature maps respectively to obtain the corresponding fourth feature maps; Each of the fourth feature maps is sequentially input into the corresponding fourth transformer layer and the fourth feature fusion network, so that the fourth feature fusion network outputs a seventh fused feature map; The first fused feature map, the third fused feature map, the fifth fused feature map, and the seventh fused feature map are sequentially input into the neck network and the decoder, so that the decoder outputs the multi-round preset segmentation results.

4. The multimodal remote sensing image segmentation method based on cross-fusion according to claim 3, characterized in that, The step of inputting each of the first feature maps into the first feature fusion network, so that the first feature fusion network outputs a first fused feature map, includes: Each of the first feature maps is sequentially input into the first 1×1 convolutional layer and the first normalization layer, so that the first normalization layer outputs each of the first intermediate feature maps, with one first feature map corresponding to one first intermediate feature map; Each of the first intermediate feature maps is input into the stitching layer so that each of the first intermediate feature maps is stitched with the four-dimensional vector along the channel dimension of the image to output multiple features; The multiple features are input into the fusion layer so that the fusion layer outputs the first fused feature map.

5. The multimodal remote sensing image segmentation method based on cross-fusion according to claim 4, characterized in that, The step of inputting the plurality of features into the fusion layer so that the fusion layer outputs the first fused feature map includes: Add dimensions to the first feature and the second feature respectively, and calculate the Cartesian product to obtain the first computed feature; The first calculated feature is sequentially rearranged in dimensions, flattened, and dimension added to obtain the first intermediate feature; The third feature is then rearranged in terms of dimensions and additional dimensions are added sequentially to obtain the first updated feature. Calculate the Cartesian product of the first updated feature and the first intermediate feature to obtain the second calculated feature, and then sequentially rearrange, flatten, and add dimensions to the second calculated feature to obtain the second intermediate feature; Accordingly, the nth feature is rearranged and dimensions are added sequentially to obtain the (n-2)th updated feature; the (n-2)th updated feature and the (n-2)th intermediate feature are calculated to obtain the (n-1)th calculated feature, and the (n-1)th calculated feature is rearranged, flattened and dimensions are added sequentially to obtain the (n-1)th intermediate feature; This process continues until all features have been processed, resulting in the first fused feature map.

6. The multimodal remote sensing image segmentation method based on cross-fusion according to claim 3, characterized in that, The first feature adaptive network, the second feature adaptive network, and the third feature adaptive network all include a second 1×1 convolutional layer and a second normalization layer; The step of inputting the first fused feature map into the first feature adaptive network, so that the first feature adaptive network outputs the second fused feature map, includes: The first fused feature map is sequentially input into the second 1×1 convolutional layer and the second normalization layer, so that the second normalization layer outputs the second fused feature map.

7. The multimodal remote sensing image segmentation method based on cross-fusion according to claim 1, characterized in that, Each first transformer layer, each second transformer layer, each third transformer layer, and each fourth transformer layer includes multiple blocks; the number of multiple blocks corresponding to each first transformer layer, each second transformer layer, each third transformer layer, and each fourth transformer layer is different; each block sequentially includes a third normalization layer, a multi-head attention layer, a fourth normalization layer, and a multilayer perceptron.

8. The multimodal remote sensing image segmentation method based on cross-fusion according to claim 1, characterized in that, The process of using the parameters from each round and the validation set to save the optimal parameters to obtain the final segmentation model, and then using the final segmentation model to segment the multimodal remote sensing image to be processed, to obtain the final segmentation result, includes: Using the parameters for each round, the validation set is processed to obtain the prediction results for each round corresponding to the validation set; Calculate the cross-union ratio (CUI) of remote sensing images of multiple categories in each round of prediction results to obtain multiple CUI corresponding to each round of prediction results; The mean of multiple crossover union ratios corresponding to the prediction results of each round is calculated, and the parameter corresponding to the highest mean is selected and saved to obtain the final segmentation model. The final segmentation model is then used to segment the multimodal remote sensing image to be processed to obtain the final segmentation result.

9. A multimodal remote sensing image segmentation device based on cross-fusion, characterized in that, The cross-fusion-based multimodal remote sensing image segmentation device includes: A partitioning module is used to acquire multiple datasets and preprocess the multiple datasets, dividing the preprocessed multiple datasets into corresponding training sets and validation sets. The multiple datasets are a collection of labeled multimodal remote sensing images, and the labels are used to mark the terrain in the multimodal remote sensing images. A construction module is used to construct a segmentation model, which includes multiple first transformer layers, a first feature fusion network, a first feature adaptive network, multiple second transformer layers, a second feature fusion network, a second feature adaptive network, multiple third transformer layers, a third feature fusion network, a third feature adaptive network, multiple fourth transformer layers, a fourth feature fusion network, a neck network, and a decoder. The number of the multiple first transformer layers, the multiple second transformer layers, the multiple third transformer layers, and the multiple fourth transformer layers are all the same as the number of categories in the multimodal remote sensing image. The first feature fusion network, the second feature fusion network, the third feature fusion network, and the fourth feature fusion network each include a first 1×1 convolutional layer, a first normalization layer, a stitching layer, and a fusion layer. The fusion layer is configured to sequentially perform operations such as adding dimensions, calculating Cartesian products, rearranging dimensions, and flattening on the input features to output the corresponding fused feature map. The training module is used to iteratively train the segmentation model using multiple training sets to obtain the parameters of the segmentation model in each round. The storage module is used to store the optimal parameters using the parameters of each round and the validation set to obtain the final segmentation model, so as to perform segmentation processing on the multimodal remote sensing image to be processed using the final segmentation model and obtain the final segmentation result.

Images