Multi-modal remote sensing image segmentation method and device based on cross fusion

Abstract

The invention provides a multi-modal remote sensing image segmentation method and device based on cross fusion, and relates to the technical field of image processing. Comprising the steps that a segmentation model is constructed, and the segmentation model comprises a plurality of first transformer layers, a first FFN, a first FAN, a plurality of second transformer layers, a second feature fusion network FFN, a second FAN, a plurality of third transformer layers, a third FFN, a third FAN, a plurality of fourth transformer layers, a fourth FFN, a neck network and a decoder; iteratively training the segmentation model by using the plurality of training sets; and storing the optimal parameters by using the parameters of each round and the verification set to obtain a final segmentation model so as to obtain a final segmentation result of the to-be-processed multi-modal remote sensing image through the final segmentation model. And the accuracy of multi-modal remote sensing image segmentation is relatively high.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a multimodal remote sensing image segmentation method and device based on cross fusion. Background Art

[0002] Multimodal remote sensing image segmentation is a method that combines data from different remote sensing sensors to improve the accuracy and details of object segmentation. This method takes advantage of the complementarity of information provided by different sensors. By fusing multiple remote sensing data, it can more comprehensively describe the characteristics of objects, thereby improving the accuracy of object segmentation. The application of multimodal remote sensing image segmentation is based on the principle that different sensors have different sensitivities to the reflection or emission characteristics of objects, so the information they provide differs in space, spectrum, time, etc. By combining this difference information, objects can be identified and segmented more effectively, especially in complex environments, such as urban areas or vegetation-covered areas, where multimodal remote sensing image segmentation is particularly important.

[0003] At present, the specific operation of multimodal remote sensing image segmentation is: first, obtain image features under different modalities separately; second, use the attention mechanism to fuse features of each scale separately, and use upsampling to gradually fuse all scale information and obtain segmentation results; finally, calculate the Dice loss and focal loss between the segmentation result and the true label, and use back propagation to complete end-to-end training. However, the existing technology uses independent feature extraction. Due to the lack of information interaction between different modalities, the independent feature extraction process leads to certain information duplication in the extracted information, and cannot effectively take into account the complementary information between modalities, which greatly affects the extraction rate and utilization rate of information, resulting in a low accuracy rate of multimodal remote sensing image segmentation. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide a multimodal remote sensing image segmentation method and device based on cross fusion to solve the problem of low accuracy of multimodal remote sensing image segmentation.

[0005] To solve the above technical problems, the embodiments of the present invention provide the following technical solutions:

[0006] A first aspect of the present invention provides a multimodal remote sensing image segmentation method based on cross fusion, the method comprising:

[0007] Acquire multiple data sets, preprocess the multiple data sets, and divide the multiple data sets after preprocessing into corresponding training sets and validation sets, respectively, wherein the multiple data sets are sets of multimodal remote sensing images with labels, and the labels are used to mark the terrain in the multimodal remote sensing images;

[0008] Constructing a segmentation model, the segmentation model includes multiple first transformer layers, a first feature fusion network, a first feature adaptive network, multiple second transformer layers, a second feature fusion network, a second feature adaptive network, multiple third transformer layers, a third feature fusion network, a third feature adaptive network, multiple fourth transformer layers, a fourth feature fusion network, a neck network and a decoder, wherein the number of the multiple first transformer layers, the number of the multiple second transformer layers, the number of the multiple third transformer layers and the number of the multiple fourth transformer layers are the same as the number of categories of the multimodal remote sensing image;

[0009] The segmentation model is iteratively trained using multiple training sets to obtain the parameters of each round of the segmentation model;

[0010] The optimal parameters are saved by using each round of parameters and the validation set to obtain the final segmentation model, so that the multimodal remote sensing image to be processed can be segmented through the final segmentation model to obtain the final segmentation result.

[0011] A second aspect of the present invention provides a multimodal remote sensing image segmentation device based on cross fusion, the device comprising:

[0012] A partitioning module is used to obtain multiple data sets, preprocess the multiple data sets, and divide the preprocessed multiple data sets into corresponding training sets and verification sets, respectively. The multiple data sets are sets of multimodal remote sensing images with labels, and the labels are used to mark the terrain in the multimodal remote sensing images;

[0013] A construction module is used to construct a segmentation model, wherein the segmentation model includes a plurality of first transformer layers, a first feature fusion network, a first feature adaptive network, a plurality of second transformer layers, a second feature fusion network, a second feature adaptive network, a plurality of third transformer layers, a third feature fusion network, a third feature adaptive network, a plurality of fourth transformer layers, a fourth feature fusion network, a neck network and a decoder, wherein the number of the plurality of first transformer layers, the number of the plurality of second transformer layers, the number of the plurality of third transformer layers and the number of the plurality of fourth transformer layers are the same as the number of categories of the multimodal remote sensing image;

[0014] A training module, used to iteratively train the segmentation model using multiple training sets to obtain parameters of each round of the segmentation model;

[0015] The saving module is used to save the optimal parameters using each round of parameters and the verification set to obtain a final segmentation model, so as to perform segmentation processing on the multimodal remote sensing image to be processed through the final segmentation model to obtain the final segmentation result.

[0016] Compared with the prior art, the multimodal remote sensing image segmentation method and device based on cross fusion provided by the present invention obtain multiple data sets, preprocess the multiple data sets, and divide the preprocessed multiple data sets into corresponding training sets and verification sets respectively, the multiple data sets are sets of multimodal remote sensing images with labels, and the labels are used to mark the terrain in the multimodal remote sensing images; construct a segmentation model, the segmentation model includes multiple first transformer layers, a first feature fusion network, a first feature adaptive network, multiple second transformer layers, a second feature fusion network, a second feature adaptive network, multiple third transformer layers, a third feature fusion network, a third feature The feature adaptive network, the plurality of fourth transformer layers, the fourth feature fusion network, the neck network and the decoder, the number of the plurality of first transformer layers, the number of the plurality of second transformer layers, the number of the plurality of third transformer layers, and the number of the plurality of fourth transformer layers are all the same as the number of categories of the multimodal remote sensing image; the segmentation model is iteratively trained using the plurality of training sets to obtain the parameters of each round of the segmentation model; the optimal parameters are saved using the parameters of each round and the validation set to obtain the final segmentation model, so as to perform segmentation processing on the multimodal remote sensing image to be processed through the final segmentation model to obtain the final segmentation result. In this way, through the first feature fusion network, the first feature adaptive network, the second feature fusion network, the second feature adaptive network, the third feature fusion network, the third feature adaptive network and the fourth feature fusion network in the segmentation model, the interactive extraction of information between the multimodal remote sensing images is realized, the complementary information between the modalities is effectively taken into account, and the fusion feature map is adaptively projected to match the feature space of each modal feature, which is conducive to the subsequent better extraction of modal features, so that the accuracy of multimodal remote sensing image segmentation is higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] By reading the detailed description below with reference to the accompanying drawings, the above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0018] Figure 1 The flowchart of the multimodal remote sensing image segmentation method based on cross fusion is schematically shown;

[0019] Figure 2 The execution diagram of the segmentation model is schematically shown;

[0020] Figure 3 The structure of a multimodal remote sensing image segmentation device based on cross fusion is schematically shown. DETAILED DESCRIPTION

[0021] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0022] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in the present invention should have the common meanings understood by those skilled in the art to which the present invention belongs.

[0023] The method in the embodiment of the present invention is described in detail below.

[0024] Figure 1 The flowchart of the multimodal remote sensing image segmentation method based on cross fusion in the embodiment of the present invention is schematically shown. Figure 1 As shown, the method may include:

[0025] S101, obtaining multiple data sets, preprocessing the multiple data sets, and dividing the preprocessed multiple data sets into corresponding training sets and validation sets.

[0026] Among them, the multiple data sets are collections of multimodal remote sensing images with labels, and the labels are used to mark the terrain in the multimodal remote sensing images.

[0027] The multiple datasets include multiple Potsdam datasets and multiple Wuhan University Optical and Synthetic Aperture Radar Dataset (WHU-OPT-SAR) datasets. Each Potsdam dataset and each WHU-OPT-SAR dataset are datasets containing multi-modal remote sensing images.

[0028] Multimodal remote sensing images include visible light remote sensing images, multispectral remote sensing images, infrared remote sensing images, near-infrared remote sensing images, SAR remote sensing images and digital surface model (DSM) remote sensing images.

[0029] The Potsdam dataset contains 28 6000*6000 remote sensing images, including four-band remote sensing tagged image file format (TIFF) files of Red Green Blue Infrared (RGBIR) and single-band DSM remote sensing images. The labels of the remote sensing images are marked with 6 types of terrain and stored in TIFF format in the form of image masks. The step size is set to 512, and each remote sensing image is cropped into 512*512 non-overlapping image blocks. The cropped remote sensing images are used as the dataset for the segmentation model.

[0030] The WHU-OPT-SAR dataset contains 100 5556*3704 remote sensing images, consisting of four bands: near infrared, red, green, and SAR. The remote sensing image labels annotate seven types of terrain and are stored in TIFF format in the form of image masks. The step size is set to 512, and each remote sensing image is cropped into 512*512 non-overlapping image blocks. The cropped remote sensing images are used as the dataset for the segmentation model.

[0031] Preprocessing of multiple datasets, including:

[0032] Step A1: Standardize and normalize multiple data sets in sequence to obtain multiple normalized data sets.

[0033] Multiple data sets are standardized to scale the multimodal remote sensing images in the multiple data sets to a range according to the calculated mean and standard deviation so that they satisfy the normal distribution.

[0034] Normalize the standardized data sets to scale them into a small specific interval, and obtain the normalized data sets. By default, the maximum and minimum normalization is used to scale them to [0,1].

[0035] Step A2: Perform data enhancement on the normalized multiple data sets to obtain multiple preprocessed data sets.

[0036] Data augmentation includes random horizontal flipping, scaling, and random crops.

[0037] Specifically, the normalized multiple data sets are randomly horizontally flipped, scaled, and randomly cropped in sequence to obtain multiple preprocessed data sets.

[0038] Random horizontal flipping is to randomly flip the normalized multimodal remote sensing images in multiple data sets along the vertical center axis, that is, to make the normalized multimodal remote sensing images in multiple data sets mirror-symmetric left and right, and the default probability is 0.5.

[0039] Scaling is to randomly scale the multimodal remote sensing images in multiple data sets that have been randomly horizontally flipped, and the size of the random scaling is not fixed.

[0040] Random cropping is to randomly crop a sub-area of ​​the multimodal remote sensing images in the scaled data sets according to the preset size as a training sample, with a default probability of 0.5. The preset size is 224*224.

[0041] After preprocessing multiple data sets, the preprocessed data sets are randomly divided into corresponding training sets and validation sets, with the training set accounting for 80% and the validation set accounting for 20%.

[0042] The multimodal remote sensing images in each data set can be extracted and saved separately according to the categories of the multimodal remote sensing images, and used as data for subsequent parallel experiments.

[0043] S102: construct a segmentation model.

[0044] The segmentation model includes a plurality of first transformer layers, a first feature fusion network, a first feature adaptive network, a plurality of second transformer layers, a second feature fusion network, a second feature adaptive network, a plurality of third transformer layers, a third feature fusion network, a third feature adaptive network, a plurality of fourth transformer layers, a fourth feature fusion network, a neck network and a decoder. The number of the plurality of first transformer layers, the number of the plurality of second transformer layers, the number of the plurality of third transformer layers and the number of the plurality of fourth transformer layers are the same as the number of categories of the multimodal remote sensing image.

[0045] Among them, each first transformer layer, each second transformer layer, each third transformer layer and each fourth transformer layer include multiple blocks. The number of multiple blocks corresponding to each first transformer layer, the number of multiple blocks corresponding to each second transformer layer, the number of multiple blocks corresponding to each third transformer layer and the number of multiple blocks corresponding to each fourth transformer layer are all different. Each block includes a third normalization layer, a multi-head attention layer, a fourth normalization layer and a multilayer perceptron (MLP) in sequence.

[0046] Exemplarily, when the number of categories of the multimodal remote sensing image is 3, the corresponding number of multiple first transformer layers, the number of multiple second transformer layers, the number of multiple third transformer layers, and the number of multiple fourth transformer layers are all 3.

[0047] Exemplarily, each first transformer layer includes 2 blocks, each second transformer layer includes 5 blocks, each third transformer layer includes 8 blocks, and each fourth transformer layer includes 11 blocks.

[0048] The neck network includes the third 1×1 convolution layer and the 3×3 convolution layer. The decoder includes the fourth 1×1 convolution layer and the fifth 1×1 convolution layer.

[0049] The global attention transform (Vision Transformer_base, ViT_base) is used as the backbone network. The backbone network, namely ViT_base, includes multiple first transformer layers, a first feature fusion network (Feature FusionNetwork, FFN), a first feature adaptive network (Feature Adaptive Network, FAN), multiple second transformer layers, a second FFN, a second FAN, multiple third transformer layers, a third FFN, a third FAN, multiple fourth transformer layers, and a fourth FFN.

[0050] FFN and FAN are called parallel perception adaptive cross-modal layers.

[0051] S103, using multiple training sets to iteratively train the segmentation model to obtain parameters of each round of the segmentation model.

[0052] Specifically, the segmentation model is iteratively trained using multiple training sets to obtain the parameters of each round of the segmentation model, including:

[0053] Step B1: Use multiple training sets to iteratively train the segmentation model to obtain corresponding multiple rounds of preset segmentation results.

[0054] Among them, each training set includes remote sensing images of multiple categories, and one category of remote sensing images corresponds to a first transformer layer, a second transformer layer, a third transformer layer and a fourth transformer layer.

[0055] Figure 2 The execution diagram of the segmentation model is schematically shown, see Figure 2 As shown, the segmentation model is iteratively trained using multiple training sets to obtain corresponding multiple rounds of preset segmentation results, including:

[0056] Step B111: input the remote sensing images of each category in each training set into the corresponding first transformer layers in sequence, so that each first transformer layer outputs the corresponding first feature maps.

[0057] Step B112: Input each first feature map into the first feature fusion network so that the first feature fusion network outputs a first fused feature map.

[0058] Among them, the first feature fusion network, the second feature fusion network, the third feature fusion network and the fourth feature fusion network all include a first 1×1 convolutional layer, a first normalization layer, a splicing layer and a fusion layer.

[0059] Specifically, inputting each first feature map into a first feature fusion network so that the first feature fusion network outputs a first fused feature map includes:

[0060] Step B1121: Input each first feature map into the first 1×1 convolutional layer and the first normalization layer in sequence, so that the first normalization layer outputs each first intermediate feature map.

[0061] Among them, one first feature map corresponds to one first intermediate feature map.

[0062] Step B1122: Input each first intermediate feature map into the splicing layer, so that each first intermediate feature map is spliced ​​with the four-dimensional vector along the channel number dimension of the image, and outputs multiple features.

[0063] The shape of a four-dimensional vector is (N, 1, H, W), and all elements of (N, 1, H, W) are 1.

[0064] Step B1123: Input multiple features into the fusion layer so that the fusion layer outputs a first fusion feature map.

[0065] Specifically, multiple features are input into the fusion layer so that the fusion layer outputs a first fusion feature map, including:

[0066] Step B11231: Add dimensions to the first feature and the second feature respectively and calculate the Cartesian product to obtain the first calculated feature.

[0067] Step B11232: Rearrange the dimensions of the first calculated feature, flatten it, and add dimensions to obtain a first intermediate feature.

[0068] Step B11233: Rearrange dimensions and add dimensions to the third feature in sequence to obtain the first updated feature.

[0069] Step B11234: Calculate the Cartesian product of the first updated feature and the first intermediate feature to obtain a second calculated feature, and sequentially rearrange, flatten, and add dimensions to the second calculated feature to obtain a second intermediate feature.

[0070] Step B11235: Correspondingly, the n-th feature is re-arranged in dimension and added in dimension to obtain the n-2th updated feature; the Cartesian product of the n-2th updated feature and the n-2th intermediate feature is calculated to obtain the n-1th calculated feature, and the n-1th calculated feature is re-arranged in dimension, flattened and added in dimension to obtain the n-1th intermediate feature.

[0071] Step B11236: until all features are processed and the first fused feature map is obtained.

[0072] For example, there are 3 categories of multimodal remote sensing images, 3 first transformer layers, 3 first intermediate feature maps, and the shape of the four-dimensional vector is (N, 1, H, W), where all elements of (N, 1, H, W) are 1. Each first intermediate feature map is concatenated with the four-dimensional vector along the channel number (C) dimension of the image, and the number of the obtained features is 3, i.e., feature X1∈R N×(C1+1)×H×W , feature X2∈R N×(C2+1)×H×W and feature X3∈R N×(C3+1)×H×W Where N represents the number of samples in a batch, H and W represent the height and width of the feature size, C1, C2, and C3 represent the number of channels of each feature, R represents the real number domain, and R N×(C1+1)×H×W Represents a real number space with dimensions (N, C1+1, H, W). Add dimensions to the first feature, feature X1, and the second feature, feature X2 (one feature adds dimensions at N, and the other feature adds dimensions at C). After adding dimensions to feature X1 and feature X2, calculate the Cartesian product to obtain the first calculated feature. Rearrange the dimensions, flatten, and add dimensions to the first calculated feature in turn to obtain the first intermediate feature, feature X 1 ∈R N ×H×W×(C1+1)*(C2+1)×1 Then, the third feature, feature X3, is re-arranged and added to obtain the first updated feature, feature X 2 ∈R N×H×W×1×(C3+1) Then update the first feature, feature X 2 and the first intermediate feature, feature X 1 Calculate the Cartesian product to obtain the second calculated feature, and then rearrange, flatten, and add dimensions to the second calculated feature to obtain the second intermediate feature, feature X 3 ∈R N×(C1+1)*(C2+1)*(C3+1)×H×W .

[0073] Step B113: input the first fused feature map into the first feature adaptive network so that the first feature adaptive network outputs a second fused feature map.

[0074] Among them, the first feature adaptive network, the second feature adaptive network and the third feature adaptive network all include a second 1×1 convolutional layer and a second normalization layer.

[0075] Specifically, inputting the first fused feature map into the first feature adaptive network so that the first feature adaptive network outputs the second fused feature map, including: inputting the first fused feature map into the second 1×1 convolutional layer and the second normalization layer in sequence so that the second normalization layer outputs the second fused feature map.

[0076] The second fusion feature map is the modality fusion feature.

[0077] Step B114: Add the second fused feature map to each first feature map respectively to obtain corresponding second feature maps.

[0078] Each second feature map is used as the input of each second transformer layer.

[0079] Step B115: Input each second feature map into the corresponding second transformer layer and the second feature fusion network in sequence, so that the second feature fusion network outputs a third fused feature map; input the third fused feature map into the second feature adaptive network, so that the second feature adaptive network outputs a fourth fused feature map.

[0080] For the specific operation of sequentially inputting each second feature map into the corresponding second transformer layer and the second feature fusion network so that the second feature fusion network outputs the third fused feature map, please refer to the above steps B111 and B112 (including steps B1121 to B1123).

[0081] For the specific operation of inputting the third fused feature map into the second feature adaptive network so that the second feature adaptive network outputs the fourth fused feature map, refer to the above step B113.

[0082] Step B116: Add the fourth fused feature map to each second feature map respectively to obtain corresponding third feature maps.

[0083] Step B117: Input each third feature map into the corresponding third transformer layer and the third feature fusion network in sequence, so that the third feature fusion network outputs a fifth fused feature map; input the fifth fused feature map into the third feature adaptive network, so that the third feature adaptive network outputs a sixth fused feature map.

[0084] The specific operation of inputting each third feature map into the corresponding third transformer layer and the third feature fusion network in sequence so that the third feature fusion network outputs the fifth fused feature map refers to the above steps B111 and B112 (including steps B1121 to B1123).

[0085] For the specific operation of inputting the fifth fused feature map into the third feature adaptive network so that the third feature adaptive network outputs the sixth fused feature map, refer to the above step B113.

[0086] Step B118: Add the sixth fused feature map to each of the third feature maps to obtain corresponding fourth feature maps.

[0087] Step B119: input each fourth feature map into the corresponding fourth transformer layer and the fourth feature fusion network in sequence, so that the fourth feature fusion network outputs a seventh fused feature map.

[0088] For the specific operation of sequentially inputting each fourth feature map into the corresponding fourth transformer layer and the fourth feature fusion network so that the fourth feature fusion network outputs the seventh fused feature map, refer to the above steps B111 and B112 (including steps B1121 to B1123).

[0089] The feature fusion network realizes the interactive extraction of information between multiple modal remote sensing images, effectively taking into account the complementary information between modalities. The feature adaptive network adaptively projects the fused feature map to match the feature space of each modal feature, which is conducive to the subsequent better extraction of modal features.

[0090] Step B120: input the first fused feature map, the third fused feature map, the fifth fused feature map and the seventh fused feature map into the neck network and the decoder in sequence, so that the decoder outputs multiple rounds of preset segmentation results.

[0091] Specifically, the fused feature maps from the four levels, namely the first fused feature map, the third fused feature map, the fifth fused feature map, and the seventh fused feature map, are respectively applied with the third 1×1 convolution layer with independent parameters to adjust their number of channels to 768. Then, bilinear interpolation is used for each layer of feature maps to generate a set of feature maps with a scaling ratio of [4, 2, 1, 0.5]. Finally, each layer of feature maps is passed through a 3×3 convolution layer with independent parameters to obtain multi-scale features with a number of channels of 768.

[0092] The multi-scale feature map output by the neck network is input to the decoder, and the decoder uses the upper perception network (UperNet). For each scale of the feature map output by the neck network, UperNet uses upsampling (such as bilinear interpolation or transposed convolution) to expand its size to the same size as the original input image. The upsampled feature map passes through a fourth 1×1 convolution layer to reduce the number of channels to reduce computational complexity. The upsampled feature map is fused with the low-level features of the corresponding scale by adding the corresponding elements. Finally, the fusion result is passed through a fifth 1×1 convolution layer to obtain multiple rounds of preset segmentation results. The number of channels of the fifth 1×1 convolution corresponds to the number of categories of semantic segmentation, which is used to predict the category of each pixel.

[0093] Step B2: Calculate the cross entropy loss function value for each round based on the preset segmentation results and corresponding labels for each round.

[0094] The cross entropy loss function is used to calculate the value of each round of cross entropy loss function. The value of each round of cross entropy loss function reflects the difference between multiple rounds of preset segmentation results and the label input by the segmentation model.

[0095] Step B3: According to the value of the cross entropy loss function in each round, the back propagation algorithm is used to update the parameters of each round of the segmentation model.

[0096] The optimizer selects the Adam optimizer with weight decay (AdamW). Different learning rates and weight decay are used for models corresponding to different transformer layers, and are set as needed. The learning rate strategy adopts the poly strategy, and the number of training rounds is set to 50 epochs.

[0097] According to the value of the cross entropy loss function in each round, the back propagation algorithm and optimizer are used to update the parameters of each round of the segmentation model so that the segmentation model can better predict the target.

[0098] S104, using each round of parameters and the validation set to save the optimal parameters to obtain a final segmentation model, so as to perform segmentation processing on the multimodal remote sensing image to be processed through the final segmentation model to obtain a final segmentation result.

[0099] Specifically, the optimal parameters are saved using each round of parameters and validation set to obtain the final segmentation model, including:

[0100] Step C1: Use the parameters of each round to process the validation set respectively to obtain the prediction results of each round corresponding to the validation set.

[0101] Step C2: Calculate the intersection-and-union ratios of multiple categories of remote sensing images in each round of prediction results to obtain multiple intersection-and-union ratios corresponding to each round of prediction results.

[0102] In addition to calculating the intersection-over-union ratio of multiple categories of remote sensing images in each round of prediction results, the F1 scores of multiple categories of remote sensing images in each round of prediction results can also be calculated.

[0103] Step C3: Calculate the mean of multiple intersection-over-union ratios corresponding to each round of prediction results, and select the parameter corresponding to the highest mean to save, so as to obtain the final segmentation model, so as to segment the multimodal remote sensing image to be processed through the final segmentation model to obtain the final segmentation result.

[0104] After using each round of parameters and the validation set to save the optimal parameters to obtain the final segmentation model, the method also includes: inputting the multimodal remote sensing image to be processed into the final segmentation model so that the final segmentation model outputs the final segmentation result.

[0105] The present invention can be used to register multi-modal remote sensing images acquired by satellites, and then extract complementary information between different modalities for image segmentation and obtain a segmentation map with ground object category labels.

[0106] Based on the above Figure 1It can be seen from the implementation method that the embodiment of the present invention obtains multiple data sets, preprocesses the multiple data sets, and divides the preprocessed multiple data sets into corresponding training sets and verification sets, respectively. The multiple data sets are sets of multimodal remote sensing images with labels, and the labels are used to mark the terrain in the multimodal remote sensing images; constructs a segmentation model, which includes multiple first transformer layers, a first feature fusion network, a first feature adaptive network, multiple second transformer layers, a second feature fusion network, a second feature adaptive network, multiple third transformer layers, a third feature fusion network, a third feature adaptive network, multiple fourth transformer layers, a fourth feature fusion network, a neck network and a decoder, and the number of the multiple first transformer layers, the number of the multiple second transformer layers, the number of the multiple third transformer layers, and the number of the multiple fourth transformer layers are all the same as the number of categories of the multimodal remote sensing images; uses multiple training sets to iteratively train the segmentation model to obtain each round of parameters of the segmentation model; uses each round of parameters and the verification set to save the optimal parameters to obtain a final segmentation model, so that the multimodal remote sensing image to be processed is segmented by the final segmentation model to obtain the final segmentation result. In this way, through the first feature fusion network, the first feature adaptive network, the second feature fusion network, the second feature adaptive network, the third feature fusion network, the third feature adaptive network and the fourth feature fusion network in the segmentation model, the interactive extraction of information between multiple modal remote sensing images is realized, the complementary information between the modalities is effectively taken into account, and the fused feature map is adaptively projected to match the feature space of each modal feature, which is conducive to the subsequent better extraction of modal features and makes the accuracy of multimodal remote sensing image segmentation higher.

[0107] Based on the same inventive concept, as an implementation of the above-mentioned multimodal remote sensing image segmentation method based on cross-fusion, an embodiment of the present invention further provides a multimodal remote sensing image segmentation device based on cross-fusion. Figure 3 is a structural diagram of a multimodal remote sensing image segmentation device based on cross fusion in an embodiment of the present invention, see Figure 3 As shown, the device may include:

[0108] A division module 301 is used to obtain multiple data sets, preprocess the multiple data sets, and divide the preprocessed multiple data sets into corresponding training sets and verification sets, respectively, where the multiple data sets are sets of multimodal remote sensing images with labels, and the labels are used to mark the terrain in the multimodal remote sensing images;

[0109] A construction module 302 is used to construct a segmentation model, wherein the segmentation model includes a plurality of first transformer layers, a first feature fusion network, a first feature adaptive network, a plurality of second transformer layers, a second feature fusion network, a second feature adaptive network, a plurality of third transformer layers, a third feature fusion network, a third feature adaptive network, a plurality of fourth transformer layers, a fourth feature fusion network, a neck network and a decoder, wherein the number of the plurality of first transformer layers, the number of the plurality of second transformer layers, the number of the plurality of third transformer layers and the number of the plurality of fourth transformer layers are the same as the number of categories of the multimodal remote sensing image;

[0110] The training module 303 is used to iteratively train the segmentation model using multiple training sets to obtain parameters of each round of the segmentation model;

[0111] The saving module 304 is used to save the optimal parameters using each round of parameters and the verification set to obtain a final segmentation model, so as to perform segmentation processing on the multimodal remote sensing image to be processed through the final segmentation model to obtain a final segmentation result.

[0112] The training module 303 is specifically used to iteratively train the segmentation model using multiple training sets to obtain corresponding multiple rounds of preset segmentation results; calculate the cross entropy loss function value of each round according to the preset segmentation results of each round and the corresponding labels; and update the parameters of each round of the segmentation model using the back propagation algorithm according to the cross entropy loss function value of each round.

[0113] In the training module 303, each training set includes multiple categories of remote sensing images, and one category of remote sensing images corresponds to a first transformer layer, a second transformer layer, a third transformer layer and a fourth transformer layer; the segmentation model is iteratively trained using multiple training sets to obtain corresponding multiple rounds of preset segmentation results, including: inputting the remote sensing images of each category in each training set into the corresponding first transformer layers in sequence, so that each first transformer layer outputs the corresponding first feature maps; inputting each first feature map into the first feature fusion network, so that the first feature fusion network outputs the first fused feature map; inputting the first fused feature map into the first feature adaptive network, so that the first feature adaptive network outputs the second fused feature map; adding the second fused feature map to each first feature map, respectively, to obtain the corresponding second feature maps; inputting each second feature map into the corresponding second transformer layer and the second feature fusion network, so that the second feature fusion network outputs a third fused feature map; the third fused feature map is input into the second feature adaptive network, so that the second feature adaptive network outputs a fourth fused feature map; the fourth fused feature map is added to each second feature map respectively to obtain corresponding third feature maps; each third feature map is sequentially input into the corresponding third transformer layer and the third feature fusion network, so that the third feature fusion network outputs a fifth fused feature map; the fifth fused feature map is input into the third feature adaptive network, so that the third feature adaptive network outputs a sixth fused feature map; the sixth fused feature map is added to each third feature map respectively to obtain corresponding fourth feature maps; each fourth feature map is sequentially input into the corresponding fourth transformer layer and the fourth feature fusion network, so that the fourth feature fusion network outputs a seventh fused feature map; the first fused feature map, the third fused feature map, the fifth fused feature map and the seventh fused feature map are sequentially input into the neck network and the decoder, so that the decoder outputs multiple rounds of preset segmentation results.

[0114] Constructing module 302, the first feature fusion network, the second feature fusion network, the third feature fusion network and the fourth feature fusion network all include a first 1×1 convolutional layer, a first normalization layer, a concatenation layer and a fusion layer.

[0115] The training module 303 sequentially inputs each first feature map into the first 1×1 convolutional layer and the first normalization layer, so that the first normalization layer outputs each first intermediate feature map, and one first feature map corresponds to one first intermediate feature map; each first intermediate feature map is input into the concatenation layer, so that each first intermediate feature map is respectively concatenated with the four-dimensional vector along the channel number dimension of the image to output multiple features; and the multiple features are input into the fusion layer, so that the fusion layer outputs the first fused feature map.

[0116] The training module 303 inputs multiple features into the fusion layer so that the fusion layer outputs a first fused feature map, including: adding dimensions to the first feature and the second feature respectively and calculating the Cartesian product to obtain a first calculated feature; rearranging the dimensions, flattening and adding dimensions to the first calculated feature in sequence to obtain a first intermediate feature; rearranging the dimensions and adding dimensions to the third feature in sequence to obtain a first updated feature; calculating the Cartesian product of the first updated feature and the first intermediate feature to obtain a second calculated feature, and rearranging the dimensions, flattening and adding dimensions to the second calculated feature in sequence to obtain a second intermediate feature; correspondingly, rearranging the dimensions and adding dimensions to the nth feature in sequence to obtain the n-2th updated feature; calculating the Cartesian product of the n-2th updated feature and the n-2th intermediate feature to obtain the n-1th calculated feature, and rearranging the dimensions, flattening and adding dimensions to the n-1th calculated feature in sequence to obtain the n-1th intermediate feature; until all features are processed to obtain the first fused feature map.

[0117] The training module 303, the first feature adaptive network, the second feature adaptive network and the third feature adaptive network, all include a second 1×1 convolution layer and a second normalization layer; the first fused feature map is input into the first feature adaptive network so that the first feature adaptive network outputs a second fused feature map, including: the first fused feature map is input into the second 1×1 convolution layer and the second normalization layer in sequence so that the second normalization layer outputs a second fused feature map.

[0118] Constructing module 302, each first transformer layer, each second transformer layer, each third transformer layer and each fourth transformer layer includes multiple blocks; the number of multiple blocks corresponding to each first transformer layer, the number of multiple blocks corresponding to each second transformer layer, the number of multiple blocks corresponding to each third transformer layer and the number of multiple blocks corresponding to each fourth transformer layer are all different; each block includes a third normalization layer, a multi-head attention layer, a fourth normalization layer and a multilayer perceptron in sequence.

[0119] The saving module 304 is specifically used to use each round of parameters to process the verification set respectively to obtain each round of prediction results corresponding to the verification set; calculate the intersection and union ratios of multiple categories of remote sensing images in each round of prediction results to obtain multiple intersection and union ratios corresponding to each round of prediction results; calculate the mean of multiple intersection and union ratios corresponding to each round of prediction results, and select the parameters corresponding to the highest mean to save, so as to obtain the final segmentation model, so as to perform segmentation processing on the multimodal remote sensing image to be processed through the final segmentation model to obtain the final segmentation result.

[0120] It should be pointed out here that: the above description of the embodiment of the multimodal remote sensing image segmentation device based on cross fusion is similar to the description of the embodiment of the multimodal remote sensing image segmentation method based on cross fusion, and has similar beneficial effects as the embodiment of the multimodal remote sensing image segmentation method based on cross fusion. For technical details not disclosed in the embodiment of the multimodal remote sensing image segmentation device based on cross fusion of the embodiment of the present invention, please refer to the description of the embodiment of the multimodal remote sensing image segmentation method based on cross fusion of the present invention for understanding.

[0121] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A multimodal remote sensing image segmentation method based on cross fusion, characterized in that: The multimodal remote sensing image segmentation method based on cross fusion includes: Acquire multiple data sets, preprocess the multiple data sets, and divide the preprocessed multiple data sets into corresponding training sets and validation sets, respectively, wherein the multiple data sets are sets of multimodal remote sensing images with labels, and the labels are used to mark the terrain in the multimodal remote sensing images; Constructing a segmentation model, the segmentation model includes a plurality of first transformer layers, a first feature fusion network, a first feature adaptive network, a plurality of second transformer layers, a second feature fusion network, a second feature adaptive network, a plurality of third transformer layers, a third feature fusion network, a third feature adaptive network, a plurality of fourth transformer layers, a fourth feature fusion network, a neck network and a decoder, wherein the number of the plurality of first transformer layers, the number of the plurality of second transformer layers, the number of the plurality of third transformer layers and the number of the plurality of fourth transformer layers are the same as the number of categories of the multimodal remote sensing image; Iteratively training the segmentation model using multiple training sets to obtain parameters of each round of the segmentation model; The optimal parameters are saved by using the parameters of each round and the verification set to obtain a final segmentation model, so as to perform segmentation processing on the multimodal remote sensing image to be processed through the final segmentation model to obtain a final segmentation result.

2. The multimodal remote sensing image segmentation method based on cross fusion according to claim 1, characterized in that: The iterative training of the segmentation model using multiple training sets to obtain each round parameter of the segmentation model includes: Iteratively training the segmentation model using the multiple training sets to obtain corresponding multiple rounds of preset segmentation results; Calculate the cross entropy loss function value of each round according to the preset segmentation results of each round and the corresponding labels; According to the cross entropy loss function value of each round, the back propagation algorithm is used to update the parameters of each round of the segmentation model.

3. The multimodal remote sensing image segmentation method based on cross fusion according to claim 2 is characterized in that: Each training set includes multiple categories of remote sensing images, and one category of remote sensing images corresponds to a first transformer layer, a second transformer layer, a third transformer layer, and a fourth transformer layer; The iterative training of the segmentation model using the multiple training sets to obtain corresponding multiple rounds of preset segmentation results includes: Inputting the remote sensing images of each category in each training set into the corresponding first transformer layers in sequence, so that each first transformer layer outputs the corresponding first feature map; Inputting each of the first feature maps into the first feature fusion network so that the first feature fusion network outputs a first fused feature map; Inputting the first fused feature map into the first feature adaptive network so that the first feature adaptive network outputs a second fused feature map; Adding the second fused feature map to each of the first feature maps respectively to obtain corresponding second feature maps; Inputting the second feature maps into the corresponding second transformer layers and the second feature fusion network in sequence, so that the second feature fusion network outputs a third fused feature map; inputting the third fused feature map into the second feature adaptive network, so that the second feature adaptive network outputs a fourth fused feature map; Adding the fourth fused feature map to each of the second feature maps respectively to obtain corresponding third feature maps; Inputting the third feature maps into the corresponding third transformer layers and the third feature fusion network in sequence, so that the third feature fusion network outputs a fifth fused feature map; inputting the fifth fused feature map into the third feature adaptive network, so that the third feature adaptive network outputs a sixth fused feature map; Adding the sixth fused feature map to each of the third feature maps respectively to obtain corresponding fourth feature maps; Inputting the fourth feature maps into the corresponding fourth transformer layers and the fourth feature fusion network in sequence, so that the fourth feature fusion network outputs a seventh fused feature map; The first fused feature map, the third fused feature map, the fifth fused feature map and the seventh fused feature map are sequentially input into the neck network and the decoder, so that the decoder outputs the multiple rounds of preset segmentation results.

4. The multimodal remote sensing image segmentation method based on cross fusion according to claim 1, characterized in that: The first feature fusion network, the second feature fusion network, the third feature fusion network and the fourth feature fusion network all include a first 1×1 convolutional layer, a first normalization layer, a splicing layer and a fusion layer.

5. The multimodal remote sensing image segmentation method based on cross fusion according to claim 4 is characterized in that: The step of inputting each of the first feature maps into the first feature fusion network so that the first feature fusion network outputs a first fused feature map includes: Inputting the first feature maps into the first 1×1 convolutional layer and the first normalization layer in sequence, so that the first normalization layer outputs first intermediate feature maps, and one first feature map corresponds to one first intermediate feature map; Inputting each of the first intermediate feature maps into the concatenation layer, so that each of the first intermediate feature maps is concatenated with the four-dimensional vector along the channel number dimension of the image, and outputting a plurality of features; The multiple features are input into the fusion layer so that the fusion layer outputs the first fusion feature map.

6. The multimodal remote sensing image segmentation method based on cross fusion according to claim 5, characterized in that: The step of inputting the plurality of features into the fusion layer so that the fusion layer outputs the first fusion feature map comprises: Add dimensions to the first feature and the second feature respectively and calculate the Cartesian product to obtain the first calculated feature; Rearranging dimensions, flattening, and adding dimensions to the first calculated features in sequence to obtain a first intermediate feature; The third feature is re-arranged and added in turn to obtain the first updated feature; Calculating a Cartesian product of the first updated feature and the first intermediate feature to obtain a second calculated feature, and sequentially re-arranging dimensions, flattening, and adding dimensions to the second calculated feature to obtain a second intermediate feature; Correspondingly, the nth feature is sequentially re-arranged in dimension and added in dimension to obtain the n-2th updated feature; the Cartesian product is calculated for the n-2th updated feature and the n-2th intermediate feature to obtain the n-1th calculated feature, and the n-1th calculated feature is sequentially re-arranged in dimension, flattened and added in dimension to obtain the n-1th intermediate feature; Until all features are processed, the first fused feature map is obtained.

7. The multimodal remote sensing image segmentation method based on cross fusion according to claim 3 is characterized in that: The first feature adaptive network, the second feature adaptive network and the third feature adaptive network each include a second 1×1 convolutional layer and a second normalization layer; The step of inputting the first fused feature map into the first feature adaptive network so that the first feature adaptive network outputs a second fused feature map comprises: The first fused feature map is sequentially input into the second 1×1 convolutional layer and the second normalization layer, so that the second normalization layer outputs the second fused feature map.

8. The multimodal remote sensing image segmentation method based on cross fusion according to claim 1, characterized in that: Each first transformer layer, each second transformer layer, each third transformer layer and each fourth transformer layer includes multiple blocks; the number of multiple blocks corresponding to each first transformer layer, the number of multiple blocks corresponding to each second transformer layer, the number of multiple blocks corresponding to each third transformer layer and the number of multiple blocks corresponding to each fourth transformer layer are all different; each block includes a third normalization layer, a multi-head attention layer, a fourth normalization layer and a multilayer perceptron in sequence.

9. The multimodal remote sensing image segmentation method based on cross fusion according to claim 1, characterized in that: The method of using the parameters of each round and the validation set to save the optimal parameters to obtain a final segmentation model, so as to perform segmentation processing on the multimodal remote sensing image to be processed by the final segmentation model to obtain a final segmentation result, includes: Using the parameters of each round, the verification set is processed respectively to obtain the prediction results of each round corresponding to the verification set; Calculating the intersection-and-union ratios of multiple categories of remote sensing images in each round of prediction results to obtain multiple intersection-and-union ratios corresponding to each round of prediction results; The means of multiple intersection-over-union ratios corresponding to each round of prediction results are calculated, and the parameters corresponding to the highest mean are selected and saved to obtain the final segmentation model, so that the multimodal remote sensing image to be processed is segmented by the final segmentation model to obtain the final segmentation result.

10. A multimodal remote sensing image segmentation device based on cross fusion, characterized in that: The multimodal remote sensing image segmentation device based on cross fusion comprises: A partitioning module, used to obtain a plurality of data sets, pre-process the plurality of data sets, and divide the pre-processed plurality of data sets into corresponding training sets and validation sets, respectively, wherein the plurality of data sets are sets of multimodal remote sensing images with labels, and the labels are used to mark the terrain in the multimodal remote sensing images; A construction module, used to construct a segmentation model, wherein the segmentation model includes a plurality of first transformer layers, a first feature fusion network, a first feature adaptive network, a plurality of second transformer layers, a second feature fusion network, a second feature adaptive network, a plurality of third transformer layers, a third feature fusion network, a third feature adaptive network, a plurality of fourth transformer layers, a fourth feature fusion network, a neck network and a decoder, wherein the number of the plurality of first transformer layers, the number of the plurality of second transformer layers, the number of the plurality of third transformer layers and the number of the plurality of fourth transformer layers are the same as the number of categories of the multimodal remote sensing image; A training module, used to iteratively train the segmentation model using multiple training sets to obtain parameters of each round of the segmentation model; The saving module is used to save the optimal parameters using the parameters of each round and the verification set to obtain a final segmentation model, so as to perform segmentation processing on the multimodal remote sensing image to be processed through the final segmentation model to obtain a final segmentation result.

Images