Multimodal remote sensing image semantic segmentation method and system

Abstract

The invention relates to the technical field of image processing, in particular to a multi-modal remote sensing image semantic segmentation method and system, and the method comprises the steps: building a data set which comprises a plurality of image pairs, and each image pair comprises an SAR image and an optical image; constructing a multi-modal semantic segmentation model, wherein the multi-modal semantic segmentation model comprises a basic feature transformation layer, a multi-scale semantic acquisition layer, a significant feature cross fusion layer, a feature deep coding layer and a feature decoding layer which are connected in sequence; training and optimizing a multi-modal semantic segmentation model based on the data set to obtain a target multi-modal semantic segmentation model; and performing joint semantic segmentation on the SAR remote sensing image and the optical remote sensing image by adopting the target multi-mode semantic segmentation model to obtain a semantic segmentation result graph. The embodiment of the invention is at least beneficial to reducing the parameter quantity and calculation quantity of semantic segmentation of the multi-modal remote sensing image.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a multimodal remote sensing image semantic segmentation method and system. Background Art

[0002] Semantic segmentation of remote sensing images can realize pixel-level classification of images. It can not only obtain information on each target category in the image, but also label the pixel areas corresponding to each category with different colors, which is conducive to image interpretation and plays an important role in land cover analysis, urban planning, agricultural monitoring and other fields.

[0003] At present, remote sensing image semantic segmentation is still mainly based on high-resolution optical images. However, since the optical image imaging process is easily affected by weather factors such as clouds, fog, rain and snow, the quality of optical images is reduced, and thus cannot meet the requirements of semantic segmentation tasks. Therefore, considering the scattering echo imaging mechanism of SAR (Synthetic Aperture Radar) and the complementarity of its image information to optical images, it is of great significance to jointly perform semantic segmentation on two different modal images.

[0004] Deep learning methods have strong feature extraction capabilities and are widely used in the current semantic segmentation of remote sensing images. However, in order to obtain higher classification accuracy, the depth and complexity of neural network models gradually increase, and the implementation of related methods can only rely on servers. How to obtain better classification effects with smaller parameters and calculations is worthy of in-depth exploration. This is also the prerequisite for deploying neural networks in embedded systems for application. Summary of the invention

[0005] In view of this, the present invention aims to provide a method and system for semantic segmentation of multimodal remote sensing images, which at least helps to reduce the number of parameters and the amount of calculation for semantic segmentation of multimodal remote sensing images.

[0006] To achieve the above object, the technical solution created by the present invention is implemented as follows: The invention provides a multimodal remote sensing image semantic segmentation method, comprising: establishing a data set, the data set comprising multiple groups of image pairs, each group of image pairs comprising a SAR image and an optical image; constructing a multimodal semantic segmentation model, the multimodal semantic segmentation model comprising a basic feature transformation layer, a multi-scale semantic acquisition layer, a significant feature cross-fusion layer, a feature deep encoding layer and a feature decoding layer connected in sequence; training and optimizing the multimodal semantic segmentation model based on the data set to obtain a target multimodal semantic segmentation model; and using the target multimodal semantic segmentation model to perform joint semantic segmentation on the SAR remote sensing image and the optical remote sensing image to obtain a semantic segmentation result map.

[0007] Furthermore, establishing a data set includes: selecting a multimodal data set for remote sensing object classification as an initial data set, cropping images in the initial data set to a uniform size, and obtaining a data set after screening, wherein the number of channels of the optical image in the data set is 4, the number of channels of the SAR image in the data set is 1, and the size of the optical image is the same as that of the SAR image.

[0008] Furthermore, the basic feature transformation layer includes: a first convolution branch, the first convolution branch is used to perform convolution processing on the optical image to convert the optical image into a first modal basic feature; and a second convolution branch, the second convolution branch is used to perform convolution processing on the SAR image to convert the SAR image into a second modal basic feature.

[0009] Furthermore, the multi-scale semantic acquisition layer includes: a first multi-scale extraction module, which uses multiple receptive field branches to deeply extract the basic features of the first modality to obtain the multi-scale features of the first modality; a second multi-scale extraction module, which has the same structure as the first multi-scale extraction module, and uses multiple receptive field branches to deeply extract the basic features of the second modality to obtain the multi-scale features of the second modality.

[0010] Furthermore, the first multi-scale extraction module includes 5 parallel receptive field branches, which include 2 conventional convolution branches, 2 hole convolution branches and 1 pooling and upsampling processing branch. The 5 receptive field branches are used to process the first modal basic features respectively, and the features output by the 5 receptive field branches are connected to obtain the first modal multi-scale features.

[0011] Furthermore, the salient feature cross-fusion layer adopts a channel attention mechanism to obtain the first channel attention weight of the first modal multi-scale feature, and the second channel attention weight of the second modal multi-scale feature; the first channel attention weight is multiplied by the first modal multi-scale feature to obtain the first modal salient feature, and the second channel attention weight is multiplied by the second modal multi-scale feature to obtain the second modal salient feature; the first channel attention weight is multiplied by the second modal multi-scale feature to obtain the second cross-guided salient feature, and the second channel attention weight is multiplied by the first modal multi-scale feature to obtain the first cross-guided salient feature; the first modal salient feature, the second modal salient feature, the second cross-guided salient feature and the first cross-guided salient feature are fused to obtain the multimodal salient feature.

[0012] Furthermore, the feature deep encoding layer uses a densely connected network to encode multimodal salient features to obtain high-level features with rich semantic information.

[0013] Furthermore, the feature decoding layer decodes high-level features through skip connections combined with upsampling to obtain semantic segmentation results.

[0014] Furthermore, during the training process, the cross entropy loss is combined with the focal loss and the IoU loss to construct a joint loss function to constrain the back propagation process of the model parameters.

[0015] On the other hand, the present invention provides a multimodal remote sensing image semantic segmentation system, including: a multimodal semantic segmentation model, and the multimodal semantic segmentation model is used to implement the above-mentioned multimodal remote sensing image semantic segmentation method.

[0016] Compared with the prior art, the invention can achieve the following beneficial effects: in the multimodal remote sensing image semantic segmentation method provided by the embodiment of the present invention, an end-to-end multimodal semantic segmentation model with dual-branch input is proposed, and good semantic segmentation results are achieved on different data sets. In the multimodal remote sensing image semantic segmentation method, emphasis is placed on deep mining of different modal features, and each modality adopts a multi-scale semantic acquisition layer to acquire semantic information of a single modality. In the multi-scale semantic acquisition layer, the recognition ability of targets of different sizes is improved through different receptive fields. In the significant feature cross-fusion layer, the acquisition of significant features of different modal images and the fusion of heterogeneous features are strengthened through a cross-attention mechanism, and the significant features of the two modalities are cross-guided to each other, thereby increasing information interaction between different modalities and further improving the multimodal feature extraction capability. The feature deep encoding layer adopts a densely connected network for deep encoding of features, and realizes feature transmission and feature reuse through jump connections, effectively improving the efficiency of back propagation and laying the foundation for feature decoding. The model structure of the multimodal remote sensing image semantic segmentation method provided by the embodiment of the present invention is concise and efficient, and the number of parameters and the amount of calculation are greatly reduced, which lays a foundation for the application deployment of neural networks in embedded systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings constituting part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments and descriptions of the present invention are used to explain the present invention and do not constitute an improper limitation on the present invention. In the drawings: Figure 1 A flowchart of the implementation steps of the multimodal remote sensing image semantic segmentation method described in the embodiment of the present invention; Figure 2 A multimodal semantic segmentation model framework diagram according to an embodiment of the present invention; Figure 3 A schematic diagram of the structure of the multi-scale extraction module according to an embodiment of the present invention; Figure 4 A schematic diagram of the structure of the attention extraction module according to an embodiment of the present invention; Figure 5 The following are the classification results of each method on the DFC2020 dataset; Figure 6 The following are the classification results of each method on the WHU-OPT-SAR dataset. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the invention more clear, the invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described here are only used to explain the invention and do not constitute a limitation of the invention.

[0019] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0020] In the description of the invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing the invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the invention, unless otherwise specified, the meaning of "multiple" is two or more.

[0021] In the description of the invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the invention can be understood according to specific circumstances.

[0022] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0023] refer to Figure 1On the one hand, the present invention provides a multimodal remote sensing image semantic segmentation method, including: establishing a data set, the data set includes multiple groups of image pairs, each group of image pairs includes a SAR image and an optical image; constructing a multimodal semantic segmentation model, the multimodal semantic segmentation model includes a basic feature transformation layer, a multi-scale semantic acquisition layer, a significant feature cross-fusion layer, a feature deep encoding layer and a feature decoding layer connected in sequence; training and optimizing the multimodal semantic segmentation model based on the data set to obtain a target multimodal semantic segmentation model; using the target multimodal semantic segmentation model to perform joint semantic segmentation on the SAR remote sensing image and the optical remote sensing image to obtain a semantic segmentation result map.

[0024] The multimodal remote sensing image semantic segmentation method provided by the embodiment of the present invention has good classification effect and generalization ability.

[0025] In some embodiments, establishing a data set includes: selecting a remote sensing ground object classification multimodal data set as an initial data set, cropping the images in the initial data set to a uniform size, and obtaining a data set after screening, wherein the number of channels of the optical image in the data set is 4, the number of channels of the SAR image in the data set is 1, and the size of the optical image is the same as the size of the SAR image.

[0026] In some examples, the selected multimodal dataset for remote sensing object classification may include the DFC2020 dataset and the WHU-OPT-SAR dataset. To maintain the consistency of the image size input to the multimodal semantic segmentation model, the large-size image pairs of 5556×3704 pixels of the WHU-OPT-SAR dataset may be moved and cropped to obtain 29,400 pairs of small image pairs of 256×256 pixels. Typically, the DFC2020 dataset contains 6114 pairs of small image pairs of 256×256 pixels. To maintain the consistency of the number of model input channels, 4 bands are extracted from the 13 bands of the optical image in the DFC2020 dataset as 4-channel optical image inputs. The 4-channel optical image has the same number of channels as the 4-channel RGBN optical image in the WHU-OPT-SAR dataset. The SAR images in the two datasets are both single-channel grayscale images. After the above processing, the sizes of the SAR images and optical images in the datasets are unified to 256×256 pixels, the number of channels of the optical image is unified to 4, and the number of channels of the SAR image is unified to 1.

[0027] It can be understood that training and optimizing the multimodal semantic segmentation model based on the data set includes: inputting the paired optical images and SAR images in the data set into the multimodal semantic segmentation model for model training.

[0028] In some embodiments, reference Figure 2In the multimodal semantic segmentation model, the basic feature transformation layer uses two convolution branches to transform the basic features of different modal images respectively. The multi-scale semantic acquisition layer deeply extracts the basic features through multiple receptive field branches in the multi-scale extraction module MDEM to improve the adaptability of the multimodal semantic segmentation model to targets of different sizes. The significant feature cross-fusion layer uses the attention extraction module MCAM to obtain important feature information of each modality. At the same time, the modal cross branch is designed in the module to increase information interaction and guidance, and promote the fusion of the two modal features. The feature deep encoding layer introduces the densely connected network DenseNet to further encode the fused significant features to obtain high-level features with rich semantic information. The feature decoding layer effectively decodes the features through skip connections combined with upsampling, and outputs the semantic segmentation results of targets of different categories.

[0029] Specifically, in some embodiments, the basic feature transformation layer includes: a first convolution branch, the first convolution branch is used to perform convolution processing on the optical image to convert the optical image into a first modal basic feature; a second convolution branch, the second convolution branch is used to perform convolution processing on the SAR image to convert the SAR image into a second modal basic feature.

[0030] In some embodiments, the multi-scale semantic acquisition layer includes: a first multi-scale extraction module, which uses multiple receptive field branches to deeply extract the basic features of the first modality to obtain the multi-scale features of the first modality; a second multi-scale extraction module, which has the same structure as the first multi-scale extraction module, and uses multiple receptive field branches to deeply extract the basic features of the second modality to obtain the multi-scale features of the second modality.

[0031] In some embodiments, the first multi-scale extraction module includes five parallel receptive field branches, including two conventional convolution branches, two hole convolution branches and one pooling and upsampling processing branch. The five receptive field branches are used to process the first modal basic features respectively, and the features output by the five receptive field branches are connected to obtain the first modal multi-scale features.

[0032] It is understandable that scale information and salient features are very important for semantic segmentation, especially for the extraction and fusion of heterogeneous features of images of different modalities. Based on this, the first multi-scale extraction module and the second multi-scale extraction module provided in the embodiment of the present invention both adopt Figure 3 The multi-scale extraction module MDEM shown in the figure is specifically referred to in Figure 3 The multi-scale extraction module MDEM includes five receptive field branches, which are the first conventional convolution branch, the second conventional convolution branch, the first dilated convolution branch, the second dilated convolution branch, and the pooling and upsampling processing branch. Figure 3middle F Base represents the basic features of each modal branch, that is, the first modal basic features or the second modal basic features, F MDEM Represents the multi-scale features of each modal branch obtained, that is, the first modal multi-scale features or the second modal multi-scale features, k , d as well as s Respectively represent the size of the convolution kernel, the dilation rate, and the pooling step size. The receptive field of the first conventional convolution branch can be 3*3, and the receptive field of the second conventional convolution branch can be 5*5. The two dilated convolution branches respectively expand the receptive field by connecting three dilated convolutions in series. d The design complies with the HDC (Hybird Dilated Convolution) criterion, as shown in the following formula: ; in, r i represents the void ratio, i =1…… n , n represents the number of dilated convolutions, M i represents the distance between two non-zero elements in the i-th layer, M i+1 Represents the distance between two non-zero elements in the (i+1)th layer. If we assume M n =r n , and can satisfy M 2 ≤K， It means that the corresponding void ratio is reasonably designed and will not produce grid effect and cause adverse effects. K represents the convolution kernel size, for example, K=3. In some examples, the dilated filter of the three dilated convolution branches connected in series in the first dilated convolution branch can be [ 1,2,3 ], the dilated filter of the three dilated convolution branches in series can be [ 1,2,5 ], so that the receptive fields can be increased to 13*13 and 17*17 respectively under the premise of meeting the criteria. Through the combination of 5 receptive field branches, the multi-scale extraction module MDEM can learn the feature information of targets of different sizes in a single modality image through different receptive fields, thereby improving the model's segmentation ability for complex scenes.

[0033] In some embodiments, the first modality multi-scale features and the second modality multi-scale features need to be input into the salient feature cross-fusion layer after corresponding convolution processing.

[0034] In some embodiments, the salient feature cross-fusion layer adopts a channel attention mechanism to obtain the first channel attention weight of the first modal multi-scale feature, and the second channel attention weight of the second modal multi-scale feature; multiplying the first channel attention weight with the first modal multi-scale feature to obtain the first modal salient feature, and multiplying the second channel attention weight with the second modal multi-scale feature to obtain the second modal salient feature; multiplying the first channel attention weight with the second modal multi-scale feature to obtain the second cross-guided salient feature, and multiplying the second channel attention weight with the first modal multi-scale feature to obtain the first cross-guided salient feature; fusing the first modal salient feature, the second modal salient feature, the second cross-guided salient feature and the first cross-guided salient feature to obtain the multimodal salient feature.

[0035] Specifically, the salient feature cross-fusion layer includes the attention extraction module MCAM, refer to Figure 4 , F Mod1 and F Mod2 Represent the first modal multi-scale features and the second modal multi-scale features respectively. First, F Mod1 and F Mod2 The attention weights of each channel in the feature graphs of different modalities are obtained through the channel attention mechanism respectively ( C SAtt1 and C SAtt2 ), the calculation process is shown in the following formula: ； ； ； in, Conv represents the convolution calculation, ReLU / Sigmoid represents the activation function, Avgpool / Maxpool represents pooled computation, n Represents the modal code, n is 1 or 2; Then, the attention weights ( C SAtt1 and C SAtt2 ) is multiplied with the original feature map to obtain the salient features of each modality image, that is, the salient features of the first modality F SAtt1 and the second modal salient features F SAtt2 , the calculation process is as follows: ; Next, the two sets of attention weights are multiplied by the feature map of the other modality to obtain the cross-guided salient features, namely the second cross-guided salient features F SAtt2 The first cross-guidance salient features F SAtt1 , the calculation process is as follows: ), ); Finally, the four sets of features are cascaded to obtain the fused multimodal salient features. F MCAM , further enrich the deep semantic information, the calculation process is as follows: .

[0036] In some embodiments, the feature deep encoding layer uses a densely connected network DenseNet to encode multimodal salient features to obtain high-level features with rich semantic information.

[0037] In some embodiments, the feature decoding layer decodes high-level features through skip connections combined with upsampling to obtain semantic segmentation results.

[0038] In some embodiments, after the multimodal semantic segmentation model is constructed, model parameter training and optimization need to be performed on the DFC2020 dataset and the WHU-OPT-SAR dataset respectively. In some examples, training can be performed on an NVIDIA RTX3090 server. Specifically, the mini-batch (gradient descent method) input method and the StepLR (learning rate scheduler) learning strategy can be adopted, and the network parameters can be optimized through the Adam (adaptive moment estimation) method. In some embodiments, the cross entropy loss is combined with the Focal loss and the IoU (Intersection over Union) loss during the training process to construct a joint loss function to constrain the model parameter back propagation process, which can effectively ensure the segmentation accuracy of the model.

[0039] In some embodiments, after training and optimization are completed, the image pairs in the validation set can be input into the trained model to obtain the numerical results of each indicator and a semantic segmentation map.

[0040] On the other hand, the present invention provides a multimodal remote sensing image semantic segmentation system, including: a multimodal semantic segmentation model, and the multimodal semantic segmentation model is used to implement the above-mentioned multimodal remote sensing image semantic segmentation method.

[0041] In order to reflect the advantages of the multimodal remote sensing image semantic segmentation method provided by an embodiment of the present invention, 6 comparative examples were selected for quantitative and qualitative analysis on two preprocessed data sets. The 6 comparative methods are DeepLabV3+, DenseASPP, DANet, CCnet, MCANet and SAMFNet. The codes of these methods can come from the original authors or public databases. Relevant experiments are all implemented on the same server, and the input and loss functions of each method are kept consistent.

[0042] The numerical results of each method tested on the DFC2020 dataset are shown in Table 1. The segmentation results are shown in Figure 5 The numerical results of each method tested on the WHU-OPT-SAR dataset are shown in Table 2, and the segmentation image results are shown in Figure 6 shown.

[0043] Table 1:

[0044] Table 2:

[0045] In the quantitative evaluation, three metrics, PA (pixel accuracy), Kappa (consistency test index) and mIoU (mean intersection over union ratio), were selected for comprehensive evaluation. The multimodal remote sensing image semantic segmentation method provided by the embodiment of the present invention achieved the optimal value (the larger the better) on both data sets. At the same time, the multimodal remote sensing image semantic segmentation method provided by the embodiment of the present invention performed outstandingly in the classification accuracy of different categories and achieved the optimal classification accuracy among multiple single-category targets.

[0046] In qualitative evaluation, the classification effect is compared through the actual segmentation map. Figure 5 and Figure 6 They represent the visualization effect of semantic segmentation on two datasets. Specifically, Figure 5 The following are the classification results of each method on the DFC2020 dataset. Group (A) and group (B) respectively represent the semantic segmentation results of different methods on two typical land cover scenes in the DFC2020 dataset. Figure 6 The following are the classification results of each method on the WHU-OPT-SAR dataset, where (A) and (B) represent the semantic segmentation results of different methods on two typical land cover scenes in the WHU-OPT-SAR dataset. Figure 5 and Figure 6, The sub-images contained in each image are defined as follows: (a) optical image, (b) SAR image, (c) label image, (d) DeepLabV3+ segmentation result, (e) DenseASPP segmentation result, (f) DANet segmentation result, (g) CCNet segmentation result, (h) MCANet segmentation result, (i) SAMFNet segmentation result, (j) segmentation result of the present invention. It can be seen from the figure that the embodiment of the present invention has a strong segmentation ability for complex scenes, and clearly depicts curved and extended targets (roads and water, etc.), interactions between similar categories (forests and green spaces, etc.), etc., which is more conducive to the segmentation of practical application scenarios.

[0047] In order to prove the effectiveness of the multi-scale extraction module MDEM and the attention extraction module MCAM designed for the multi-modal remote sensing image semantic segmentation method provided by the embodiment of the present invention, three groups of ablation experiments are designed for comparative analysis. At the same time, in order to illustrate the computational advantages of the multi-modal remote sensing image semantic segmentation method, the parameter amount and computational overhead of each method (DeepLabV3+, DenseASPP, DANet, CCnet, MCANet and SAMFNet) under the same input are statistically analyzed. Specifically, in Experiment 1 (Test1), the five branches in the multi-scale extraction module MDEM are replaced by conventional convolutions; in Experiment 2 (Test2), the serial hole convolutions in the multi-scale extraction module MDEM that meet the HDC criterion are replaced by single hole convolutions; in Experiment 3 (Test3), the cross attention in the attention extraction module MCAM is removed, and only the channel attention features of the two branches are retained. It can be seen from Table 3 that the multi-scale extraction module MDEM and the attention extraction module MCAM provided by the embodiment of the present invention have a significant effect on improving the segmentation results of the entire method.

[0048] Table 3:

[0049] Table 4 shows the multimodal remote sensing image semantic segmentation method provided by the embodiment of the present invention and the comparison of the parameter amount and the amount of calculation of each method. The multimodal remote sensing image semantic segmentation method provided by the embodiment of the present invention uses the least number of parameters, and the amount of calculation is only higher than the DeepLabV3+ and DANet methods, but the multimodal remote sensing image semantic segmentation method provided by the embodiment of the present invention has a more obvious performance advantage over the two methods. The multimodal remote sensing image semantic segmentation method provided by the embodiment of the present invention is more conducive to deployment in an embedded system, laying the foundation for its application in actual segmentation scenarios.

[0050] Table 4:

[0051] The multimodal remote sensing image semantic segmentation method provided by the embodiment of the present invention is trained and verified based on a deep convolutional neural network model, focuses on the effectiveness of feature encoding at the front end of the model, obtains multiscale semantic information by designing a multi-scale extraction module, completes deep extraction of different modal features with multiple receptive field branches, thereby improving the adaptability of the segmentation model to targets of different sizes, and uses an attention extraction module to obtain significant features of each modality. At the same time, modal cross branches are designed in the module to increase information interaction and guidance, and promote the fusion of different modal features. In addition, a densely connected network is introduced to transmit and reuse the features extracted by the front end, effectively improving the efficiency of back propagation. The model structure provided by the embodiment of the present invention is simple and efficient, with less calculation and parameter amounts, laying a foundation for its deployment and application in embedded systems.

[0052] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the disclosure of the present invention can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.

[0053] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multimodal remote sensing image semantic segmentation method, characterized in that: include: Establishing a data set, wherein the data set includes a plurality of groups of image pairs, each group of image pairs includes a SAR image and an optical image; Constructing a multimodal semantic segmentation model, wherein the multimodal semantic segmentation model includes a basic feature transformation layer, a multi-scale semantic acquisition layer, a significant feature cross-fusion layer, a feature deep encoding layer, and a feature decoding layer connected in sequence; Training and optimizing the multimodal semantic segmentation model based on the data set to obtain a target multimodal semantic segmentation model; The target multimodal semantic segmentation model is used to perform joint semantic segmentation on the SAR remote sensing image and the optical remote sensing image to obtain a semantic segmentation result map.

2. The multimodal remote sensing image semantic segmentation method according to claim 1, characterized in that: The data set establishment includes: selecting a remote sensing ground object classification multimodal data set as an initial data set, cropping images in the initial data set to a uniform size, and obtaining the data set after screening, wherein the number of channels of the optical image in the data set is 4, the number of channels of the SAR image in the data set is 1, and the size of the optical image is the same as the size of the SAR image.

3. The multimodal remote sensing image semantic segmentation method according to claim 1, characterized in that: The basic feature transformation layer includes: A first convolution branch, wherein the first convolution branch is used to perform convolution processing on the optical image to convert the optical image into a first modal basic feature; A second convolution branch, wherein the second convolution branch is used to perform convolution processing on the SAR image to convert the SAR image into a second modality basic feature.

4. The multimodal remote sensing image semantic segmentation method according to claim 3, characterized in that: The multi-scale semantic acquisition layer includes: A first multi-scale extraction module, wherein the first multi-scale extraction module uses a plurality of receptive field branches to perform deep extraction on the first modality basic features to obtain first modality multi-scale features; A second multi-scale extraction module, wherein the second multi-scale extraction module has the same structure as the first multi-scale extraction module, and the second multi-scale extraction module uses multiple receptive field branches to perform deep extraction on the second modality basic features to obtain second modality multi-scale features.

5. The multimodal remote sensing image semantic segmentation method according to claim 4, characterized in that: The first multi-scale extraction module includes five parallel receptive field branches, including two conventional convolution branches, two hole convolution branches and one pooling and upsampling processing branch. The five receptive field branches are used to process the first modal basic features respectively, and the features output by the five receptive field branches are connected to obtain the first modal multi-scale features.

6. The multimodal remote sensing image semantic segmentation method according to claim 4, characterized in that: The salient feature cross-fusion layer adopts a channel attention mechanism to obtain a first channel attention weight of the first modality multi-scale feature, and obtains a second channel attention weight of the second modality multi-scale feature; Multiplying the first channel attention weight by the first modality multi-scale feature to obtain a first modality salient feature, and multiplying the second channel attention weight by the second modality multi-scale feature to obtain a second modality salient feature; Multiplying the first channel attention weight by the second modality multi-scale feature to obtain a second cross-guided salient feature, and multiplying the second channel attention weight by the first modality multi-scale feature to obtain a first cross-guided salient feature; The first modality salient feature, the second modality salient feature, the second cross-guidance salient feature and the first cross-guidance salient feature are fused to obtain a multi-modality salient feature.

7. The multimodal remote sensing image semantic segmentation method according to claim 6, characterized in that: The feature deep encoding layer uses a densely connected network to encode the multimodal salient features to obtain high-level features with rich semantic information.

8. The multimodal remote sensing image semantic segmentation method according to claim 7, characterized in that: The feature decoding layer decodes the high-level features through skip connections combined with upsampling to obtain semantic segmentation results.

9. The multimodal remote sensing image semantic segmentation method according to claim 1, characterized in that: In the training process, the cross entropy loss is combined with the focal loss and the IoU loss to construct a joint loss function to constrain the back propagation process of the model parameters.

10. A multimodal remote sensing image semantic segmentation system, characterized in that: include: A multimodal semantic segmentation model, wherein the multimodal semantic segmentation model is used to implement the multimodal remote sensing image semantic segmentation method according to any one of claims 1 to 9.

Images