Multimodal remote sensing image semantic segmentation method and device combining optical image and SAR (Synthetic Aperture Radar)

Abstract

The invention discloses a multi-mode remote sensing image semantic segmentation method and device combining an optical image and an SAR (Synthetic Aperture Radar). The method comprises the following steps: inputting an optical image and an SAR image into a multi-mode remote sensing image semantic segmentation network to obtain an image segmentation result; the processing process of the multi-mode remote sensing image semantic segmentation network comprises the following steps: inputting an optical image into an optical branch to obtain an optical image segmentation result, and inputting an SAR image into an SAR branch to obtain an SAR image segmentation result; the features corresponding to the optical images output by the optical branches and the features corresponding to the SAR images output by the SAR branches are fused to obtain optical-SAR fusion features, and the optical-SAR fusion features are input into the fusion feature branches to obtain a fusion semantic segmentation result. According to the method, the ability of the model to express semantic and detail features by using optical-SAR cross-modal features is significantly improved, and a more efficient solution is provided for improvement of prediction accuracy of a remote sensing image semantic segmentation method.

Description

Technical Field

[0001] This application relates to the field of remote sensing image processing technology, and in particular to a method, apparatus, storage medium and electronic device for semantic segmentation of multimodal remote sensing images combining optical images and SAR. Background Technology

[0002] Optical imagery, with its higher resolution, provides richer details about the Earth's surface, vividly reflecting features such as shape, color, and texture. Therefore, it is widely used in semantic segmentation of remote sensing images and applied in research fields such as urban planning, land cover surveys, disaster change detection, and maritime situational awareness. However, its passive imaging method also makes it highly susceptible to weather conditions. Clouds, fog, and other weather phenomena can affect data acquisition quality, resulting in unclear representation of surface information. Furthermore, the usability of optical imagery is limited at night or in low-light conditions. In addition, because optical imagery passively receives surface reflection information, it is sensitive to surface material and lighting conditions, but cannot reflect the true physical characteristics of ground features.

[0003] As sensor technology continues to develop, the data sources for semantic segmentation are diversifying from single-source to multi-source. Synthetic Aperture Radar (SAR) acquires data through active imaging, possessing rich all-weather, all-time, and polarization information, and exhibiting unique responses to ground structure and moisture content. Its complementarity with optical imagery in imaging mechanisms and data characteristics has led many researchers to use SAR imagery as an effective supplement to single-modal optical segmentation methods. Existing research has also demonstrated that semantic segmentation methods combining optical imagery and SAR can fully integrate the advantages of different sensors, providing more comprehensive information on ground features and scenes, and playing a significant role in improving model robustness.

[0004] Existing semantic segmentation methods for joint optical-SAR imagery can be mainly categorized into data-based, feature-based, and decision-based approaches. Data-based methods combine SAR imagery with high-quality optical imagery data, then utilize traditional single-modal semantic segmentation methods. However, data-based methods are inefficient due to the need to process all sensor data before segmentation. Furthermore, the processed optical imagery is highly susceptible to noise in SAR, resulting in poor overall anti-interference capabilities. Feature-fusion-based methods extract features from both optical and SAR images separately, then fuse cross-modal features. While these methods improve computational efficiency and noise resistance, they may compromise the semantic integrity of the original image features during feature fusion. Decision-based methods rely solely on the segmentation results from different images for judgment. While simple and efficient, these methods may lose some information from the original images.

[0005] While semantic segmentation methods combining optical and SAR imagery can provide models with more diverse ground feature features, existing multimodal segmentation methods still fall short in extracting and fusing optical-SAR features due to inherent differences in imaging geometry, ground feature radiometric properties, and texture representation between the two types of data. This results in issues such as ground feature confusion, false positives and false negatives of small ground features, and blurred ground feature outlines. Furthermore, neglecting single-modal prediction results during segmentation can also affect semantic integrity. These shortcomings pose challenges to high-precision semantic segmentation of remote sensing images and hinder the further promotion of multimodal semantic segmentation technology. Currently, deeper and more targeted multimodal feature fusion strategies, more comprehensive model training optimization objectives, and multi-level modal fusion methods hold promise for further improving the semantic segmentation accuracy of combining optical and SAR imagery, providing more accurate technical means for surface monitoring. Summary of the Invention

[0006] This application provides a method, apparatus, storage medium, and electronic device for semantic segmentation of multimodal remote sensing images that combine optical imagery and SAR. It can fully extract and fuse optical-SAR features, significantly improving the model's ability to express semantic and detailed features using cross-modal optical-SAR features, and providing a more efficient solution for improving the prediction accuracy of remote sensing image semantic segmentation methods.

[0007] This application provides a multimodal remote sensing image semantic segmentation method that combines optical imagery and SAR, including: Acquire optical and SAR images; Optical and SAR images are input into a multimodal remote sensing image semantic segmentation network to obtain image segmentation results; The multimodal remote sensing image semantic segmentation network includes an optical branch, a SAR branch, and a fusion feature branch. The processing steps of the multimodal remote sensing image semantic segmentation network include: The optical image is input into the optical branch to obtain the optical image segmentation result; The SAR image is input into the SAR branch to obtain the SAR image segmentation result; The features corresponding to the optical image output by the optical branch and the features corresponding to the SAR image output by the SAR branch are fused to obtain the optical-SAR fused features. The optical-SAR fused features are then input into the fused feature branch to obtain the fused semantic segmentation result.

[0008] Furthermore, according to the above-mentioned multimodal remote sensing image semantic segmentation method combining optical imagery and SAR, the optical branch includes an encoder and a decoder; the processing procedure of the optical branch includes: Convolution operations are performed on optical images to obtain low-level optical features; The low-level optical features are obtained by processing the low-level optical features based on the attention mechanism. The optical low-level aggregated features are passed through a convolutional layer and an activation function to obtain the optical image segmentation result.

[0009] Furthermore, according to the above-mentioned multimodal remote sensing image semantic segmentation method combining optical imagery and SAR, the SAR branch includes an encoder and a decoder, and the processing procedure of the SAR branch includes: Convolutional operations are performed on SAR images to obtain low-level SAR features; The low-level SAR is processed based on an attention mechanism to obtain low-level SAR aggregated features.

[0010] The SAR low-level aggregated features are processed through a convolutional layer and an activation function to obtain the SAR image segmentation result.

[0011] Furthermore, according to the above-mentioned multimodal remote sensing image semantic segmentation method combining optical imagery and SAR, the fusion feature branch includes a local feature fusion module, a global feature fusion module, and a decoder; The optical low-level features and the SAR low-level features are input into the local feature fusion module, and processed through a gating mechanism and lightweight attention to obtain the local fused features. The optical low-level features and the SAR low-level features are input into the global feature fusion module and processed through convolutional attention and self-attention mechanisms to obtain the global fused features. The local fusion features are added to the global fusion features and then passed through a convolutional layer and an activation function to obtain the optical-SAR fusion semantic segmentation result.

[0012] Furthermore, according to the aforementioned multimodal remote sensing image semantic segmentation method combining optical imagery and SAR, the low-level optical features and the low-level SAR features are input into a local feature fusion module, and processed through a gating mechanism and lightweight attention to obtain local fused features, including: The optical low-level features and the SAR low-level features are merged to obtain merged features; Global average pooling and linear projection operations are performed on the merged features to obtain the attention weights of the optical feature channel and the SAR feature channel. The optical feature attention weights are multiplied by the optical low-level features, and the SAR feature attention weights are multiplied by the SAR low-level features. The multiplication results are then concatenated along the channel dimension to obtain local preliminary fusion features. Global average pooling and linear projection operations are performed on the local preliminary fusion features to obtain the attention weights of the optical feature channel and the SAR feature channel; The optical feature channel attention weights are multiplied by the optical low-level features to obtain the filtered optical features, and the SAR feature channel attention weights are multiplied by the SAR low-level features to obtain the filtered SAR features. The filtered optical features and the filtered SAR features are added together to calculate the lightweight attention, resulting in local fusion features.

[0013] Furthermore, according to the above-mentioned multimodal remote sensing image semantic segmentation method combining optical imagery and SAR, the lightweight attention is calculated by adding the selected optical features and the selected SAR features to obtain local fusion features, including: The filtered optical features and the filtered SAR features are added together and then global average pooling is performed to obtain the local feature weights. Based on the local feature weights, the size of the filtered optical features and the filtered SAR features is adjusted and normalized to obtain the fused feature result after calculating attention; The size of the fusion feature results after attention calculation is restored to obtain the local fusion features.

[0014] Furthermore, according to the above-mentioned multimodal remote sensing image semantic segmentation method combining optical imagery and SAR, the low-level optical features and the low-level SAR features are input into a global feature fusion module and processed through convolutional attention and self-attention mechanisms to obtain global fused features, including: Global average pooling, global max pooling, and linear transformation operations are performed on the optical low-level features and the SAR low-level features respectively to obtain the attention weights of the optical feature channels and the SAR feature channels. After max pooling and average pooling of the attention weights of the optical feature channels and the attention weights of the SAR feature channels, the channels are concatenated to obtain preliminary calculation results of optical features and preliminary calculation results of SAR features. The preliminary calculation results of the optical features and the preliminary calculation results of the SAR features are linearized after passing through a convolution kernel to obtain the spatial importance weight matrix of the optical features and the spatial importance weight matrix of the SAR features. After applying activation functions to the spatial importance weight matrix of the optical features and the spatial importance weight matrix of the SAR features, they are added together to obtain the preliminary global fusion features; The global preliminary fusion features are processed based on the self-attention mechanism to obtain the global fusion features.

[0015] This application also provides a multimodal remote sensing image semantic segmentation device that combines optical imagery and SAR, including: The acquisition module is used to acquire optical and SAR images; The image segmentation module is used to input optical images and SAR images into a multimodal remote sensing image semantic segmentation network to obtain image segmentation results; The multimodal remote sensing image semantic segmentation network includes an optical branch, a SAR branch, and a fusion feature branch. The processing steps of the multimodal remote sensing image semantic segmentation network include: The optical image is input into the optical branch to obtain the optical image segmentation result; The SAR image is input into the SAR branch to obtain the SAR image segmentation result; The features corresponding to the optical image output by the optical branch and the features corresponding to the SAR image output by the SAR branch are fused to obtain the optical-SAR fused features. The optical-SAR fused features are then input into the fused feature branch to obtain the fused semantic segmentation result.

[0016] This application also provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute any of the above-described multimodal remote sensing image semantic segmentation methods combining optical imagery and SAR.

[0017] This application also provides an electronic device, including a processor and a memory, wherein the processor is electrically connected to the memory, the memory is used to store instructions and data, and the processor is used in the steps of the multimodal remote sensing image semantic segmentation method of combined optical imagery and SAR as described in any of the above claims.

[0018] This application provides a method, apparatus, storage medium, and electronic device for semantic segmentation of multimodal remote sensing images using combined optical imagery and SAR. The multimodal remote sensing image semantic segmentation network includes an encoder-decoder, a global feature fusion module, and a local feature fusion module. The encoder and decoder, based on a self-attention mechanism, capture semantic and detail features of the image, and the local feature fusion module, based on convolutional attention and gating mechanisms, filters and optimizes the multimodal features, reducing noise in the image representation. The global feature fusion module, based on a self-attention mechanism, further enhances the model's ability to model global contextual relationships and improves understanding of complex remote sensing scenes. This application overcomes the low robustness of single-modal semantic segmentation models while significantly improving the model's ability to express semantic and detail features using optical-SAR cross-modal features, providing a more efficient solution for improving the prediction accuracy of remote sensing image semantic segmentation methods. Attached Figure Description

[0019] The technical solution and other beneficial effects of this application will become apparent from the following detailed description of specific embodiments in conjunction with the accompanying drawings.

[0020] Figure 1 A flowchart of a multimodal remote sensing image semantic segmentation method combining optical imagery and SAR provided in this application embodiment.

[0021] Figure 2 The segmentation comparison results are shown in the embodiment of this application.

[0022] Figure 3 A schematic diagram of the structure of the multimodal remote sensing image semantic segmentation device combining optical imagery and SAR provided in the embodiments of this application.

[0023] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] This application provides a method, apparatus, storage medium, and electronic device for semantic segmentation of multimodal remote sensing images combining optical imagery and SAR. The multimodal remote sensing image semantic segmentation apparatus provided in this application can be integrated into an electronic device, such as a terminal or server. The terminal can include a tablet computer, laptop computer, personal computer (PC), microprocessor box, or other devices.

[0026] Please see Figure 1 , Figure 1 The flowchart illustrates a multimodal remote sensing image semantic segmentation method combining optical imagery and SAR provided in this application embodiment. This method, applied in an electronic device, includes the following steps: S1, acquire optical and SAR images.

[0027] S2 inputs optical and SAR images into a multimodal remote sensing image semantic segmentation network to obtain image segmentation results.

[0028] Multimodal remote sensing image semantic segmentation networks include optical branches, SAR branches, and feature fusion branches. The processing steps of multimodal remote sensing image semantic segmentation networks include: S21, input the optical image into the optical branch to obtain the optical image segmentation result.

[0029] In one embodiment, the optical branch includes an encoder and a decoder; step S21 includes: S211 performs a convolution operation on the optical image to obtain low-level optical features.

[0030] The optical image is input into a 16×16 convolutional layer with a stride of 16. This convolutional layer is used to extract low-level features from the optical image. The formula for the convolution operation is expressed as:

[0031] In the formula, It is a convolution kernel. It is the bias term, the output of the convolution operation. The size is .

[0032] S212, based on the attention mechanism, processes the low-level optical data to obtain low-level optical aggregated features.

[0033] The feature map after convolutional layer processing The input is fed into the attention layer. This layer's role is to further enhance attention to important regions by learning weighting coefficients for different locations. This invention uses a self-attention mechanism to assign weights to each spatial location, expressed by the following formula:

[0034] In the formula, , , These are query, key, and value matrices, respectively, through... Obtained by performing a linear transformation; It is the dimension of the key vector, used to scale the dot product; It is a normalization operation used to ensure that the sum of the weights is 1; This involves weighting the output feature map, including weight adjustments for each spatial location. Ultimately, the weighted output feature map after the attention mechanism is represented as follows: . It is the final attention-weighted output of the shallow feature aggregation module in step 1, which contains low-level feature information extracted and aggregated from optical and SAR images.

[0035] S213 passes the low-level optical aggregated features through a convolutional layer and activation function to obtain the optical image segmentation result.

[0036] S22, input the SAR image into the SAR branch to obtain the SAR image segmentation result.

[0037] In one embodiment, the SAR branch includes an encoder and a decoder. The encoder and decoder structures are similar when optical and SAR single-mode predictions are performed independently. Step S22 includes: S221, perform convolution operation on SAR image to obtain low-level SAR features; S222, based on the attention mechanism, processes the low-level SAR to obtain the low-level aggregated features of SAR.

[0038] S223 processes the low-level aggregated features of SAR through a convolutional layer and an activation function to obtain the SAR image segmentation result.

[0039] S23, fuse the features corresponding to the optical image output by the optical branch and the features corresponding to the SAR image output by the SAR branch to obtain the optical-SAR fused features, and input the optical-SAR fused features into the fused feature branch to obtain the fused semantic segmentation result.

[0040] In one embodiment, the fusion feature branch includes a local feature fusion module, a global feature fusion module, and a decoder; S231 inputs low-level optical features and low-level SAR features into the local feature fusion module, and processes them through a gating mechanism and lightweight attention to obtain local fused features.

[0041] In one embodiment, step S231 includes: S2311 performs a merging operation on optical low-level features and SAR low-level features to obtain merged features.

[0042] This process can be represented by the following formula:

[0043] In the formula, This indicates that two tensors are concatenated along the channel dimension. In this invention, the number of feature channels is 768 in both instances when this module is used for fusion; therefore, the merged tensor... The size is .

[0044] S2312 performs global average pooling and linear projection operations on the merged features to obtain optical feature attention weights and SAR feature attention weights.

[0045] Cross-modal channel attention is obtained through global average pooling, and channel weights for the two image features are obtained through linear projection. These weights are used for preliminary screening and adjustment of cross-modal features along the channel dimension. The formula is as follows:

[0046]

[0047] In the formula, and These represent the attention weights for the optical feature channel and the SAR feature channel, respectively, both with a size of [missing information]. .

[0048] S2313 multiplies the optical feature attention weights with the low-level optical features, and multiplies the SAR feature attention weights with the low-level SAR features. The results of the multiplication are then concatenated along the channel dimension to obtain the preliminary local fusion features.

[0049] This can be expressed by the following formula:

[0050] S2314 performs global average pooling and linear projection operations on the local preliminary fusion features to obtain the attention weights of the optical feature channel and the SAR feature channel.

[0051] The initial local fusion features are processed through global average pooling along the channel dimension to obtain cross-modal spatial attention, and the spatial weights of the two image features are obtained through linear projection. The cross-modal features are then further adjusted, as expressed by the following formula:

[0052]

[0053] In the formula, and These represent the attention weights for the optical feature channel and the SAR feature space, respectively, both with a size of [size missing]. .

[0054] S2315, multiply the attention weights of the optical feature channels with the low-level optical features to obtain the filtered optical features, and multiply the attention weights of the SAR feature channels with the low-level SAR features to obtain the filtered SAR features.

[0055] This can be expressed by the following formula:

[0056]

[0057] In the formula, and These represent the results of the initial screening based on the gating mechanism, and the size of the two data sets remains the same as the input.

[0058] S2316 calculates lightweight attention by adding the filtered optical features and filtered SAR features to obtain local fusion features.

[0059] In one embodiment, step S2316 includes: S23161, after adding the filtered optical features and the filtered SAR features, performs global average pooling calculation to obtain the local feature weights.

[0060] This can be expressed by the following formula:

[0061] In the formula, The weights of the local features learned by the model are represented by the following value: ,in This indicates the size of the receptive field of attention.

[0062] S23162, based on local feature weights, adjusts and normalizes the size of the filtered optical features and filtered SAR features to obtain the fused feature result after calculating attention.

[0063] This can be expressed by the following formula:

[0064] In the formula, Indicates in The result of fusion feature calculation after attention is calculated within the pixel.

[0065] S23163 restores the size of the fusion feature results after attention calculation to obtain local fusion features.

[0066] This can be expressed by the following formula:

[0067] S232 inputs low-level optical features and low-level SAR features into the global feature fusion module, and processes them through convolutional attention and self-attention mechanisms to obtain global fused features.

[0068] Spatial and channel information of optical and SAR features are optimized using convolutional attention and self-attention mechanisms. In one embodiment, step S232 includes the following steps: S2321 performs global average pooling, global max pooling, and linear transformation operations on optical low-level features and SAR low-level features respectively to obtain the attention weights of optical feature channels and SAR feature channels.

[0069] By using global average pooling and global max pooling to obtain channel attention information in two dimensions, and adjusting the corresponding feature dimensions using linear transformation, it can be expressed as follows:

[0070]

[0071] In the formula, and These represent the attention weights of the optical feature channel and the SAR feature channel obtained after linear transformation, respectively.

[0072] S2322, after max pooling and average pooling of the attention weights of the optical feature channel and the SAR feature channel, the channels are connected to obtain the preliminary calculation results of the optical features and the preliminary calculation results of the SAR features.

[0073] This can be expressed by the following formula:

[0074]

[0075] In the formula, and The figures represent the preliminary calculation results for optical features and SAR features, respectively. The width and height of both remain unchanged, and the number of channels is 2.

[0076] S2323 performs a linearization operation on the preliminary calculation results of optical features and SAR features after passing them through a convolution kernel, resulting in the spatial importance weight matrix of optical features and the spatial importance weight matrix of SAR features.

[0077] The preliminary calculation results are then processed through a 7×7 large kernel convolution to obtain the spatial importance information of the features, and adjusted through corresponding linearization operations, as shown in the following formula:

[0078]

[0079] In the formula, and These represent the spatial importance weight matrices for optical features and SAR features, respectively.

[0080] S2324: After applying an activation function to the spatial importance weight matrix of the optical features and the spatial importance weight matrix of the SAR features, the two matrices are summed to obtain the preliminary global fused features.

[0081] use The activation function introduces nonlinearity and converts the feature importance weight matrices of the spatial and channel dimensions to values ​​between 0 and 1. These values ​​are then multiplied by the input optical and SAR features. The sum of the two results yields a preliminary fusion result, expressed by the following formula:

[0082]

[0083] +

[0084] in, This represents the preliminary fusion result of fusing optical features and SAR features using convolutional attention.

[0085] S2325, based on the self-attention mechanism, processes the preliminary global fusion features to obtain the global fusion features.

[0086] This step is similar to the process in S212, and can be referred to step S212. It will not be repeated here.

[0087] S233 adds the local fusion features to the global fusion features and then passes them through a convolutional layer and activation function to obtain the optical-SAR fusion semantic segmentation result.

[0088] Existing multimodal optical-SAR semantic segmentation methods often use the fused feature prediction result as the sole calculation result, which to some extent affects the semantic integrity of the original single-modal prediction. Therefore, in the loss calculation process, the fused semantic segmentation result corresponding to optical-SAR feature fusion is used as the final output loss of the model. At the same time, the two modal prediction results (i.e., optical image segmentation result and SAR image segmentation result) are also upsampled to the original resolution to calculate the auxiliary loss. The three loss values ​​are then added together as the comprehensive loss for model optimization. Therefore, the mathematical expression of the loss function for multimodal optical-SAR semantic segmentation proposed in this invention is as follows:

[0089] in, Let cross-entropy be the loss function. This is the result of segmenting labels. It is a fusion of semantic segmentation results. It is the result of optical image segmentation. This is the SAR segmentation result.

[0090] The cross-entropy loss function is defined as follows:

[0091] Where N is the number of categories, For real labels, Predict probabilities for each category. Since semantic segmentation is a intensive prediction task, every pixel in the image needs to be considered when calculating the cross-entropy loss.

[0092] The following is a specific embodiment. The experiment in this embodiment was conducted in an NVIDIA Tesla P100 hardware environment and a Python software environment.

[0093] The constructed optical-SAR semantic segmentation system was optimized using the AdamW optimizer. The initial learning rate was set to 1×10-3, and the learning rate was adjusted using a cosine annealing strategy. A total of 100 iterations were trained.

[0094] The dataset used in this embodiment is the WHU-OPT-SAR dataset released by Wuhan University. This dataset contains 29,400 image pairs, each pair including one Gaofen-1 optical image, one Gaofen-3 SAR image, and corresponding semantic segmentation labels. The training, validation, and test sets are divided in a 6:2:2 ratio, meaning there are 17,640 training pairs, 5,880 validation pairs, and 5,880 test pairs.

[0095] This invention was compared with five existing methods published in authoritative journals for the restoration of cloud-containing optical remote sensing images, namely: [Hazirbas C, Ma L, Domokos C, et al. FuseNet: Incorporating Depth into Semantic Segmentation via Fusion-Based CNN Architecture[M] / / Lai S H, Lepetit V, Nishino K, et al., Computer Vision – ACCV 2016: Vol. 10111. Cham: Springer International Publishing: 213-228, 2017.] (Comparison Method 1), [Audebert N, LeSaux B, Lefèvre S. Beyond RGB: Very high resolution urban remote sensing with multimodal deep networks[J]. ISPRS journal of photogrammetry and remote sensing, 140: 20-32, 2018.] (Comparison Method 2), [Ma X, Zhang X, Pun M O, et al., 2024. A Multilevel Multimodal Fusion Transformer for Remote Sensing Semantic Segmentation[J]. IEEE Transactions on Geoscience and Remote Sensing, 62: 1-15.] (Comparison Method 3), [Li Xue, Zhang G, Cui H, et al., 2022. MCANet: A joint semantic segmentation framework of optical and SAR images for land use classification[J]. International Journal of Applied Earth Observation and Geoinformation, 106: 102638.] (Comparison Method 4), [Ren B, Ma S, Hou B, et al., 2022.Adual-stream high resolution network: Deep fusion of GF-2 and GF-3 data forland cover classification[J]. International Journal of Applied EarthObservation and Geoinformation, 112: 102896.] (Compare method 5). .

[0096] Figure 2 The segmentation comparison result diagram provided for the embodiments of this application is as follows: Figure 2 As shown in the figure. The comparison results show that, compared with other methods, the multimodal remote sensing image semantic segmentation system combining optical imagery and SAR designed in this invention has a more comprehensive understanding of the overall scene of the remote sensing image and is superior in terms of information extraction capabilities at both the semantic and detail levels. Other methods all suffer from varying degrees of land cover category confusion and blurred land cover boundaries during segmentation, while this invention can significantly improve these shortcomings and effectively enhance the intelligent interpretation capability of images.

[0097] Based on the method described in the above embodiments, this embodiment will further describe it from the perspective of a multimodal remote sensing image semantic segmentation device that combines optical imagery and SAR. Specifically, the multimodal remote sensing image semantic segmentation device that combines optical imagery and SAR can be implemented as an independent entity or integrated into an electronic device. The electronic device can be a terminal, server, or other devices. The terminal can include a tablet computer, a laptop computer, a personal computer (PC), a microprocessor box, or other devices.

[0098] Please see Figure 3 , Figure 3 This application provides a detailed description of a multimodal remote sensing image semantic segmentation device combining optical imagery and SAR, applied in an electronic device. This device may include: The acquisition module is used to acquire optical and SAR images; The image segmentation module is used to input optical images and SAR images into a multimodal remote sensing image semantic segmentation network to obtain image segmentation results; The multimodal remote sensing image semantic segmentation network includes an optical branch, a SAR branch, and a fusion feature branch. The processing steps of the multimodal remote sensing image semantic segmentation network include: The optical image is input into the optical branch to obtain the optical image segmentation result; The SAR image is input into the SAR branch to obtain the SAR image segmentation result; The features corresponding to the optical image output by the optical branch and the features corresponding to the SAR image output by the SAR branch are fused to obtain the optical-SAR fused features. The optical-SAR fused features are then input into the fused feature branch to obtain the fused semantic segmentation result.

[0099] In specific implementation, the above modules and / or units can be implemented as independent entities, or they can be arbitrarily combined and implemented as the same or several entities. For the specific implementation of the above modules and / or units, please refer to the previous method embodiments. For the specific beneficial effects that can be achieved, please also refer to the beneficial effects in the previous method embodiments, which will not be repeated here.

[0100] In addition, this application also provides an electronic device, which may be a computer, tablet computer, or other similar device. This electronic device can implement the steps of any embodiment of the multimodal remote sensing image semantic segmentation method combining optical imagery and SAR provided in this application. Therefore, it can achieve the beneficial effects that any of the multimodal remote sensing image semantic segmentation methods combining optical imagery and SAR provided in this invention can achieve, as detailed in the preceding embodiments, and will not be repeated here.

[0101] Figure 4 A specific structural block diagram of an electronic device provided in an embodiment of the present invention is shown. This electronic device can be used to implement the multimodal remote sensing image semantic segmentation method combining optical imagery and SAR provided in the above embodiments. The electronic device 500 can be a terminal, server, or other device. The terminal can include a tablet computer, laptop computer, personal computer (PC), microprocessor box, or other devices.

[0102] RF circuit 510 is used to receive and transmit electromagnetic waves, converting electromagnetic waves into electrical signals and vice versa, thereby enabling communication with communication networks or other devices. RF circuit 510 may include various existing circuit elements used to perform these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, subscriber identity modules (SIM cards), memory, etc. RF circuit 510 can communicate with various networks such as the Internet, corporate intranets, and wireless networks, or communicate with other devices via wireless networks. The aforementioned wireless networks may include cellular telephone networks, wireless local area networks (WLANs), or metropolitan area networks (MANs). The aforementioned wireless networks may use various communication standards, protocols, and technologies, including but not limited to Global System for Mobile Communication (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (WCDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Wireless Fidelity (Wi-Fi) (such as IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Worldwide Interoperability for Microwave Access (Wi-Max), other protocols for email, instant messaging, and short messages, and any other suitable communication protocols, including those that have not yet been developed.

[0103] The memory 520 can be used to store software programs and modules, such as the program instructions / modules corresponding to those in the above embodiments. The processor 580 executes various functional applications and data processing by running the software programs and modules stored in the memory 520, such as taking pictures with the front-facing camera, processing the captured images, and switching the display colors of the content displayed on the screen. The memory 520 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 520 may further include memory remotely located relative to the processor 580, and these remote memories can be connected to the electronic device 500 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0104] The input unit 530 can be used to receive input numeric or character information, and to generate a keyboard and mouse related to user settings and function control. Display unit 540 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces, which can be composed of graphics, text, icons, video, and any combination thereof. Display unit 540 may include display panel 541, which may optionally be configured in the form of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), or other similar forms.

[0105] Audio circuitry 560, speaker 561, and microphone 562 provide an audio interface between the user and electronic device 500. Audio circuitry 560 converts received audio data into electrical signals and transmits them to speaker 561, where speaker 561 converts them into sound signals for output. Conversely, microphone 562 converts collected sound signals into electrical signals, which are then received by audio circuitry 560, converted back into audio data, and processed by processor 580. The audio data is then transmitted via RF circuitry 510 to, for example, another terminal, or output to memory 520 for further processing. Audio circuitry 560 may also include an earphone jack to facilitate communication between external headphones and electronic device 500.

[0106] Electronic device 500, through transmission module 570 (e.g., Wi-Fi module), can help users receive requests, send information, etc., providing users with wireless broadband internet access. Although transmission module 570 is shown in the figure, it is understood that it is not an essential component of electronic device 500 and can be omitted as needed without changing the essence of the invention.

[0107] The processor 580 is the control center of the electronic device 500. It connects to various parts of the phone via various interfaces and lines, and performs various functions and processes data of the electronic device 500 by running or executing software programs and / or modules stored in the memory 520, and by calling data stored in the memory 520, thereby providing overall monitoring of the electronic device. Optionally, the processor 580 may include one or more processing cores; in some embodiments, the processor 580 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 580.

[0108] Electronic device 500 also includes a power supply 590 (such as a battery) that supplies power to various components. In some embodiments, the power supply may be logically connected to processor 580 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 590 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0109] Although not shown, the electronic device 500 also includes cameras (such as front-facing cameras and rear-facing cameras), Bluetooth modules, etc., which will not be described in detail here. Specifically, in this embodiment, the display unit of the electronic device is a touch screen display, and the mobile terminal also includes a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors. One or more programs contain instructions for performing the following operations: Acquire optical and SAR images; Optical and SAR images are input into a multimodal remote sensing image semantic segmentation network to obtain image segmentation results; The multimodal remote sensing image semantic segmentation network includes an optical branch, a SAR branch, and a fusion feature branch. The processing steps of the multimodal remote sensing image semantic segmentation network include: The optical image is input into the optical branch to obtain the optical image segmentation result; The SAR image is input into the SAR branch to obtain the SAR image segmentation result; The features corresponding to the optical image output by the optical branch and the features corresponding to the SAR image output by the SAR branch are fused to obtain the optical-SAR fused features. The optical-SAR fused features are then input into the fused feature branch to obtain the fused semantic segmentation result.

[0110] In practice, the above modules can be implemented as independent entities or combined in any way to be implemented as the same or several entities. For the specific implementation of the above modules, please refer to the previous method implementation examples, which will not be repeated here.

[0111] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. Therefore, embodiments of the present invention provide a storage medium storing multiple instructions that can be loaded by a processor to execute the steps of any embodiment of the multimodal remote sensing image semantic segmentation method combining optical imagery and SAR provided by the present invention.

[0112] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0113] Since the instructions stored in the storage medium can execute the steps in any embodiment of the multimodal remote sensing image semantic segmentation method of combined optical imagery and SAR provided in the embodiments of the present invention, the beneficial effects that any multimodal remote sensing image semantic segmentation method of combined optical imagery and SAR provided in the embodiments of the present invention can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0114] The foregoing has provided a detailed description of a multimodal remote sensing image semantic segmentation method, apparatus, storage medium, and electronic device combining optical imagery and SAR, as provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A multimodal remote sensing image semantic segmentation method combining optical imagery and SAR, characterized in that, The method includes: Acquire optical and SAR images; Optical and SAR images are input into a multimodal remote sensing image semantic segmentation network to obtain image segmentation results; The multimodal remote sensing image semantic segmentation network includes an optical branch, a SAR branch, and a fusion feature branch. The processing steps of the multimodal remote sensing image semantic segmentation network include: The optical image is input into the optical branch to obtain the optical image segmentation result; The SAR image is input into the SAR branch to obtain the SAR image segmentation result; The features corresponding to the optical image output by the optical branch and the features corresponding to the SAR image output by the SAR branch are fused to obtain the optical-SAR fused features. The optical-SAR fused features are then input into the fused feature branch to obtain the fused semantic segmentation result.

2. The multimodal remote sensing image semantic segmentation method combining optical imagery and SAR according to claim 1, characterized in that, The optical branch includes an encoder and a decoder; the processing of the optical branch includes: Convolution operations are performed on optical images to obtain low-level optical features; The low-level optical features are obtained by processing the low-level optical features based on the attention mechanism. The optical low-level aggregated features are passed through a convolutional layer and an activation function to obtain the optical image segmentation result.

3. The multimodal remote sensing image semantic segmentation method combining optical imagery and SAR according to claim 2, characterized in that, The SAR branch includes an encoder and a decoder, and the processing procedure of the SAR branch includes: Convolutional operations are performed on SAR images to obtain low-level SAR features; The low-level SAR is processed based on an attention mechanism to obtain low-level SAR aggregated features; The SAR low-level aggregated features are processed through a convolutional layer and an activation function to obtain the SAR image segmentation result.

4. The multimodal remote sensing image semantic segmentation method combining optical imagery and SAR according to claim 3, characterized in that, The fusion feature branch includes a local feature fusion module, a global feature fusion module, and a decoder; The optical low-level features and the SAR low-level features are input into the local feature fusion module, and processed through a gating mechanism and lightweight attention to obtain the local fused features. The optical low-level features and the SAR low-level features are input into the global feature fusion module and processed through convolutional attention and self-attention mechanisms to obtain the global fused features. The local fusion features are added to the global fusion features and then passed through a convolutional layer and an activation function to obtain the optical-SAR fusion semantic segmentation result.

5. The multimodal remote sensing image semantic segmentation method combining optical imagery and SAR according to claim 4, characterized in that, The optical low-level features and the SAR low-level features are input into the local feature fusion module, and processed through a gating mechanism and lightweight attention to obtain the local fused features, including: The optical low-level features and the SAR low-level features are merged to obtain merged features; Global average pooling and linear projection operations are performed on the merged features to obtain the attention weights of the optical feature channel and the SAR feature channel. The optical feature attention weights are multiplied by the optical low-level features, and the SAR feature attention weights are multiplied by the SAR low-level features. The multiplication results are then concatenated along the channel dimension to obtain local preliminary fusion features. Global average pooling and linear projection operations are performed on the local preliminary fusion features to obtain the attention weights of the optical feature channel and the SAR feature channel; The optical feature channel attention weights are multiplied by the optical low-level features to obtain the filtered optical features, and the SAR feature channel attention weights are multiplied by the SAR low-level features to obtain the filtered SAR features. The filtered optical features and the filtered SAR features are added together to calculate the lightweight attention, resulting in local fusion features.

6. The multimodal remote sensing image semantic segmentation method combining optical imagery and SAR according to claim 5, characterized in that, The filtered optical features and the filtered SAR features are added together to calculate the lightweight attention, resulting in local fusion features, including: The filtered optical features and the filtered SAR features are added together and then global average pooling is performed to obtain the local feature weights. Based on the local feature weights, the size of the filtered optical features and the filtered SAR features is adjusted and normalized to obtain the fused feature result after calculating attention; The size of the fusion feature results after attention calculation is restored to obtain the local fusion features.

7. The multimodal remote sensing image semantic segmentation method combining optical imagery and SAR according to claim 2, characterized in that, The optical low-level features and the SAR low-level features are input into the global feature fusion module and processed through convolutional attention and self-attention mechanisms to obtain the globally fused features, including: Global average pooling, global max pooling, and linear transformation operations are performed on the optical low-level features and the SAR low-level features respectively to obtain the attention weights of the optical feature channels and the SAR feature channels. After max pooling and average pooling of the attention weights of the optical feature channels and the attention weights of the SAR feature channels, the channels are concatenated to obtain preliminary calculation results of optical features and preliminary calculation results of SAR features. The preliminary calculation results of the optical features and the preliminary calculation results of the SAR features are linearized after passing through a convolution kernel to obtain the spatial importance weight matrix of the optical features and the spatial importance weight matrix of the SAR features. After applying activation functions to the spatial importance weight matrix of the optical features and the spatial importance weight matrix of the SAR features, they are added together to obtain the preliminary global fusion features; The global preliminary fusion features are processed based on the self-attention mechanism to obtain the global fusion features.

8. A semantic segmentation device for multimodal remote sensing images combining optical imagery and SAR, characterized in that, include: The acquisition module is used to acquire optical and SAR images; The image segmentation module is used to input optical images and SAR images into a multimodal remote sensing image semantic segmentation network to obtain image segmentation results; The multimodal remote sensing image semantic segmentation network includes an optical branch, a SAR branch, and a fusion feature branch. The processing steps of the multimodal remote sensing image semantic segmentation network include: The optical image is input into the optical branch to obtain the optical image segmentation result; The SAR image is input into the SAR branch to obtain the SAR image segmentation result; The features corresponding to the optical image output by the optical branch and the features corresponding to the SAR image output by the SAR branch are fused to obtain the optical-SAR fused features. The optical-SAR fused features are then input into the fused feature branch to obtain the fused semantic segmentation result.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted to be loaded by a processor to execute the multimodal remote sensing image semantic segmentation method of combined optical imagery and SAR as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, The method includes a processor and a memory, the processor being electrically connected to the memory, the memory being used to store instructions and data, and the processor being used to execute the steps in the multimodal remote sensing image semantic segmentation method of combined optical imagery and SAR as described in any one of claims 1 to 7.

Images