Typical ground object target segmentation method based on multi-source data attention feature fusion

Abstract

The invention discloses a typical ground object target segmentation method based on multi-source data attention feature fusion, and the method comprises the following steps: (1), obtaining a data set containing a high-resolution optical image, an SAR image and a corresponding semantic tag image, and carrying out the preprocessing of the data set; (2) constructing a typical ground object target segmentation model based on multi-source data attention feature fusion, and realizing semantic segmentation of a typical ground object target; and (3) taking the preprocessed data set obtained in the step (1) as the input of the model constructed in the step (2), training the model based on a weighted cross entropy loss function, and realizing typical ground object target segmentation based on multi-source data. The invention provides an adaptive fusion algorithm based on an attention mechanism by comprehensively considering the observation advantages of SAR and optical data for different ground object targets, and better combines the respective advantage characteristics of an optical image and an SAR image, thereby achieving a better semantic segmentation effect in the field of ground object classification.

Description

Technical Field

[0001] The invention belongs to the field of remote sensing image processing and ground object target segmentation and extraction, and specifically relates to a typical ground object target segmentation method based on multi-source data attention feature fusion. Background Art

[0002] Land object classification usually uses semantic segmentation, that is, to represent the category for each label in the remote sensing image. Remote sensing images include optical images and synthetic aperture radar images. Land object classification based on remote sensing images depends on the spatial information and semantic information provided in the image. The research results can be applied to land resource planning, mineral exploration, environmental protection and urban planning. Synthetic Aperture Radar (SAR), as an active microwave imaging sensor, can observe and image the observation area all day and all day, and has strong penetration ability while not being affected by external environmental factors such as lighting conditions and climate change. SAR images can provide rich electromagnetic scattering characteristics and geometric characteristics of land object targets. However, SAR is grayscale data obtained in a single band, and there are defects such as blurred target details. The advantage of optical imaging is that it can provide high spatial resolution and rich spectral and texture information, but it is easily affected by the observation environment such as clouds, fog and rain. Since optical imaging and SAR imaging show excellent complementarity in some practical applications, optical SAR fusion has good application prospects in improving the accuracy of remote sensing target recognition.

[0003] The optical SAR fusion methods based on deep learning are divided into decision-level fusion and feature-level fusion. Decision-level fusion obtains semantic features of interest in a variety of ways, but it is difficult to extract high-dimensional feature representations after fusion; feature-level fusion includes pseudo-Siamese Convolutional Neural Network (PSCNN) and multisource remote sensing data classification (MRSDC). PSCNN is a network architecture proposed by Hughes et al. in 2018. It can extract corresponding image slices from high-resolution optical and SAR images and achieve high accuracy in complex remote sensing data containing a large number of buildings. MRSDC has developed an architecture based on a dual-tunnel convolutional neural network, which uses dual-tunnel branches and cascade networks to achieve classification fusion of hyperspectral images and other multi-sensor data. However, the above methods do not fully utilize the distribution characteristics of multi-source data and the performance is not stable enough. Therefore, a ground object segmentation method with more stable performance and guaranteed classification accuracy is needed. Summary of the invention

[0004] Purpose of the invention: Based on the semantic consistency constraints of multi-source data, the purpose of the present invention is to provide a typical land object segmentation method based on the fusion of multi-source data attention features, which is used in the field of land object classification and can obtain good semantic segmentation results.

[0005] Technical solution: The typical ground object target segmentation method based on multi-source data attention feature fusion described in the present invention specifically includes the following steps:

[0006] (1) Obtain a dataset containing high-resolution optical images, SAR images, and corresponding semantically labeled images, divide the dataset into a training set, a validation set, and a test set, and perform data preprocessing;

[0007] (2) constructing a typical ground object target segmentation model based on multi-source data attention feature fusion to achieve semantic segmentation of typical ground object targets; the model includes an encoder and a decoder; the encoder includes a data prior feature construction and guidance module, a feature extraction backbone network and a multimodal cross attention fusion module;

[0008] (3) The preprocessed data set obtained in step (1) is used as the input of the model constructed in step (2), and the model parameters are trained and optimized based on the weighted cross entropy loss function to achieve typical ground object segmentation based on multi-source data.

[0009] Furthermore, the data preprocessing process in step (1) is as follows:

[0010] First, the dataset is cut into non-overlapping pixel blocks of 256×256 pixels, and then the images are standardized, the mean is set to [0.485, 0.456, 0.406], the standard deviation is set to [0.229, 0.224, 0.225], and finally the training set images are enhanced.

[0011] Furthermore, the implementation process of the typical ground object target segmentation model based on multi-source data attention feature fusion in step (2) is as follows:

[0012] Firstly, the data prior feature construction and guidance module extracts the texture features of optical images and the scattering features of SAR images as the neural network prior information for the special feature extraction and second-order hidden feature mining of optical images and SAR images. Then, an encoder with the same structure is used to extract the feature vectors of optical images and SAR images, and their corresponding query set feature vectors Q, index set feature vectors K, and content feature vectors V. Secondly, the multimodal cross-attention fusion module calculates the optical image attention feature vectors A, B, C, and C respectively. OPT =Attention(Q OPT ,K SAR ,V SAR), and the SAR image attention feature vector A SAR =Attention(Q SAR ,K OPT ,V OPT ), and then concatenate to obtain the multimodal cross-attention feature vector A cros =concat(A OPT ,A SAR ); Further, the optical image feature vector obtained by the tandem encoder, the SAR image feature vector, and the multimodal cross-attention feature vector A cros , as the model decoder input, and obtain the model prediction result.

[0013] Furthermore, the feature extraction backbone network adopts ResNet50, and performs feature extraction on the input of optical images and SAR images respectively to obtain representative and effective multi-level features of the two types of images; the 2nd and 5th layer features extracted by the feature extraction backbone network are selected as the low-dimensional features and high-dimensional features of the image respectively, the high-dimensional features of the optical image and SAR image are used as the input of the high-dimensional cross-attention feature fusion module, and the low-dimensional features of the two are used as the input of the low-dimensional cross-attention feature fusion module, and finally the cross-attention feature vectors output by the high- and low-dimensional modules are connected in series to obtain multi-dimensional optical SAR cross-attention fusion features; the multi-dimensional optical SAR cross-attention fusion features are input into the decoder, and the final semantic segmentation result is obtained after convolution and upsampling operations.

[0014] Furthermore, the feature extraction backbone network in step (2) extracts features from the image, performs two-dimensional convolution operations on the cut optical image and SAR image using a 3×3 convolution kernel, respectively, and performs 1 pixel padding around the edge of the input image, and normalizes the data of each channel through batch normalization; the result passes through a maximum pooling layer to achieve downsampling of the image data; after two times of the above processing, two types of 64×64 pixel blocks are obtained as low-dimensional features as input to the inter-domain fusion module; after three times of the above operation, two types of 32×32 pixel blocks are obtained, which are no longer passed through the maximum pooling layer, and after two times of convolution and batch normalization operations, two types of 32×32 pixel blocks with a channel number of 2048 are obtained as high-dimensional features as input to the inter-domain fusion module.

[0015] Furthermore, the implementation process of the multimodal cross attention fusion module in step (2) is as follows:

[0016] The high-dimensional features and low-dimensional features are taken as input respectively. The cross-attention mechanism can simultaneously consider the feature data information of the optical and SAR domains and calculate their attention weights. First, the query vector Q, key vector K, and value vector V are obtained through the input matrix X. The formula is as follows:

[0017] {Q,K,V}={XW Q ,XW K ,XW V}

[0018] Then multiply the Q and K matrices by After normalization, the formula is as follows:

[0019]

[0020] In the formula, Q, K, and V are matrices obtained after linear mapping of the input matrix, d k is the K matrix dimension, B is the learnable relative position code, and Softmax is the activation function, which converts the input real number vector into a probability distribution vector;

[0021] The query vector Q, key vector K, and value vector V from different sources are cross-calculated for relevance to obtain the weighted value of the attention mechanism. The calculation formula of the multimodal cross-attention fusion module is as follows:

[0022]

[0023] Z1=LN(MCA(Q1,K2,V2))+Q1

[0024] Z2=LN(MCA(Q2,K1,V1))+Q2

[0025] Where X1 is the embedded representation of the optical input; X2 is the embedded representation of the SAR input; are the weight matrices of query vectors Q1 and Q2, key vectors K1 and K2, and value vectors V1 and V2 corresponding to optical input and SAR input respectively; MCA represents the multi-head cross attention module; LN represents layer regularization, that is, normalizing all features of a single sample;

[0026] Convolve Z1 and Z2 to obtain the high-dimensional features and low-dimensional features after optical SAR fusion. First, the low-dimensional features and the prior features are connected to obtain the connected low-dimensional features. Then the high-dimensional features and the prior features are connected, and the connected high-dimensional features are obtained through the atrous spatial pyramid pooling ASPP. Finally, the connected high- and low-dimensional features are connected in series as the input of the decoder.

[0027] Furthermore, the atrous spatial pyramid pooling ASPP first adopts atrous convolution branches with different dilation rates to capture feature information at different scales, increase the receptive field without increasing network parameters, and improve the feature extraction capability; then, the feature map of each branch is spatially pyramid pooled to obtain a feature vector of a fixed size; finally, the feature vectors of each scale are connected and the final ASPP feature representation is obtained after dimensionality reduction.

[0028] Furthermore, the decoder in step (2) includes two 1×1 convolutions and one upsampling.

[0029] Furthermore, the weighted cross entropy loss function in step (3) is used to train and optimize the model parameters, and the calculation method is as follows:

[0030]

[0031] Where x is the input, y is the target, w is the weight, C is the number of categories, and N is the minimum block dimension.

[0032] Beneficial effects: Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention proposes a multimodal cross-attention feature fusion architecture, which has the function of fusing multi-domain feature information and can effectively suppress redundant information interference; the present invention proposes to use optical image texture features and SAR image scattering features as network optimization prior information, which can provide richer image detail feature information for subsequent deep learning and object segmentation, and improve the accuracy of object segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A schematic diagram of the structure of a typical ground object segmentation model based on multi-source data attention feature fusion constructed by the present invention;

[0034] Figure 2 Schematic diagram of the structure of the multimodal cross-attention fusion module proposed in the present invention;

[0035] Figure 3 The calculation flow chart of the multi-head cross attention mechanism proposed in the present invention;

[0036] Figure 4 These are examples of optical images, SAR images, and semantically labeled images used in the experiments of the present invention;

[0037] Figure 5 This is a diagram showing the segmentation results of seven types of typical ground objects using the algorithm proposed in this invention. DETAILED DESCRIPTION

[0038] The present invention is further described in detail below with reference to the accompanying drawings.

[0039] This paper proposes a typical ground object target segmentation method based on multi-source data attention feature fusion, comprehensively considers the observation advantages of SAR and optical data for different ground objects and designs an adaptive fusion algorithm, which is then applied to the semantic segmentation task of typical ground objects. Specifically, it includes the following steps:

[0040] Step 1: Obtain a dataset containing high-resolution optical images, SAR images, and corresponding semantically labeled images, divide the dataset into training set, validation set, and test set, perform sub-image cutting first, and then perform data enhancement on the training set images to obtain the preprocessed dataset.

[0041] Based on the WHU-OPT-SAR dataset containing pixel-level optical images and SAR images of the same area, the present invention first cuts the dataset into non-overlapping pixel blocks of 256×256 pixels, then standardizes the images, sets the mean to [0.485, 0.456, 0.406], sets the standard deviation to [0.229, 0.224, 0.225], and finally performs data enhancement on the training dataset.

[0042] Step 2: Construct a typical ground object segmentation model based on multi-source data attention feature fusion, such as Figure 1 As shown in the figure, it includes two main parts: encoder and decoder. The encoder part includes data prior feature construction and guidance module, feature extraction backbone network and multimodal cross attention fusion module, and the decoder part includes upsampling interpretation module.

[0043] Among them, the data prior feature construction and guidance module is used to extract the texture features of optical images and the scattering features of SAR images as the neural network prior information, which can realize the special feature extraction and second-order hidden feature mining of optical images and SAR images; then, the encoder with the same structure is used to extract the optical image feature vector and the SAR image feature vector, and their corresponding query set feature vector Q, index set feature vector K, and content feature vector V; secondly, a multimodal cross-attention fusion module is constructed to calculate the optical image attention feature vector A respectively. OPT =Attention(Q OPT ,K SAR ,V SAR ), and the SAR image attention feature vector A SAR =Attention(Q SAR ,K OPT ,V OPT ), and then concatenate to obtain the multimodal cross-attention feature vector A cros =concat(A OPT ,A SAR ); Further, the optical image feature vector obtained by the tandem encoder, the SAR image feature vector, and the multimodal cross-attention feature vector A cros , as the model decoder input, and obtain the model prediction result.

[0044] The feature extraction backbone network adopts ResNet50, and the feature extraction of the input of optical image and SAR image is performed respectively to obtain representative and effective multi-level features of the two types of images; in the multimodal cross-attention fusion module, the query set feature vector Q, the index set feature vector K, and the content feature vector V are cross-calculated to obtain the fusion features with semantic consistency; the second and fifth layer features extracted by the feature extraction backbone network are selected as the low-dimensional features and high-dimensional features of the image respectively, the high-dimensional features of the optical image and SAR image are used as the input of the high-dimensional multimodal cross-attention feature fusion module, and the low-dimensional features of the two are used as the input of the low-dimensional multimodal cross-attention feature fusion module. Finally, the cross-attention feature vectors output by the high- and low-dimensional modules are connected in series to obtain the multi-dimensional optical SAR cross-attention fusion features; the multi-dimensional optical SAR cross-attention fusion features are input into the decoder, and the final semantic segmentation results are obtained after convolution and upsampling operations.

[0045] First, the image features are extracted through the feature extraction backbone network. The cut optical image and SAR image are respectively subjected to two-dimensional convolution operations using 3×3 convolution kernels, and 1 pixel is padded around the edge of the input image. The data of each channel is normalized by batch normalization (Batch Normalization, BN); the result is passed through the maximum pooling layer to achieve downsampling of the image data. After the above processing twice, two types of 64×64 pixel blocks are obtained as low-dimensional features as the input of the inter-domain fusion module. After the above operation three times, two types of 32×32 pixel blocks are obtained, which are no longer passed through the maximum pooling layer. After two convolution and batch normalization operations, two types of 32×32 pixel blocks with 2048 channels are obtained as high-dimensional features as the input of the multimodal cross-attention fusion module.

[0046] The BN layer normalizes each batch of data in each hidden layer of the neural network so that the data of each feature dimension is in a standard normal distribution, which helps solve problems such as gradient disappearance and gradient explosion, and helps accelerate the convergence of the model; the maximum pooling layer can reduce the size of the feature map and obtain its most significant features.

[0047] The structure of the multimodal cross-attention fusion module is as follows: Figure 2 As shown, the purpose is to achieve the fusion of optical image features and SAR image features.

[0048] The high-dimensional features and low-dimensional features obtained in the previous step are used as inputs respectively. Through the cross-attention mechanism, the feature data information of the optical and SAR domains can be considered at the same time and their attention weights can be calculated. First, the query vector Q (Query), key vector K (Key), and value vector V (Value) are obtained by inputting the matrix X as follows:

[0049] {Q,K,V}={XW Q ,XW K ,XW V}

[0050] Then multiply the Q and K matrices by To avoid the problem of too small gradient after normalization by Softmax function, the formula is as follows after normalization:

[0051]

[0052] In the formula, Q, K, and V are matrices obtained after linear mapping of the input matrix, d k is the K matrix dimension, B is the learnable relative position encoding, and Softmax is the activation function, which converts the input real number vector into a probability distribution vector.

[0053] Based on the above attention mechanism principle, in order to better integrate feature data from different sources, we consider cross-calculating the correlation between query vectors Q, key vectors K, and value vectors V from different sources to obtain the weighted value of the attention mechanism. The calculation formula of the multimodal cross attention fusion module is as follows:

[0054]

[0055] Z1=LN(MCA(Q1,K2,V2))+Q1,

[0056] Z2=LN(MCA(Q2,K1,V1))+Q2,

[0057] In the formula, MCA represents the multi-head cross attention module, and its calculation process is as follows Figure 3 As shown; LN represents layer regularization, which is to normalize all features of a single sample.

[0058] Furthermore, Z1 is convolved with Z2 to obtain the high-dimensional features and low-dimensional features after optical SAR fusion. Figure 1 As shown in the figure, firstly, the low-dimensional features and the prior features are connected to obtain the connected low-dimensional features, then the high-dimensional features and the prior features are connected, and the connected high-dimensional features are obtained through atrous spatial pyramid pooling (ASPP), and finally the connected high- and low-dimensional features are connected in series as the input of the decoder. Among them, the ASPP module first uses atrous convolution branches with different expansion rates to capture feature information at different scales, increase the receptive field without increasing network parameters, and improve the ability of feature extraction; then, the feature map of each branch is spatially pyramid pooled to obtain a feature vector of fixed size; finally, the feature vectors of each scale are connected, and the final ASPP feature representation is obtained after dimensionality reduction.

[0059] The decoder part is structured as follows: Figure 1 As shown in Figure 2, the output of the encoder is used as the input of the decoder, and after two 1×1 convolutions and one upsampling, the image is reconstructed to obtain the final semantic segmentation result.

[0060] Step 3: Use the preprocessed data set obtained in step 1 as the input of the model constructed in step 2, train and optimize the model parameters based on the weighted cross entropy loss function, and realize the segmentation of typical ground objects based on multi-source data.

[0061] The present invention uses the WHU-OPT-SAR data set provided by Wuhan University as the experimental data set. The data set contains 100 optical remote sensing images and SAR remote sensing images with a size of 5556×3704 pixels, covering an area of ​​about 50,000 square kilometers in Hubei Province. Figure 4 The example images of SAR images, optical images and real labels in the experiment are shown in turn, including 7 main categories: farmland, city, village, water body, forest, road and others. This area has a subtropical monsoon climate, with the lowest altitude of about 50 meters and the highest altitude of about 3,000 meters. It has rich and diverse terrain and vegetation, which can better simulate the ground object segmentation task in actual application scenarios.

[0062] The WHU-OPT-SAR dataset is divided into training set, validation set and test set. The Adam optimizer is used to train the model, and the hyperparameters β1 and β2 are set to 0.900 and 0.999, which are used to calculate the moving average of momentum and squared gradient; the weighted cross entropy loss function is used, which is calculated as follows:

[0063]

[0064] Where x is the input, y is the target, w is the weight, C is the number of categories, and N is the minimum block dimension.

[0065] The present invention uses accuracy, single category accuracy, category average accuracy, single category intersection-over-union ratio, average intersection-over-union ratio and kappa coefficient as indicators to measure the segmentation effect.

[0066] The WHU-OPT-SAR dataset is taken as an example to verify the present invention. Figure 5 This is the segmentation result of the typical ground object segmentation method proposed in this invention on the WHU-OPT-SAR dataset for a total of 7 types of targets, including farmland, city, village, water body, forest, road and others.

[0067] Table 1. Object segmentation accuracy based on WHU-OPT-SAR dataset

[0068]

[0069]

[0070] Table 2 Kappa values ​​for object segmentation based on WHU-OPT-SAR dataset

[0071] Model farmland City village Water forest the way other Kappa DeepLabv3+ 0.685 0.600 0.501 0.417 0.111 0.265 0.039 0.519 UNet 0.694 0.616 0.488 0.415 0.146 0.267 0.042 0.524 MCANet 0.715 0.597 0.516 0.457 0.149 0.337 0.039 0.552 The present invention 0.717 0.589 0.512 0.470 0.218 0.341 0.044 0.567

[0072] Tables 1 and 2 show the accuracy and Kappa coefficient comparison of the prediction results of 7 typical ground object targets by the proposed ground object segmentation method and DeepLabv3+, Unet, and MCANet algorithms, respectively. The experimental results show that compared with the traditional semantic segmentation algorithm, the proposed method achieves an average 1.83% improvement in accuracy and an average 3.53% improvement in Kappa coefficient, and has a more superior segmentation performance.

[0073] The technical solutions provided by the embodiments of the present invention are described in detail above. The principles and implementation methods of the embodiments of the present invention are described in this patent using specific examples. The above embodiments and features in the embodiments can be combined with each other without conflict. The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

Claims

1. A typical ground object segmentation method based on multi-source data attention feature fusion, It is characterized in that The following steps are involved: (1) Obtain a dataset containing high-resolution optical images, SAR images, and corresponding semantically labeled images, divide the dataset into a training set, a validation set, and a test set, and perform data preprocessing; (2) constructing a typical ground object target segmentation model based on multi-source data attention feature fusion to achieve semantic segmentation of typical ground object targets; the model includes an encoder and a decoder; the encoder includes a data prior feature construction and guidance module, a feature extraction backbone network and a multimodal cross attention fusion module; (3) The preprocessed data set obtained in step (1) is used as the input of the model constructed in step (2), and the model parameters are trained and optimized based on the weighted cross entropy loss function to achieve typical ground object segmentation based on multi-source data.

2. According to claim 1, a typical ground object target segmentation method based on multi-source data attention feature fusion, It is characterized in that The data preprocessing process in step (1) is as follows: First, the dataset is cut into non-overlapping pixel blocks of 256×256 pixels, and then the images are standardized, the mean is set to [0.485, 0.456, 0.406], the standard deviation is set to [0.229, 0.224, 0.225], and finally the training set images are enhanced.

3. According to claim 1, a typical ground object target segmentation method based on multi-source data attention feature fusion, It is characterized in that The implementation process of the typical ground object target segmentation model based on multi-source data attention feature fusion in step (2) is as follows: Firstly, the data prior feature construction and guidance module extracts the texture features of optical images and the scattering features of SAR images as the neural network prior information for the special feature extraction and second-order hidden feature mining of optical images and SAR images; then, an encoder with the same structure is used to extract the optical image feature vector and the SAR image feature vector, and their corresponding query set feature vector Q, index set feature vector K, and content feature vector V respectively; Secondly, the multimodal cross-attention fusion module calculates the optical image attention feature vector A OPT =Attention(Q OPT ,K SAR ,V SAR ), and the SAR image attention feature vector A SAR =Attention(Q SAR ,K OPT ,V OPT ), and then concatenate to obtain the multimodal cross-attention feature vector A cros =concat(A OPT ,A SAR ); Further, the optical image feature vector obtained by the tandem encoder, the SAR image feature vector, and the multimodal cross-attention feature vector A cros , as the model decoder input, and obtain the model prediction result.

4. According to claim 1, a typical ground object target segmentation method based on multi-source data attention feature fusion, It is characterized in that The feature extraction backbone network adopts ResNet50, and performs feature extraction on the input of optical images and SAR images respectively to obtain representative and effective multi-level features of the two types of images; The 2nd and 5th layer features extracted by the feature extraction backbone network are selected as the low-dimensional features and high-dimensional features of the image respectively. The high-dimensional features of the optical image and SAR image are used as the input of the high-dimensional cross-attention feature fusion module, and the low-dimensional features of the two are used as the input of the low-dimensional cross-attention feature fusion module. Finally, the cross-attention feature vectors output by the high- and low-dimensional modules are connected in series to obtain the multi-dimensional optical SAR cross-attention fusion features; the multi-dimensional optical SAR cross-attention fusion features are input into the decoder, and the final semantic segmentation results are obtained after convolution and upsampling operations.

5. According to claim 1, a typical ground object target segmentation method based on multi-source data attention feature fusion, It is characterized in that In step (2), the feature extraction backbone network extracts features from the image, performs two-dimensional convolution operations on the cut optical image and SAR image using a 3×3 convolution kernel, fills the edges of the input image with 1 pixel, and normalizes the data of each channel through batch normalization; the result passes through a maximum pooling layer to achieve downsampling of the image data; after two times of the above processing, two types of 64×64 pixel blocks are obtained as low-dimensional features as input to the inter-domain fusion module; after three times of the above operation, two types of 32×32 pixel blocks are obtained, which are no longer passed through the maximum pooling layer, and after two times of convolution and batch normalization operations, two types of 32×32 pixel blocks with a channel number of 2048 are obtained as high-dimensional features as input to the inter-domain fusion module.

6. According to claim 1, a typical ground object target segmentation method based on multi-source data attention feature fusion, It is characterized in that The implementation process of the multimodal cross attention fusion module in step (2) is as follows: The high-dimensional features and low-dimensional features are taken as input respectively. The cross-attention mechanism can simultaneously consider the feature data information of the optical and SAR domains and calculate their attention weights. First, the query vector Q, key vector K, and value vector V are obtained through the input matrix X. The formula is as follows: {Q,K,V}={XW Q ,XW K ,XW V ) Then multiply the Q and K matrices by After normalization, the formula is as follows: In the formula, Q, K, and V are matrices obtained after linear mapping of the input matrix, d k is the K matrix dimension, B is the learnable relative position code, and Softmax is the activation function, which converts the input real number vector into a probability distribution vector; The query vector Q, key vector K, and value vector V from different sources are cross-calculated for relevance to obtain the weighted value of the attention mechanism. The calculation formula of the multimodal cross-attention fusion module is as follows: Z 1 =LN(MCA(Q 1 ,K 2 ,V 2 ))+Q 1 Z 2 =LN(MCA(Q 2 ,K 1 ,V 1 ))+Q 2 Where, X 1 is the embedded representation of the optical input; X 2 is the embedded representation of SAR input; The optical input and SAR input correspond to the query vector Q 1 , Q 2 , the key vector K 1 , K 2 , value vector V 1 、V 2 The weight matrix of ; MCA represents the multi-head cross attention module; LN stands for layer normalization, which normalizes all features of a single sample; Z 1 With Z 2 Convolution is performed to obtain the high-dimensional and low-dimensional features after optical SAR fusion. First, the low-dimensional features and the prior features are connected to obtain the connected low-dimensional features. Then the high-dimensional features and the prior features are connected, and the connected high-dimensional features are obtained through the atrous spatial pyramid pooling ASPP. Finally, the connected high- and low-dimensional features are connected in series as the input of the decoder.

7. A typical ground object target segmentation method based on multi-source data attention feature fusion according to claim 6, It is characterized in that The atrous spatial pyramid pooling (ASPP) first uses atrous convolution branches with different dilation rates to capture feature information at different scales, increase the receptive field without increasing network parameters, and improve the ability of feature extraction; then, the feature map of each branch is spatially pyramid pooled to obtain a feature vector of a fixed size; finally, the feature vectors of each scale are connected and the final ASPP feature representation is obtained after dimensionality reduction.

8. According to claim 1, a typical ground object target segmentation method based on multi-source data attention feature fusion, It is characterized in that The decoder in step (2) includes two 1×1 convolutions and one upsampling.

9. According to claim 1, a typical ground object target segmentation method based on multi-source data attention feature fusion, It is characterized in that The weighted cross entropy loss function in step (3) is used to train and optimize the model parameters, and the calculation method is as follows: Where x is the input, y is the target, w is the weight, C is the number of categories, and N is the minimum block dimension.

Images