An improved method for road extraction from remote sensing images based on dual encoders

Abstract

The present invention discloses an improved remote sensing image road extraction method based on a dual encoder, comprising the following steps: preparing a data set; building a remote sensing image road extraction network; training the remote sensing image road extraction network; and testing the remote sensing image road extraction network. The present invention uses a dual encoder combining Swin Transformer and pre-trained Resnet‑34 in the encoder to perform feature extraction to improve the connectivity of road extraction, and combines a cascaded parallel hole convolution block to realize multi-scale feature representation and reduce the fragmentation of the extracted roads. The present invention designs a spatial pixel module in the encoder to capture the spatial information of pixel-level features, thereby alleviating the inaccurate semantic information caused by occlusion. And enhances the global modeling capability of Swin Transformer due to the limitation of the window mechanism. The present invention not only maintains the complete structure of the road network, but also enhances the connectivity and integrity of the road.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing image segmentation and deep learning technology, and in particular to a remote sensing image road extraction method based on an improved dual encoder. Background Art

[0002] Road data, as an important component of geographic information data, plays a key role in urban planning, autonomous driving, vehicle navigation, and geographic information system construction. However, manually labeling road data is a time-consuming and arduous task, especially when a large area of ​​road network coverage is present in the image. Therefore, these fields urgently need efficient automated road extraction methods to provide accurate and reliable road information. Research on automated road extraction methods can significantly save time and labor costs, and is of great significance for improving traffic safety, supporting autonomous driving, and promoting the development of smart cities. In recent years, various automated road extraction methods have been proposed, which are mainly divided into traditional methods and deep learning-based methods.

[0003] Traditional road extraction methods, such as threshold segmentation, edge detection, texture analysis, and mathematical morphology algorithms, use shallow features (such as grayscale, edge, spectrum, texture, and geometric features) to extract roads from remote sensing images. These methods are often limited by fixed parameters and, when processing large-scale, high-resolution satellite imagery, cannot fully capture road features at different scales, resulting in low efficiency and poor results.

[0004] With the continuous advancement of deep learning technology in image segmentation, road extraction methods using deep learning have become mainstream. Road extraction methods from remote sensing images based on convolutional neural networks (CNNs) have attracted significant interest among researchers in the field of remote sensing image processing. A series of advanced deep convolutional neural networks have been applied to various remote sensing image segmentation tasks. Compared with traditional road extraction methods, these methods, based on deep convolutional neural networks, have attracted extensive attention and research due to their advantages, such as high automation, the ability to automatically learn and accurately extract road features from large-scale remote sensing imagery, and strong adaptability and generalization. These methods overcome the shortcomings of traditional methods and offer superior performance. However, roads in remote sensing images may be partially obscured by buildings, trees, or other objects. Furthermore, shadows can cause variations in the brightness and color of road areas. These occlusions and shadows can lead to discontinuities in road areas. To address this issue, algorithms are required that can comprehensively leverage global information to identify and fill in missing sections. However, these CNN-based methods primarily focus on extracting local features, and their limited receptive field limits their ability to discern road integrity, resulting in inaccurate segmentation results.

[0005] In recent years, Transformer-based methods have been used in computer vision and have achieved promising results. Unlike previous CNN-based methods, Transformer has a strong ability to capture long-range dependencies, stronger contextual semantic relevance, a wider receptive field, and excellent transferability in downstream tasks. However, the road extraction task faces challenges posed by complex scenes and background interference such as buildings, trees, and vehicles. These interfering objects are similar to roads in texture and color, making them difficult to distinguish. To improve the accuracy of distinction, the key is to more effectively utilize the correlation information and channel information between pixels in the feature map to accurately segment the road. Although Transformer-based methods have significant advantages in capturing long-range dependencies, they have limitations in capturing local details and integrating spatial pixel information and channel information, resulting in poor performance in distinguishing roads from interfering objects and poor connectivity of the extracted roads. Summary of the Invention

[0006] In summary, in order to solve the problem of automatic road extraction in high-resolution remote sensing images, the present invention proposes a remote sensing image road extraction method based on an improved dual encoder, which aims to improve the connectivity and integrity of roads and maintain the integrity of the road network to the greatest extent.

[0007] In order to achieve the above-mentioned object of the invention, the present invention adopts the following technical solutions:

[0008] A road extraction method for remote sensing images based on an improved dual encoder includes the following steps:

[0009] A. Prepare the dataset

[0010] Collect high-resolution remote sensing image data taken by remote sensing satellites and prepare a remote sensing image road extraction dataset by following the steps below:

[0011] A1. Label and crop the image

[0012] A2. Image Enhancement

[0013] B. Build a remote sensing image road extraction network

[0014] The remote sensing image road extraction network is an improved dual-encoder architecture, employing an encoder-decoder structure. The encoder consists of a primary encoder and a secondary encoder. The primary encoder is a ResNet block composed of a ResNet-34, while the secondary encoder is a SwinConv block composed of a Swin Transformer module and a spatial pixel module. Each time, a batch of enhanced remote sensing images is fed into a Conv block (which includes convolution and pooling operations) and the first SwinConv block. The feature maps output by the Conv block are fed into the first ResNet block. The feature maps output by the first SwinConv block are input into the first feature fusion module and the second SwinConv block respectively, the feature maps output by the first ResNet block are input into the first feature fusion module, the feature maps output by the first feature fusion module are input into the second ResNet block and the first jump connection unit respectively, the feature maps output by the second SwinConv block are input into the second feature fusion module and the third SwinConv block respectively, the feature maps output by the second ResNet block are input into the second feature fusion module, the feature maps output by the second feature fusion module are input into the third ResNet block and the second jump connection unit respectively, the feature maps output by the third SwinConv block are input into the third feature fusion module and the fourth SwinConv block respectively, the feature maps output by the third ResNet block are input into the third feature fusion module, and the third feature fusion module is input into the fourth SwinConv block. The feature maps output by the block are respectively input into the fourth ResNet block and the third skip connection unit. The feature maps output by the fourth SwinConv block and the feature maps output by the fourth ResNet block are both input into the fourth feature fusion module. The feature maps output by the fourth feature fusion module are input into the dilated convolution module. The feature maps output by the dilated convolution module are input into the first decoding block. The feature maps output by the first decoding block are fused with the feature maps from the third skip connection unit and then input into the second decoding block. The feature maps output by the second decoding block are fused with the feature maps from the second skip connection unit and then input into the third decoding block. The feature maps output by the third decoding block are fused with the feature maps from the first skip connection unit and then input into the fourth decoding block. The feature maps output by the fourth decoding block pass through the Conv block containing transposed convolution and convolution operations to output a prediction map of the same size as the input remote sensing image.

[0015] C. Training remote sensing image road extraction network

[0016] The enhanced remote sensing image dataset prepared in step A is fed into the remote sensing image road extraction network for training. During training, a composite loss function consisting of binary cross entropy loss and dice loss is used as the final loss function. The initial learning rate is set to 2e-4, and the adaptive moment estimation optimization algorithm is used to adaptively adjust the learning rate. Training is terminated when the learning rate falls below 5e-7.

[0017] D. Testing the Remote Sensing Image Road Extraction Network

[0018] Select the road extraction network parameters with the smallest loss value on the training set from step C, and test the road extraction performance of the remote sensing image road extraction network on the test set.

[0019] Furthermore, the method for marking and cropping the image in step A1 includes the following steps:

[0020] Download the original RGB three-band color remote sensing image dataset from the official website and annotate it using dataset annotation software. Then, crop the original remote sensing images and the manually annotated label images to 1024×1024 pixels using the same method, obtaining a dataset of no fewer than 6,000 labeled 1024×1024 pixel remote sensing images, referred to as the labeled remote sensing image dataset.

[0021] Furthermore, the image enhancement method in step A2 includes the following steps:

[0022] Image enhancement operations are performed on the labeled remote sensing image dataset. The image enhancement operations include horizontal flipping, vertical flipping, diagonal flipping, color dithering, and translation. The dataset after image enhancement operations is called enhanced remote sensing image dataset.

[0023] Furthermore, the method for building a remote sensing image road extraction network in step B includes the following steps:

[0024] B1. Build a dual encoder

[0025] In the encoder module, a dual encoder is used to extract feature information using a fusion of the Swin Transformer and a ResNet-34 pre-trained on the ImageNet dataset. The ResNet-34 serves as the primary encoder, while the SwinConv block, a combination of the Swin Transformer and a spatial pixel module, serves as the auxiliary encoder. A feature fusion module is designed for the fusion of the primary and auxiliary encoders. In the auxiliary encoder SwinConv block, a spatial pixel module is designed that spans the W-Trans and SW-Trans blocks in the Swin Transformer.

[0026] B2. Introducing a dilated convolutional module into the central part of the remote sensing image road extraction network

[0027] The dilated convolution module is composed of dilated convolutions with different dilated ratios stacked in a cascade mode. The dilated ratios of the stacked dilated convolutions are 1, 2, 4, and 8, and the corresponding receptive fields are 3, 7, 15, and 31, respectively.

[0028] B3. Design a spatial pixel module in the Swin Transformer part of the encoder of the remote sensing image road extraction network

[0029] When the image enters the SwinConv block, for the features in the nth SwinConv block, the input features s of SwinTransformer are first converted to l-1 Reshape into z, where c1=2 n-1 C1,h=(H / (2 n+1 )) and w=(W / (2 n+1 )).

[0030] The reshaped feature z is fed into a dilated convolution with a size of 3×3 and a dilation rate of 2, and the number of channels becomes c1 / 2. Then, a global average pooling operation is performed to obtain statistical features in the three dimensions of height h, width w, and channel c. Specifically, the element calculations in the three dimensions are expressed as follows:

[0031]

[0032] Where GAP h 、GAP w and GAP c They are global average pooling in three dimensions: height h, width w, and channel c. is the dilated convolution layer, batch normalization and GELU activation. The tensors in three dimensions are v h 、v w and v c , v h 、v w and v c are the weights of feature z convergence in height h, width w and channel c, respectively. and The three weight information are combined to obtain the spatial attention information I, which is calculated as follows:

[0033]

[0034] in, represents matrix multiplication,

[0035] Finally, the spatial attention information I is combined with the Swin Transformer block s l+1 The output of the SwinConv block is added to obtain the output feature L, which is matched by a 1×1 convolution layer. l+1 The output feature L is expressed as follows:

[0036]

[0037] in, represents matrix addition, represents a 1×1 convolutional layer, batch normalization, and GELU activation.

[0038] B4. Design a feature fusion module in the encoder part of the remote sensing image road extraction network

[0039] Assume that the outputs of the main encoder and auxiliary encoder at the nth stage are A n With S n . n Input to the 1×1 convolution layer to change the size and obtain features

[0040] First, apply the maximum pooling to calculate the statistical features and the fully connected layer to obtain At the same time, the global weight is calculated using average pooling and put into a fully connected layer. The result is calculated using P A This process is expressed as follows:

[0041]

[0042] Where σ represents the ReLu function.

[0043] Then, use P A Optimize P for each channel M , which is expressed as follows:

[0044] P = δ(FC(P M ⊙P A )) (5)

[0045] Where δ represents the Sigmoid function and ⊙ represents element-wise multiplication. n Multiply to get the extracted features.

[0046] Finally, the output feature F of the feature fusion module n It is expressed as follows:

[0047]

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] 1. In the encoder part, the present invention uses a dual encoder combining Swin Transformer and pre-trained Resnet-34 for feature extraction to improve the connectivity of road extraction, and combines a cascaded parallel void convolution block to realize multi-scale feature representation and reduce the fragmentation of extracted roads.

[0050] 2. This paper designs a spatial pixel module within the Swin Transformer part of the encoder to capture the spatial information of pixel-level features, thereby alleviating the inaccuracy of semantic information caused by occlusion. It also enhances the global modeling capabilities of the Swin Transformer, which is limited by the window mechanism.

[0051] 3. This paper designs a feature fusion module in the fusion part of the two encoders, extracting channel-related information from the Swin Transformer as global clues to guide the ResNet. It can effectively distinguish roads blocked by obstacles.

[0052] 4. In summary, the present invention not only maintains the complete structure of the road network, but also enhances the connectivity and integrity of the roads, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is an architectural diagram of the present invention.

[0054] Figure 2 This is the structural diagram of the dilated convolution module.

[0055] Figure 3 This is a diagram of the SwinConv block and its internal module structure.

[0056] Figure 4 It is the structural diagram of the feature fusion module.

[0057] Figure 5 This is a comparison chart of the road extraction effects of the present invention and other road segmentation networks. DETAILED DESCRIPTION

[0058] The present invention will be further described below with reference to the accompanying drawings. Figure 1 As shown in the figure, a remote sensing image road extraction method based on dual encoder improvement is shown in the figure. The specific steps are as follows:

[0059] Step 1: Collect remote sensing image data taken by remote sensing satellites. Download the original remote sensing image dataset from the official website of remote sensing image dataset. The original remote sensing image dataset is an RGB three-band color remote sensing image dataset.

[0060] Step 2: Annotate the remote sensing image dataset. Use a remote sensing image annotation tool to annotate the road areas in the RGB three-channel color remote sensing image collected from a remote sensing satellite, generating a label image of the same size as the original image. The label image is a black and white binary image, with the road markings in black and the background in white.

[0061] Step 3: Image cropping. The original RGB color remote sensing image and its corresponding label image are cropped into 1024×1024 pixel images in the same way, resulting in a dataset of 6226 labeled remote sensing images of 1024×1024 pixel size.

[0062] Step 4: Image enhancement. The 6226 cropped 1024×1024 pixel remote sensing images were enhanced using operations such as horizontal flip, vertical flip, diagonal flip, color dithering, and translation.

[0063] Step 5: Dataset Partitioning. The 6226 labeled remote sensing images after image enhancement are randomly divided into training and test sets in a ratio of 4:1, resulting in 4981 training images and 1245 test images.

[0064] Step 6: Build a remote sensing image road extraction network. The remote sensing image road extraction network is an improved remote sensing image road extraction network based on dual encoders. The overall architecture adopts an encoder-decoder architecture. The encoder includes a main encoder and an auxiliary encoder. The main encoder is a ResNet block composed of ResNet-34. The auxiliary encoder is a SwinConv block composed of a SwinTransformer and a designed spatial pixel module. The SwinConv block structure is as follows: Figure 3 In addition, it also includes a feature fusion module and a hole convolution block, as shown in Figure 4 and Figure 2As shown. The enhanced remote sensing image is input into the Conv block containing convolution and pooling operations and the first SwinConv block respectively, and the feature map output by the Conv block is input into the first ResNet block. The feature maps output by the first SwinConv block are input into the first feature fusion module and the second SwinConv block respectively, the feature maps output by the first ResNet block are input into the first feature fusion module, the feature maps output by the first feature fusion module are input into the second ResNet block and the first jump connection part respectively, the feature maps output by the second SwinConv block are input into the second feature fusion module and the third SwinConv block respectively, the feature maps output by the second ResNet block are input into the second feature fusion module, the feature maps output by the second feature fusion module are input into the third ResNet block and the second jump connection part respectively, the feature maps output by the third SwinConv block are input into the third feature fusion module and the fourth SwinConv block respectively, the feature maps output by the third ResNet block are input into the third feature fusion module, and the third feature fusion module The feature maps output by the block are respectively input into the fourth ResNet block and the third skip connection part. The feature maps output by the fourth SwinConv block and the feature maps output by the fourth ResNet block are both input into the fourth feature fusion module. The feature maps output by the fourth feature fusion module are input into the hole convolution module; the feature maps output by the hole convolution module are input into the first decoding block, the feature maps output by the first decoding block are fused with the feature maps from the third skip connection part and then input into the second decoding block, the feature maps output by the second decoding block are fused with the feature maps from the second skip connection part and then input into the third decoding block, the feature maps output by the third decoding block are fused with the feature maps from the first skip connection part and then input into the fourth decoding block, and the feature maps output by the fourth decoding block are output as a prediction map of the same size as the input remote sensing image after passing through the Conv block containing transposed convolution and convolution operations.

[0065] Step 7: Train the remote sensing image road extraction network. The 4981 training images obtained after the partitioning in Step 5 were input into the remote sensing image road extraction network. Training was performed simultaneously on two NVIDIA GeForce RTX 3090 graphics cards, with eight images processed per training session. During training, a composite loss function consisting of binary cross entropy loss and dice loss was used as the final loss function. The initial learning rate was set to 2e-4, and the adaptive moment estimation optimization algorithm was used to adaptively adjust the learning rate. Training was terminated when the learning rate fell below 5e-7. The road extraction network parameters that minimized the loss on the training set were selected, and the road extraction performance of the remote sensing image road extraction network was tested on the test set.

[0066] Step 8: Test the performance of the remote sensing image road extraction network. Input the 1245 test images obtained after the division in Step 5 into the trained remote sensing image road extraction network and output the corresponding 1245 prediction images.

[0067] Step 9: Evaluation of the prediction results of the road extraction network based on remote sensing images. The prediction results are evaluated using evaluation indicators such as F1 score and intersection-over-union ratio, and a comprehensive evaluation is performed in combination with the visual comparison chart of the prediction results. Figure 5 As shown in the figure, some mainstream road extraction networks have shown good road extraction effects. However, these mainstream methods do not show good detection performance for roads in some complex scenes. The main reason is that they do not effectively extract and fuse the global and local information of the image and lack attention to channel information. Figure 5 The results of road extraction from these images using eight different methods are shown. Figure 5 The first and second rows of the image may be similar in color and texture to the surrounding farmland or land. This similarity makes it difficult for general road extraction algorithms to clearly distinguish road boundaries. However, the remote sensing image road extraction network of the present invention can accurately distinguish between road and non-road areas compared to other methods. Figure 5 In the third row, the road area is blocked by trees. In this case, the other seven methods cannot recognize or can only recognize part of the blocked road, while the remote sensing image road extraction network of the present invention can extract complete road information. Figure 5 The other seven methods missed more road information and the extracted roads were fragmented, while the remote sensing image road extraction network of the present invention can maintain a relatively complete road structure. Figure 5 The other seven methods cannot completely extract the road structure, while the remote sensing image road extraction network of the present invention can accurately extract the road area.

[0068] The present invention is not limited to this embodiment, and any equivalent concepts or modifications within the technical scope disclosed by the present invention are included in the protection scope of the present invention.

Claims

1. A method for extracting roads from remote sensing images based on an improved dual encoder, characterized by: The following steps are involved: A. Prepare the dataset Collect high-resolution remote sensing image data taken by remote sensing satellites and prepare a remote sensing image road extraction dataset by following the steps below: A1. Label and crop the image; A2, image enhancement; B. Build a remote sensing image road extraction network The remote sensing image road extraction network is an improved remote sensing image road extraction network with dual encoders. The overall architecture adopts an encoder-decoder architecture. The encoder includes a main encoder and an auxiliary encoder. The main encoder is a ResNet block composed of ResNet-34; the auxiliary encoder is a SwinConv block composed of Swin The network consists of a Transformer module and a spatial pixel module; each time, the batch size of enhanced remote sensing images are input into the Conv block containing convolution and pooling operations and the first SwinConv block, and the feature map output by the Conv block is input into the first ResNet block; the feature map output by the first SwinConv block is input into the first feature fusion module, the feature map output by the first feature fusion module is input into the second ResNet block and the first jump connection unit, the feature map output by the second SwinConv block is input into the second feature fusion module and the third SwinConv block, the feature map output by the second ResNet block is input into the second feature fusion module, the feature map output by the second feature fusion module is input into the third ResNet block and the second jump connection unit, and the feature map output by the third SwinConv block is input into the third feature fusion module and the fourth SwinConv blocks, the feature map output by the third ResNet block is input to the third feature fusion module, the feature map output by the third feature fusion module is input to the fourth ResNet block and the third jump connection unit respectively, the feature map output by the fourth SwinConv block and the feature map output by the fourth ResNet block are both input to the fourth feature fusion module, and the feature map output by the fourth feature fusion module is input to the hole convolution module; the feature map output by the hole convolution module is input to the first decoding block, the feature map output by the first decoding block is fused with the feature map from the third jump connection unit and then input to the second decoding block, the feature map output by the second decoding block is fused with the feature map from the second jump connection unit and then input to the third decoding block, the feature map output by the third decoding block is fused with the feature map from the first jump connection unit and then input to the fourth decoding block, and the feature map output by the fourth decoding block is output after the Conv block containing transposed convolution and convolution operations, and a prediction map of the same size as the input remote sensing image is output; C. Training remote sensing image road extraction network The enhanced remote sensing image dataset prepared in step A is input into the remote sensing image road extraction network for training. During training, a composite loss function consisting of binary cross entropy loss and dice loss is used as the final loss function. The initial learning rate is set to 2e-4, and the adaptive moment estimation optimization algorithm is used to adaptively adjust the learning rate. When the learning rate is less than 5e-7, training is stopped. D. Testing the Remote Sensing Image Road Extraction Network Select the road extraction network parameters with the smallest loss value on the training set from step C, and test the road extraction performance of the remote sensing image road extraction network on the test set.

2. The method for extracting roads from remote sensing images based on an improved dual encoder according to claim 1, characterized in that: The method for labeling and cropping an image in step A1 includes the following steps: The original remote sensing image dataset was downloaded from the official website of the remote sensing image dataset. The original remote sensing image dataset is an RGB three-band color remote sensing image dataset and annotated using the dataset annotation software. Then, the original remote sensing images and the manually annotated label images were uniformly cropped to 1024×1024 pixel images in the same way, obtaining a remote sensing image dataset of no less than 6,000 labeled 1024×1024 pixel images, referred to as the labeled remote sensing image dataset.

3. The method for extracting roads from remote sensing images based on an improved dual encoder according to claim 1, characterized in that: The image enhancement method described in step A2 includes the following steps: An image enhancement operation is performed on a labeled remote sensing image dataset, wherein the image enhancement operation includes horizontal flipping, vertical flipping, diagonal flipping, color jittering, and translation. The dataset after the image enhancement operation is called an enhanced remote sensing image dataset.

4. The method for extracting roads from remote sensing images based on an improved dual encoder according to claim 1, characterized in that: The method for building a remote sensing image road extraction network in step B includes the following steps: B1. Build a dual encoder In the encoder module, a dual encoder is used to extract feature information by fusing the Swin Transformer with a ResNet-34 pre-trained on the ImageNet dataset. ResNet-34 serves as the primary encoder, and the SwinConv block, which combines the Swin Transformer with a spatial pixel module, serves as the auxiliary encoder. A feature fusion module is designed for the fusion of the primary and auxiliary encoders. In the auxiliary encoder SwinConv block, a spatial pixel module is designed that spans the W-Trans and SW-Trans blocks in the Swin Transformer. B2. Introducing a dilated convolutional module into the central part of the remote sensing image road extraction network The dilated convolution module is composed of dilated convolutions with different dilation rates stacked in a cascade mode. The dilation rates of the stacked dilated convolutions are 1, 2, 4, and 8, and the corresponding receptive fields are 3, 7, 15, and 31, respectively. B3. Design a spatial pixel module in the Swin Transformer part of the encoder of the remote sensing image road extraction network When the image enters the SwinConv block, for the features in the nth SwinConv block, the input features s of SwinTransformer are first converted to l-1 Reshape into z, where c1=2 n-1 C1,h=(H / (2 n+1 )) and w=(W / (2 n+1 )); The reshaped feature z is fed into a dilated convolution with a size of 3×3 and a dilation rate of 2, and the number of channels becomes c1 / 2. Then, a global average pooling operation is performed to obtain statistical features in the three dimensions of height h, width w, and channel c. Specifically, the element calculations in the three dimensions are expressed as follows: Where GAP h 、GAP w and GAP c They are global average pooling in three dimensions: height h, width w, and channel c; features is the dilated convolution layer, batch normalization and GELU activation; the tensors in three dimensions are v h 、v w and v c , v h 、v w and v c are the weights of feature z convergence in height h, width w and channel c, respectively. and The three weight information are combined to obtain the spatial attention information I, which is calculated as follows: in, represents matrix multiplication, Finally, the spatial attention information I is combined with the Swin Transformer block s l+1 The output of the SwinConv block is added to obtain the output feature L, which is matched by a 1×1 convolution layer. l+1 The dimension of ; the output feature L is expressed as follows: in, represents matrix addition, represents a 1×1 convolutional layer, batch normalization, and GELU activation; B4. Design a feature fusion module in the encoder part of the remote sensing image road extraction network Assume that the outputs of the main encoder and auxiliary encoder at the nth stage are A n With S n ; S n Input to the 1×1 convolution layer to change the size and obtain features First, apply the maximum pooling to calculate the statistical features and the fully connected layer to obtain At the same time, the global weight is calculated using average pooling and put into a fully connected layer. The result is calculated using P A This process is expressed as follows: Where, σ represents the ReLu function; Then, use P A Optimize P for each channel M , which is expressed as follows: P=δ(FC(P M ⊙P A )) (5) Where δ represents the Sigmoid function, ⊙ represents element-wise multiplication; the channel-dependent weight P is used with the output A of the main encoder n Multiply to get the extracted features; Finally, the output feature F of the feature fusion module n It is expressed as follows:

Images