Multi-branch multi-task multi-modal remote sensing image road extraction method and system

Abstract

The present application belongs to the technical field of remote sensing image road extraction, and particularly relates to a multi-branch multi-task multi-modal remote sensing image road extraction method and system. First, an encoding-decoding network with the same network structure but independent parameters is constructed to respectively extract features of optical images and SAR images, and a road surface segmentation label is used for supervised training. Second, the encoding layer features of the SAR image are introduced for road edge detection, and the intermediate features are input into the decoding layer features of the SAR image. Finally, the channel-strip spatial attention CA-SSA is used to fully fuse the shallow features and deep features of the optical images and the SAR images to predict the final road extraction result. The present application takes the optical images and the SAR images as the data basis, designs the network model around the road shape and topological structure, thereby fully mining the advantage information of the two modal images and improving the accuracy of the road extraction result.

Description

Technical Field

[0001] This invention belongs to the field of road extraction technology in remote sensing images, and specifically relates to a multi-branch, multi-task, multimodal remote sensing image road extraction method and system. Background Technology

[0002] Roads, as the backbone and core hub of transportation infrastructure, play a crucial role in map updates, vehicle navigation, urban planning, and disaster relief through rapid and accurate extraction. Road extraction refers to dividing each pixel of an image into roads and non-road (background) segments. Benefiting from advancements in semantic segmentation using convolutional neural networks, the intuitiveness of optical images, and the richness of their datasets, current research on automatic road interpretation primarily focuses on deep learning methods driven by optical images. This method combines optical images and labeled data, fully mining the deep features of the image through convolutional neural networks. The network framework serves as a mapping function, and its parameters are continuously updated and optimized based on a set similarity measurement standard (loss function).

[0003] Optical imagery visualizes surface information of ground features by relying on the electromagnetic energy reflected from them in the visible light band. It boasts advantages such as high spatial resolution and rich spectral and textural information. However, because optical imagery is acquired passively, with solar radiation as its primary energy source, its limitations are significant. Figure 1 As shown in (a) and (b), the image quality of optical images is often greatly affected by weather conditions. Furthermore, as... Figure 1 As shown in (c) and (d), due to factors such as imaging angle, projection errors and shadow features can cause discrepancies between the attribute characteristics of the road surface and the actual labels, leading to incorrect information guidance for road extraction. Conversely, SAR, as an active imaging system, possesses all-weather, all-day data acquisition capabilities and can provide scattering and geometric features of ground objects from a side-view perspective, but it suffers from problems such as patch noise and semantic difficulties. Overall, thanks to the continuous advancement of application demands, systematic, diversified, and mature remote sensing data acquisition systems are constantly emerging, and significant breakthroughs have been achieved in the granularity and adaptability of remote sensing data in electromagnetic spectrum description. Acquiring multimodal remote sensing images of the same area is easier and can provide diverse complementary information for ground object interpretation, such as spectral information from optical images and scattering information from SAR images. Furthermore, many studies have shown that combining the complementary information from optical and SAR images is beneficial for improving the effectiveness and accuracy of ground object interpretation.

[0004] Currently, multimodal data fusion, based on its origin, mainly falls into three categories: data-level, feature-level, and decision-level. Data-level fusion operates at the raw data level, treating multimodal data as a whole during subsequent interpretation, making it suitable for homogeneous data. Feature-level fusion extracts features from each data set and employs strategies (such as cascading, summation, and attention mechanisms) to achieve complementary advantages. Decision-level fusion analyzes and makes decisions based on the interpretation results of different data sets, but it doesn't fully consider the relative strengths of each. Although significant progress has been made in multimodal data-driven ground feature interpretation, the fundamentally different imaging mechanisms of optical and SAR images lead to inherent heterogeneity in their features. Therefore, constructing effective fusion strategies remains a key research area. Furthermore, most existing multimodal image interpretation technologies focus on full-element ground feature classification, neglecting the structure and distribution patterns of roads themselves, thus requiring improvement in accuracy and generalization capabilities. Summary of the Invention

[0005] The purpose of this invention is to solve the problems of the prior art and propose a multi-branch, multi-task, multimodal remote sensing image road extraction method and system. Based on optical imagery and SAR imagery, a network model is designed around the shape and topology of the road itself, so as to fully explore the advantages of the two modalities of imagery and improve the accuracy of road extraction results.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A multi-branch, multi-task, multimodal remote sensing image road extraction method, comprising:

[0008] We constructed encoder-decoder networks with identical structures but independent parameters to extract features from optical and SAR images, and supervised training using road surface segmentation labels.

[0009] The coding layer features of SAR images are introduced for road edge detection, and their intermediate features are input into the decoding layer features of SAR images.

[0010] Channel-strip spatial attention (CA-SSA) is used to fully fuse shallow and deep features of optical and SAR images to predict the final road extraction results.

[0011] According to the multi-branch, multi-task, multimodal remote sensing image road extraction method of the present invention, the coding layers of both the optical image branch and the SAR image branch use ResNet34 for feature extraction, the decoding layer uses a combination of transposed convolution and convolution for upsampling, and at the same time, a skip connection is used to perform pixel-by-pixel addition of the coding layer features and the corresponding decoding layer features.

[0012] According to the multi-branch, multi-task, multimodal remote sensing image road extraction method of the present invention, the step of introducing the coding layer features of SAR image for road edge detection and inputting its intermediate features into the decoding layer features of SAR image includes: obtaining four levels of features in the branch coding layer of SAR image by relying on independent convolution and upsampling, and then obtaining edge features by channel superposition of these four levels of features. The edge features are then processed by two more convolutions to output the road edge, and are also fused with the branch decoding layer features of SAR image by pixel-by-pixel addition.

[0013] According to the multi-branch, multi-task, multimodal remote sensing image road extraction method of the present invention, the input of the channel-strip spatial attention (CA-SSA) is the decoding layer features of the optical image branch and the SAR image branch, and the output is the road surface segmentation result; furthermore, features are input at both the deep and shallow layers of the decoding layer, and the fusion of the two layers is achieved by using convolution + upsampling to dynamically optimize the parameters.

[0014] According to the multi-branch, multi-task, multimodal remote sensing image road extraction method of the present invention, the channel-strip spatial attention (CA-SSA) implementation process is further as follows:

[0015] (1) Channel overlay, which hard-fuses features of different modalities;

[0016] (2) Conv processing: perform preliminary refinement and interaction on data of different modalities, and change the number of feature channels;

[0017] (3) Channel attention: learn effective features and suppress ineffective features in the channel dimension;

[0018] (4) Strip spatial attention, capturing effective features in the spatial dimension along four directions (0°, 45°, 90°, 135°);

[0019] (5) Channel superposition to achieve attention fusion in strip-shaped spaces from different directions;

[0020] (6) Conv processing to obtain the final feature fusion result.

[0021] According to the multi-branch, multi-task, multimodal remote sensing image road extraction method of the present invention, the channel-strip spatial attention (CA-SSA) implementation process is further expressed by the following formula:

[0022] F CA =CA(Conv(Cat(F) Opt ,F SAR )))

[0023] F fus=Conv(Cat(SSA0(F CA SSA 45 (F CA SSA 90 (F CA SSA 135 (F CA )))

[0024] In the formula, CA represents the channel attention operation, Conv includes 1*1 convolution, normalization, and ReLU activation operations, Cat is channel stacking, and F... Opt F represents the characteristics of the optical image branch decoding layer. SAR SSA0 and SSA represent the branch decoding layer features of SAR images. 45 SSA 90 SSA 135 These represent the strip-shaped spatial attention processing in the directions of 0°, 45°, 90° and 135°, respectively.

[0025] According to the multi-branch, multi-task, multimodal remote sensing image road extraction method of the present invention, the implementation process of the strip spatial attention is as follows: First, the input features are processed by average pooling and max pooling respectively, and the processing results are superimposed by channels; second, strip convolutions in four directions of 0°, 45°, 90° and 135° are used to extract features in specific directions; finally, the extracted features are processed by the Sigmoid activation function to obtain the weight of each position in the spatial dimension.

[0026] According to the multi-branch, multi-task, multimodal remote sensing image road extraction method of the present invention, the network model further calculates the loss function using road surface segmentation labels and road edge labels. The formula for the overall loss function is as follows:

[0027] Loss = Loss seg +α×Loss edge

[0028] In the formula, Loss, Loss seg Loss edge These represent the total loss, road surface segmentation loss, and road edge loss, respectively, with α representing the scaling factor.

[0029] According to the multi-branch, multi-task, multimodal remote sensing image road extraction method of the present invention, the road surface segmentation loss and road edge loss are further defined using the sum of BCE loss and dice coefficient loss as the loss value, wherein the formulas for BCE loss and dice coefficient loss are as follows:

[0030]

[0031]

[0032] In the formula, P and Y represent the prediction result and label data, respectively, W and H are the width and height of the image, and the prediction and label of the image position (i,j) are p and p, respectively. ij and y ij .

[0033] A multi-branch, multi-task, multimodal remote sensing image road extraction system is provided to implement the multi-branch, multi-task, multimodal remote sensing image road extraction method described above. The system includes a feature extraction module, an edge support module, and a feature fusion module, wherein:

[0034] The feature extraction module is used to construct an encoder-decoder network with the same network structure but independent parameters, to extract features from optical images and SAR images respectively, and to supervise training using road surface segmentation labels.

[0035] The edge support module is used to introduce the coding layer features of SAR images for road edge detection and input its intermediate features into the decoding layer features of SAR images.

[0036] The feature fusion module is used to fully fuse shallow and deep features of optical and SAR images using channel-strip spatial attention (CA-SSA) to predict the final road extraction result.

[0037] Compared with the prior art, the present invention has the following advantages:

[0038] 1. This invention combines the difficulties of ground feature interpretation and data foundation to construct a multi-branch, multi-task network architecture. It trains the road extraction network collaboratively using optical images, SAR images, fusion features, and road surface segmentation labels, and strengthens the road boundary cutting capability through road edge labels, thus forming a stable and efficient road extraction method.

[0039] 2. Considering the different imaging mechanisms of optical and SAR images, data-level fusion cannot fully leverage their respective advantages. Therefore, mainstream research focuses on feature-level fusion, especially on how to form an effective fusion module. This invention uses the multimodal features of optical and SAR images as data foundation, combined with road shape patterns, to construct Channel Attention-Strip Spatial Attention (CA-SSA), thereby fully mining the advantageous information of both modalities and improving the accuracy of road extraction results.

[0040] 3. Through experiments, using the Dongying dataset, it was verified that the method of the present invention outperforms the comparative methods in quantitative evaluation indicators, has significant advantages in key and difficult areas such as road intersections and low-grade roads, and achieves the best road extraction effect when optical images are affected by clouds and fog. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is an example diagram illustrating the limitations of optical imaging;

[0043] Figure 2 This is a structural diagram of the multi-branch, multi-task, multimodal remote sensing image road extraction system according to an embodiment of the present invention;

[0044] Figure 3 This is a network model structure diagram of the multi-branch, multi-task, multimodal remote sensing image road extraction method according to an embodiment of the present invention;

[0045] Figure 4 This is a structural diagram of the CA-SSA according to an embodiment of the present invention;

[0046] Figure 5 This is a diagram of the strip-shaped spatial attention structure according to an embodiment of the present invention;

[0047] Figure 6 These are example images and label data from the experiment;

[0048] Figure 7 These are example images showing the road surface segmentation results using different methods;

[0049] Figure 8 This is a statistical table of the degree of IoU decrease for each method under cloud and fog conditions (unit: %). Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] Optical and SAR imagery possess rich complementary attributes, and effective fusion strategies can provide a solid information foundation for ground feature interpretation. Roads, as strip-shaped ground features, often present challenges to interpretation due to their topological structure, distribution patterns, and application scenarios. Therefore, this embodiment proposes a multi-branch, multi-task, multimodal remote sensing imagery road extraction system, such as... Figure 2 As shown (“E” and “D” correspond to the encoding layer and decoding layer, respectively), this system includes a Feature Extraction Module (FEM), an Edge Support Module (ESM), and a Feature Fusion Module (FFM), wherein:

[0052] The feature extraction module is used to construct an encoder-decoder network with the same network structure but independent parameters to extract features from optical images and SAR images respectively. Its supervision data is road surface segmentation labels.

[0053] The edge support module is used to introduce the coding layer features of SAR images for road edge detection and input its intermediate features into the decoding layer features of SAR images. Its supervision data is road edge labels.

[0054] The feature fusion module is used to fully fuse shallow and deep features of optical and SAR images using channel-strip spatial attention (CA-SSA) to predict the final road extraction result. Its supervision data is also the road surface segmentation label.

[0055] Corresponding to the aforementioned multi-branch, multi-task, multimodal remote sensing image road extraction system, this embodiment also proposes a multi-branch, multi-task, multimodal remote sensing image road extraction method, comprising the following steps:

[0056] Step S101: Construct an encoding and decoding network with the same network structure but independent parameters, extract features from optical images and SAR images respectively, and supervise training using road surface segmentation labels.

[0057] Step S102: Introduce the coding layer features of SAR imagery for road edge detection, and input its intermediate features into the decoding layer features of SAR imagery to optimize the cutting effect between road and background.

[0058] Step S103: Channel Attention-Strip Spatial Attention (CA-SSA) is used to fully fuse the shallow and deep features of optical and SAR images to predict the final road extraction result.

[0059] The network model of the method in this embodiment is presented in the form of multiple branches and modules. The branches include an optical image branch, a SAR image branch, and a fusion branch. The structure of the network model is as follows: Figure 3 As shown. "Transposed Convolution Unit" includes convolution, transposed convolution, and convolution (each followed by normalization and ReLU activation); "Convolution Unit 1" includes two convolutions, the first followed by ReLU activation, and the second by sigmoid activation; "Convolution Unit 2" includes convolution and ReLU activation; "Convolution Unit 3" includes convolution, normalization, and ReLU activation; "Addition 1" indicates pixel-wise accumulation; "Opt" and "SAR" represent the optical image branch and SAR image branch, respectively; "E" and "D" correspond to encoded features and decoded features, respectively; "Addition 2" indicates skip connections, also using pixel-wise accumulation; "i" indicates the object of the skip connection, such as E1-Opt connecting D1-Opt, etc. Figure 3 (The identifier for the skip connection is omitted).

[0060] The optical image branch and the SAR image branch correspond to FEM. The SAR image branch, except for having ESM, has the same network structure as the optical image branch. However, considering the differences in imaging methods between the two types of images, this example adopts a non-shared parameter strategy. Specifically, the inputs to the two branches are optical images and SAR images, respectively, and the outputs are both road surface segmentation results. Both branches use ResNet34 for feature extraction in their coding layers, and the decoding layers use a combination of transposed convolutions and convolutions for upsampling. Skip connections are used to add the coding layer features and the corresponding decoding layer features pixel-by-pixel, avoiding excessive information loss.

[0061] The Channel-Strip Spatial Attention (CA-SSA) algorithm takes as input features the decoding layers of both the optical and SAR image branches and outputs road surface segmentation results. To ensure that the input features contain both semantic and detail information, features are input at both deep and shallow levels of the decoding layer. The fusion of these two levels is achieved through dynamic parameter optimization using convolution and upsampling.

[0062] Considering that current ground feature interpretation often results in confusion at boundary lines, ESM (Enhanced Supervisory Management) is introduced for strengthened supervision. Simultaneously, taking into account the limited penetration capability of optical images and their susceptibility to edge information loss due to occlusion by other ground features, this support module is introduced into the SAR image branch. The input data for this module is the coding layer features of the SAR image branch. The features at four levels are independently processed through convolution and upsampling to achieve channel and size uniformity, and then edge features are obtained through channel stacking. These edge features are further processed through convolution and other methods to output road edges, and are also incorporated into the decoding layer features of the SAR image branch through pixel-by-pixel addition.

[0063] Optical and SAR images have significantly different influencing factors, and the two types of images have certain complementary advantages. However, directly superimposing the two types of data through channels or adding attributes often leads to the stacking of redundant features and the loss of advantageous features. Therefore, this embodiment introduces an attention mechanism from both channel and spatial dimensions, and constructs a strip-shaped convolutional space in the spatial attention mechanism based on road shape patterns, ultimately forming CA-SSA. See [link to specific structure] for details. Figure 4 .

[0064] From a formulaic perspective, the CA-SSA process involves sequentially going through equations (1) and (2):

[0065] F CA =CA(Conv(Cat(F) Opt ,F SAR ))) (1)

[0066] F fus =Conv(Cat(SSA0(F CA SSA 45 (F CA SSA 90 (F CA SSA 135 (F CA (2)

[0067] In the formula, CA represents the channel attention operation, Conv includes 1*1 convolution, normalization, and ReLU activation operations, Cat is channel stacking, and F... Opt F represents the characteristics of the optical image branch decoding layer. SAR SSA0 and SSA represent the branch decoding layer features of SAR images. 45 SSA 90 SSA 135 These represent the strip-shaped spatial attention processing in the directions of 0°, 45°, 90° and 135°, respectively.

[0068] Therefore, the CA-SSA implementation process is as follows:

[0069] (1) Channel overlay, which hard-fuse the features of different modes.

[0070] (2)Conv( Figure 4 The ConvBNReLu process in the model performs preliminary refinement and interaction on data from different modalities and changes the number of feature channels.

[0071] (3) Channel attention: learn effective features and suppress ineffective features in the channel dimension.

[0072] (4) Strip spatial attention, capturing effective features in the spatial dimension along four directions (0°, 45°, 90°, 135°).

[0073] (5) Channel superposition to perform attention fusion in strip-shaped spaces in different directions.

[0074] (6) Conv processing to obtain the final feature fusion result.

[0075] The core structure of CA-SSA is a strip-shaped spatial attention that conforms to the shape of a road. The structure of this attention is as follows: Figure 5 As shown, firstly, the input features are processed by average pooling and max pooling respectively, and the processing results are stacked by channels; secondly, strip convolutions in four directions (0°, 45°, 90°, and 135°) are used to extract features in specific directions; finally, the extracted features are activated by the Sigmoid activation function to obtain the weights at each position in the spatial dimension.

[0076] The deep learning network in this embodiment involves two tasks, using road surface segmentation labels and road edge labels to calculate the loss function, and the overall loss function is shown in equation (3):

[0077] Loss = Loss seg +α×Loss edge (3)

[0078] In the formula, Loss, Loss seg Loss edgeLet represent the total loss, road surface segmentation loss, and road edge loss, respectively. The road surface segmentation loss is composed of the optical image branch, SAR image branch, and fusion branch in equal proportions. α is set to 1 / 3 here as a coefficient for the auxiliary task. Considering that both tasks have binary labels and there is an imbalance between positive and negative samples, the sum of the BCE loss and the die coefficient loss is used as the loss value for each task. The BCE loss treats each pixel equally. When there are few positive samples, the network will focus on negative samples, which weakens the ability to recognize positive samples. The die coefficient loss focuses on information mining of positive samples (foreground region), which can better solve the problem of imbalance between positive and negative samples. However, the training loss is prone to instability. Therefore, combining the two losses can achieve better results. The calculation formulas for the BCE loss and the die coefficient loss are shown in equations (4) and (5).

[0079]

[0080]

[0081] In the formula, P and Y represent the prediction result and label data, respectively, W and H are the width and height of the image, and the prediction and label of the image position (i,j) are p and p, respectively. ij and y ij .

[0082] The superiority of the method of the present invention will be verified through experiments below.

[0083] ① Experimental data

[0084] The Dongying dataset includes optical imagery, SAR imagery, and label data for land cover classification. To verify the road extraction capability of the proposed method, this paper preprocesses the label data by classifying the road categories as the foreground and other land cover categories as the background. The experimental images used in this paper are 256*256 pixels in size, and the spatial resolution of the preprocessed images is 1m. The optical images are GF-2, containing three RGB channels, and the SAR images are GF-3 VV polarization data. Regarding the size of the training and testing data, the dataset is randomly divided in a 7:1 ratio, resulting in 6852 training images and 979 testing images. Furthermore, considering that the imaging quality of optical images is greatly affected by weather conditions, and that the advantage of multimodal data interpretation is that it can avoid the limitation of insufficient information from a single data source, this paper uses the "cloud" rendering function in Photoshop to simulate optical images acquired under cloud and fog conditions, and conducts targeted analysis in subsequent experiments. Example images and label data of the experimental data are shown below. Figure 6 As shown.

[0085] ②Experimental details

[0086] The experimental environment consisted of two NVIDIA Tesla V100 graphics cards with a total of 64GB of video memory. Adam was chosen as the optimizer for network training, with an initial learning rate of 2e-4. The learning rate was reduced by a factor of 5 whenever the loss value exceeded the current optimal loss value three consecutive times. The training data block size was 32, and the number of epochs was 100. Simultaneously, to augment the samples, 50% of the training data was randomly subjected to vertical, horizontal, and diagonal flipping and radial transformations.

[0087] ③ Evaluation indicators

[0088] To ensure the comprehensiveness and consistency of the quantitative evaluation, this paper selects a total of 7 evaluation indicators from two levels to compare the accuracy of the road extraction results. (1) Pixel-level evaluation indicators: accuracy (P), recall (R), F1 score, overall precision (OA) and intersection-union ratio (IoU); (2) Connectivity evaluation indicators: completeness (Com) and error rate (Eor).

[0089] ④ Experimental Results

[0090] 4.1 Method Comparison Experiment

[0091] This section of the experiments primarily uses a selected dataset to compare the proposed method with existing classic and state-of-the-art methods. A total of 11 methods are compared, including 7 single-modal methods suitable for input data of only one type, such as UNet (2015), ASPP-UNet (2019), DeepLabv3+ (2018), D-LinkNet (2018), SGCN (2022), DT-Net (2023), and MANet (2021); and 4 multi-modal methods supporting road extraction under two modal data conditions, including MCANet (2022), DDHRNet (2022), JoiTriNet_e (2023), and JoiTriNet_d (2023).

[0092] The quantization accuracy statistics of the proposed method and the comparison methods are shown in Table 1. Considering that the input of the single-mode method only contains one type of data, in order to comprehensively compare the extraction effects of each method, this paper independently uses optical images and SAR images to conduct road extraction experiments. "Single-mode method-Opt" indicates that the input data is optical images, and "Single-mode method-SAR" indicates that the input data is SAR images.

[0093] The following conclusions can be drawn by comparing the accuracy indicators: (1) For single-modal conditions, if the quality of optical images is normal, the extraction effect is higher than that of SAR images in both pixel-level and connectivity evaluation indicators. In particular, the recall and completeness are significantly better, indicating that optical images can provide more support for road extraction in road extraction operations. This is also consistent with the visual judgment rules of human eyes. (2) Among the single-modal methods, DT-Net and MANet have higher accuracy, especially MANet's comprehensive accuracy indicator is second only to the method in this paper. (3) Among the multimodal methods, except for DDHRNet, the road extraction accuracy of other comparative methods is among the best, proving that multimodal methods are more stable and efficient. (4) The Com of the five pixel-level evaluation indicators and connectivity evaluation indicators of the method in this paper is the best. Only the connectivity evaluation indicator Eor is slightly worse than DDHRNet (DDHRNet's Com is much lower than that of the method in this paper), proving that the completeness and accuracy of the road extraction results of the method in this paper are in a leading position.

[0094] Table 1 Comparison of accuracy of road extraction results using different methods

[0095] unit:%

[0096]

[0097]

[0098] Note: Bold text is preferred. underline Second best, italics is the worst.

[0099] In addition to the quantitative accuracy index comparison analysis mentioned above, to more vividly and intuitively compare the road extraction effects of various methods, road surface segmentation results from a selection of test images were compared and analyzed. The five selected images came from different scenes and basically covered the difficult areas of road extraction, thus possessing a certain degree of representativeness. Furthermore, this section of the analysis no longer covers all comparison methods, but instead selects the top six methods based on the comprehensive accuracy evaluation index—"IoU".

[0100] Specific details are as follows: Figure 7As shown, Image 1 contains a large area of ​​water with relatively simple land cover, but the overall tone is dark. Due to limited image coverage, the water shoreline and roads form almost parallel strips. MANet and MCANet suffer from severe omissions in extraction, resulting in obvious "broken paths." JoiTriNet_e, JoiTriNet_d, and the proposed method also have some omissions, but the road topology is complete. DT-Net's extraction results have the highest completeness, but like JoiTriNet_e and JoiTriNet_d, it has mis-extraction problems at road boundaries. In Image 2, the road is vertically and vertically cut, and only a small part is shown in the image, equivalent to a low-level road. Due to the high spatial resolution, some traffic signs can be clearly seen in the optical image. In addition, the shadows formed by the linear fences above the road are basically consistent with the road attributes in the SAR image. From a visual perspective, all six methods successfully eliminated the interference of shadows. DT-Net, MANet, MCANet, and the proposed method have varying degrees of omissions, with MANet and MCANet showing the most significant omissions. Image 3 has relatively severe interference issues. While JoiTriNet_e and JoiTriNet_d achieved relatively complete road extraction, they also incorrectly introduced some other features. Image 3 consists of two intersecting roads. The latter five methods accurately and completely extracted the intersection information. DT-Net had a problem of missing extraction, causing the two connected roads to become disconnected. JoiTriNet_e and JoiTriNet_d had a relatively obvious problem of mis-extraction at the north-south road boundary. Image 4 is a scene of farmland, with relatively uniform features and tones. The road tones are close to the background. All six methods had varying degrees of missing extraction issues, but the extraction results of the proposed method were still relatively complete in terms of connectivity. The other five comparison methods failed to accurately restore the topology of the road. Image 5 is a typical problem of extracting main roads and auxiliary roads. MANet, MCANet, and the proposed method had relatively ideal extraction results. DT-Net, JoiTriNet_e, and JoiTriNet_d had varying degrees of missing extraction issues, especially DT-Net, which missed extracting one complete auxiliary road (low-level road).

[0101] 4.2 Ablation Experiment

[0102] This section of the experiments primarily verifies the effectiveness of the multi-branch, multi-task network architecture constructed in this paper, as well as the designed CA-SSA module. Specific scenarios include the following four types, marked with a "√" to indicate the presence of the corresponding units: "No multi-branch" indicates that only single-modal data is used for road extraction, without multi-modal branch fusion processing. Considering that the extraction effect of optical imagery is comprehensively superior to SAR imagery, optical imagery was chosen for the ablation experiment; "No CA-SSA" indicates that in the multi-branch state, the features of optical and SAR imagery are directly added pixel-by-pixel, without using an attention mechanism for quality trade-offs; "No edge detection task" means there is no ESM. The specific accuracy statistics for the four scenarios are shown in Table 2.

[0103] The comparison between scenarios 1 and 2 in Table 2 shows that the multi-branch network architecture can comprehensively improve various accuracy indicators, proving that this network structure is suitable for the joint interpretation of optical and SAR images and can effectively integrate the advantages of two heterogeneous remote sensing data. The comparison between scenarios 2 and 3 shows that the introduction of the "CA-SSA" module has a positive effect on road extraction, especially in terms of recall and completeness, proving that this module can more fully explore the features of linear features. The comparison between scenarios 3 and 4 shows that the introduction of the edge detection task comprehensively improves various accuracy indicators, proving that the enhanced supervision of road boundary information is conducive to improving road extraction results.

[0104] Table 2. Statistics on the Accuracy of Ablation Experiments

[0105] unit:%

[0106]

[0107] Furthermore, to further demonstrate the effectiveness of the multi-branch structure, the accuracy of the extraction results of the optical image branch and the SAR image branch in this paper is statistically analyzed. Considering the influence of factors such as natural weather and sensor conditions, modality loss is a common problem in practical applications. Therefore, this section conducts a simulation experiment of modality loss. The training model and parameters remain consistent with the proposed method, but in the testing phase, completely black images are used to replace optical and SAR images, respectively, to represent the loss of relevant modal data. The accuracy of the extraction results of the SAR image branch and the optical image branch is then statistically analyzed, and the specific results are shown in Table 3. It can be seen that even with modality loss, the other modality branch can still maintain its road extraction capability. This is because the network structure in this paper does not directly link the two single modal branches during forward propagation; therefore, the loss of one modality image does not affect the road prediction of the other modality branch. Furthermore, comparing Table 2 reveals that the single-modal branch obtained through fusion processing has higher extraction accuracy, with the SAR image branch even outperforming the road extraction from the optical image alone. Comparing Table 1 shows that, in terms of IoU, the optical image branch extraction accuracy of our proposed method is higher than all the compared single-modal methods, while the SAR image branch is slightly lower than DT-Net, further demonstrating the effectiveness and advancement of our proposed method.

[0108] Table 3. Statistics on branch accuracy under different conditions

[0109] unit:%

[0110]

[0111] ⑤ Discussion

[0112] Optical images rely on the reflection of visible light electromagnetic waves for imaging. The image quality is greatly affected by weather conditions. Therefore, we conducted an experimental comparison using image data under cloud and fog conditions. The specific results are shown in Table 4. By comparing Table 1, we can draw the following conclusions: (1) When affected by clouds and fog, the extraction accuracy of all methods showed a downward trend. Among them, in terms of the overall evaluation index (IoU), D-LinkNet, SGCN, DT-Net and MANet were even lower than the road extraction results of SAR images; (2) Except for Eor, the accuracy index of the proposed method is better than the comparison methods, which proves the stability of the proposed method; (3) Combining Figure 8 As can be seen, the IoU decrease is more significant with the single-modal method (orange for single-modal methods and light green for multimodal methods), indicating that the multimodal method can reduce the requirements for image quality to a greater extent.

[0113] Table 4 Comparison of accuracy of road extraction results under cloud and fog conditions

[0114] unit:%

[0115]

[0116]

[0117] In summary, the continuous advancement of remote sensing technology has created a full-space, multi-dimensional remote sensing data environment, presenting diverse development trends and providing a solid foundation for multimodal information mining. Roads, as important transportation facilities, have shape patterns closely related to their functional attributes, which are also key to their accurate interpretation. Furthermore, the acquisition conditions for optical images are relatively stringent, and SAR images possess scattering information that optical images lack. Therefore, road extraction by combining optical and SAR images can not only compensate for image quality issues but also provide richer supporting information at the feature level.

[0118] Unless otherwise specifically stated, the relative steps, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the invention.

[0119] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0120] The units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations are not considered to be beyond the scope of this invention.

[0121] Those skilled in the art will understand that all or part of the steps in the above methods can be implemented by a program instructing related hardware, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk. Optionally, all or part of the steps in the above embodiments can also be implemented using one or more integrated circuits. Accordingly, each module / unit in the above embodiments can be implemented in hardware or as a software functional module. This invention is not limited to any particular combination of hardware and software.

[0122] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for road extraction from multimodal remote sensing images with multiple branches and tasks, characterized in that, Include: We constructed encoder-decoder networks with identical structures but independent parameters to extract features from optical and SAR images, and supervised training using road surface segmentation labels. The coding layer features of SAR images are introduced for road edge detection, and their intermediate features are input into the decoding layer features of SAR images. Channel-strip spatial attention CA-SSA is used to fully fuse shallow and deep features from optical and SAR images to predict the final road extraction result. The input to CA-SSA is the decoding layer features of the optical and SAR image branches, and the output is the road surface segmentation result. Further, features are input at both the deep and shallow levels of the decoding layer, and the fusion of these two levels is achieved through dynamic parameter optimization using convolution and upsampling. The implementation process of CA-SSA is as follows: (1) Channel overlay, which performs hard fusion of features from different modalities; (2) The processing involves preliminary refinement and interaction of data from different modalities, and adjustment of the number of feature channels. (3) Channel attention: learn effective features and suppress ineffective features in the channel dimension; (4) Strip spatial attention, capturing effective features in the spatial dimension along four directions: 0°, 45°, 90°, and 135°; (5) Channel superposition to achieve strip-shaped spatial attention fusion in different directions; (6) The process involves processing the features to obtain the final fusion result. The channel-strip spatial attention CA-SSA implementation process is expressed by the following formula: , In the formula, This indicates a channel attention operation. Includes 1x1 convolution, normalization, and ReLU activation operations. It is channel overlay. This represents the characteristics of the optical image branch decoding layer. This represents the branch decoding layer features of SAR images. , , , These represent the strip-shaped spatial attention processing in the directions of 0°, 45°, 90° and 135°, respectively.

2. The multi-branch, multi-task, multimodal remote sensing image road extraction method according to claim 1, characterized in that, Both the optical image branch and the SAR image branch use ResNet34 for feature extraction in the coding layer. The decoding layer uses a combination of transposed convolution and convolution for upsampling, and uses skip connections to add the coding layer features and the corresponding decoding layer features pixel by pixel.

3. The multi-branch, multi-task, multimodal remote sensing image road extraction method according to claim 1, characterized in that, The method of introducing the coding layer features of SAR imagery for road edge detection and inputting its intermediate features into the decoding layer features of SAR imagery includes: obtaining four levels of features in the SAR imagery branch coding layer by relying on independent convolution and upsampling, and then obtaining edge features by superimposing these four levels of features in the form of channels. The edge features are then processed by two more convolution processes to output the road edge, and are also fused with the SAR imagery branch decoding layer features by adding them pixel by pixel.

4. The multi-branch, multi-task, multimodal remote sensing image road extraction method according to claim 1, characterized in that, The implementation process of the strip spatial attention is as follows: First, the input features are processed by average pooling and max pooling respectively, and the processing results are stacked by channels; second, strip convolutions in four directions of 0°, 45°, 90° and 135° are used to extract features in specific directions; finally, the extracted features are processed by the Sigmoid activation function to obtain the weight of each position in the spatial dimension.

5. The multi-branch, multi-task, multimodal remote sensing image road extraction method according to claim 1, characterized in that, The network model uses road surface segmentation labels and road edge labels to calculate the loss function. The formula for the overall loss function is as follows: In the formula, , , These represent total loss, road surface segmentation loss, and road edge loss, respectively. This represents the proportionality coefficient.

6. The multi-branch, multi-task, multimodal remote sensing image road extraction method according to claim 5, characterized in that, The road surface segmentation loss and road edge loss are calculated using the sum of the BCE loss and the dice coefficient loss as the loss value, where the formulas for the BCE loss and the dice coefficient loss are as follows: , In the formula, and These represent the prediction results and label data, respectively. and It refers to the width and height of the image. Location prediction and labeling are respectively and .

7. A multi-branch, multi-task, multimodal remote sensing image road extraction system, characterized in that, For implementing the multi-branch, multi-task, multimodal remote sensing image road extraction method as described in any one of claims 1 to 6, the system comprises a feature extraction module, an edge support module, and a feature fusion module, wherein: The feature extraction module is used to construct an encoder-decoder network with the same network structure but independent parameters, to extract features from optical images and SAR images respectively, and to supervise training using road surface segmentation labels. The edge support module is used to introduce the coding layer features of SAR images for road edge detection and input its intermediate features into the decoding layer features of SAR images. The feature fusion module is used to fully fuse shallow and deep features of optical and SAR images using channel-strip spatial attention (CA-SSA) to predict the final road extraction result.

Images