Remote sensing image road segmentation method combining bidirectional multi-level road feature dynamic fusion and double-context dynamic extraction

Abstract

The invention provides a remote sensing image road segmentation method combining bidirectional multi-level road feature dynamic fusion and double-context dynamic extraction, and belongs to the field of road segmentation. The problems of shielding and difficulty in distinguishing road levels existing in the current road segmentation network are solved; comprising the following steps of data set production, model construction, model training and result prediction. The invention designs a bidirectional multi-level road feature dynamic fusion and double-context dynamic extraction module. The bidirectional multi-level road feature dynamic fusion module effectively compensates the problem of spatial information and edge detail loss caused by continuous down-sampling through bidirectional interactive transmission of multi-level feature information, and enhances the understanding of the model for complex scenes and multi-level road structures. The double-context dynamic extraction module realizes dynamic fusion of long-distance and local multi-scale context information, is helpful for suppressing background interference, shielding effect and confusion of similar road structures, and ensures the integrity and continuity of extracted roads.

Description

Technical Field

[0001] The invention provides a remote sensing image road segmentation method combining bidirectional multi-level road feature dynamic fusion and dual-context dynamic extraction, belonging to the technical field of remote sensing image road segmentation. Background Art

[0002] The road extraction task from remote sensing images is a typical application scenario of semantic segmentation tasks, and plays a core role in many application scenarios such as urban planning, map updating, vehicle navigation, autonomous driving, military reconnaissance, and disaster warning.

[0003] Due to the development of algorithms and computing power, remote sensing image road extraction methods based on deep learning have become mainstream. Most deep learning methods have improved the ability to restore occluded roads in remote sensing images to varying degrees, but it is difficult to take into account the ability to distinguish different road levels. The reason is that when most studies improve jump connections, features at different stages use fixed receptive fields, which lack flexibility; in addition, when extracting multi-scale contexts, the traditional attention mechanism is directly added to the convolutional network, but the effect is poor due to the different mechanisms of the two. Taking the above problems and reasons into account, the present invention proposes a remote sensing image road extraction algorithm based on bidirectional multi-level road feature dynamic fusion and dual-context dynamic extraction strategy, which improves the road extraction capability while increasing the ability to distinguish different road levels. Summary of the invention

[0004] In order to solve the problem that the current road segmentation network is easily disturbed by background, occlusion and road-like information and difficult to distinguish roads of different levels, the present invention proposes a remote sensing image road segmentation method combining bidirectional multi-level road feature dynamic fusion and dual context dynamic extraction.

[0005] The technical solution adopted by the present invention is: a remote sensing image road segmentation method combining bidirectional multi-level road feature dynamic fusion and dual context dynamic extraction, comprising the following steps:

[0006] Step 1: Dataset preparation: randomly divide the training set and the test set from the high-resolution remote sensing image dataset; the remote sensing image dataset is acquired through remote sensing satellites, including original images and manually labeled image data; the remote sensing images of the training set are preprocessed and data enhanced to form an enhanced remote sensing image dataset;

[0007] Step 2: Construction of remote sensing image road segmentation network model: including the following steps:

[0008] Step 2.1: Construction of feature encoding module;

[0009] Step 2.2: Construction of a bidirectional multi-level road feature dynamic fusion module: The bidirectional multi-level road feature dynamic fusion module is used to implement bidirectional step-by-step dynamic fusion of multi-level features by designing a dynamic fusion module, and the dynamic fusion module includes a strip-shaped large core spatial attention with a dynamic receptive field and an effective channel attention for spatial and channel dimension selection;

[0010] This module implements bidirectional and step-by-step dynamic fusion of multi-level features, alleviating the problem of edge detail information loss caused by continuous downsampling, reducing the semantic gap between the encoder and decoder, and improving the model's ability to understand complex scenes and road features at different levels.

[0011] Step 2.3: Construction of dual-context dynamic extraction module: The dual-context dynamic extraction module designs a strip-hole input-dependent convolution, and based on this, designs a dilated spatial pyramid pooling based on the strip-hole input-dependent convolution and a dual-axis attention mechanism based on the strip-hole input-dependent convolution, which are used for local multi-scale context dynamic extraction and global context dynamic extraction respectively, and further feature screening and fusion are performed in combination with effective channel attention; effectively distinguishing road and background information, making the extracted road structure more complete and continuous;

[0012] Step 2.4: Construction of the feature decoding module; this module aims to convert the abstract feature map output by the dual-context dynamic extraction module back to the same spatial resolution as the original image;

[0013] The feature encoding module and the feature decoding module realize bidirectional and step-by-step dynamic fusion of features through a bidirectional and multi-level road feature dynamic fusion module; the fused features are extracted by a dual-context dynamic extraction module for global and local multi-scale context features; the extraction results are used for decoding by a feature decoding module;

[0014] Step 3: Model training: input the preprocessed training set data into the remote sensing image road segmentation network; use the ResNet-34 model weights pre-trained on the ImageNet-1K dataset to initialize the feature encoding module, and randomly initialize the remaining network parameters; train the remote sensing image road segmentation network until the model converges; after the training is completed, save the trained remote sensing image road segmentation network model parameters;

[0015] Step 4: Result prediction: Input the preprocessed remote sensing road image into the trained remote sensing image road segmentation network to obtain accurate segmentation results of the remote sensing image data.

[0016] The feature encoding module and the feature decoding module are each composed of five stages. The feature encoding module includes a downsampling module, encoder 1, encoder 2, encoder 3 and encoder 4 in sequence; the feature decoding module includes decoder 1, decoder 2, decoder 3, decoder 4 and a segmentation head in sequence.

[0017] The downsampling module is a 7×7 convolution operation with a step size of 2 and a 3×3 maximum pooling with a step size of 2; the four encoders are respectively stacked by 3, 4, 6, and 3 residual blocks, and finally output feature maps of 4 stages as the input of the multi-level road feature dynamic fusion module for bidirectional dynamic fusion; each residual block is composed of two 3×3 convolution operations and residual connections.

[0018] Each of the four decoders includes a 1×1 convolution that reduces the channel to 1 / 4, a 3×3 transposed convolution with a stride of 2, and a 1×1 convolution that doubles the channel. All three convolutions include batch normalization and activation function ReLU; finally, the decoder output is converted into a road extraction result through a segmentation head, which consists of a 3×3 transposed convolution with a stride of 2, a 3×3 convolution, and a 2×2 convolution.

[0019] The bidirectional multi-level road feature dynamic fusion module implements bidirectional step-by-step dynamic fusion of the multi-level features output by the encoder through three down-sampling dynamic fusion modules and three up-sampling dynamic fusion modules.

[0020] The strip-like large core spatial attention includes two parts: strip-like large core selection and spatial selection; the strip-like large core selection enables the model to adaptively find the receptive field that is most suitable for the current stage; the spatial selection selects the feature map obtained after the strip-like large core selection in the spatial dimension;

[0021] The structure of the strip-shaped macronucleus selection is as follows:

[0022] By combining two depth-wise separable convolutions and four groups of strip-shaped depth-wise separable convolutions with different receptive fields, and using one 1×1 convolution for weighted fusion; during training, the dynamic fusion modules at different stages will adjust the weights of the corresponding 1×1 convolutions so that the model can adaptively find the receptive field that best suits the current stage.

[0023] The structure of the space selection is as follows:

[0024] The feature map obtained in the strip large kernel selection stage is subjected to maximum pooling, average pooling, and strip adaptive pooling that conforms to the road characteristics. Four different spatial attention maps are obtained through 7×7 convolution and Sigmod function activation. The final spatial attention map is obtained by adding them together, and spatial selection is performed on the feature map obtained after the strip large kernel selection.

[0025] The strip depth-wise separable convolution is composed of two strip-shaped hole depth-wise separable convolution operations with different dilation rates, one horizontal and one vertical, which can better adapt to the shape characteristics of the road while maintaining a larger receptive field, and also reduce the computational overhead of the network.

[0026] The dual-context dynamic extraction module is designed based on strip-hole input-dependent convolution, and is composed of a dilated spatial pyramid pooling based on strip-hole input-dependent convolution, a dual-axis attention mechanism based on strip-hole input-dependent convolution, and effective channel attention;

[0027] The implementation process of the strip hole input dependent convolution is as follows:

[0028] By generating multiple sets of static learnable parameters and performing strip adaptive pooling, the corresponding strip spatial attention maps are dynamically generated for multiple sets of static learnable parameters according to the current input. The static learnable parameters are multiplied and summed with the corresponding dynamically generated strip spatial attention maps to obtain the parameters of the strip hole input dependent convolution. The static learnable parameters remain unchanged after training, which is equivalent to a set of traditional static convolutions.

[0029] The structure of the dilated spatial pyramid pooling module based on strip-dilated input-dependent convolution is as follows:

[0030] It consists of 1 1×1 convolution, 1 global average pooling and 4 groups of strip hole input-dependent convolutions with expansion rates of 1, 3, 6, and 12 respectively. Each group of strip hole input-dependent convolutions contains 1 horizontal, 1 vertical and 1 full-size strip hole input-dependent convolutions. The convolution kernel size of each strip hole input-dependent convolution is 3, and it contains 4 groups of static learnable parameters.

[0031] The structure of the dual-axis attention mechanism based on strip hole input dependent convolution is as follows:

[0032] First, two groups of Q and K for horizontal and vertical directions and one group of shared V are generated by inputting feature maps, and the horizontal and vertical axial attention operations are completed respectively;

[0033] A 3×3 strip hole input-dependent convolution is used to perform a convolution operation on the shared V and add it to the horizontal and vertical attention maps; while retaining the input-dependent characteristics, a beneficial inductive bias is introduced into the dual-axis attention mechanism based on the strip hole input-dependent convolution, which supplements the local position information and promotes in-depth interaction and aggregation between adjacent information on the same axis;

[0034] Finally, the results of different axes are fused through 1×1 convolution to deepen the hierarchy and richness of feature representation.

[0035] The beneficial effects of the present invention compared with the prior art are as follows:

[0036] (1) The method of the present invention uses the residual block of the ResNet-34 network in the encoder part and initializes it with the weights pre-trained on the ImageNet-1K dataset. This strategy not only effectively utilizes the existing knowledge and significantly improves the performance of the encoder, but also lays a solid foundation for the subsequent remote sensing image road segmentation task. This method enables the model to have a certain local feature extraction capability at the beginning of training, thereby accelerating the convergence speed of the model and improving the training efficiency.

[0037] (2) The present invention designs a bidirectional multi-level road feature dynamic fusion module, which alleviates the problem of edge detail information loss caused by continuous downsampling by implementing bidirectional and step-by-step dynamic fusion of multi-level features, reduces the semantic gap between the encoder and the decoder, and improves the model's ability to understand complex scenes and road features of different levels.

[0038] (3) The present invention designs a dual-context dynamic extraction module, which effectively distinguishes road and background information by dynamically extracting and integrating global and local multi-scale context information, making the extracted road structure more complete and continuous.

[0039] (4) The present invention designs a strip-shaped large kernel spatial attention, which includes two parts: strip-shaped large kernel selection and spatial selection; the strip-shaped large kernel selection enables the model to adaptively find the receptive field that is most suitable for the current stage; the spatial selection selects the feature map obtained after the strip-shaped large kernel selection in the spatial dimension.

[0040] (5) The present invention designs a strip-shaped void input-dependent convolution, which generates multiple sets of static learnable parameters and performs strip-shaped adaptive pooling. It dynamically generates corresponding strip-shaped spatial attention maps for multiple sets of static learnable parameters according to the current input. The convolution parameters are obtained by using the static learnable parameters and the strip-shaped spatial attention maps. This enables the model to adaptively aggregate feature information at different scales and in different regions, thereby improving the accuracy and robustness of semantic segmentation.

[0041] (6) The present invention designs a spatial selection, including maximum pooling, average pooling, and strip adaptive pooling that conforms to road characteristics, which can perform spatial selection on the feature map that is more consistent with road characteristics.

[0042] (7) In the feature decoding module, the present invention upsamples the abstract feature map of the dual-context dynamic extraction module to the original image resolution through transposed convolution. Specifically, this process not only effectively restores the details and spatial resolution of the feature map, but also combines the output of the bidirectional multi-level road feature dynamic fusion module, significantly enhancing the retention of road spatial details and contextual information, and improving the model's understanding of multi-level road features. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The present invention will be further described below in conjunction with the accompanying drawings:

[0044] Figure 1 It is a flow chart of a remote sensing image road segmentation method that combines bidirectional multi-level road feature dynamic fusion and dual-context dynamic extraction, provided by the present invention.

[0045] Figure 2 It is a structural schematic diagram of a remote sensing image road segmentation network model that combines bidirectional multi-level road feature dynamic fusion and dual-context dynamic extraction provided by the present invention.

[0046] Figure 3 It is a structural schematic diagram of a dynamic fusion module in a remote sensing image road segmentation method that combines bidirectional multi-level road feature dynamic fusion and dual-context dynamic extraction provided by the present invention.

[0047] Figure 4 It is a structural schematic diagram of a strip-shaped large-core space selection mechanism in a remote sensing image road segmentation method that combines bidirectional multi-level road feature dynamic fusion and dual-context dynamic extraction provided by the present invention.

[0048] Figure 5 It is a schematic diagram of the structure of a dual dynamic context extraction module in a remote sensing image road segmentation method that combines bidirectional multi-level road feature dynamic fusion and dual context dynamic extraction provided by the present invention.

[0049] Figure 6 This is an example of a remote sensing image of a DeepGlobe dataset used in a remote sensing image road segmentation method that combines bidirectional multi-level road feature dynamic fusion and dual-context dynamic extraction provided by the present invention.

[0050] Figure 7 This is an example of a remote sensing image with labels in the DeepGlobe dataset used in a remote sensing image road segmentation method that combines bidirectional multi-level road feature dynamic fusion and dual-context dynamic extraction provided by the present invention.

[0051] Figure 8 It is a schematic diagram of a road structure extracted from a remote sensing image road segmentation method that combines bidirectional multi-level road feature dynamic fusion with dual-context dynamic extraction provided by the present invention. DETAILED DESCRIPTION

[0052] like Figures 1 to 8 As shown, the present invention provides a remote sensing image road segmentation method combining bidirectional multi-level road feature dynamic fusion and dual context dynamic extraction, comprising the following steps:

[0053] Step 1: Dataset preparation: A high-resolution remote sensing image dataset with a size of 1024×1024 pixels is collected using remote sensing satellites, and it is subdivided into a training set and a test set at a ratio of 8:2; the dataset covers unprocessed original remote sensing images and data accurately annotated by professionals. Next, a series of preprocessing and data enhancement measures are implemented on the images in the training set and their corresponding labels, including but not limited to horizontal flipping, vertical flipping, horizontal and vertical joint flipping, rotation, deformation scaling, random noise injection, and chromaticity transformation. These enhancement methods effectively increase the diversity of image perspectives and enhance the robustness of the model to changes in road structures under different orientations and conditions, thereby improving the accuracy and reliability of the remote sensing image road segmentation network. The images after the above preprocessing and data enhancement are used to train the network model, thereby achieving efficient and accurate segmentation of road structures in remote sensing images, providing a solid foundation for subsequent remote sensing image analysis applications.

[0054] Step 2: Model construction: The remote sensing image road segmentation network model includes a feature encoding module (Encoder), a bidirectional multi-level road feature dynamic fusion module (Bidirecrtional Multi-level Road Feafure Dynamic Fusion, BMDF), a dual context dynamic extraction module (Dual Context Dynamic Extraction, DCDE) and a feature decoding module (Dncoder). Among them, the feature encoding module and the feature decoding module are composed of five stages respectively. The feature encoding module includes a downsampling module, encoder 1, encoder 2, encoder 3 and encoder 4 in sequence; the feature decoding module includes decoder 1, decoder 2, decoder 3, decoder 4 and segmentation head in sequence; the feature encoding module and the feature decoding module realize bidirectional step-by-step dynamic fusion of features through a bidirectional multi-level road feature dynamic fusion module; the fused features are extracted by the dual context dynamic extraction module for global and local multi-scale context features; the extraction module results are used for decoding in the feature decoding module.

[0055] The specific steps for building the remote sensing image road segmentation network model are as follows:

[0056] Step 2.1: Construction of feature encoding module: The ResNet-34 network pre-trained on the ImageNet-1K dataset is used to construct the downsampling module, encoder 1, encoder 2, encoder 3 and encoder 4 for feature extraction, and the feature maps of four stages are obtained, which will be used as the input of the bidirectional multi-level road feature dynamic fusion module for bidirectional dynamic fusion.

[0057] The structure of the feature encoding module is as follows:

[0058] The feature encoding module includes a downsampling module, encoder 1, encoder 2, encoder 3 and encoder 4 in sequence; the downsampling module is a 7×7 convolution operation with a stride of 2 and a 3×3 maximum pooling with a stride of 2; the four encoders are respectively stacked by 3, 4, 6, and 3 residual blocks (ResBlock); each residual block is composed of two 3×3 convolution operations and residual connections. The four encoders generate four feature maps respectively, each of which reflects the feature representation of the image at different levels of abstraction. Specifically, the feature map generated by encoder 1 is a shallow feature map, which mainly captures the basic and local detail information in the image; while the feature map generated by encoder 4 is a deep feature map, which contains more abstract and global information, which is particularly critical for performing fine semantic segmentation tasks.

[0059] Step 2.2: Construction of a bidirectional multi-level road feature dynamic fusion module: Skip connections are intended to compensate for information loss in the decoding stage, but simple skip connections produce semantic differences between the codec, which highlights the importance of understanding and narrowing the semantic gap between the encoder and decoder. Especially in road detection, continuous downsampling leads to the loss of key edge details, and a single scale feature is not sufficient to capture the rich semantic information required to distinguish the background and roads at all levels. To solve the above problems, and taking into account that features at different encoding stages have different roles in the expression of decoded features, the present invention implements bidirectional step-by-step dynamic fusion of the multi-level features output by the feature encoding module, replacing the commonly used skip connections, thereby alleviating the problem of edge detail information loss caused by continuous downsampling, reducing the semantic gap between the encoder and decoder, and improving the model's ability to understand complex scenes and road features at different levels.

[0060] The structure of the bidirectional multi-level road feature dynamic fusion module is as follows:

[0061] The multi-level features output by the encoder are bidirectionally and step by step dynamically fused through three downsampling dynamic fusion modules and three upsampling dynamic fusion modules. In the downward fusion stage, the feature fusion operation of two adjacent layers is realized from shallow to deep along the encoder hierarchy. When the three downsampling dynamic fusion operations are completed, the feature information from all stages of the encoder has been integrated into the output feature map, thereby realizing the supplement of the deep abstract semantic information with the shallow detail information, so that the deep layer of the network can capture richer context information; in the upward fusion stage, the feature fusion path is reversed, and the feature fusion operation of two adjacent layers is realized from deep to shallow along the encoder hierarchy. The upsampling dynamic fusion effectively feeds back the learned deep rich semantics to the shallow feature representation, closing the bidirectional circulation loop of feature information, realizing the deep penetration and cross-layer sharing of feature information, and significantly enhancing the expressiveness and accuracy of the model in semantic segmentation tasks. Finally, the four outputs of the bidirectional multi-level road feature dynamic fusion module will be used as part of the input of the corresponding decoder. The bidirectional multi-level road feature dynamic fusion module utilizes the hierarchical dependency between encoders to strictly limit the feature fusion of each stage to the feature maps in a specific direction and directly adjacent to each other. The strip-shaped large-core spatial attention in the dynamic fusion module is used to realize the strip-shaped adaptive receptive field. The effective channel attention in the dynamic fusion module is used to suppress interference information such as background. While maintaining computational efficiency, it ensures the fluidity and complementarity of multi-level and multi-scale semantic information within the network. Ultimately, it realizes the deep integration of features between different levels, alleviates the loss of road spatial information and edge detail information caused by continuous downsampling, plays a key role in identifying road error segmentation areas, and comprehensively improves the model's comprehensive understanding and expression capabilities of the input image.

[0062] The structure of the dynamic fusion module is as follows:

[0063] The dynamic fusion module is mainly used in the bidirectional multi-level road feature dynamic fusion module to realize the step-by-step fusion of the encoder output. The dynamic fusion module is different in different stages. In the upward fusion stage, the downsampling dynamic fusion module is used. It first downsamples the output of the previous stage through a convolution operation of size 7×7 and step size 2, and finally makes its size and number of channels the same as the encoder output feature map of the corresponding level; in the downward fusion stage, the upsampling dynamic fusion module is used. It first downsamples the output of the previous stage through a transposed convolution operation of size 3×3 and step size 2, and finally makes its size and number of channels the same as the encoder output feature map of the corresponding level. The subsequent structure of the two dynamic fusion modules is the same, that is, after the weighted summation of the two, the strip-shaped large kernel spatial attention (SLSA) with dynamic receptive field is used to dynamically adjust the receptive field size to capture spatial information, and then the effective channel attention (ECA) is used to suppress interference information such as background. Feature fusion is effectively completed through the selection of spatial and channel dimensions.

[0064] The structure of the strip-shaped large core spatial attention is as follows:

[0065] Strip Large Kencel Spatial Attention includes two parts: Strip Large Kencel Selection and Spatial Selection. Strip Large Kencel Selection enables the model to adaptively find the most suitable receptive field for the current stage. Spatial Selection selects the feature map obtained after Strip Large Kencel Selection in the spatial dimension. The Strip Large Kencel Spatial Attention mechanism is used in the dynamic fusion module in each stage of the bidirectional multi-level road feature dynamic fusion module. Through Strip Large Kencel Selection and Spatial Selection, the long-distance context extraction and spatial selection of the adaptive receptive field are realized, so that the model can flexibly adjust when facing different scale feature inputs, so as to adaptively focus on multi-scale road targets in different stages, and accurately capture the road contours of different levels, which significantly enhances the model's ability to understand complex scenes and multi-level roads, and provides strong support for more accurate road extraction.

[0066] The structure of the strip-shaped macronucleus selection is as follows:

[0067] The strip-shaped large kernel selection is achieved by combining two depthwise separable convolutions (DWConv) with different receptive fields and four groups of strip-shaped depthwise separable convolutions (SDWConv), and using one 1×1 convolution for weighted fusion. In the bidirectional multi-level road feature dynamic fusion module, the weights of the corresponding 1×1 convolutions will be adjusted during training for the dynamic fusion modules at different feature fusion stages, so that the model can adaptively find the receptive field that best suits the current stage.

[0068] The structure of strip depthwise separable convolution is as follows:

[0069] like Figure 4 As shown in the figure, the strip depthwise separable convolution operation consists of two strip hole depthwise separable convolution operations, one horizontal and one vertical (i.e., strip depthwise separable convolutions with different dilation rates), which can better adapt to the shape characteristics of the road while maintaining a larger receptive field, and also reduce the computational overhead of the network;

[0070] The structure of a space selection is as follows:

[0071] The spatial selection performs maximum pooling, average pooling, and strip adaptive pooling that conforms to the road characteristics on the feature map obtained in the strip large kernel selection stage, and obtains four different spatial attention maps through 7×7 convolution and Sigmod function activation. The final spatial attention map is added together, and spatial selection is performed on the feature map obtained after the strip large kernel selection.

[0072] Step 2.3: Construction of dual-context dynamic extraction module: In semantic segmentation tasks, the synergistic integration of attention mechanism and convolution operation has been widely adopted as a strategy to enhance feature expression ability. However, standard convolution operation cannot dynamically adjust its own weight according to the input like the attention mechanism, and it is difficult to effectively utilize the high-level features of attention output. Although the inductive bias is introduced, it has limited improvement on the model's feature fusion and representation capabilities. In order to solve the above problems, the present invention combines the advantages of dynamic convolution, strip convolution and dilated convolution to design a strip atrous input-dependent convolution (Srtip Atrous Input-dependent Convolution, SIDConv), and based on the strip atrous input-dependent convolution, designs a dilated spatial pyramid pooling module based on strip atrous input-dependent convolution (SID-ASPP) and a dual-axis attention mechanism based on strip atrous input-dependent convolution (SID-DA), which are respectively used to dynamically extract and integrate local multi-scale and global contextual information, effectively distinguish road and background information, and make the extracted road structure more complete and continuous; then, combined with effective channel attention to filter road interference information, suppress background and other interference information, and finally construct a dual-context dynamic extraction module.

[0073] The structure of the dual-context dynamic extraction module is as follows:

[0074] The dual-context dynamic extraction module is mainly divided into three parts: local multi-scale context dynamic extraction (Local Dynamic Context Information Extraction), global dynamic context information extraction (Global Dynamic Context Information Extraction) and channel attention (Channel Aftention). Its input comes from the deep features that have completed the full-stage feature fusion in the bidirectional multi-level road feature dynamic fusion module. First, a 7×7 depth-separable convolution is used to encode its position, and its result is divided into two independent branches along the channel dimension to extract local multi-scale context and global context respectively. The local multi-scale context dynamic extraction is realized through the dilated spatial pyramid pooling module based on the strip-based dilated input-dependent convolution, which can obtain 14 groups of local features of different scales. After splicing, they are selected and fused through effective channel attention and 1×1 convolution to enhance the model's sensitivity to local details. The global context dynamic extraction is mainly completed based on the dual-axis attention mechanism of the strip-based dilated input-dependent convolution. First, the input is sequentially passed through the dual-axis attention mechanism based on the strip-based dilated input-dependent convolution with four axial window sizes of 2, 4, 8, and 32 to obtain 4 groups of global features. After splicing these global features with the original input, they are further screened and fused through effective channel attention and 3×3 convolution. Finally, the channel attention operation is used to aggregate the local multi-scale context with the global context, and the interference of background noise and irrelevant information is effectively suppressed through the joint processing of 1×1 convolution and effective channel attention. The dual-context dynamic extraction module is used to extract and fuse the global and local multi-scale contexts of the deep features that have completed the full-stage feature fusion in the bidirectional multi-level road feature dynamic fusion module, which efficiently realizes the deep integration and optimization of local multi-scale features and global features, further improving the segmentation performance of the model. At the same time, the final output of the dual-context dynamic extraction module will be used as the input of the decoder to promote more refined reconstruction of prediction results and ensure the accuracy and consistency of the segmentation results.

[0075] The principle of strip hole input dependent convolution is as follows:

[0076] The convolution operation generates multiple sets of static learnable parameters, and through strip adaptive pooling, dynamically generates corresponding strip space attention maps for multiple sets of static learnable parameters according to the current input, and multiplies and sums the static learnable parameters with the corresponding dynamically generated strip space attention maps to obtain the parameters of the strip hole input dependent convolution. The static learnable parameters remain unchanged after the training is completed, which is equivalent to a set of traditional static convolutions. The strip hole input dependent convolution combines the advantages of dynamic convolution, strip convolution and hole convolution, so that the model can adaptively aggregate feature information at different scales and in different regions, and can be better combined with the attention mechanism, thereby improving the accuracy and robustness of semantic segmentation.

[0077] The structure of the dilated spatial pyramid pooling module based on strip-dilated input-dependent convolution is as follows:

[0078] The dilated spatial pyramid pooling module based on strip-hole input-dependent convolution is mainly used for dynamic extraction of local multi-scale context. It consists of 1 1×1 convolution, 1 global average pooling and 4 groups of strip-hole input-dependent convolutions with expansion rates of 1, 3, 6, and 12, respectively. Each group of strip-hole input-dependent convolutions contains 1 horizontal, 1 vertical and 1 full-size strip-hole input-dependent convolutions. The convolution kernel size of each strip-hole input-dependent convolution is 3, and it contains 4 groups of static learnable parameters. The dilated spatial pyramid pooling module based on strip-hole input-dependent convolution has excellent performance in capturing multi-scale context information, while avoiding the problem of poor effect caused by the combination of static convolution and attention, and better adapting to the shape characteristics of the road and reducing computational overhead.

[0079] The structure of the dual-axis attention mechanism based on strip hole input dependent convolution is as follows:

[0080] The dual-axis attention mechanism based on strip hole input-dependent convolution is mainly used for global context dynamic extraction. First, two groups of Q, K for horizontal and vertical directions and one group of shared V are generated through the input feature map, and the horizontal and vertical axial attention operations are completed respectively. In particular, a 3×3 strip hole input-dependent convolution is used to perform a convolution operation on the shared V, and it is added to the horizontal and vertical attention maps. While retaining the input dependency characteristics, a beneficial inductive bias is introduced to the dual-axis attention mechanism based on strip hole input-dependent convolution, which supplements the local position information and promotes the in-depth interaction and aggregation between adjacent information on the same axis. Finally, the results of different axes are fused through 1×1 convolution, which deepens the hierarchy and richness of feature representation and can well balance the needs of global dependency modeling and computational efficiency.

[0081] Step 2.4: Construction of feature decoding module: The feature decoding module aims to convert the abstract feature map output by the dual context dynamic extraction module back to the same spatial resolution as the original image, so as to achieve accurate semantic segmentation of remote sensing images. In this step, starting from the global-local context extraction features aggregated by the dual context dynamic extraction module, up-sampled by transposed convolution operation, and combined with the output from the corresponding stage of the bidirectional multi-level road feature dynamic fusion module to supplement the detailed boundary information and context information of the road, thereby improving the accuracy and refinement of semantic segmentation.

[0082] The structure of the feature decoding module is as follows:

[0083] Each of the four decoders includes a 1×1 convolution that reduces the channel to 1 / 4, a 3×3 transposed convolution with a stride of 2, and a 1×1 convolution that doubles the channel. All three convolutions include batch normalization and activation function ReLU; finally, the decoder output is converted into a road extraction result through a segmentation head, which consists of a 3×3 transposed convolution with a stride of 2, a 3×3 convolution, and a 2×2 convolution.

[0084] Step 3: Model training: input the preprocessed training set data into the remote sensing image road segmentation network; use the ResNet-34 model weights pre-trained on the ImageNet-1K data set to initialize the feature encoding module, and randomly initialize the remaining network parameters; train the remote sensing image road segmentation network until the model converges; after the training is completed, save the trained remote sensing image road segmentation network model parameters.

[0085] Step 4: Result prediction: Input the preprocessed remote sensing road image into the trained remote sensing image road segmentation network to obtain accurate segmentation results of the remote sensing image data.

[0086] The training program is built on the PyTorch framework. The optimizer uses the Adam algorithm, and the loss function combines binary cross entropy and Dice coefficient to alleviate the problem of category imbalance and enhance the model's ability to segment small-scale targets. The initial learning rate of the model is set to 2e-4, the batch size is set to 4, and the maximum training batch is set to 500. The learning rate adjustment mechanism uses the Plateau Learning Rate Scheduler, and sets the attenuation factor to 0.5 and the tolerance period to 5, so as to dynamically adjust the learning rate according to the learning process and avoid local optimal solutions. In addition, an early stopping condition is set in the experiment, that is, when the loss value does not decrease significantly during 8 consecutive epochs, it is determined that the model has reached a convergence state and the training process is terminated in advance. This is to curb the tendency of overfitting in a timely manner and reasonably control the computing cost.

[0087] Among them, the calculation formula of the binary cross entropy loss function is:

[0088]

[0089] In the above formula: y is the true pixel label value, y' is the predicted label pixel value, and N is the number of label categories;

[0090] The calculation formula of the Dice coefficient loss function is:

[0091]

[0092] In the above formula: X is the generated prediction graph, Y is the real label, |X∩Y| is the intersection between the label and the prediction, |X| is the number of elements in the label, and |Y| is the number of elements in the prediction;

[0093] The final semantic segmentation loss function is the sum of the weighted coefficients of the cross entropy loss function and the Dice coefficient loss function, and the calculation formula is:

[0094] Ls=dice+L cr0ss

[0095] The present invention will be further described below based on specific embodiments.

[0096] The experimental dataset uses the DeepGlobe road extraction dataset, which comes from the remote sensing road extraction challenge held by CVPR2018. The dataset contains 6226 sets of remote sensing images with a size of 1024×1024 pixels and road labels, with an image resolution of 0.5m / pixel. The images are collected by Digital Globe satellites. These images not only cover a wide range of geographical areas such as cities and villages in many countries, but also include various types of roads and street networks. In addition, in order to maintain the authenticity and practicality of the data, the author deliberately did not mark the paths in the farmland. During the recognition, the author did not want the model to make incorrect extractions, making the dataset more challenging and practical, and also providing a broader space for exploration.

[0097] When evaluating the road segmentation performance of the model, the present invention uses five evaluation indicators commonly used in semantic segmentation to evaluate the model, and the evaluation results are shown in Table 1. The five evaluation indicators are accuracy, precision, recall, F1 score and intersection-over-union. Precision means that the extracted road objects are more accurate, and recall means that the extracted road objects are more complete, but the two are contradictory, and the F1 score is the harmonic mean of recall and precision. The accuracy measures the proportion of correct predictions made by the model, but it cannot solve the problem of category imbalance very well, while the intersection-over-union reflects the degree of overlap between the model prediction results and the true labels. These indicators can comprehensively and objectively reflect the performance of the model in the road segmentation task. By Figure 6 (original remote sensing road image), Figure 7 (manually annotated remote sensing road segmentation reference image) and Figure 8 The comparison and analysis of (the predicted road segmentation result image obtained after applying the method of the present invention) shows that the method of the present invention has demonstrated excellent performance in road extraction from remote sensing images. Specifically, Figure 8 The predicted segmentation results shown are consistent with Figure 7 This fact strongly confirms the accuracy and reliability of the method in the road segmentation task.

[0098] Accuracy Accuracy Recall F1 score Intersection and Union 0.9882 0.8541 0.8498 0.8520 0.7421

[0099] Table 1 Specific indicators on the DeepGlobe road extraction dataset.

[0100] The present invention discloses an innovative remote sensing image road segmentation method, which integrates the bidirectional multi-level road feature dynamic interaction and dual-context dynamic extraction mechanism. The present invention aims to achieve highly accurate segmentation of road areas in high-resolution remote sensing images through advanced deep learning technology, combined with specially designed network architecture and modules. First, a detailed manually annotated label image is created based on the high-resolution remote sensing image of RGB three-channel, which is used as a standard reference for model training. Then, the complete image data set is divided into a training set and a test set in proportion, and a series of preprocessing operations such as denoising and feature enhancement are performed on the images in the training set to optimize the image quality and lay a good foundation for model training. In the model building stage, the present invention creatively designs a remote sensing image road extraction algorithm based on bidirectional multi-level road feature dynamic fusion and dual-context dynamic extraction strategy. Among them, the bidirectional multi-level road feature dynamic fusion module effectively compensates for the loss of spatial information and edge details caused by continuous downsampling through the bidirectional interactive transmission of multi-level feature information, and enhances the model's understanding of complex scenes and multi-level road structures. On the other hand, the dual-context dynamic extraction module realizes the dynamic fusion of long-distance and local multi-scale context information, which helps to suppress background interference, occlusion effects and confusion of similar road structures, and ensures that the extracted road structure has integrity and continuity. During the model training process, the present invention inputs the preprocessed training image and its corresponding label image into the road segmentation model, and uses the optimization algorithm to continuously adjust the network parameters until the model can accurately identify and segment the road area. When the model reaches a convergence state, the optimal parameter settings are saved for subsequent applications. Finally, the present invention can automatically generate a predicted label image, that is, a segmented road image, by inputting a new remote sensing image into the trained model. Compared with the prior art, the uniqueness of the present invention lies in its bidirectional multi-level road feature dynamic fusion and the application of dual-context dynamic extraction mechanism, which not only improves the connectivity and accuracy of road segmentation, but also enhances the model's ability to distinguish roads of different levels and restore edge details.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A remote sensing image road segmentation method combining bidirectional multi-level road feature dynamic fusion and dual context dynamic extraction, characterized by: The following steps are involved: Step 1: Dataset preparation: randomly divide the training set and test set from the high-resolution remote sensing image dataset; The remote sensing image data set is acquired through a remote sensing satellite, and includes original images and manually labeled image data; the remote sensing images of the training set are preprocessed and data enhanced to form an enhanced remote sensing image data set; Step 2: Construction of remote sensing image road segmentation network model: including the following steps: Step 2.1: Construction of feature encoding module; Step 2.2: Construction of a bidirectional multi-level road feature dynamic fusion module: The bidirectional multi-level road feature dynamic fusion module is used to implement bidirectional step-by-step dynamic fusion of multi-level features by designing a dynamic fusion module, and the dynamic fusion module includes a strip-shaped large core spatial attention with a dynamic receptive field and an effective channel attention for spatial and channel dimension selection; Step 2.3: Construction of dual-context dynamic extraction module: The dual-context dynamic extraction module designs a strip-hole input-dependent convolution, and based on this, designs a dilated spatial pyramid pooling based on the strip-hole input-dependent convolution and a dual-axis attention mechanism based on the strip-hole input-dependent convolution, which are used for local multi-scale context dynamic extraction and global context dynamic extraction respectively, and further feature screening and fusion are performed in combination with effective channel attention; Step 2.4: Construction of feature decoding module; The feature encoding module and the feature decoding module realize bidirectional and step-by-step dynamic fusion of features through a bidirectional and multi-level road feature dynamic fusion module; the fused features are extracted by a dual-context dynamic extraction module for global and local multi-scale context features; the extraction results are used for decoding by a feature decoding module; Step 3: Model training: input the preprocessed training set data into the remote sensing image road segmentation network; use the ResNet-34 model weights pre-trained on the ImageNet-1K dataset to initialize the feature encoding module, and randomly initialize the remaining network parameters; train the remote sensing image road segmentation network until the model converges; after the training is completed, save the trained remote sensing image road segmentation network model parameters; Step 4: Result prediction: Input the preprocessed remote sensing road image into the trained remote sensing image road segmentation network to obtain accurate segmentation results of the remote sensing image data.

2. The remote sensing image road segmentation method combining bidirectional multi-level road feature dynamic fusion and dual context dynamic extraction according to claim 1, characterized in that: The feature encoding module and the feature decoding module are each composed of five stages. The feature encoding module includes a downsampling module, encoder 1, encoder 2, encoder 3 and encoder 4 in sequence; the feature decoding module includes decoder 1, decoder 2, decoder 3, decoder 4 and a segmentation head in sequence.

3. The remote sensing image road segmentation method combining bidirectional multi-level road feature dynamic fusion and dual context dynamic extraction according to claim 2, characterized in that: The downsampling module is a 7×7 convolution operation with a step size of 2 and a 3×3 maximum pooling with a step size of 2; the four encoders are respectively stacked by 3, 4, 6, and 3 residual blocks; each residual block is composed of two 3×3 convolution operations and residual connections.

4. The remote sensing image road segmentation method combining bidirectional multi-level road feature dynamic fusion and dual context dynamic extraction according to claim 2, characterized in that: Each of the four decoders includes a 1×1 convolution that reduces the channel to 1 / 4, a 3×3 transposed convolution with a stride of 2, and a 1×1 convolution that doubles the channel. All three convolutions include batch normalization and activation function ReLU; finally, the decoder output is converted into a road extraction result through a segmentation head, which consists of a 3×3 transposed convolution with a stride of 2, a 3×3 convolution, and a 2×2 convolution.

5. The remote sensing image road segmentation method combining bidirectional multi-level road feature dynamic fusion and dual context dynamic extraction according to claim 1, characterized in that: The bidirectional multi-level road feature dynamic fusion module implements bidirectional step-by-step dynamic fusion of the multi-level features output by the encoder through three down-sampling dynamic fusion modules and three up-sampling dynamic fusion modules.

6. The remote sensing image road segmentation method combining bidirectional multi-level road feature dynamic fusion and dual context dynamic extraction according to claim 1, characterized in that: The strip-like large core spatial attention includes two parts: strip-like large core selection and spatial selection; the strip-like large core selection enables the model to adaptively find the receptive field that is most suitable for the current stage; the spatial selection selects the feature map obtained after the strip-like large core selection in the spatial dimension; The structure of the strip-shaped macronucleus selection is as follows: By combining two depth-wise separable convolutions with different receptive fields and four groups of strip-shaped depth-wise separable convolutions, and using one 1×1 convolution for weighted fusion; The structure of the space selection is as follows: The feature map obtained in the strip large kernel selection stage is subjected to maximum pooling, average pooling, and strip adaptive pooling that conforms to the road characteristics. Four different spatial attention maps are obtained through 7×7 convolution and Sigmod function activation. The final spatial attention map is obtained by adding them together, and spatial selection is performed on the feature map obtained after the strip large kernel selection.

7. The remote sensing image road segmentation method combining bidirectional multi-level road feature dynamic fusion and dual context dynamic extraction according to claim 6, characterized in that: The strip depth-wise separable convolution is composed of two strip-shaped hole depth-wise separable convolution operations with different dilation rates, one horizontal and one vertical.

8. The remote sensing image road segmentation method combining bidirectional multi-level road feature dynamic fusion and dual context dynamic extraction according to claim 1, characterized in that: The dual-context dynamic extraction module is designed based on strip-hole input-dependent convolution, and is composed of a dilated spatial pyramid pooling based on strip-hole input-dependent convolution, a dual-axis attention mechanism based on strip-hole input-dependent convolution, and effective channel attention; The implementation process of the strip hole input dependent convolution is as follows: By generating multiple sets of static learnable parameters and performing strip adaptive pooling, corresponding strip spatial attention maps are dynamically generated for multiple sets of static learnable parameters according to the current input. The static learnable parameters are multiplied by the dynamically generated corresponding strip spatial attention maps and the summed up can obtain the parameters of the strip void input dependent convolution.

9. The remote sensing image road segmentation method combining bidirectional multi-level road feature dynamic fusion and dual context dynamic extraction according to claim 8, characterized in that: The structure of the dilated spatial pyramid pooling module based on strip-dilated input-dependent convolution is as follows: It consists of 1 1×1 convolution, 1 global average pooling and 4 groups of strip hole input-dependent convolutions with expansion rates of 1, 3, 6, and 12 respectively. Each group of strip hole input-dependent convolutions contains 1 horizontal, 1 vertical and 1 full-size strip hole input-dependent convolutions. The convolution kernel size of each strip hole input-dependent convolution is 3, and it contains 4 groups of static learnable parameters.

10. The remote sensing image road segmentation method combining bidirectional multi-level road feature dynamic fusion and dual context dynamic extraction according to claim 8, characterized in that: The structure of the dual-axis attention mechanism based on strip hole input dependent convolution is as follows: First, two groups of Q and K for horizontal and vertical directions and one group of shared V are generated by inputting feature maps, and the horizontal and vertical axial attention operations are completed respectively; A 3×3 strip-hole input-dependent convolution is used to perform a convolution operation on the shared V and add it to the horizontal and vertical attention maps; Finally, the results of different axes are fused through 1×1 convolution to deepen the hierarchy and richness of feature representation.

Images