A remote sensing image road segmentation method combining dynamic fusion of bidirectional multi-level road features and double-context dynamic extraction

Abstract

The application provides a kind of remote sensing image road segmentation method combining bidirectional multi-level road feature dynamic fusion and double context dynamic extraction, belongs to the field of road segmentation;Solve the problem of current road segmentation network exists and is difficult to distinguish road level;Including the following steps: data set production, model construction, model training and result prediction;The application designs bidirectional multi-level road feature dynamic fusion and double context dynamic extraction module.Bidirectional multi-level road feature dynamic fusion module effectively compensates the problem of spatial information and edge detail loss caused by continuous downsampling through bidirectional interaction transmission of multi-level feature information, enhances the understanding of complex scenes and multi-level road structure by the model.Double context dynamic extraction module realizes the dynamic fusion of long-distance and local multi-scale context information, which helps to suppress background interference, occlusion effect and confusion of similar road structure, and ensures the integrity and continuity of the extracted road.

Description

TECHNICAL FIELD

[0001] The application provides a remote sensing image road segmentation method combining bidirectional multi-level road feature dynamic fusion and double-context dynamic extraction, and belongs to the technical field of remote sensing image road segmentation. BACKGROUND

[0002] The remote sensing image road extraction task is a typical semantic segmentation task application scenario, and plays a core role in many application scenarios such as city planning, map updating, vehicle navigation, automatic driving, military reconnaissance, and disaster warning.

[0003] Due to the development of algorithms and computing power, the current remote sensing image road extraction method based on deep learning has become the mainstream. Most deep learning methods improve the recovery ability of the occluded road in the remote sensing image to different degrees, but it is difficult to balance the resolution ability of different road levels. The reason is that most studies improve the skip connection, and the fixed receptive field of the features at different stages is adopted, which lacks flexibility. In addition, when extracting multi-scale context, the traditional attention mechanism is directly added to the convolutional network, and due to the difference between the two mechanisms, the effect is not good. Considering the above problems and reasons, the application proposes a remote sensing image road extraction algorithm based on bidirectional multi-level road feature dynamic fusion and double-context dynamic extraction strategy, which improves the road extraction ability and increases the resolution ability of different road levels. SUMMARY

[0004] In order to solve the problem that the current road segmentation network is easily disturbed by background, occlusion and road-like information and is difficult to distinguish different levels of roads, the application proposes a remote sensing image road segmentation method combining bidirectional multi-level road feature dynamic fusion and double-context dynamic extraction.

[0005] The technical scheme adopted by the application is as follows: a remote sensing image road segmentation method combining bidirectional multi-level road feature dynamic fusion and double-context dynamic extraction, comprising the following steps:

[0006] Step 1: Data set making: randomly dividing a training set and a test set from a high-resolution remote sensing image data set; the remote sensing image data set is obtained by a remote sensing satellite, including an original image and an image data marked by a human being; the remote sensing image of the training set is preprocessed and data enhanced to form an enhanced remote sensing image data set;

[0007] Step 2: Construction of a remote sensing image road segmentation network model: comprising the following steps:

[0008] Step 2.1: Construction of a feature encoding module;

[0009] Step 2.2: Construction of the bidirectional multi-level road feature dynamic fusion module: The bidirectional multi-level road feature dynamic fusion module is designed to implement bidirectional hierarchical dynamic fusion of multi-level features. The dynamic fusion module includes a strip-shaped large kernel spatial attention with a dynamic receptive field and an effective channel attention for spatial and channel dimension selection.

[0010] This module alleviates the loss of edge detail information caused by continuous downsampling, reduces the semantic gap between the encoder and the decoder, and improves the model's understanding of complex scenes and different levels of road features.

[0011] Step 2.3: Construction of the dual-context dynamic extraction module: The dual-context dynamic extraction module is designed with a strip-dilated input-dependent convolution. Based on this, a strip-dilated input-dependent convolution-based hollow spatial pyramid pooling and a dual-axis attention mechanism based on strip-dilated input-dependent convolution are designed for local multi-scale context dynamic extraction and global context dynamic extraction, respectively. Effective channel attention is used for further feature selection and fusion. It effectively distinguishes road and background information, making the extracted road structure more complete and continuous.

[0012] Step 2.4: Construction of the feature decoding module: This module aims to map the abstract features output by the dual-context dynamic extraction module back to the same spatial resolution as the original image.

[0013] The feature encoding module and the feature decoding module are connected through the bidirectional multi-level road feature dynamic fusion module to realize bidirectional hierarchical dynamic fusion of features. The fused features are extracted by the dual-context dynamic extraction module for global and local multi-scale context features. The extraction results are used for decoding by the feature decoding module.

[0014] Step three: Model training: input the preprocessed training set data into the remote sensing image road segmentation network; use the pre-trained ResNet-34 model weight on the ImageNet-1K dataset to initialize the feature encoding module, and randomly initialize the remaining network parameters; train the remote sensing image road segmentation network until the model converges; after training is completed, save the model parameters of the remote sensing image road segmentation network obtained by training;

[0015] Step four: result prediction: input the preprocessed remote sensing road image into the trained remote sensing image road segmentation network to obtain the accurate segmentation result of the remote sensing image data.

[0016] The feature encoding module and the feature decoding module are each composed of five stages, the feature encoding module sequentially includes a downsampling module, an encoder 1, an encoder 2, an encoder 3 and an encoder 4; the feature decoding module sequentially includes a decoder 1, a decoder 2, a decoder 3, a decoder 4 and a segmentation head.

[0017] The downsampling module is a 7x7 convolution operation with a step of 2 and a 3x3 maximum pooling with a step of 2; the four encoders are respectively stacked by 3, 4, 6 and 3 residual blocks, and finally output four-stage feature maps as the input of the multi-level road feature dynamic fusion module for bidirectional dynamic fusion; each residual block is composed of two 3x3 convolution operations and residual connection.

[0018] The four decoders each include a 1x1 convolution reducing the channel to 1 / 4, a 3x3 transpose convolution with a step of 2, and a 1x1 convolution doubling the channel, and each of the three convolutions includes batch normalization and an activation function ReLU; finally, the decoder output is converted into a road extraction result through a segmentation head, and the segmentation head is composed of a 3x3 transpose convolution with a step of 2, a 3x3 convolution and a 2x2 convolution.

[0019] The bidirectional multi-level road feature dynamic fusion module realizes bidirectional step-by-step dynamic fusion of multi-level features of the encoder output through three downsampling dynamic fusion modules and three upsampling dynamic fusion modules.

[0020] The strip large core spatial attention includes two parts of strip large core selection and spatial selection; the strip large core selection enables the model to adaptively find the most suitable receptive field for the current stage; the spatial selection selects the feature map obtained after the strip large core selection in the spatial dimension;

[0021] The structure of the strip large core selection is as follows:

[0022] The 2 depth separable convolutions with different receptive fields and the 4 groups of strip depth separable convolutions are combined, and a 1x1 convolution is used for weighted fusion; in the training, the dynamic fusion module of different stages adjusts the weight of the corresponding 1x1 convolution, so that the model can adaptively find the most suitable receptive field for the current stage.

[0023] The structure of the spatial selection is as follows:

[0024] The feature map obtained after the strip large core selection stage is subjected to maximum pooling, average pooling and strip adaptive pooling conforming to road features, and 4 different spatial attention maps are obtained through 7x7 convolution and Sigmod function activation, which are added to obtain the final spatial attention map, and the feature map obtained after the strip large core selection is subjected to spatial selection.

[0025] The strip-shaped depth separable convolution is composed of two strip-shaped hole depth separable convolution operations with different expansion rates in the horizontal direction and the vertical direction, can better adapt to the shape characteristics of the road while maintaining a larger receptive field, and reduces the calculation overhead of the network.

[0026] The double-context dynamic extraction module is designed based on a strip hole input dependent convolution, and is composed of a strip hole input dependent convolution based on a strip hole input dependent convolution, a double-axis attention mechanism based on a strip hole input dependent convolution, and effective channel attention;

[0027] The implementation process of the strip hole input dependent convolution is as follows:

[0028] A plurality of groups of static learnable parameters are generated, and a strip-shaped spatial attention map corresponding to the plurality of groups of static learnable parameters is dynamically generated according to the current input through strip-shaped adaptive pooling; the static learnable parameters are multiplied by the dynamically generated corresponding strip-shaped spatial attention map and summed to obtain the parameters of the strip hole input dependent convolution. The static learnable parameters remain unchanged after training, which is equivalent to a group of traditional static convolution.

[0029] The structure of the strip hole input dependent convolution based on a strip hole input dependent convolution based on a strip hole input dependent convolution is as follows:

[0030] The strip hole input dependent convolution based on a strip hole input dependent convolution based on a strip hole input dependent convolution is composed of 1 1x1 convolution, 1 global average pooling and 4 groups of strip hole input dependent convolution with expansion rates of 1, 3, 6 and 12, respectively, wherein each group of strip hole input dependent convolution comprises 1 horizontal, 1 vertical and 1 full-size strip hole input dependent convolution, and the convolution kernel size of each strip hole input dependent convolution is 3, comprising 4 groups of static learnable parameters.

[0031] The structure of the double-axis attention mechanism based on the strip hole input dependent convolution is as follows:

[0032] First, 2 groups of Q, K for horizontal and vertical directions and 1 group of shared V are generated from the input feature map, and horizontal and vertical axis attention operations are respectively completed;

[0033] A 3x3 strip hole input dependent convolution is used to perform convolution operation on the shared V, and the convolution operation is added to the horizontal and vertical attention maps; while retaining the input dependent characteristics, the double-axis attention mechanism based on the strip hole input dependent convolution is introduced to introduce beneficial induction bias, supplement local position information, and promote in-depth interaction and aggregation between adjacent information in the same axis;

[0034] Finally, 1x1 convolution is used to fuse the results of different axes, deepening the hierarchy and richness of feature representation.

[0035] The present application has the beneficial effects of the prior art:

[0036] (1) The method of the application adopts the residual block of ResNet-34 network in the encoder part, and initializes by using the pre-trained weight on the ImageNet-1K dataset. This strategy not only effectively utilizes the existing knowledge, but also significantly improves the performance of the encoder, laying a solid foundation for the subsequent remote sensing image road segmentation task. This method enables the model to have certain local feature extraction ability at the beginning of training, thereby accelerating the convergence speed of the model and improving the training efficiency.

[0037] (2) The application designs a bidirectional multi-level road feature dynamic fusion module. The module implements bidirectional and step-by-step dynamic fusion on multi-level features, alleviates the loss of edge detail information caused by continuous downsampling, reduces the semantic gap between the encoder and the decoder, and improves the understanding ability of the model for complex scenes and different levels of road features.

[0038] (3) The application designs a double-context dynamic extraction module, which dynamically extracts and integrates global and local multi-scale context information, effectively distinguishes road and background information, and makes the extracted road structure more complete and continuous.

[0039] (4) The application designs a strip large core spatial attention, which includes strip large core selection and spatial selection. The strip large core selection enables the model to adaptively find the most suitable receptive field for the current stage. The spatial selection selects the feature map obtained after strip large core selection in the spatial dimension.

[0040] (5) The application designs a strip hollow input dependent convolution. By generating multiple groups of static learnable parameters, and through strip adaptive pooling, the corresponding strip spatial attention map is dynamically generated for multiple groups of static learnable parameters according to the current input. Through static learnable parameters and strip spatial attention map, the convolution parameter is obtained, so that the model can adaptively aggregate feature information in different scales and different regions, thereby improving the accuracy and robustness of semantic segmentation.

[0041] (6) The application designs a spatial selection, including maximum pooling, average pooling, and strip adaptive pooling conforming to road features, which can perform spatial selection on the feature map more conforming to road features.

[0042] (7) The application maps the abstract features of the double-context dynamic extraction module to the original image resolution through transposed convolution in the feature decoding module part. Specifically, this process not only effectively restores the details and spatial resolution of the feature map, but also combines the output of the bidirectional multi-level road feature dynamic fusion module, significantly enhances the preservation of road spatial details and context information, and improves the understanding of the model for multi-level road features. BRIEF DESCRIPTION OF DRAWINGS

[0043] The application is further described below in conjunction with the accompanying drawings:

[0044] Figure 1 is a flowchart of a remote sensing image road segmentation method provided by the application, which combines dynamic fusion of bidirectional multi-level road features and dynamic extraction of double contexts.

[0045] Figure 2 is a structural diagram of a remote sensing image road segmentation network model provided by the application, which combines dynamic fusion of bidirectional multi-level road features and dynamic extraction of double contexts.

[0046] Figure 3 is a structural diagram of a dynamic fusion module in a remote sensing image road segmentation method provided by the application, which combines dynamic fusion of bidirectional multi-level road features and dynamic extraction of double contexts.

[0047] Figure 4 is a structural diagram of a strip large core space selection mechanism in a remote sensing image road segmentation method provided by the application, which combines dynamic fusion of bidirectional multi-level road features and dynamic extraction of double contexts.

[0048] Figure 5 is a structural diagram of a double dynamic context extraction module in a remote sensing image road segmentation method provided by the application, which combines dynamic fusion of bidirectional multi-level road features and dynamic extraction of double contexts.

[0049] Figure 6 is an example of a remote sensing image in a DeepGlobe dataset used in a remote sensing image road segmentation method provided by the application, which combines dynamic fusion of bidirectional multi-level road features and dynamic extraction of double contexts.

[0050] Figure 7 is an example of a remote sensing image with labels in a DeepGlobe dataset used in a remote sensing image road segmentation method provided by the application, which combines dynamic fusion of bidirectional multi-level road features and dynamic extraction of double contexts.

[0051] Figure 8 is a structural diagram of a road extracted in a remote sensing image road segmentation method provided by the application, which combines dynamic fusion of bidirectional multi-level road features and dynamic extraction of double contexts. DETAILED DESCRIPTION

[0052] As shown in Figures 1 to 8 , the application provides a remote sensing image road segmentation method combining dynamic fusion of bidirectional multi-level road features and dynamic extraction of double contexts, which includes the following steps:

[0053] Step one: dataset making: collect 1024x1024 pixel size high resolution remote sensing image dataset using remote sensing satellite, and subdivide it into training set and test set according to the ratio of 8:2; the dataset covers the original remote sensing image without processing and the data labeled by professionals. Next, a series of preprocessing and data enhancement measures are implemented on the images in the training set and their corresponding labels, including but not limited to image horizontal flip, vertical flip, horizontal and vertical joint flip, rotation, deformation scaling, random noise injection and chroma transformation, etc. These enhancement measures effectively increase the diversity of image perspectives, enhance the robustness of the model to changes in road structure in different directions and conditions, and thus improve the accuracy and reliability of the remote sensing image road segmentation network. The images after the above preprocessing and data enhancement are used to train the network model, so as to realize efficient and accurate segmentation of the road structure in the remote sensing image, and provide a solid foundation for subsequent remote sensing image analysis applications.

[0054] Step two: model building: the remote sensing image road segmentation network model includes a feature encoding module (Encoder), a bidirectional multi-level road feature dynamic fusion module (BMDF), a dual context dynamic extraction module (DCDE) and a feature decoding module (Dencoder). Among them, the feature encoding module and the feature decoding module are each composed of five stages, the feature encoding module includes down-sampling module, encoder 1, encoder 2, encoder 3 and encoder 4 in turn; the feature decoding module includes decoder 1, decoder 2, decoder 3, decoder 4 and segmentation head in turn; the feature encoding module and the feature decoding module realize the bidirectional step-by-step dynamic fusion of features through the bidirectional multi-level road feature dynamic fusion module; the fused features are extracted by the dual context dynamic extraction module for global and local multi-scale context feature extraction; the extraction module result is used for feature decoding module for decoding.

[0055] The specific steps of building the remote sensing image road segmentation network model are as follows:

[0056] Step 2.1: construction of feature encoding module: ResNet-34 network pre-trained on ImageNet-1K dataset is used to build down-sampling module, encoder 1, encoder 2, encoder 3 and encoder 4 for feature extraction, to obtain four-stage feature maps, which will be used as input for bidirectional dynamic fusion in the bidirectional multi-level road feature dynamic fusion module.

[0057] The structure of the feature encoding module is as follows:

[0058] The feature encoding module sequentially comprises a downsampling module, an encoder 1, an encoder 2, an encoder 3 and an encoder 4; the downsampling module is a 7x7 convolution operation with a step of 2 and a 3x3 max pooling with a step of 2; the four encoders are respectively stacked by 3, 4, 6 and 3 residual blocks (ResBlock); each residual block is composed of two 3x3 convolution operations and residual connection. The four encoders respectively generate four feature maps, and each feature map reflects the feature representation of the image at different abstraction levels. Specifically, the feature map generated by the encoder 1 is a shallow feature map, which mainly captures the basic and local detailed information in the image; while the feature map generated by the encoder 4 is a deep feature map, which contains more abstract and global information, and is particularly crucial for performing fine semantic segmentation tasks.

[0059] Step 2.2: Construction of bidirectional multi-level road feature dynamic fusion module: The skip connection aims to compensate for the information loss in the decoding stage, but the simple skip connection produces semantic differences between the encoder and the decoder, which highlights the importance of understanding and narrowing the semantic gap between the encoder and the decoder. Especially in road detection, continuous downsampling leads to the loss of key edge details, and single-scale features are insufficient to capture the rich semantic information needed to distinguish backgrounds and roads at different levels. To solve the above problems, while considering that different encoding stage features have different roles in decoding feature expression, the present application implements bidirectional step-by-step dynamic fusion on the multi-level features output by the feature encoding module, instead of the commonly used skip connection, which alleviates the problem of edge detail information loss caused by continuous downsampling, reduces the semantic gap between the encoder and the decoder, and improves the understanding ability of the model for complex scenes and different levels of road features.

[0060] The structure of the bidirectional multi-level road feature dynamic fusion module is as follows:

[0061] The bidirectional multi-level road feature dynamic fusion module is used for implementing the bidirectional multi-level road feature dynamic fusion. The bidirectional multi-level road feature dynamic fusion module is used for implementing the bidirectional multi-level road feature dynamic fusion. In the downward fusion stage, along the encoder level structure, the feature fusion operation of the adjacent two layers is implemented from shallow to deep. After three times of down-sampling dynamic fusion operation, the feature map is output, and the feature information from all stages of the encoder is integrated, so as to realize the supplement of the shallow layer detail information to the deep layer abstract semantic information, and the deep layer of the network can capture more rich context information. In the upward fusion stage, the feature fusion path is reversed, along the encoder level structure, the feature fusion operation of the adjacent two layers is implemented from deep to shallow. The learned deep rich semantics is effectively fed back to the shallow feature representation, the bidirectional circulation loop of the feature information is closed, the depth penetration and cross-layer sharing of the feature information are realized, and the expressiveness and accuracy of the model in the semantic segmentation task are significantly enhanced. Finally, the four outputs of the bidirectional multi-level road feature dynamic fusion module are used as part of the input of the corresponding decoder. The bidirectional multi-level road feature dynamic fusion module uses the level dependence relationship between the encoders, strictly limits the feature fusion of each stage to the directly adjacent feature maps in a specific direction, realizes the strip adaptive receptive field through the strip large kernel spatial attention in the dynamic fusion module, and suppresses the background and other interference information through the effective channel attention in the dynamic fusion module. While maintaining the calculation efficiency, the semantic information liquidity and complementarity of the network are ensured, and finally the deep integration of the features between different levels is realized. The loss of road space information and edge detail information caused by continuous down-sampling is relieved, which plays a key role in identifying the error segmentation area of the road, and comprehensively improves the comprehensive understanding and expression ability of the model to the input image.

[0062] The structure of the dynamic fusion module is as follows:

[0063] The dynamic fusion module is mainly used for implementing the step-by-step fusion of the encoder output in the bidirectional multi-level road feature dynamic fusion module. The dynamic fusion module is different according to the stage. In the upward fusion stage, the down-sampling dynamic fusion module is used. Firstly, the convolution operation with the size of 7*7 and the step of 2 is used to down-sample the output of the previous stage, so that the size and the number of channels are the same as the encoder output feature map of the corresponding level. In the downward fusion stage, the up-sampling dynamic fusion module is used. Firstly, the transposed convolution operation with the size of 3*3 and the step of 2 is used to down-sample the output of the previous stage, so that the size and the number of channels are the same as the encoder output feature map of the corresponding level. The subsequent structures of the two dynamic fusion modules are the same, that is, after the weighted sum of the two, the strip large kernel spatial attention (SLSA) with dynamic receptive field is used to dynamically adjust the receptive field size to capture spatial information, and the effective channel attention (ECA) is used to suppress the background and other interference information. Through the selection of spatial and channel dimensions, the feature fusion is effectively completed.

[0064] The structure of the strip large kernel spatial attention is as follows:

[0065] The strip large kernel spatial attention includes strip large kernel selection and spatial selection. The strip large kernel selection enables the model to adaptively find the receptive field most suitable for the current stage. The spatial selection selects the feature map obtained after the strip large kernel selection in the spatial dimension. The strip large kernel spatial attention mechanism is applied to the dynamic fusion module in each stage of the bidirectional multi-level road feature dynamic fusion module. Through the strip large kernel selection and the spatial selection, the long-distance context extraction of the adaptive receptive field and the spatial selection are realized, so that the model can flexibly adjust when facing different scale feature inputs, adaptively focus on multi-scale road targets in different stages, accurately capture road outlines of different levels, and significantly enhance the understanding ability of the model to complex scenes and multi-level roads, thereby providing strong support for more accurate road extraction.

[0066] The structure of the strip large kernel selection is as follows:

[0067] The strip large kernel selection is obtained by combining 2 depthwise separable convolutions (DWConv) with different receptive fields and 4 groups of strip depthwise separable convolutions (SDWConv), and then performing weighted fusion through a 1x1 convolution. In the bidirectional multi-level road feature dynamic fusion module, the weights of the corresponding 1x1 convolution are adjusted in the training of the dynamic fusion module in different feature fusion stages, so that the model can adaptively find the receptive field most suitable for the current stage.

[0068] The structure of the strip depthwise separable convolution is as follows:

[0069] As shown in Figure 4 , the strip depthwise separable convolution operation is composed of a horizontal and a vertical strip hole depthwise separable convolution (i.e. a strip depthwise separable convolution with different expansion rates), which can better adapt to the shape features of the road while maintaining a large receptive field, and also reduces the computational overhead of the network.

[0070] The structure of the spatial selection is as follows:

[0071] The spatial selection performs maximum pooling, average pooling, and strip adaptive pooling conforming to road features on the feature map obtained after the strip large kernel selection stage, and obtains 4 different spatial attention maps through a 7x7 convolution and a Sigmod function activation. The final spatial attention map is obtained by adding the 4 different spatial attention maps, and the spatial selection is performed on the feature map obtained after the strip large kernel selection.

[0072] Step 2.3: Construction of the dual-context dynamic extraction module: In the semantic segmentation task, attention mechanisms have been widely used in conjunction with convolution operations as a strategy to enhance feature expression capabilities. However, standard convolution operations cannot dynamically adjust their weights like attention mechanisms, making it difficult to effectively utilize the advanced features of attention outputs. Despite the introduction of bias, the model's feature fusion and representation capabilities have been limited. To address the above problems, the present invention combines the advantages of dynamic convolution, strip convolution, and atrous convolution to design a strip atrous input-dependent convolution (SIDConv) and based on the strip atrous input-dependent convolution to design a strip atrous input-dependent convolution-based atrous spatial pyramid pooling module (SID-ASPP) and a dual-axis attention mechanism based on strip atrous input-dependent convolution (SID-DA), which are used to dynamically extract and integrate local multi-scale and global context information, effectively distinguish road and background information, and make the extracted road structure more complete and continuous. Then, combined with effective channel attention filtering, the road interference information is filtered, and the background interference information is suppressed, and finally the dual-context dynamic extraction module is constructed.

[0073] The structure of the dual-context dynamic extraction module is as follows:

[0074] The dual-context dynamic extraction module is mainly divided into three parts: local multi-scale context dynamic extraction, global context dynamic extraction and channel attention. The input comes from the deep features that have completed the full-stage feature fusion in the bidirectional multi-level road feature dynamic fusion module. First, a 7x7 depth separable convolution is used for position coding, and the result is divided into two independent branches along the channel dimension, and local multi-scale context and global context extraction are extracted respectively. The local multi-scale context dynamic extraction is realized by the strip-dilated input-dependent convolution-based spatial pyramid pooling module, which can obtain 14 groups of local features of different scales. After splicing, effective channel attention and 1x1 convolution are used for selection and fusion to enhance the sensitivity of the model to local details; the global context dynamic extraction is mainly based on the strip-dilated input-dependent convolution-based dual-axis attention mechanism. First, the input is sequentially passed through four axis window sizes of 2, 4, 8, and 32 based on the strip-dilated input-dependent convolution-based dual-axis attention mechanism, obtaining four groups of global features. After splicing these global features with the original input, effective channel attention and 3x3 convolution are used for further feature selection and fusion; finally, the channel attention operation is performed, which aggregates the local multi-scale context and the global context, and is processed by the joint processing of 1x1 convolution and effective channel attention, effectively suppressing the interference of background noise and irrelevant information. The dual-context dynamic extraction module is used to extract and fuse the global and local multi-scale context of the deep features that have completed the full-stage feature fusion in the bidirectional multi-level road feature dynamic fusion module, efficiently realizing the deep integration and optimization of local multi-scale features and global features, and further improving the segmentation performance of the model. At the same time, the output of the dual-context dynamic extraction module will be used as the input of the decoder to promote the reconstruction of more detailed prediction results, ensuring the accuracy and continuity of the segmentation results.

[0075] The principle of strip-dilated input-dependent convolution is as follows:

[0076] The convolution operation generates multiple groups of static learnable parameters, and dynamically generates corresponding bar space attention maps for the multiple groups of static learnable parameters according to the current input through bar adaptive pooling, multiplies the static learnable parameters and the dynamically generated corresponding bar space attention maps, and sums up to obtain the parameters of the strip hollow input dependent convolution. Wherein the static learnable parameters remain unchanged after training, which is equivalent to a group of traditional static convolution. The strip hollow input dependent convolution combines the advantages of dynamic convolution, bar convolution and hollow convolution, so that the model can adaptively aggregate feature information in different scales and different regions, can better combine with the attention mechanism, and thus improve the accuracy and robustness of semantic segmentation.

[0077] The structure of the strip hollow input dependent convolution based on the hollow space pyramid pooling module is as follows:

[0078] The strip hollow input dependent convolution based on the hollow space pyramid pooling module is mainly used for local multi-scale context dynamic extraction, which is composed of 1 1x1 convolution, 1 global average pooling and 4 groups of strip hollow input dependent convolution with expansion rates of 1, 3, 6 and 12 respectively. Each group of strip hollow input dependent convolution includes 1 horizontal, 1 vertical and 1 full-size strip hollow input dependent convolution, and the convolution kernel size of each strip hollow input dependent convolution is 3, which includes 4 groups of static learnable parameters. The strip hollow input dependent convolution based on the hollow space pyramid pooling module has excellent performance in capturing multi-scale context information, while avoiding the problem of poor effect caused by the combination of static convolution and attention, and better adapting to the shape features of the road and reducing the computational overhead.

[0079] The structure of the strip hollow input dependent convolution based on the double-axis attention mechanism is as follows:

[0080] The strip hollow input dependent convolution based on the double-axis attention mechanism is mainly used for global context dynamic extraction. First, 2 groups of Q, K for horizontal and vertical directions and 1 group of shared V are generated from the input feature map, and horizontal and vertical axis attention operations are performed respectively. In particular, a 3x3 strip hollow input dependent convolution is used to perform convolution operation on the shared V, and the result is added to the horizontal and vertical attention maps, which introduces beneficial inductive bias for the strip hollow input dependent convolution based on the double-axis attention mechanism, supplements local position information, and promotes in-depth interaction and aggregation between adjacent information on the same axis. Finally, 1x1 convolution is used to fuse the results of different axes, which deepens the hierarchy and richness of feature representation, and can well balance the demand for global dependency modeling and computational efficiency.

[0081] Step 2.4: Construction of feature decoding module: The feature decoding module aims to map the abstract features output by the dual-context dynamic extraction module back to the same spatial resolution as the original image, thereby achieving accurate semantic segmentation of remote sensing images. In this step, starting with the global-local context extraction features aggregated by the dual-context dynamic extraction module, upsampling is performed through transpose convolution operations, and the output from the corresponding stage of the bidirectional multi-level road feature dynamic fusion module is combined to supplement the detailed boundary information and contextual information of the road, thereby improving the accuracy and fineness of semantic segmentation.

[0082] The structure of the feature decoding module is as follows:

[0083] Each of the 4 decoders includes a 1x1 convolution that reduces the number of channels by 1 / 4, a 3x3 transpose convolution with a step size of 2, and a 1x1 convolution that doubles the number of channels. After each of the three convolutions, batch normalization and the activation function ReLU are included. Finally, the decoder output is converted into a road extraction result through a segmentation head composed of a 3x3 transpose convolution with a step size of 2, a 3x3 convolution, and a 2x2 convolution.

[0084] Step 3: Model training: input the preprocessed training set data into the remote sensing image road segmentation network; initialize the feature encoding module using the pre-trained ResNet-34 model weights on the ImageNet-1K dataset, and randomly initialize the remaining network parameters; train the remote sensing image road segmentation network until the model converges; after training is complete, save the model parameters of the remote sensing image road segmentation network obtained by training.

[0085] Step 4: Result prediction: input the preprocessed remote sensing road image into the trained remote sensing image road segmentation network to obtain the accurate segmentation result of the remote sensing image data.

[0086] The training program is built based on the PyTorch framework, the optimizer uses the Adam algorithm, and the loss function combines binary cross-entropy and Dice coefficient to alleviate the class imbalance problem and enhance the segmentation ability of small-scale targets. The initial learning rate of the model is set to 2e-4, the batch size is set to 4, and the maximum training batch is set to 500. The learning rate adjustment mechanism uses the Plateau Learning Rate Scheduler, with a decay factor of 0.5 and a tolerance period of 5, which dynamically adjusts the learning rate during the learning process to avoid local optimal solutions. In addition, an early stopping condition is set in the experiment, i.e., when the loss value does not decrease significantly during the last 8 epochs, the model is considered to have reached convergence and the training process is terminated early. This is aimed at timely suppressing the tendency of overfitting and reasonably controlling the computational cost.

[0087] The calculation formula of the binary cross-entropy loss function is as follows:

[0088]

[0089] In the above formula, y is the real pixel label value, y' is the predicted label pixel value, and N is the number of label categories.

[0090] The calculation formula of the Dice coefficient loss function is as follows:

[0091]

[0092] In the above formula, X is the generated prediction map, Y is the real label, |X∩Y| is the intersection between the label and the prediction, |X| is the number of elements of the label, and |Y| is the number of elements of the prediction.

[0093] The final semantic segmentation loss function is the weighted sum of the cross-entropy loss function and the Dice coefficient loss function, and the calculation formula is as follows:

[0094] Ls = dice + L cr0ss

[0095] The application will be further described below according to specific embodiments.

[0096] The experimental data set is selected as the DeepGlobe road extraction data set. The data set is from the remote sensing road extraction challenge competition held by CVPR2018. The data set contains 6226 groups of remote sensing images with a size of 1024x1024 pixels and road labels. The image resolution is 0.5m / pixel, and the image is collected by the satellite of Digital Globe. These images not only cover a wide range of geographical areas such as cities and rural areas in multiple countries, but also contain various types of roads and street networks. In addition, in order to maintain the authenticity and practicality of the data, the authors deliberately do not label the small roads in the farmland, and do not want the model to extract errors during identification, so that the data set is more challenging and practical, and also provides a wider exploration space.

[0097] When evaluating the road segmentation performance of the model, the application uses five commonly used evaluation indicators in semantic segmentation to evaluate the model, and the evaluation results are shown in Table 1. The five evaluation indicators are accuracy, precision, recall, F1 score and intersection over union. The precision represents that the extracted road object is more accurate, the recall represents that the extracted road object is more complete, and the two are contradictory to each other, and the F1 score is the harmonic mean of the recall and the precision. The accuracy measures the proportion of correct predictions of the model, but cannot well solve the problem of class imbalance, while the intersection over union reflects the overlap between the model prediction results and the real label. These indicators can comprehensively and objectively reflect the performance of the model in the road segmentation task. Through the comparison of the experimental results, it can be seen that the model of the application has better performance in the road segmentation task.Figure 6 (original remote sensing road image), Figure 7 (human-labeled remote sensing road segmentation reference image), and Figure 8 (predicted road segmentation result image obtained after applying the method of the present application) can be seen, the method of the present application exhibits excellent performance in road extraction of remote sensing images. Specifically, Figure 8 the predicted segmentation result shown is highly consistent with the human-labeled standard image in Figure 7 This fact powerfully proves the accuracy and reliability of the method of the present application in the road segmentation task.

[0098] Accuracy Precision Recall F1 Score Intersection over Union 0.9882 0.8541 0.8498 0.8520 0.7421

[0099] Table 1 Specific indicators on the DeepGlobe road extraction dataset.

[0100] The application discloses an innovative remote sensing image road segmentation method, which fuses a bidirectional multi-level road feature dynamic interaction and a double-context dynamic extraction mechanism. The application aims to realize highly accurate segmentation of road regions in high-resolution remote sensing images by means of advanced deep learning technology, in combination with a specially designed network architecture and modules. First, a detailed manual labeling label image is created based on a high-resolution remote sensing image of an RGB three-channel, which serves as a standard reference for model training. Then, the complete image dataset is proportionally divided into a training set and a test set, and a series of preprocessing operations such as denoising and feature enhancement are performed on the images in the training set to optimize image quality and lay a good foundation for model training. In the model construction stage, the application creatively designs a remote sensing image road extraction algorithm based on a bidirectional multi-level road feature dynamic fusion and a double-context dynamic extraction strategy. The bidirectional multi-level road feature dynamic fusion module effectively compensates for the loss of spatial information and edge details caused by continuous downsampling through bidirectional interaction of multi-level feature information, and enhances the model's understanding of complex scenes and multi-level road structures. On the other hand, the double-context dynamic extraction module realizes dynamic fusion of long-distance and local multi-scale context information, which helps to suppress background interference, occlusion effects and confusion of similar road structures, and ensures that the extracted road structure has integrity and continuity. During model training, the preprocessed training images and their corresponding label images are input into the road segmentation model, and the network parameters are continuously adjusted using an optimization algorithm until the model can accurately identify and segment the road region. When the model reaches a convergent state, the best parameter settings are saved for subsequent applications. Finally, the application can automatically generate a predicted label image by inputting a new remote sensing image into the trained model, which is the segmented road image. Compared with the prior art, the application has the unique feature of applying the bidirectional multi-level road feature dynamic fusion and double-context dynamic extraction mechanism, which not only improves the connectivity and accuracy of road segmentation, but also enhances the model's ability to distinguish different levels of roads and restore edge details.

[0101] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the application, and not to limit them; although the application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for part or all of the technical features; and these modifications or substitutions do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the application.

Claims

1. A remote sensing image road segmentation method combining bidirectional multi-level road feature dynamic fusion and dual-context dynamic extraction, characterized in that: Includes the following steps: Step 1: Dataset Creation: Randomly divide the high-resolution remote sensing image dataset into training and test sets; The remote sensing image dataset is acquired by remote sensing satellites and includes raw images and manually labeled image data; the remote sensing images in the training set are preprocessed and augmented to form an augmented remote sensing image dataset; Step 2: Construction of the remote sensing image road segmentation network model: including the following steps: Step 2.1: Construction of the feature encoding module; Step 2.2: Construction of the bidirectional multi-level road feature dynamic fusion module: The bidirectional multi-level road feature dynamic fusion module is designed to perform bidirectional step-by-step dynamic fusion of multi-level features. The dynamic fusion module includes a strip-shaped large kernel spatial attention with dynamic receptive field and an effective channel attention for spatial and channel dimension selection. Step 2.3: Construction of the dual-context dynamic extraction module: The dual-context dynamic extraction module designs striped dilated input dependency convolution, and based on this, designs dilated spatial pyramid pooling based on striped dilated input dependency convolution and a dual-axis attention mechanism based on striped dilated input dependency convolution, which are used for local multi-scale dynamic context extraction and global dynamic context extraction, respectively, and combined with effective channel attention for further feature filtering and fusion; Step 2.4: Construction of the feature decoding module; The feature encoding module and the feature decoding module achieve bidirectional, step-by-step dynamic fusion of features through a bidirectional, multi-level road feature dynamic fusion module; the fused features are then extracted by a dual-context dynamic extraction module to extract global and local multi-scale context features; the extraction results are used by the feature decoding module for decoding. Step 3: Model Training: Input the preprocessed training set data into the remote sensing image road segmentation network; initialize the feature encoding module with the weights of the ResNet-34 model pre-trained on the ImageNet-1K dataset, and randomly initialize the remaining network parameters; train the remote sensing image road segmentation network until the model converges; after training, save the trained remote sensing image road segmentation network model parameters. Step 4: Result Prediction: Input the preprocessed remote sensing road image into the trained remote sensing image road segmentation network to obtain accurate segmentation results of the remote sensing image data.

2. The remote sensing image road segmentation method according to claim 1, which combines bidirectional multi-level road feature dynamic fusion and dual-context dynamic extraction, is characterized in that: The feature encoding module and the feature decoding module each consist of five stages. The feature encoding module includes, in sequence, a downsampling module, encoder 1, encoder 2, encoder 3 and encoder 4; the feature decoding module includes, in sequence, decoder 1, decoder 2, decoder 3, decoder 4 and a segmentation head.

3. The remote sensing image road segmentation method according to claim 2, which combines bidirectional multi-level road feature dynamic fusion and dual-context dynamic extraction, is characterized in that: The downsampling module consists of a 7×7 convolution operation with a stride of 2 and a 3×3 max pooling operation with a stride of 2; the four encoders are composed of stacked residual blocks of 3, 4, 6 and 3 respectively; each residual block is composed of two 3×3 convolution operations and residual concatenation.

4. The remote sensing image road segmentation method combining bidirectional multi-level road feature dynamic fusion and dual-context dynamic extraction as described in claim 2, characterized in that: Each of the four decoders includes a 1×1 convolution that reduces the number of channels to 1 / 4, a 3×3 transposed convolution with a stride of 2, and a 1×1 convolution that doubles the number of channels. Each of the three convolutions is followed by batch normalization and the ReLU activation function. Finally, the decoder output is converted into road extraction results by a segmentation head, which consists of a 3×3 transposed convolution with a stride of 2, a 3×3 convolution, and a 2×2 convolution.

5. The remote sensing image road segmentation method according to claim 1, which combines bidirectional multi-level road feature dynamic fusion and dual-context dynamic extraction, is characterized in that: The bidirectional multi-level road feature dynamic fusion module achieves bidirectional, step-by-step dynamic fusion of the multi-level features output by the encoder through three downsampling dynamic fusion modules and three upsampling dynamic fusion modules.

6. The remote sensing image road segmentation method according to claim 1, which combines bidirectional multi-level road feature dynamic fusion and dual-context dynamic extraction, is characterized in that: The strip-shaped macrokernel spatial attention consists of two parts: strip-shaped macrokernel selection and spatial selection. Strip-shaped macrokernel selection enables the model to adaptively find the receptive field that is most suitable for the current stage. Spatial selection selects the feature map obtained after strip-shaped macrokernel selection in the spatial dimension. The structure of the strip-shaped large nucleus is selected as follows: By combining two depthwise separable convolutions with different receptive fields and four sets of strip-shaped depthwise separable convolutions, and then using a 1×1 convolution for weighted fusion; The structure for spatial selection is as follows: The feature maps obtained in the strip large kernel selection stage are subjected to max pooling, average pooling, and strip adaptive pooling that conforms to road features. Four different spatial attention maps are obtained by 7×7 convolution and Sigmoid activation. The final spatial attention map is obtained by summing these maps. Spatial selection is then performed on the feature maps obtained after the strip large kernel selection.

7. The remote sensing image road segmentation method according to claim 6, which combines bidirectional multi-level road feature dynamic fusion and dual-context dynamic extraction, is characterized in that: The strip depth-separable convolution consists of two strip depth-separable convolution operations with different dilation rates, one horizontal and one vertical.

8. The remote sensing image road segmentation method according to claim 1, which combines bidirectional multi-level road feature dynamic fusion and dual-context dynamic extraction, is characterized in that: The dual-context dynamic extraction module is designed based on striped dilated input-dependent convolution and consists of dilated spatial pyramid pooling based on striped dilated input-dependent convolution, a dual-axis attention mechanism based on striped dilated input-dependent convolution, and effective channel attention. The implementation process of the striped dilated input-dependent convolution is as follows: By generating multiple sets of static learnable parameters and dynamically generating corresponding bar spatial attention maps based on the current input as multiple sets of static learnable parameters through bar adaptive pooling, the parameters of the bar-drilled input-dependent convolution can be obtained by multiplying the static learnable parameters with the dynamically generated corresponding bar spatial attention maps and summing them.

9. The remote sensing image road segmentation method according to claim 8, which combines bidirectional multi-level road feature dynamic fusion and dual-context dynamic extraction, is characterized in that: The structure of the dilated spatial pyramid pooling module based on striped dilated input-dependent convolution is as follows: It consists of one 1×1 convolution, one global average pooling, and four sets of striped dilated input dependency convolutions with dilation rates of 1, 3, 6, and 12. Each set of striped dilated input dependency convolutions contains one horizontal, one vertical, and one full-size striped dilated input dependency convolution. The kernel size of each striped dilated input dependency convolution is 3, and it contains four sets of statically learnable parameters.

10. A remote sensing image road segmentation method combining bidirectional multi-level road feature dynamic fusion and dual-context dynamic extraction as described in claim 8, characterized in that: The structure of the biaxial attention mechanism based on striped dilated input-dependent convolution is as follows: First, two sets of Q and K are generated from the input feature map for the horizontal and vertical directions respectively, and V is shared by the first set. Then, axial attention operations are performed for the horizontal and vertical directions respectively. A 3×3 striped dilated input-dependent convolution was performed on the shared V and then added to the horizontal and vertical attention maps. Finally, by fusing the results from different axes using 1×1 convolution, the hierarchy and richness of the feature representation are enhanced.

Images