A Road Extraction Method Based on High-Resolution Remote Sensing Image Fusion of Multi-Attention

Abstract

This invention relates to the field of remote sensing image processing technology, specifically to a method for road extraction from high-resolution remote sensing images that integrates multiple attention mechanisms. Based on a dilated residual collaborative attention network, and combined with a dilated convolution module, it extracts higher-dimensional road feature information. A lightweight self-attention module is used in the skip connection portion of the U-shaped structure to capture long-range dependencies of feature information. A channel-oriented cross-attention module eliminates ambiguity between the self-attention mechanism module and the decoder features, improving the feature map reconstruction capability. Finally, training weights are used to predict unlabeled remote sensing images, entropy values ​​are calculated, and the high-entropy prediction results are used to label the unlabeled images, which are then mixed with the original labels and input into the training network. This invention, through a deep learning road extraction method integrating multiple attention mechanisms, effectively establishes long-range channel and location information relationships for roads, improving the efficiency of road extraction.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing technology, and more specifically to a method for road extraction from high-resolution remote sensing images that integrates multiple attention. Background Technology

[0002] Automatic road extraction from remote sensing imagery plays a crucial role in urban planning, georeferenced data, intelligent transportation navigation, geospatial data integration, and intelligent transportation systems. However, the noise, occlusion, and complexity of road structures in remote sensing imagery make automatic road extraction extremely challenging. Traditional road extraction methods, including support vector machines, mathematical morphology methods, and template matching, are generally effective for extracting information from small amounts of remote sensing data. With the widespread application of deep learning technology in practical production and daily life, the ability to quickly and efficiently extract remote sensing imagery information from massive amounts of data has significant guiding value.

[0003] Deep learning-based road extraction methods from remote sensing images have improved the efficiency of road extraction, but they cannot simultaneously take into account information between channels and location information. Although later attention mechanisms attempted to extract location attention information through convolution after reducing the number of channels, the receptive field of convolution is limited, lacking the utilization of long-range feature information. At the same time, deep learning requires the support of a large amount of labeled data, which requires a lot of manual annotation. Summary of the Invention

[0004] The purpose of this invention is to provide a road extraction method for high-resolution remote sensing images that integrates multiple attention mechanisms. This method aims to integrate multiple attention mechanisms, establish the relationship between long-distance road channels and location information, recover detailed road information, improve existing remote sensing image road extraction methods, and increase the efficiency of road extraction.

[0005] To achieve the above objectives, the present invention provides a method for road extraction from high-resolution remote sensing images that incorporates multi-attention processing, comprising the following steps:

[0006] Based on the dilated residual collaborative attention neural network, with the encoder-decoder architecture as the basic network model;

[0007] The input data is preprocessed and combined with a dilated convolution module to extract high-dimensional road feature information;

[0008] Use a lightweight self-attention module to capture long-range dependencies of feature information;

[0009] Add a channel-oriented cross-attention module and train the model until convergence;

[0010] Using the weights obtained from training, road prediction is performed on unlabeled images.

[0011] Among them, the dilated residual collaborative attention neural network is based on the encoder-decoder architecture as the basic network model;

[0012] The input data is preprocessed and combined with a dilated convolution module to extract high-dimensional road feature information;

[0013] Use a lightweight self-attention module to capture long-range dependencies of feature information;

[0014] Add a channel-oriented cross-attention module and train the model until convergence;

[0015] Using the weights obtained from training, road prediction is performed on unlabeled images.

[0016] The preprocessing of input data includes cropping and data augmentation. The image is cropped using a sliding window with a size of 256, and the data augmentation includes flipping, random rotation, and scaling. The image is then input into the network in a 256*256*3 format.

[0017] The preprocessed input data is divided into a training set and a validation set. The validation set is used to verify the segmentation accuracy of the training of the convolutional neural network, while the training set is used to adjust the parameters of the training of the convolutional neural network. The ratio of the training set to the validation set is 8:2.

[0018] Specifically, the dilated convolution module is used to input a dilated spatial pyramid in the last layer, and then input a 1*1 convolution and three 3*3 convolutions with dilation rates of 1, 2, 4 and 8 respectively. The feature maps of the four layers with different dilation rates are concatenated and input into a convolution with a kernel of 1 for channel adjustment.

[0019] The process of adding a channel-oriented cross-attention module involves the decoding network gradually restoring the original image size through linear interpolation upsampling operations, with a channel-oriented cross-attention module added at the end of each upsampling module layer.

[0020] During the training process until convergence, the network convergence is determined by the training loss and validation loss. If convergence is achieved, training is stopped, and the trained MAG-Unet segmentation model is obtained. The test set is then input into the trained MAG-Unet segmentation model, and the segmentation results are output.

[0021] In the process of predicting roads in unlabeled images using the weights obtained from training, the prediction results are labeled according to the pixel entropy value, and the original labels are mixed and input into the network for training.

[0022] The process of labeling the prediction results involves using pixel entropy values ​​to determine the reliability of the output results, labeling the high-entropy parts, cutting the predicted output labels, and then mixing them with the original labels to enhance the data.

[0023] This invention provides a high-resolution remote sensing image road extraction method incorporating multiple attention mechanisms. Based on a dilated residual collaborative attention network, it combines a dilated convolutional module to extract higher-dimensional road feature information. A lightweight self-attention module is used in the U-shaped skip connection portion to capture long-range dependencies of feature information. A channel-oriented cross-attention module eliminates ambiguity between the self-attention mechanism and the decoder features, improving feature map reconstruction capabilities. Finally, training weights are used to predict unlabeled remote sensing images, entropy values ​​are calculated, and the high-entropy prediction results are used to label the unlabeled images, which are then mixed with the original labels and input into the training network. This invention, through its deep learning road extraction method incorporating multiple attention mechanisms, effectively establishes long-range channel and location information relationships for roads, improving the efficiency of road extraction. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart illustrating a high-resolution remote sensing image road extraction method that incorporates multi-attention processing according to the present invention.

[0026] Figure 2 This is a structural diagram of the MAG-Unet convolutional neural network in a specific embodiment of the present invention.

[0027] Figure 3 This is a diagram of the collaborative attention structure in a specific embodiment of the present invention.

[0028] Figure 4 This is a diagram of a lightweight self-attention structure in a specific embodiment of the present invention.

[0029] Figure 5 This is a schematic diagram of a channel-oriented cross-attention module in a specific embodiment of the present invention. Detailed Implementation

[0030] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0031] Please see Figure 1 This invention provides a method for road extraction from high-resolution remote sensing images that incorporates multiple attention, comprising the following steps:

[0032] S1: Based on the dilated residual collaborative attention neural network, with the encoder-decoder architecture as the basic network model;

[0033] S2: Preprocess the input data and combine it with the dilated convolution module to extract high-dimensional road feature information;

[0034] S3: Uses a lightweight self-attention module to capture long-range dependencies of feature information;

[0035] S4: Add a channel-oriented cross-attention module and train the model until convergence;

[0036] S5: Use the weights obtained from training to predict roads in unlabeled images.

[0037] The following provides further explanation with reference to specific embodiments and implementation steps:

[0038] Please see Figures 2 to 5 The network uses an encoder-decoder structure as its basic network model. The encoder part employs an improved dilated ResNet101, with the first layer replacing max pooling with convolution to effectively eliminate grid artifacts. The first three residual structures are consistent with ResNet101, but a collaborative attention branch is added after the first 3×3 convolution in each residual structure. The collaborative attention module first uses two 1D vector encoding operations. For the input, pooling operations X pooling and Y pooling are used to encode horizontal and vertical features, effectively capturing long-distance relationships in one direction while preserving spatial information in the other direction, helping the network to more accurately locate the target. The feature output can be represented by the following formula:

[0039]

[0040] Horizontal and vertical features are concatenated, passed through a regularized convolutional layer, and then channel-separated to maintain consistency with the structure after two one-dimensional encodings. The output is then activated by a sigmoid function and merged into a weight matrix. Each weight contains inter-channel, lateral, and vertical spatial information, helping the network to more accurately locate target information and enhance recognition capabilities. The fourth residual layer uses dilated convolution with a dilation rate of 2, maintaining resolution while having a receptive field similar to the original structure. The outputs of the first four layers are input to a lightweight self-attention layer. Similar to the self-attention structure, the lightweight self-attention layer has three branches: Q, K, and V. Unlike the self-attention layer where Q is directly multiplied by the transpose of K, a depthwise separable convolution with a stride of k and a kernel size of k×k is first used to reduce the spatial size of K and V before the attention operation.

[0041]

[0042] The corresponding lightweight Attention definition is:

[0043] B is a randomly initialized and learnable relative positional bias. Lightweight self-attention replaces skip connections, fully utilizing long-distance features between feature maps. The final layer is an input void spatial pyramid, with one 1×1 convolution and three 3×3 convolutions, using dilation rates of 1, 2, 4, and 8 respectively. The four feature maps with different dilation rates are concatenated and input into a convolution with a kernel of 1 for channel adjustment, extracting road information at different scales, ensuring road recognition accuracy while extracting road details. First, global average pooling is performed to generate a vector G(X)∈R. C×1×1 and its kth channel This operation is used to embed global spatial information, and then an attention mask is generated. This is followed by entropy;

[0044]

[0045] The reliability of the prediction results is measured, the most reliable parts are selected, and then pseudo-labels are applied using conventional methods.

[0046]

[0047] Next, the preprocessed training dataset is input, the model hyperparameters are initialized, and the improved road segmentation network is trained. In this invention, both the intersection-union loss function and the cross-entropy loss function are used as the loss function. Specifically, in the binary classification case, the cross-entropy loss function is calculated as follows:

[0048]

[0049] Where y represents the actual pixel label value, y′ represents the predicted label pixel value, and N represents the number of label categories. The crossover ratio loss is calculated as follows:

[0050] Where X and Y represent the generated prediction map and the actual label, respectively.

[0051] The final loss function is the sum of the cross-entropy function and the cross-union loss: L Loss =L cross +L IoU .

[0052] The specific operation steps of this embodiment are as follows:

[0053] S1~S2: The high-resolution image road extraction network MAG-Unet was built. The image was cropped using a sliding window with a size of 256. The high-resolution data of the cropped 256×256×3 band was used as input data. A neural network architecture was built based on encoder-decoder.

[0054] The preprocessing process includes cropping and data augmentation of the high-resolution data. Data augmentation mainly includes flipping the data vertically and horizontally, random rotation, and scaling. Then, the high-resolution data of 256×256×3 is input into the network.

[0055] Furthermore, the preprocessed high-resolution images are divided into training and validation sets. The validation set is used to verify the segmentation accuracy of the training convolutional neural network, while the training set is used to adjust the parameters of the training convolutional neural network. The ratio of the training set to the validation set is 8:2.

[0056] The cropped image is input into the encoding network. The max pooling operation following the initial 7×7 convolution in the original ResNet101 is removed and replaced with two dilated convolution operations with a stride of 2. The subsequent three downsampling stages use a structure similar to ResNet101, with the difference being the addition of a collaborative attention branch after the first 3×3 convolution in each residual block. In the fourth downsampling convolution, the dilation coefficient is changed to 2, and the size of the output feature map remains consistent with the previous layer, maintaining the same field of view as the residual operation. Finally, the output feature dimension is 16×16×512.

[0057] The feature maps after the convolutional layers contain rich spatial semantic information. The feature maps are input into the improved ASPP module, and after dilated convolutions with dilation rates of 1, 2, 4, and 8, they are input into a 1×1 network to extract road information at different scales, ensuring the accuracy of road recognition while extracting road details.

[0058] The formula for calculating the receptive field is: S = k + (k-1)(n-1), where k represents the kernel size and n represents the dilated convolution rate.

[0059] S3: Input the previously output 64×64×128, 32×32×256, 16×16×512, and 16×16×512 into the lightweight self-attention module to establish long-distance dependencies.

[0060] S4: The decoding network gradually restores the original image size through linear interpolation upsampling operations. Each upsampling module is followed by a channel-oriented cross-attention module. The channel-oriented cross-attention module uses average pooling operations on the sampled features and the features after skip connections to eliminate the ambiguity between the self-attention mechanism module and the decoder features. Linear operations are used to help the upsampling layer effectively restore the detailed information of the road target.

[0061] During the training process until convergence, the network convergence is determined by the training loss and validation loss. If convergence is achieved, training is stopped, and the trained MAG-Unet segmentation model is obtained. The test set is then input into the trained MAG-Unet segmentation model, and the segmentation results are output.

[0062] Specifically, this experiment uses the Adam optimization algorithm, which, compared to SGD, RMSprop, and AdaGrad optimizers, enables the model to converge to optimal performance faster. The default parameters in Adam are used, with the learning rate set to 0.001, exponential decay rates β1 = 0.9, β2 = 0.999, and a constant ε = 10⁻⁸. Every 10 iterations, the learning rate is reduced to half its original value. Cross-entropy loss and cross-union loss are selected to update network parameters during backpropagation. The training iterations are 300.

[0063] S5: Using the weights obtained from training, road prediction is performed on unlabeled images. Based on the pixel entropy value, the prediction results with higher reliability are labeled using conventional methods, and the original labels are mixed and input into the network for training.

[0064] In summary, the significant advantages of this invention compared to existing algorithms are as follows:

[0065] (1) This invention uses an improved dilated residual neural network ResNet101 for preprocessing, cancels the max pooling operation after the 7×7 convolution in the original residual network, and replaces it with two dilated convolution operations with a stride of 2. The following three downsampling stages use a structure similar to the residual network. The improvement is that a collaborative attention branch is added after the first 3×3 convolution of the residual block. In the convolution process of the fourth downsampling, the dilation coefficient is changed to 2 to perform initial classification of road targets; (2) This invention adopts an encoder-decoder network structure. This structure model shows good performance in road feature extraction and detail restoration; (3) This invention adopts Using a dilated spatial convolution pooling pyramid, a dilated spatial convolution pooling pyramid is added between the encoder and decoder, which will help extract the feature semantic information of different levels of road images; (4) The skip connection uses a lightweight self-attention module, which captures long-distance dependencies without increasing the number of parameters and floating-point calculations. The encoder part uses a channel-oriented cross-attention module to eliminate the ambiguity between the self-attention mechanism module and the decoder features, improve the feature map restoration ability, and thus improve the integrity of road segmentation; (5) This invention uses high entropy value prediction to label unlabeled images, which can make full use of remote sensing images and expand the remote sensing image road dataset.

[0066] The above description discloses only one preferred embodiment of the present invention, and should not be construed as limiting the scope of the present invention. Those skilled in the art will understand that all or part of the processes of the above embodiments can be implemented, and equivalent changes made in accordance with the claims of the present invention are still within the scope of the invention.

Claims

1. A method for road extraction from high-resolution remote sensing images fused with multi-attention, characterized in that, Includes the following steps: Based on the dilated residual collaborative attention neural network, with the encoder-decoder architecture as the basic network model; The process of building a high-resolution image road extraction network MAG-Unet based on a dilated residual collaborative attention neural network and using an encoder-decoder architecture as the basic network model is as follows: The encoder part adopts an improved dilated ResNet101. The first layer uses convolution to replace max pooling. The first three residual structures are consistent with ResNet101. A collaborative attention branch is added after the first 3*3 convolution of each residual block. In the fourth downsampling convolution process, the dilation coefficient is changed to 2. The size of the output feature map is consistent with the previous layer. Add a dilated spatial convolutional pooling pyramid between the encoder and decoder; The input data is preprocessed and combined with a dilated convolution module to extract high-dimensional road feature information; Combined with the dilated convolution module, specifically, the last layer inputs a dilated spatial convolution pooling pyramid, and inputs a 1*1 convolution and three 3*3 convolutions respectively, using convolutions with dilation rates of 1, 2, 4, and 8 respectively. The four feature maps with different dilation rates are concatenated and input into a convolution with a kernel of 1 for channel adjustment. A lightweight self-attention module is used in the jump connection part of the U-shaped structure to capture long-range dependencies of feature information; The lightweight self-attention module has three branches Q, K, and V. Unlike the direct multiplication of Q and the transpose of K in self-attention, it first uses a step size of V. The convolution kernel is Depth-separable convolutions can reduce the space size of K and V before attention operations. ； The corresponding lightweight Attention definition is: ; Add a channel-oriented cross-attention module and train the model until convergence; The process of adding a channel-oriented cross-attention module is as follows: the decoding network gradually restores the original image size through linear interpolation upsampling operation. At the end of each upsampling module, a channel-oriented cross-attention module is added. The channel-oriented cross-attention module uses average pooling operation on the sampled features and the features after skip connections to eliminate the ambiguity between the self-attention mechanism module and the decoder features. Linear operation is used to help the upsampling layer effectively restore the road target details. Using the weights obtained from training, road prediction is performed on unlabeled images.

2. The high-resolution remote sensing image road extraction method fused with multi-attention as described in claim 1, characterized in that, The process of preprocessing the input data includes cropping and data augmentation. The image is cropped using a sliding window with a size of 256, and the data augmentation includes flipping it horizontally and vertically, random rotation, and scaling. The image is then input into the network in a 256*256*3 format.

3. The high-resolution remote sensing image road extraction method fused with multi-attention as described in claim 2, characterized in that, The preprocessed input data is divided into a training set and a validation set. The validation set is used to verify the segmentation accuracy of the training convolutional neural network, while the training set is used to adjust the parameters of the training convolutional neural network. The ratio of the training set to the validation set is 8:

2.

4. The high-resolution remote sensing image road extraction method fused with multi-attention as described in claim 1, characterized in that, During the training process until convergence, the network convergence is determined by the training loss and validation loss. If convergence is achieved, training is stopped, and the trained MAG-Unet segmentation model is obtained. The test set is then input into the trained MAG-Unet segmentation model, and the segmentation results are output.

5. The high-resolution remote sensing image road extraction method fused with multi-attention as described in claim 1, characterized in that, In the process of predicting roads in unlabeled images using the weights obtained during training, the prediction results are labeled according to the pixel entropy value, and the original labels are mixed and input into the network for training.

6. The high-resolution remote sensing image road extraction method fused with multi-attention as described in claim 5, characterized in that, The process of labeling the prediction results involves using pixel entropy values ​​to calculate and determine the reliability of the output results, labeling the high-entropy parts, cutting the predicted output labels, and then mixing them with the original labels to enhance the data.

Images