Road remote sensing extraction method based on optimized multi-head attention mechanism neural network

By using a neural network based on an optimized multi-head attention mechanism, and leveraging a hierarchical Swing Transformer and a cross-scale attention fusion module, we have addressed multiple challenges in road extraction from high-resolution remote sensing images, achieving high-precision and efficient road extraction results.

CN121616965BActive Publication Date: 2026-11-10CHINA INST OF WATER RESOURCES & HYDROPOWER RES
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202511826087.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2025-12-05
Publication Date
2026-11-10
Estimated Expiration
2045-12-05

AI Technical Summary

Technical Problem

Existing technologies struggle to effectively extract roads from high-resolution remote sensing images, especially given the challenges of diverse road appearances, narrow and elongated characteristics, complex spatial structures, interference factors, and the large scale of high-resolution data. Furthermore, traditional methods exhibit poor robustness, while deep learning methods struggle to model long-range dependencies and multi-scale features.

Method used

We employ a neural network based on an optimized multi-head attention mechanism, using a hierarchical Swing Transformer structure for feature extraction and cross-scale attention fusion, combined with a location-aware decoder for road extraction, and utilizing a cross-scale attention fusion module and a context aggregation module to achieve multi-scale feature interaction and adaptive weighting.

Benefits of technology

It achieves high-precision and high-connectivity extraction of complex road scenes, especially curved and narrow roads, improving the accuracy and efficiency of road extraction and significantly improving the segmentation effect of narrow roads and complex scenes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN121616965B_ABST
    Figure CN121616965B_ABST
Patent Text Reader

Abstract

This invention discloses a road remote sensing extraction method based on an optimized multi-head attention mechanism neural network. The neural network model includes four parts: an encoder, a fusion module, a decoder, and a task output head. The encoder includes a 4-stage Swing Transformer structure, responsible for progressively extracting and downsampling features. The fusion module includes a cross-scale attention fusion module and a context aggregation module, which perform feature fusion between encoding stages and are responsible for feature interaction and aggregation at different scales. The decoder is a location-aware decoder, responsible for progressively upsampling to restore spatial resolution.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of computer vision and remote sensing image processing technology, and particularly relates to a high-resolution image road extraction method based on a hierarchical visual transformer. It also relates to a road remote sensing extraction method based on an optimized multi-head attention mechanism neural network. Background Technology

[0002] Road extraction from remote sensing imagery is crucial for urban planning, traffic navigation, and disaster assessment. However, automatically extracting roads from high-resolution imagery faces multiple challenges. First, there's the diversity of road appearances; roads in different areas are paved with different materials, resulting in significant differences in spectral characteristics. Second, the narrow and elongated nature of roads, with widths of only a few pixels and extreme aspect ratios, makes them difficult for standard convolutional networks to perceive effectively. Third, complex spatial structures, such as winding road sections and intersections, require a global understanding of spatial relationships. Fourth, various interfering factors, such as building shadows, vegetation occlusion, parked vehicles, and road surface damage, alter the road's appearance. Fifth, there are issues of intra-class variation and inter-class similarity; the same road can vary greatly due to aging and maintenance, while roads and parking lots with similar spectra are prone to misclassification. Finally, high-resolution data is large in scale, making processing difficult.

[0003] Traditional image processing-based methods, such as edge detection and thresholding, are computationally simple but sensitive to changes in lighting and have poor robustness. Machine learning-based methods use SVM, random forests, and other methods combined with handcrafted features, but these features are insufficient to fully characterize the complexity of roads.

[0004] Deep learning methods based on CNNs have achieved good results, with representative methods such as FCN, U-Net, and DeepLabv3+. However, they have obvious limitations: the receptive field of convolution operations is small, requiring stacking multiple layers to obtain the global view, which leads to an increase in parameters and training difficulties; it is difficult to model long-range dependencies, as strong dependencies exist between pixels that are far apart on curved roads, but CNNs are difficult to capture; and translation and other variability are insufficient.

[0005] The Vision Transformer establishes global relationships through self-attention, but the standard ViT's O(N²) complexity is impractical for high-resolution images. The Swin Transformer reduces this to O(N) through moving window self-attention, achieving success in tasks such as image classification and detection. However, its design involves independent processing at each stage, lacking direct interaction between features at different resolutions. This is detrimental to road extraction requiring multi-scale context, especially for thin roads where feature performance is inconsistent across different stages, lacking consistency constraints. Summary of the Invention

[0006] Based on the above technical problems, this invention proposes a high-resolution road remote sensing extraction method based on an optimized multi-head attention mechanism neural network, specifically including the following steps: S1 remote sensing image data collection, preprocessing and training template creation: acquire high-resolution satellite remote sensing raw images, and after preprocessing, mark roads in the images, crop the images, perform data augmentation and standardization to obtain training samples, and divide all samples into training set, validation set and test set according to a certain ratio; S2 Network Model Construction: The neural network model consists of four parts: encoder, fusion module, decoder, and task output head. The encoder includes a 4-stage Swing Transformer structure, responsible for progressive feature extraction and downsampling. The fusion module includes a cross-scale attention fusion module and a context aggregation module, which perform feature fusion between encoding stages and are responsible for feature interaction and aggregation at different scales. The decoder is a position-aware decoder, responsible for progressive upsampling to restore spatial resolution. The overall network data flow is: input image → patch embedding → encoder (4 Swing Transformer stages + fusion) → aggregation → decoder → task output.

[0007] S3 Model Training: The model built in S2 is trained, validated, and tested based on the samples obtained in S1; S4 extracts roads from the remote sensing image under test by inputting the model trained in S3. Further optimization involves using a resolution of 0.3~0.1m for the original high-resolution satellite remote sensing image described in S1, and 0.5m for the embodiment of this invention.

[0008] Furthermore, in S1, the image is labeled by manual visual interpretation, and the labeling result is a binary mask, where a pixel value of 255 represents a road pixel and a pixel value of 0 represents a non-road pixel.

[0009] Furthermore, in S2, the first stage of the encoder includes a patch embedding layer to convert the raw image pixels into a sequence. In S2, the first stage of the encoder downsamples by 4x and uses two Swin Transformer modules, each containing three attention heads, primarily capturing high-resolution detail features such as subtle textures at road boundaries. The second stage downsamples by 8x and uses two Swin Transformer modules, each containing six attention heads; the reduced resolution decreases computational cost, allowing for the use of more attention heads. The third stage downsamples by 16x and uses six Swin Transformer modules, each containing 12 attention heads; the resolution is moderate, computational cost is controllable, and it utilizes the most Swin Transformer modules for thorough global feature learning, making it the deepest part of the encoder. The fourth stage downsamples by 32x and uses two Swin Transformer modules, each containing 24 attention heads; further downsampling yields limited benefits, and the number of blocks is relatively small. Downsampling is performed between stages using PatchMerging. The Swin Transformer module internally includes: layer normalization, multi-head self-attention, residual connections, layer normalization, MLP feedforward network, and residual connections.

[0010] In this invention: The encoder adopts the standard Swin Transformer structure, which is divided into four stages with different resolutions. The core innovation of SwinTransformer is the moving window self-attention mechanism, which reduces the complexity of standard self-attention to O(N²) to O(N).

[0011] The working principle of moving window self-attention is as follows: First, the feature map is divided into non-overlapping windows of size 7×7. Self-attention is calculated independently within each window, significantly reducing computational cost. To enable information interaction between different windows, windows are shifted (moved 3.5 pixels to the right and 3.5 pixels down) within adjacent Transformer blocks, allowing pixels within different windows of adjacent blocks to interact. Through a relative position bias mechanism, the network learns the relative positional relationships between pixels.

[0012] The cross-scale attention fusion module is inserted between the four stages of the encoder. It allows multi-head cross-attention computation of features from different stages, meaning that the query of stage i can perform attention computation with the key and value of stage j.

[0013] This module addresses the challenges of multi-scale fusion. First, it calculates the "information content" score for features at each scale, evaluating it based on the variance of the feature gradient (representing boundary richness) and the entropy of the feature distribution (representing information complexity). Second, it uses an MLP network to predict pixel-level contextual relevance weights, learning fusion weights for different scales through fully connected layers. Finally, it uses Softmax normalization weights to perform a weighted summation and fusion of features from different scales.

[0014] The module explicitly introduces relative position encoding and distance transform constraints during the decoding process. First, it performs deconvolution upsampling, with each upsampling factor being 2. It uses a PixelShuffle layer instead of simple deconvolution to reduce checkerboard artifacts. Second, it fuses skip connection features, enabling the network to adaptively select shallow details or deep structures.

[0015] Relative position encoding is incorporated. This encoding encodes the relative coordinates of each location in the feature map, maps them to the feature dimension using an MLP, and adds them to the decoded features. During decoding, predicting the distance transformation map serves as an auxiliary task.

[0016] The advantages and beneficial effects of this invention are: This invention enables high-precision, high-connectivity extraction of complex road scenes, especially curved and slender roads.

[0017] (1) Layered Swin Transformer encoder: adopts the standard 4-stage pyramid structure, performs feature extraction and downsampling step by step, and ensures O(N) linear complexity.

[0018] (2) Cross-scale attention fusion module: Breaking the stage isolation, introducing multi-head cross attention between stages, enabling direct interaction of features at different resolutions, which is particularly beneficial for capturing multi-scale features of narrow roads.

[0019] (3) Context aggregation module: Adaptively weights features at different scales based on the context importance of pixels to avoid noise caused by simple average fusion.

[0020] (4) Location-aware decoder: Explicitly introduces relative position encoding and distance transformation constraints to help the network understand spatial location relationships and road connectivity. Attached Figure Description

[0021] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0022] Figure 1 This is a schematic diagram of the overall structure of the model; Figure 2 Swin Transformer structure diagram; Figure 3 Multiple points of cross-attention; Figure 4 Context aggregation module; Figure 5 Feature encoding fusion; Figure 6 Single-image extraction results during implementation; Figure 7 The example shows the results of a large-scale extraction in Jingzhou city. Detailed Implementation

[0023] Example 1: Jingzhou City, a vital transportation hub in Hubei Province, is located in the middle reaches of the Yangtze River, occupying a crucial position in central China. In recent years, with the acceleration of urbanization and economic development, Jingzhou's road network has continuously expanded, placing increasing pressure on road construction, maintenance, and management in both urban and rural areas. Traditional road survey methods, relying on on-site measurements and manual mapping, are resource-intensive, inefficient, and unable to reflect real-time changes in the road network. Remote sensing technology, with its advantages of fast acquisition speed, wide coverage, and low cost, has become an important means of acquiring road information. However, automatically and accurately extracting roads from high-resolution remote sensing images faces multiple challenges. First, there is the diversity of road appearances; roads in different areas of Jingzhou are paved with different materials, including asphalt, cement, and village roads, resulting in significant differences in spectral characteristics. Second, the narrow and elongated nature of roads; city streets are only a few meters to tens of meters wide, corresponding to only a few pixels in remote sensing images. This extreme aspect ratio makes it difficult for standard convolutional neural networks to effectively perceive and extract information. Third, the complex spatial structure of Jingzhou's roads, including straight urban arterial roads, winding rural roads, various intersections, and complex junctions, requires the network to have a deep understanding of global spatial relationships. Fourth, various interference factors, such as building shadows, tree and vegetation obstruction, parked vehicles, and road surface damage, can alter the appearance of roads, increasing the difficulty of extraction. Fifth, the massive scale of high-resolution remote sensing image data presents technical challenges for processing and storage. Therefore, it is necessary to develop a method that can both process high-resolution remote sensing images and effectively extract the features of slender roads, providing accurate basic geographic information for the planning, construction, and management of urban and rural roads in Jingzhou.

[0024] The road remote sensing extraction method based on an optimized multi-head attention mechanism neural network of the present invention is specifically operated as follows: The method includes the following steps: S1: Data Collection, Preprocessing, and Sample Preparation S1.1: Acquisition and Preliminary Processing of Remote Sensing Image Data This embodiment utilizes multispectral imagery data from the GF-2 remote sensing satellite acquired by the China Center for Resources Satellite Data and Application. The GF-2 satellite provides high-resolution remote sensing images with a 1m resolution, encompassing four bands: blue, green, red, and near-infrared, clearly showcasing the road features of Jingzhou City. A total of 30 images were acquired, each covering an area of ​​approximately 200 square kilometers, for a total coverage area of ​​approximately 6,000 square kilometers. These images were collected in spring (April), summer (July), autumn (October), and winter (January) of 2023 and 2024, forming a complete seasonal sequence that reflects the changes in the visual characteristics of roads under different seasonal conditions.

[0025] The acquired raw images underwent a series of preprocessing steps, including orthorectification, atmospheric correction, and image fusion. Orthorectification utilizes DEM (Digital Elevation Model) data and satellite orbital parameters to eliminate geometric distortions caused by terrain undulations in the raw images, ensuring geometric accuracy. Atmospheric correction establishes an atmospheric radiative transfer model to eliminate the influence of the atmosphere on remote sensing images, making images acquired at different times and locations comparable. Image fusion combines 1m resolution multispectral images with 0.5m resolution panchromatic images, improving the spatial resolution and spectral accuracy of the entire dataset. After these preprocessing steps, the raw images are transformed into processed images with higher geometric accuracy and spectral precision.

[0026] S1.2: Acquisition and Quality Control of Road Marking Data This embodiment employs a manual visual interpretation method. Three personnel with extensive remote sensing interpretation experience independently annotated the imagery of the same area, and then cross-referenced the annotations. For areas with inconsistent annotations, the project leader organized further discussions and confirmations to ensure the accuracy and reliability of the final annotation results. The annotation results are presented as a binary mask, where a pixel value of 255 represents road pixels, and a pixel value of 0 represents non-road pixels.

[0027] Road labeling needs to fully consider the diverse characteristics of roads in Jingzhou City. For main urban roads, the labeling should include the entire width of the road surface, from the curb on one side to the curb on the other. For urban secondary roads and alleys, although they are narrower, they still need to be accurately labeled. For county and township roads connecting different areas, even if they are only a few meters wide, they need to be included in the labeling. For some large parking lots and squares, although their appearance is similar to roads, they should not be labeled as roads based on their actual use. For key roads such as highways, special care needs to be taken to ensure completeness. During the labeling process, special attention should be paid to handling the boundaries between roads and other features, such as the road edge and the shadows of buildings, the boundaries of vegetation, and the accurate division of road forks.

[0028] S1.3: Image cropping and sample creation To accommodate the input requirements of the neural network, the large original image needs to be cropped into appropriately sized samples. The cropping window size is set to 512×512 pixels, a size that includes sufficient spatial context without causing individual samples to become too large and increase computational burden. A sliding window cropping method is used, moving in 256-pixel steps in both the horizontal and vertical directions. This means that adjacent cropping blocks have 50% overlap in both the horizontal and vertical directions.

[0029] For image boundary processing, when the cropping window exceeds the range of the original image, zero-padding or mirror expansion is used. Specifically, for images or cropping blocks smaller than 512×512, zero-padding is used to bring them to a size of 512×512. This process ensures that all samples have a uniform size of 512×512, facilitating batch input into the network.

[0030] This embodiment uses this cropping method to generate approximately 18,000 sample blocks from 30 original images. These sample blocks cover various environments, including urban, suburban, and rural areas, and include a variety of road types such as straight main roads, winding rural roads, and complex intersections and junctions. The 50% overlap between sample blocks ensures the diversity and integrity of the samples, avoiding information loss due to cropping.

[0031] S1.4: Data Augmentation To improve the generalization ability and robustness of the model, various data augmentation processes are needed for the samples. The purpose of data augmentation is to simulate various changing conditions that may be encountered in real-world applications, enabling the model to learn to cope with these changes.

[0032] Geometric transformation enhancements include three operations: random rotation, random flipping, and random scaling. Random rotation ranges from 0° to 360°, rotating the image and corresponding mask within the same sample block by the same angle to ensure consistency between the rotated image and mask. This rotation process simulates the diversity of image acquisition orientations. Random flipping is divided into horizontal flipping (probability 0.5) and vertical flipping (probability 0.5), performed independently, producing four different flipping results. Random scaling has a factor ranging from 0.8 to 1.2, using bilinear interpolation to scale the image and nearest-neighbor interpolation to scale the mask, avoiding interpolation of intermediate values ​​in the mask's pixel values.

[0033] Noise enhancement methods include Gaussian noise and salt-and-pepper noise. Gaussian noise has a mean of 0 and a standard deviation of 5, simulating the noise characteristics of a remote sensing sensor. Salt-and-pepper noise performs extreme value replacement on randomly selected pixels with a probability of 0.01 (i.e., the pixel value is replaced with 0 or 255), simulating bad pixels that may appear in remote sensing images.

[0034] The application probabilities of various augmentation operations were set as follows: rotation 0.5, flip 0.5, scaling 0.3, hue adjustment 0.3, Gaussian noise 0.2, and salt-and-pepper noise 0.1. This probability setting ensures the appropriate application of augmentation operations, increasing sample diversity while avoiding excessive data distortion. After random combinations of various augmentation operations, the original approximately 18,000 sample blocks were expanded into approximately 72,000 effective training samples.

[0035] S1.5: Sample set partitioning and data standardization All samples were divided into three parts: a training set, a validation set, and a test set, in a ratio of 8:1:1. Therefore, the training set contained approximately 57,600 samples, the validation set contained approximately 7,200 samples, and the test set contained approximately 7,200 samples. This division ensured that the model used non-overlapping samples during the training, validation, and testing phases, avoiding data leakage issues.

[0036] The enhanced samples are standardized. A channel-level normalization method is used, calculating the mean and standard deviation of each band of the image across the entire training set, and then normalizing each pixel. Specifically, for the pixel value of the c-th band, the normalized value is obtained by dividing the pixel value by the standard deviation of that band. This process ensures the input data has a zero-mean and unit-variance distribution, which is beneficial for stable training of the neural network and accelerates convergence.

[0037] The processed image data and labeled mask are converted into PyTorch tensor format Tensor(C,H,W), where C represents the number of channels (4 channels for the image and 1 channel for the mask), and H and W represent the height and width (both 512).

[0038] S2: Model Building The road extraction network constructed in this embodiment adopts an encoder-decoder architecture, integrating multi-head attention mechanism, residual connections, and a multi-task learning framework. The overall process of the network is as follows: the input image is first converted into a feature sequence through a patch embedding layer, and then enters a 4-stage hierarchical Swin Transformer encoder for feature extraction and multi-scale downsampling. During the encoding process, a cross-scale attention fusion module enables features of different resolutions to interact, and then a context aggregation module performs adaptive weighted fusion of multi-scale features. The network then enters a location-aware decoder for progressive upsampling and feature recovery, and finally outputs the road segmentation result through the task output head.

[0039] The road extraction network constructed in this embodiment adopts an encoder-decoder architecture, integrating multi-head attention mechanism, residual connections, and a multi-task learning framework. The overall process of the network is as follows: the input image is first converted into a feature sequence through a patch embedding layer, and then enters a 4-stage hierarchical Swin Transformer encoder for feature extraction and multi-scale downsampling. During the encoding process, a cross-scale attention fusion module enables features of different resolutions to interact, and then a context aggregation module performs adaptive weighted fusion of multi-scale features. The network then enters a location-aware decoder for progressive upsampling and feature recovery, and finally outputs the road segmentation result through the task output head.

[0040] The network consists of four main parts: an encoder, a fusion module, a decoder, and a multi-task output head. The encoder is responsible for progressively extracting features and downsampling them, the fusion module is responsible for feature interaction and aggregation at different scales, and the decoder is responsible for progressively restoring the spatial resolution to generate the final prediction result.

[0041] Layered Swing Transformer encoder: The input road remote sensing image is 512×512×4 in size (height, width, and four spectral bands). First, it is processed through a patch embedding layer to convert the raw image pixels into a sequence format to meet the input requirements of the Transformer module. Specifically, the 512×512 image is divided into 16×16 patches, generating a total of 32×32=1024 patches. All pixels in each patch are flattened, and then each patch is mapped to a 96-dimensional feature space through a linear projection layer (fully connected layer). After this step, the feature dimension is transformed from 4×16×16=1024 dimensions to 96 dimensions, forming 1024 96-dimensional feature vector sequences.

[0042] Simultaneously, positional encoding information is added. Positional encoding is used to allow the network to learn the location information of the patch. This embodiment uses a learnable relative positional encoding, initialized with random values, which is learned along with the network parameters during training. The positional encoding has the same dimension as the feature dimension, both being 96-dimensional, and is then added element-wise with the patch features.

[0043] An encoder employing a standard 4-stage pyramid-shaped Swin Transformer structure progressively performs feature extraction and spatial downsampling. The first stage processes 1024 96-dimensional feature vectors embedded from the patch. First, the features are reconstructed back into a two-dimensional spatial form, resulting in a 128×128×96 feature map. Then, two Swin Transformer blocks are used for processing, each containing three attention heads. The internal structure of a Swin Transformer block includes: layer normalization, multi-head self-attention (using a 7×7 window), residual connections, layer normalization, MLP feedforward network, and residual connections. In this embodiment, the Swin Transformer structure is as follows: Figure 2 As shown, the Swing Transformer feature extraction block employs a locally enhanced window self-attention mechanism. It reduces complexity through non-overlapping window computation and introduces a window offset strategy to achieve cross-window connections, effectively capturing global context information while maintaining computational efficiency. After processing by two Swing blocks, the feature dimension remains at 96. Then, downsampling is performed using PatchMerging, grouping adjacent 2×2 pixels into four groups. The features of these four pixels are then concatenated along the channel dimension (resulting in four times the number of channels), and finally mapped back to twice the original dimension through a fully connected layer. This reduces the spatial resolution of the feature map from 128×128 to 64×64, while increasing the feature dimension from 96 to 192.

[0044] The second stage processes the 64×64×192 feature map. It contains two Swin Transformer blocks, each with six attention heads. Due to the reduced resolution and computational cost, more attention heads can be used to enhance feature representation. Each block still includes structures such as moving window self-attention (window size remains 7×7), residual connections, and MLP feedforward networks. After processing, the feature dimension remains 192. Then, downsampling is performed using Patch Merging, reducing the feature map resolution from 64×64 to 32×32 and increasing the feature dimension from 192 to 384. The outputs of the first and second stages enter the cross-scale attention module, where the resulting output is fused with the output of the second stage.

[0045] The third stage processes the 32×32×384 feature map. This is the deepest layer of the encoder, containing 6 SwinTransformer blocks, each with 12 attention heads. A large number of blocks and attention heads are used to ensure sufficient global feature learning. This stage is the most comprehensive part of the encoder's feature extraction; the features learned in this stage represent the most important semantic information of the input image. After processing, the feature dimension remains at 384. Then, downsampling is performed using PatchMerging, reducing the feature map resolution from 32×32 to 16×16 and increasing the feature dimension from 384 to 768. The outputs of the second and third stages are fed into the cross-scale attention module, and the resulting output is then fused with the output of the third stage.

[0046] The fourth stage processes the 16×16×768 feature map. Further downsampling offers limited benefit, so this stage contains only two Swin Transformer blocks, each with 24 attention heads. Although the number of blocks is small, the number of attention heads is the highest, which is beneficial for learning the highest-level global semantic features. After processing, the feature dimension remains 768. Since this is the final encoding stage, no further downsampling is performed. Through these four stages of progressive processing, the encoder transforms the original 512×512×4 input image into a 16×16×768 deep feature representation. The feature space resolution is reduced by 64 times, but the feature dimension increases significantly from 4 dimensions to 768 dimensions. The outputs of the third and fourth stages are fed into the cross-scale attention module.

[0047] The features from the first, second, third, and fourth stages of fusion are combined and then subjected to contextual aggregation to obtain a feature pyramid. The feature pyramid is then subjected to positional encoding, distance transformation constraints, and upsampled to the original output size.

[0048] The output is a 512×512×1 single-channel feature map, representing the probability that each pixel belongs to a road. A 1×1 convolution maps the decoded features from the number of decoder channels to 1, and then a sigmoid activation function constrains the output to between 0 and 1. The road segmentation prediction result is compared with the true road binary mask to calculate the segmentation loss function.

[0049] Loss function: To effectively optimize the road remote sensing extraction task in Jingzhou City, this embodiment designs a loss function suitable for road segmentation. Since road pixels typically account for a small proportion of remote sensing images, class imbalance occurs. Therefore, a combination of Dice loss and cross-entropy loss is used, calculated as follows: The total loss function consists of several parts: The road segmentation loss uses a combination of Dice loss and cross-entropy loss: Where X represents the predicted set of road pixels, and Y represents the set of ground truth labeled road pixels. The Dice loss is closely related to the Intersection over Union (IoU). In the road extraction scenario of Jingzhou City, road pixels account for a relatively small proportion of the total number of pixels. This class imbalance causes the standard cross-entropy loss to favor the predicted background class. The Dice loss effectively addresses this imbalance by directly optimizing the ratio of intersection to union, enabling the network to fully learn road features.

[0050] The cross-entropy loss is calculated as follows: Where N is the total number of pixels, y i p is the true label of the i-th pixel. i This represents the predicted probability.

[0051] like Figure 3 As shown, in this embodiment, a multi-head cross-attention module is used to calculate the interaction relationship between two different sequence data in parallel using multiple attention heads and weighted aggregate value vectors to capture the complex dependencies and semantic alignment between cross-domain features.

[0052] Figure 4 The context aggregation module structure captures and fuses global context information. By constructing long-range dependencies or integrating multi-scale features, it effectively fuses scattered local features with global semantic cues, enhancing the model's ability to understand complex scenes and improving its feature representation.

[0053] Figure 5 For feature encoding fusion, positional encoding injects relative positional information into sequence elements to compensate for the positional insensitivity of the self-attention mechanism. Then, the geodesic distance between points in the feature space is calculated through distance transformation to highlight the contours and spatial structure of key targets. The spatial resolution of the feature map is gradually restored using an upsampling decoder, and multi-level features are fused in the process.

[0054] S3, Model Training S3.1 Optimizer and Learning Rate Settings Model training was performed using the AdamW optimizer. AdamW is an improved version of the Adam optimizer that explicitly separates the weight decay term during parameter updates, avoiding the coupling problem between weight decay and the learning rate in standard Adam, thus resulting in better regularization performance. The initial learning rate was set to 1×10⁻. 4 The weight decay coefficient is set to 0.01 to prevent the model from overfitting.

[0055] A cosine annealing learning rate decay strategy is employed. The learning rate remains relatively stable in the early stages, begins to decrease in the middle stages, and gradually approaches its minimum value in the later stages. A warm-up phase of 5000 iterations is included. During the warm-up phase, the learning rate increases linearly from 0 to an initial value of 1×10⁻⁻⁻⁶. 4 The purpose of warming up is to prevent the model from becoming unstable due to using an excessively large learning rate in the early stages of training, so that the network parameters can start learning from relatively reasonable initial values.

[0056] S3.2 Training Configuration Different numerical precisions are used in different parts of the neural network. During forward propagation, the main computations are performed using float16 to reduce memory usage and leverage the GPU's Tensor Cores for acceleration. For operations requiring higher precision, float32 (single-precision floating-point numbers) is retained. Parameter updates are performed every two batches, effectively resulting in a batch size of 32. Pre-trained weights are used for model initialization. The encoder is initialized using ImageNet pre-trained Swin Transformer weights. The training epochs are set to 200, with a physical batch size of 16 (2 gradient accumulation steps, effective batch size 32). A checkpoint is saved every 10 epochs, containing complete model weights and optimizer state for easy recovery after training interruptions.

[0057] Model training was performed on a high-performance computing platform. The platform was configured with four NVIDIA A100 graphics processors, each with 80GB of video memory, for a total of 320GB. The CPU was an Intel Xeon Platinum 8380 processor running at 2.30GHz, and the system memory was 256GB. Storage consisted of 4TB NVIDIA NVMe solid-state drives, providing high-speed data read and write capabilities. The operating system was Ubuntu 20.04 LTS, and the deep learning framework was PyTorch 2.0.0 with Python 3.9. CUDA version 12.1 was used.

[0058] S3.3 Training Monitoring and Verification During training, a full validation set evaluation is performed on the model every two training epochs. The evaluation process employs a multi-dimensional metric system, specifically including mean Intersection over Union (mIoU), frequency-weighted Intersection over Union (FWIoU), precision, recall, and F1 score. The definitions of each metric are as follows: Mean Intersection over Union (IoU): As a core evaluation metric for image segmentation tasks, in the context of binary road extraction, this metric is equivalent to the IoU of road categories. It is calculated as the ratio between the set of correctly predicted road pixels and the union of the set of actual road pixels and the set of predicted road pixels. The value ranges from [0,1], with higher values ​​indicating better model segmentation accuracy.

[0059] Frequency-weighted Intersection over Union (IoU): This is a weighted and optimized version of the average IoU, with weights determined based on the frequency of each category in the sample set. In application scenarios where road areas constitute a relatively small portion of the image, this metric can more accurately reflect the model's segmentation performance on the overall image. Accuracy: Defined as the proportion of real road pixels in the set of pixels predicted as roads by the model. The higher the value of this indicator, the fewer false positives the model has misclassified as roads.

[0060] Recall rate: Defined as the percentage of pixels in the set of real road pixels that are correctly predicted as roads by the model. The higher the value of this metric, the fewer false negative examples of real roads the model misses. F1 score: Calculated by the harmonic mean of precision and recall, used to comprehensively measure the model's precision and recall performance.

[0061] S4. Results Evaluation In the task of road extraction from high-resolution remote sensing images in Jingzhou City, the model in this embodiment achieved significant performance improvement, fully verifying the effectiveness of the proposed method.

[0062] This model was systematically compared with several representative deep learning methods. Compared with the standard SwinTransformer encoder (excluding cross-scale attention fusion and context aggregation modules), this model improved mIoU by 3.2%, from 84.5% to 87.7%. This 3.2% improvement may seem small, but it is significant for fine-grained tasks like road extraction, as it represents the correct classification of approximately 3800 road pixels (based on a 512×512 input). Simultaneously, FWIoU improved by 2.8%, from 83.2% to 86.0%. This demonstrates that the innovative cross-scale fusion and aggregation modules significantly improve the accuracy of road segmentation, particularly for extracting roads at different scales.

[0063] Compared to traditional deep learning methods based on convolutional neural networks, the performance advantage is more significant. U-Net++, a classic method in image segmentation, achieves an mIoU of only 82.6% on this task, a decrease of 5.1% compared to our model. Although DeepLabv3+ expands the receptive field through dilated convolutions, achieving an mIoU of 81.5%, it is still 6.2% lower than our model. This fully demonstrates the advantage of the Transformer architecture based on multi-head attention and multi-scale fusion over traditional CNN methods when handling complex road features. The convolutional operations of CNNs are inherently local, requiring multiple layers to obtain a global view, which leads to increased parameters and training difficulties. In contrast, the Transformer, through its self-attention mechanism, can directly establish global dependencies, which is particularly advantageous for extracting slender linear features such as roads.

[0064] Jingzhou City has a large number of narrow roads, including urban alleys and mountain trails. These roads are typically less than 30 pixels wide, making them the most challenging type of road extraction. Our model demonstrates a particularly significant improvement in detecting these types of roads. The recall rate for narrow roads is improved by 8-12% compared to the benchmark Swin Transformer, a much larger improvement than the overall performance. Specifically, our model achieves a recall rate of 85.3% for narrow roads, while the benchmark method only achieves 73-77%. This significant improvement is mainly due to the cross-scale attention fusion module, which effectively integrates features from different resolutions. The high-resolution stage captures the boundary details and linear features of narrow roads, while the low-resolution stage perceives the global direction and connectivity of the roads. This comprehensive integration allows the network to gain a more complete understanding of narrow roads.

[0065] The road network in Jingzhou City contains numerous complex scenarios, such as intersections, Y-shaped intersections, and roundabouts, where multiple roads intersect, easily leading to segmentation confusion. Our model improves segmentation accuracy in these complex areas by 5-8% compared to benchmark methods. In typical complex scenarios such as intersections, our model achieves an accuracy of 83.2%, while the benchmark method only reaches 75-80%. This improvement is the result of multiple factors. First, the relative position encoding in the location-aware decoder helps the network understand spatial relationships, enabling it to understand the adjacency and connectivity relationships between different road segments.

[0066] Table 1 As shown in Table 1, our model outperforms the comparison methods in all metrics. Specifically, the precision reaches 89.2%, indicating that our model effectively avoids misclassifying background as roads. The recall reaches 86.5%, demonstrating that the model captures most of the true road pixels. The combined F1 score is 87.8%, indicating that our model achieves a good balance between precision and recall.

[0067] Finally, it should be noted that the above description is only used to illustrate the technical solutions of the present invention and is not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention.

Claims

1. A road remote sensing extraction method based on an optimized multi-head attention mechanism neural network, characterized in that: Includes the following steps: S1 remote sensing image data collection, preprocessing and training template creation: acquire high-resolution satellite remote sensing raw images, and after preprocessing, mark roads in the images, crop the images, perform data augmentation and standardization to obtain training samples, and divide all samples into training set, validation set and test set according to a certain ratio; The S2 network model is constructed as follows: The neural network model consists of four parts: an encoder, a fusion module, a decoder, and a task output head. The encoder includes a 4-stage Swing Transformer structure, responsible for progressive feature extraction and downsampling. The fusion module includes a cross-scale attention fusion module and a context aggregation module. During the encoding process, the cross-scale attention fusion module enables features of different resolutions to interact, and the context aggregation module performs adaptive weighted fusion of multi-scale features. The cross-scale attention fusion module is inserted between the four stages of the encoder. It allows features from different stages to perform multi-head cross-attention calculations, i.e., the query of stage i can perform attention calculations with the key and value of stage j. The cross-scale attention fusion module solves the problem in multi-scale fusion. First, it calculates the "information content" score of features at each scale, based on the variance of the feature gradient and the entropy of the feature distribution. Second, it uses an MLP network to predict pixel-level context relevance weights and learns the fusion weights of different scales through fully connected layers. Finally, it uses Softmax normalization weights to perform weighted summation and fusion of features at different scales. The decoder is a position-aware decoder, responsible for progressive upsampling to restore spatial resolution. Relative position encoding and distance transformation constraints are explicitly introduced during the decoding process; S3 Model Training: The model built in S2 is trained, validated, and tested based on the samples obtained in S1; S4 extracts roads from the remote sensing images under test based on the model trained in S3.

2. The road remote sensing extraction method based on an optimized multi-head attention mechanism neural network according to claim 1, characterized in that: The high-resolution satellite remote sensing raw image described in S1 has a resolution of 0.5m.

3. The road remote sensing extraction method based on an optimized multi-head attention mechanism neural network according to claim 1, characterized in that: In S1, the image is labeled by manual visual interpretation. The labeling result is a binary mask, where a pixel value of 255 represents a road pixel and a pixel value of 0 represents a non-road pixel.

4. The road remote sensing extraction method based on an optimized multi-head attention mechanism neural network according to claim 1, characterized in that: In S2, the encoder's first stage includes a patch embedding layer, which converts the raw image pixels into a sequence.

5. The road remote sensing extraction method based on an optimized multi-head attention mechanism neural network according to claim 1, characterized in that: In S2, the encoder downsamples by 4 times in the first stage and is processed using 2 Swing Transformer modules, each containing 3 attention heads; the encoder downsamples by 8 times in the second stage and is processed using 2 Swing Transformer modules, each containing 6 attention heads; the encoder downsamples by 16 times in the third stage and is processed using 6 Swing Transformer modules, each containing 12 attention heads. The encoder downsamples by 32x in the fourth stage and uses two SwinTransformer modules for processing, with each block containing 24 attention heads.

6. The road remote sensing extraction method based on an optimized multi-head attention mechanism neural network according to claim 5, characterized in that: Downsampling is performed between stages using Patch Merging. The Swin Transformer module internally includes: layer normalization, multi-head self-attention, residual connections, layer normalization, MLP feedforward network, and residual connections.

7. The road remote sensing extraction method based on an optimized multi-head attention mechanism neural network according to claim 1, characterized in that: The total loss in S2 is based on Dice loss. and cross-entropy loss combination: ; Where: X represents the predicted set of road pixels, and Y represents the set of actually labeled road pixels; ; Where: N is the total number of pixels, y i p is the true label of the i-th pixel. i This represents the predicted probability.

8. The road remote sensing extraction method based on an optimized multi-head attention mechanism neural network according to claim 1, characterized in that: S3 uses the AdamW optimizer for model training and employs a cosine annealing learning rate decay strategy.

Citation Information

Patent Citations

  • Remote sensing image road extraction method and system based on multi-scale context sensing network

    CN119339239A

  • Remote sensing image semantic segmentation method and system based on prototype learning

    CN120953992A