A multi-task remote sensing image road damage assessment method and system

Abstract

The application relates to a multi-task remote sensing image road damage evaluation method and system, and relates to the field of remote sensing image analysis.The method comprises the following steps: acquiring a road damage double-time image, and labeling the road damage double-time image; inputting the labeled road damage double-time image, extracting multi-scale features through a multi-scale feature extraction module, and acquiring a multi-scale double-time feature map; modeling the multi-scale double-time feature map in space-time global information; performing feature reconstruction through step-by-step upsampling and dense connection; setting a loss function, inputting the reconstructed feature image into two prediction heads respectively, using the reconstructed features to generate a road change detection image, and simultaneously using class embedding to predict a road damage level.The application fuses road damage evaluation on the basis of road change detection, promotes the multi-tasks, and achieves better road evaluation effect; meanwhile, the application can effectively extract a continuously changed road area and accurately identify the subtle continuous features of road change.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image analysis technology, specifically to a method and system for assessing road damage using multi-task remote sensing images. Background Technology

[0002] Road damage assessment based on remote sensing imagery is a task built upon the change detection task in the field of remote sensing. It involves comparing pre-disaster and post-disaster images to identify damaged areas and assess the level of road damage.

[0003] Currently, existing change detection and damage assessment methods mainly focus on buildings, land cover, and forest area. For example, Chinese patent CN115861703A provides a method and device for remote sensing image change detection based on multi-scale CNN-Transformer. By inputting dual-temporal remote sensing images into a multi-scale CNN-Transformer encoder, the encoder extracts features from the dual-temporal remote sensing images to obtain multi-scale global-local features. Then, a cascaded decoder is used to fuse and reconstruct the multi-scale global-local features of the dual-temporal remote sensing images layer by layer to obtain a change feature map. Finally, a classifier classifies the change feature map pixel by pixel to obtain the change detection result map. For example, Chinese patent CN115690002A discloses a method and system for detecting changes in remote sensing images based on Transformer and dense feature fusion. It uses a ResNeT18-based backbone network for initial feature extraction, then generates two sets of token sequences through semantic tokenization. These sequences are concatenated and input into a Transformer encoder. An improved multi-head attention mechanism, while maintaining diversity, enables interaction between different heads, helping the network model global image information in a token-based space. The context-rich tokens are then segmented and input into a Siamese Transformer decoder, reprojected back into pixel space, resulting in two enhanced feature maps. A dense feature fusion module helps the network generate change regions and enhance edge information. Finally, feature fusion and a shallow convolutional network generate the resulting change map.

[0004] For change detection tasks, existing methods mainly target features such as buildings, land cover, and forest area. These features are geographically dispersed and fixed, while roads are continuous and tortuous. Therefore, existing methods have some problems in extracting change areas of roads. For example, they cannot effectively extract continuous road change areas, and the detection results often show road discontinuities or false detections, thus failing to accurately capture the fine-grained and continuous features of road changes. Furthermore, for road damage assessment tasks, existing methods neglect the important semantic information of damage level. Existing datasets lack semantic information with damage levels, and existing methods have not fully explored road features, nor have they established a research paradigm for this multi-task approach of change detection and damage assessment. In addition, existing change detection and damage assessment methods suffer from loss of detail and low accuracy in road change detection, leading to inaccurate damage assessment results. Therefore, accurately extracting features of road changes and damage levels is crucial for achieving road damage assessment. Summary of the Invention

[0005] To accurately extract features of road changes and damage levels for road damage assessment, this invention provides a multi-task remote sensing imagery road damage assessment method and system. This invention integrates transformer and UET technologies, establishing a complete end-to-end multi-task model. Utilizing pre-disaster and post-disaster images, it learns rich multi-scale road change and damage features to detect road change areas and assess damage levels, thereby reducing errors in road damage assessment tasks.

[0006] The technical solution adopted by this invention to solve the technical problem is as follows:

[0007] The present invention provides a multi-task remote sensing image road damage assessment system, comprising:

[0008] The road image data preprocessing module is used to annotate the dual-temporal images of road damage and input the annotated dual-temporal images of road damage into the multi-scale feature extraction module;

[0009] The multi-scale feature extraction module is used to extract multi-scale features from the labeled dual-temporal images of road damage, and input the obtained multi-scale dual-temporal feature map into the image feature space information modeling module;

[0010] The image feature spatial information modeling module is used to perform spatiotemporal global information modeling on multi-scale dual-temporal feature maps and output the road damage level.

[0011] The feature map reconstruction module is used to reconstruct the feature map after modeling the spatiotemporal global information, realizing the reconstruction process from the feature map to the change map, and outputting the road change detection image.

[0012] The present invention provides a method for assessing road damage using multi-task remote sensing imagery, implemented using the aforementioned multi-task remote sensing imagery road damage assessment system. The method includes the following steps:

[0013] Step 1: Obtain two temporal images of road damage and annotate them;

[0014] Step 2: Input labeled dual-temporal images of road damage, extract multi-scale features through the multi-scale feature extraction module, and obtain a multi-scale dual-temporal feature map;

[0015] Step 3: Perform spatiotemporal global information modeling on multi-scale dual-temporal feature maps at different scales;

[0016] Step 4: Reconstruct features through progressive upsampling and dense connections;

[0017] Step 5: Set the loss function, input the reconstructed feature images into two prediction heads respectively, use the reconstructed features to generate road change detection images, and use class embedding to predict the road damage level.

[0018] Furthermore, in step two, when two single road damage bitemporal image pairs are input, four bitemporal feature maps of different scales are generated through convolution operations. The larger-scale bitemporal feature map contains low-level information on changes in road geographical location, while the smaller-scale bitemporal feature map contains high-level information on the degree of road damage.

[0019] Furthermore, in step two, the multi-scale feature extraction module employs a ResNet-18 network, a ResNet-50 network, or a VGGNet network.

[0020] Furthermore, in step three, the spatiotemporal global information of the multi-scale dual-temporal feature map is modeled using transformation and residual jump connections. The transformation and residual jump connections include a transformation connection module and a residual connection module. First, the multi-scale dual-temporal feature map is input to the transformation connection module and mapped into two vectors of length L. After concatenation, the vectors are fed into the Transformer model for overall modeling, fully learning the differences in geographic information before and after the disaster, and obtaining a single feature map containing rich contextual information. At the same time, the multi-scale dual-temporal feature map is input to the residual connection module for processing to obtain a low-level geographic feature map that retains the original features. The single feature map and the low-level geographic feature map are then fused to obtain a complete feature image.

[0021] Furthermore, the Transformer model adopts either the DeiT architecture or the Swing-Transformer architecture.

[0022] Furthermore, in step four, the feature map reconstruction module is used to realize the reconstruction process from the feature map to the transformation map. The feature map reconstruction module is implemented by a dense upsampling module.

[0023] Furthermore, in step five, the loss function is set by combining Focal loss and Dice loss.

[0024] Furthermore, in step five, the two prediction heads are a change detection prediction head and a damage assessment prediction head. For the change detection prediction head, the reconstructed feature image is directly upsampled through multiple convolutional layers and the output channels are adjusted. Then, the softmax and argmax functions are used to visualize the binary change map of the road and generate a road change detection image. For the damage assessment prediction head, in the modeling process of the maximum-scale dual-temporal feature map, class embedding is introduced. After fully learning the damage features through the Encoder module in the Transformer model, the features are split and input into the multilayer perceptron to predict the road damage level.

[0025] The beneficial effects of this invention are:

[0026] 1. Multi-task Fusion: This invention integrates road change detection with road damage assessment, establishing a unified end-to-end multi-task model that combines these two tasks. Unlike existing technologies, most focus on a single task, namely either road change detection or road damage assessment. This invention is the first to tightly integrate road change detection and road damage assessment tasks into a complete framework. By combining these two tasks, the road change information generated by road change detection and the road damage information generated by road damage assessment mutually reinforce each other, achieving better road assessment results and providing more practical applications in real-world scenarios.

[0027] Specifically, the road change detection task provides valuable contextual and change information, helping to accurately assess the extent of road damage; while the damage assessment task provides detailed information about the degree of road damage, helping to improve the accuracy of road change detection. By combining these two tasks to construct an end-to-end multi-task model, the accuracy and robustness of road damage assessment can be effectively improved. By comprehensively considering both road change detection and road damage assessment tasks, a more comprehensive understanding and assessment of road changes can be achieved, providing accurate decision support for related fields. This approach, which comprehensively considers multiple tasks, is the first of its kind in existing road-related technologies and is innovative and unique in the field of road damage assessment.

[0028] 2. For damage assessment tasks, this invention divides the degree of road damage into three categories (no or minor / moderate / severe damage). This classification can provide a quantitative assessment of the degree of road damage and output the level of road damage during the actual inference stage of the model.

[0029] 3. For road change detection tasks, deep learning-based methods typically employ CNNs (Convolutional Neural Networks) and Transformer models. CNNs extract higher-level feature representations step by step through multiple layers of convolution and pooling operations, forming a hierarchical structure, with the main advantage being the capture of visual and spatial relationships.

[0030] Transformers, through their self-attention mechanism, capture the relationships between different positions in the input sequence and possess a global perspective. Their main advantage lies in effectively capturing global contextual information and modeling long-range dependencies. Based on these advantages, this invention uses CNNs for initial image feature extraction to obtain multi-scale features. Furthermore, by combining Transformers and UET for spatial information modeling, features with rich contextual information can be obtained at different scales. By combining these two approaches, this invention can better facilitate road damage assessment tasks and help improve assessment results.

[0031] 4. In addition, for road change detection tasks, this invention proposes modules targeting road geographic features (road image data preprocessing module, multi-scale feature extraction module, image feature spatial information modeling module, and feature map reconstruction module), which can more comprehensively extract and utilize the features of dual-temporal remote sensing images, and realize dense feature fusion and upsampling processes. At the same time, an appropriate loss function is used to help training, which can effectively extract continuously changing road areas, accurately identify subtle continuous features of road changes, improve the accuracy of road damage assessment, and thus more accurately detect road changes and assess road damage.

[0032] 5. This invention introduces Transformer and residual connection, which can better utilize spatiotemporal global information. In the design of the loss function, Focal loss and Dice loss are combined to better adapt to imbalanced datasets and regression problems. It has stronger model expressive power and higher accuracy in road change detection and road damage assessment tasks. Attached Figure Description

[0033] Figure 1 This is a flowchart of a multi-task remote sensing image road damage assessment method according to the present invention.

[0034] Figure 2 This is a block diagram illustrating the structural components of a multi-task remote sensing image road damage assessment system according to the present invention. Detailed Implementation

[0035] The present invention will be further described in detail below with reference to the accompanying drawings.

[0036] In a first aspect, the present invention provides a multi-task remote sensing image road damage assessment system, such as... Figure 2 As shown, the system includes: a road image data preprocessing module, a multi-scale feature extraction module, an image feature spatial information modeling module, and a feature map reconstruction module. The road image data preprocessing module is mainly used to annotate dual-temporal images of road damage. The multi-scale feature extraction module (MFE) is used to extract multi-scale features from the annotated dual-temporal images of road damage. The image feature spatial information modeling module is used to perform spatiotemporal global information modeling on the obtained multi-scale dual-temporal feature maps, outputting the road damage level. After performing spatiotemporal global information modeling on the multi-scale dual-temporal feature maps, the feature map reconstruction module uses progressive upsampling and dense connections to reconstruct features, realizing the reconstruction process from the feature map to the change map, and outputting a road change detection image.

[0037] Secondly, this invention provides a method for assessing road damage using multi-task remote sensing imagery, such as... Figure 1 As shown, firstly, dual-temporal images of road damage (including pre-disaster and post-disaster images) are acquired, and the dual-temporal images are labeled using a road image data preprocessing module. Then, a multi-scale feature extraction module (MFE) is used to obtain multi-scale features of the dual-temporal images of road damage. Next, a spatiotemporal global information modeling module is used to model the obtained multi-scale dual-temporal feature maps at different scales. In addition, a feature map reconstruction module is used to reconstruct the feature maps into change maps. Finally, a loss function is set by combining Focal loss and Dice loss, and the reconstructed feature images are input into two prediction heads respectively. The reconstructed features are used to generate road change detection images, and class embedding is used to predict the road damage level.

[0038] This invention presents a multi-task remote sensing imagery-based road damage assessment method that combines road change detection and road damage assessment, expanding the application scenarios of road-related technologies and defining a new technological paradigm. This integrated task approach not only provides a more comprehensive understanding of the characteristics of road changes and damage but also effectively improves the accuracy and efficiency of road assessment. Furthermore, this invention establishes an end-to-end multi-task model that fully utilizes the multi-scale characteristics of pre- and post-disaster road changes to achieve the detection of road change areas and the assessment of damage levels.

[0039] The present invention provides a method for assessing road damage using multi-task remote sensing imagery, which specifically includes the following steps:

[0040] Step 1: Obtain dual-temporal images of road damage, including pre-disaster and post-disaster images, and annotate the dual-temporal images of road damage using the road image data preprocessing module.

[0041] During the annotation process, this invention utilizes pre-disaster images to locate road positions and post-disaster images to detect and annotate changed road areas. Then, by comparing the road damage characteristics between pre- and post-disaster images, the road damage is categorized into three types (no or minor damage / moderate / severe damage). Specifically, by comparing pre- and post-disaster images, the characteristics of road changes are observed and analyzed: for no or minor damage, the road may still maintain a relatively good shape with only some minor ground damage; in cases of moderate damage, the road may show more obvious damage, such as road surface collapse or slope landslides; and in cases of severe damage, the road has suffered significant damage, is impassable, and requires large-scale repair and restoration work. Therefore, this invention categorizes the degree of road damage into three types: no or minor damage, moderate damage, and severe damage.

[0042] Step 2: Input labeled road damage images in two temporal phases, and use the multi-scale feature extraction module (MFE) to obtain multi-scale two-temporal feature maps.

[0043] Specifically, the multi-scale feature extraction module (MFE) can be implemented using networks such as ResNet-18, ResNet-50, and VGGNet, but is not limited to these.

[0044] When using the ResNet-18 network, taking a single road damage bitemporal image pair as an example, for two input images of size w*h (w is the image width and h is the image height), the first three layers of the ResNet-18 network and a regular convolutional block are used to generate four bitemporal feature maps of different scales (w / 2*h / 2, w / 4*h / 4, w / 8*h / 8, w / 16*w / 16) through convolution operations. The larger-scale bitemporal feature map contains low-level road geographic location information, while the smaller-scale bitemporal feature map contains high-level road damage information.

[0045] This invention designs a method to extract multi-scale features from dual-temporal images of road damage. These features contain more information about geographical location and damage severity at different scales than single-scale features, enabling the model to more comprehensively understand and process remote sensing images. Simultaneously, through multi-scale feature extraction and multi-modal feature fusion, the model can obtain multi-scale features containing geographical location and damage severity information, accurately capturing fine-grained and continuous features of road changes, thereby improving the accuracy and robustness of road segmentation and analysis.

[0046] Step 3: Use the image feature spatial information modeling module to perform spatiotemporal global information modeling on the multi-scale dual-temporal feature maps obtained above at different scales.

[0047] Specifically, the image feature spatial information modeling module can employ Trans and Res Skip Connection (TRSC) to model the spatiotemporal global information of multi-scale dual-temporal feature maps at different scales. The Trans and Res Skip Connection (TRSC) mainly includes a Trans.Block module and a Residual Connection module.

[0048] (Res.Block); First, the multi-scale dual-temporal feature map is input into the Transformer Connector module (Trans.Block). The Transformer Connector module (Trans.Block) maps the multi-scale dual-temporal feature map into two vectors of length L. After concatenation, they are fed into the Transformer model for overall modeling, fully learning the differences in geographic information before and after the disaster, and obtaining a single feature map containing rich contextual information. At the same time, the multi-scale dual-temporal feature map is input into the Residual Connector module (Res.Block). After processing by the Residual Connector module (Res.Block), a low-level geographic feature map that retains the original is obtained. Finally, the single feature map and the low-level geographic feature map obtained above are fused to obtain a complete feature image.

[0049] This invention, through the Transformer and Residual Connectivity modules (Trans.Block and Res.Block) and feature fusion, can fully utilize Transformer and Residual Connectivity to model features holistically, while preserving the original low-level geographic features and extracting the differences in geographic information before and after a disaster. This improves the accuracy and robustness of road analysis, which helps enhance the model's ability to learn from the differences in geographic information before and after a disaster.

[0050] In this invention, spatial information modeling of feature maps can also be implemented using architectures such as DeiT (Data-efficient imageTransformers) and Swin-Transformer.

[0051] Step 4: Reconstruct features through progressive upsampling and dense connections.

[0052] Specifically, the complete feature image obtained above is input into the feature map reconstruction module. Feature fusion is performed through stepwise upsampling and dense connections to reconstruct the feature map from the feature map into a transformation map. This allows each scale of the feature map to receive feature maps from other scales, and this dense upsampling method better preserves the complexity and diversity of features. The feature map reconstruction module can specifically be a dense cased upsample module (DCU). The stepwise upsampling and dense connections used in the feature map reconstruction stage offer advantages in feature reuse and reducing gradient vanishing. To achieve feature fusion, each scale of the feature map can receive feature maps from all previous scales, ultimately resulting in the reconstructed feature image.

[0053] In this invention, the reconstruction of feature maps can also be achieved by direct upsampling without interconnection.

[0054] Step 5: Set the loss function, input the reconstructed feature images into two prediction heads respectively, use the reconstructed features to generate road change detection images, and use class embedding to predict the road damage level.

[0055] Regarding the setting of the loss function, the commonly used cross-entropy loss function has some problems. First, it is not sensitive to the distribution of positive and negative samples. For road change detection, the distribution of the actual changed area and the unchanged area is usually uneven, which makes it impossible for the cross-entropy loss to reasonably adjust the weight ratio between the two, thus reducing accuracy. Second, the cross-entropy loss function does not focus on the regression problem. In the road change detection task, IoU is a key indicator that directly measures the overlap between the predicted change area and the actual change area. Therefore, it is very important to perform regression analysis on the predicted change area.

[0056] Therefore, to address the aforementioned problems, this invention employs a combination of Focal loss and Dice loss to set the loss function. Focal loss is used to solve the imbalanced data problem by assigning higher weights to samples that are difficult to classify, thereby reducing the impact of easily classified samples on the overall loss. Dice loss exhibits good performance when dealing with highly imbalanced data and is highly sensitive to small differences between the predicted and true masks.

[0057] In addition, this invention employs a loss function that combines Focal loss and Dice loss, which is optimized for the characteristics of road change detection and road damage assessment tasks. Compared with simple pixel-by-pixel classification, the more detailed loss function design can better handle imbalanced datasets and regression problems, thereby improving the model's performance in road change detection and road damage assessment.

[0058] Specifically, for the change detection prediction head in the two prediction heads, the reconstructed feature image is directly upsampled through multiple convolutional layers and the output channels are adjusted. Then, the softmax and argmax functions are used to visualize the binary change map of the road, thereby generating the road change detection image.

[0059] Among them, for the damage assessment prediction head in the two prediction heads, class embedding is introduced in the modeling process of the maximum scale dual temporal feature map. After fully learning the damage features through the Encoder module in the Transformer model, the features are split and input into the Multi-Layer Perceptron (MLP) to predict the road damage level.

[0060] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for assessing road damage using multi-task remote sensing imagery, characterized in that, A multi-task remote sensing imagery road damage assessment system is used, comprising: The road image data preprocessing module is used to annotate the dual-temporal images of road damage and input the annotated dual-temporal images of road damage into the multi-scale feature extraction module; The multi-scale feature extraction module is used to extract multi-scale features from the labeled dual-temporal images of road damage, and input the obtained multi-scale dual-temporal feature map into the image feature space information modeling module; The image feature spatial information modeling module is used to perform spatiotemporal global information modeling on multi-scale dual-temporal feature maps and output the road damage level. The feature map reconstruction module is used to reconstruct the feature map after modeling the spatiotemporal global information, realize the reconstruction process from the feature map to the change map, and output the road change detection image; The method includes the following steps: Step 1: Obtain two temporal images of road damage and annotate them; Step 2: Input labeled dual-temporal images of road damage, extract multi-scale features through the multi-scale feature extraction module, and obtain a multi-scale dual-temporal feature map; Step 3: Perform spatiotemporal global information modeling on multi-scale dual-temporal feature maps at different scales; The spatiotemporal global information of multi-scale dual-temporal feature maps is modeled using transformation and residual jump connections. The transformation and residual jump connections include a transformation connection module and a residual connection module. First, the multi-scale dual-temporal feature map is input to the transformation connection module, which maps it into two vectors of length L. These vectors are then concatenated and fed into the Transformer model for overall modeling, fully learning the differences in geographic information before and after the disaster to obtain a single feature map containing rich contextual information. Simultaneously, the multi-scale dual-temporal feature map is input to the residual connection module for processing, resulting in a low-level geographic feature map that retains the original features. Finally, the single feature map and the low-level geographic feature map are fused to obtain a complete feature image. Step 4: Reconstruct features through progressive upsampling and dense connections; Step 5: Set the loss function by combining Focal loss and Dice loss, input the reconstructed feature image into two prediction heads respectively, use the reconstructed features to generate road change detection images, and use class embedding to predict the road damage level. The two prediction heads are a change detection prediction head and a damage assessment prediction head. For the change detection prediction head, the reconstructed feature image is directly upsampled through multiple convolutional layers and the output channels are adjusted. Then, the softmax and argmax functions are used to visualize the binary change map of the road to generate a road change detection image. For the damage assessment prediction head, in the modeling process of the maximum-scale dual-temporal feature map, class embedding is introduced. After fully learning the damage features through the Encoder module in the Transformer model, the features are split and input into the multilayer perceptron to predict the road damage level.

2. The method for assessing road damage using multi-task remote sensing imagery according to claim 1, characterized in that, In step two, when two single road damage bitemporal image pairs are input, four bitemporal feature maps of different scales are generated through convolution operations. The larger-scale bitemporal feature map contains low-level information on changes in road geographical location, while the smaller-scale bitemporal feature map contains high-level information on the degree of road damage.

3. The method for assessing road damage using multi-task remote sensing imagery according to claim 1, characterized in that, In step two, the multi-scale feature extraction module uses a ResNet-18 network, a ResNet-50 network, or a VGGNet network.

4. The method for assessing road damage using multi-task remote sensing imagery according to claim 1, characterized in that, The Transformer model adopts either the DeiT architecture or the Swing-Transformer architecture.

5. The method for assessing road damage using multi-task remote sensing imagery according to claim 1, characterized in that, In step four, the feature map reconstruction module is used to reconstruct the feature map into a transformation map. The feature map reconstruction module is implemented using a dense upsampling module.

Images