Method for constructing road crack segmentation system fusing infrared and visible light images

Abstract

Disclosed is a method for constructing a road crack segmentation system fusing infrared and visible light images, which belongs to the technical field of crack detection. The present invention aims at solving the problem of insufficient identification of little cracks under different lighting and weather conditions. The method includes the following steps: improving an RTFormer model; proposing a visible light and infrared fusion module IFF; obtaining a DDAPPM: improving a deep aggregation pyramid pooling module (DAPPM); and performing training by using a knowledge distillation method. According' to the present invention, by introducing an infrared image branch, environmental constraints of relying solely on the visible light image are overcome effectively, and. a high identification rate can still be maintained under the condition of insufficient light or reflection.

Description

TECHNICAL FIELD [Ol] The present invention relates to a method for constructing a crack segmentation system fusing infrared and visible light images, which belongs to the technical field of crack detection. BACKGROUND ART

[02] With the rapid development of transportation infrastructure, the detection of apparent harms of roads and bridges has become increasingly important. Early detection of cracks plays an important role in prolonging the service life of roads and ensuring the traffic safety. In recent years, the image segmentation technology based. on deep learning has made remarkable achievements in the field of crack detection. An image segmentation technology in the field of computer vision is a basic key technology. Especially in the routing inspection of roads and bridges, highaccuracy image segmentation tasks such as the crack detection are of great significance to identify and prevent potential structural problems in advance. Although the application of the deep learning technology, especially the convolutional neural network (CNN) and Transformer models, has significantly improved the image segmentation accuracy, the detection accuracy and robustness under complex lighting and diverse environment conditions still face challenges. These models still have defects in processing images in complex environments, such as identifying little cracks under different lighting and weather conditions.

[03] Current studies pay little attention to how to effectively fuse infrared and visible light images to improve the crack detection performance under changing lighting and different weather conditions. Relying solely on the visible light image may be limited by the environment such as shadow occlusion, and texture similarity, leading to the decrease of identification rate. Furthermore, traditional feature fusion. methods such as direct image mosaic are easy to cause information redundancy of a feature space, which increases the computational burden of subsequent processing.

[04] Ma, a former scholar, fused infrared and visible light images by using a gradient transfer and total variation minimization method. to improve the visibility and information richness of the images under various lighting conditions; DenseFuse (a novel deep learning framework for fusing infrared and visible light images) performs feature extraction and fusion by using the deep learning framework to improve the feature integration capacity of the infrared and visible light images; and although DenseFuse and RTFormer are both advanced technologies in the field of image fusion, their technological paths are different; DenseFuse focuses on the optimization of feature extraction process by using' a DenseBlock structure, while RTFormer processes image data by using an attention mechanism of Transformer; and DenseFuse and RTFormer both have own advantages in the field of the image fusion, and are suitable for different application scenes.

[05] Therefore, it is urgent to put forward to a method for constructing a road crack segmentation system fusing infrared and visible light images to solve the above technical problems. SUMMARY

[06] A purpose of the present invention is to provide a method for constructing a road crack segmentation system fusing infrared and visible light images to solve the problem. of insufficient identification of little cracks under different lighting and weather conditions. The present invention is briefly described below to facilitate the basic understanding of some aspects of the present invention. It should be noted that the brief description is not exhaust description of the present invention. The description is not intended to determine a key or important part of the present invention, nor is it intended to limit the scope of the present invention.

[07] The present invention adopts a technical solution as follows:

[08] A method for constructing a road crack segmentation system fusing infrared and visible light images includes the following steps:

[09] step 1: improving an RTFormer model;

[10] step 2: proposing a visible light and infrared fusion module IFF;

[11] step 3: obtaining a DDAPPM: improving a deep aggregation pyramid pooling module (DAPPM); and

[12] step 4: performing training' by using a knowledge distillation method.

[13] Preferably, step 1 includes the following steps:

[14] step 11: introducing an infrared branch:

[15] adding a branch specially for processing an infrared image into the RTFormer model, where the infrared image can provide different information from the visible light image, such as temperature distribution, and is especially suitable for identifying cracks in low light or under other complex lighting conditions;

[16] step 12: optimizing feature fusion:

[17] fusing infrared and visible light image features at layer3 of the RTFormer model, where layer3, as a deeper network layer, processes advanced feature information; and experiments demonstrate that fusing at this layer maximizes the synergy between the two image features, thereby improving crack detection accuracy; and

[18] step 13: deepening a network structure: adding extra layers compression3e and layer31e respectively into the infrared. image branch and the visible light branch to enhance the depth and feature extraction capacity of the network; and these new layers enable the network to better capture and fuse complex features from different sources while maintaining relative complexity.

[19] Preferably, step 2 includes the following steps:

[20] step 21: performing feature enhancement:

[21] preprocessing and enhancing, by an IFF, features from the visible light and infrared images by using an attention mechanism, where the attention mechanism can automatically identify and emphasize the most important information in the two images, for example, in the crack detection, the attention is concentrated on edges of cracks or in an area with obvious contrast with the environment;

[22] to be specific, the visible light image is preprocessed to improve the contrast, and at the same time, operations such as denoising and enhancement are performed on the infrared image; and in the feature fusion process, a channel attention operation is used to amplify important features and suppress irrelevant features;

[23] step 22: performing fusion strategy:

[24] fusing the enhanced features by means of an add operation rather than simple mosaic, and a fusion mode is adding by elements, which facilitates the integration of multiscale features, enhancing overall detection performance; and this method effectively combines the local and global information, keeps the number of channels unchanged, reduces the complexity and computational burden of the model, and avoids the problem of feature redundancy brought by the traditional fusion method;

[25] step 23: integrating modules:

[26] integrating the IFF into a backbone of an RTFormer network to ensure the effective fusion of features of the visible light and infrared data in the whole network, thereby improving the capacity of the model for identifying the cracks; and

[27] to be specific, a processed feature fusion result is used as a part of the IFF to further remove interference features and enhance the fused features; and an output of the IFF is integrated into the backbone of the RTFormer network to realize more efficient feature fusion and target detection.

[28] Preferably, step 3 includes the following steps:

[29] step 31: improving a DAPPM structure:

[30] based on the original DAPPM, adding denser connection and pooling operations to better aggregate the features from different network depths; and this design is helpful for the model to capture richer context information, especially when processing the large-scale images with complex backgrounds, and thus obtaining a DDAPPM;

[31] step 32: reusing the features:

[32] through an improved dense connection mode, each layer is connected not only with its immediate predecessor but also with all previous layers, ensuring full interconnectivity, for example, a third layer receiving the feature from the first layer is connected.to the first layer, that is, the extracted information is re-transmitted to the first layer, i.e. feature reuse, and in this way, the information stream and feature reuse can be maximized; this method is helpful to improve the characterization capacity of features, especially' the distinction between cracked areas and non-cracked areas in the images; and the layer refers to a layer inside the RTFormer; and

[33] step 33: performing multiscale information fusion:

[34] DDAPPM adopts different scales of pooling layers, which can process the information of multiple scales; the multiscale fusion strategy' enables the model to better understand an overall scene structure, while retaining key details; the pooling layers reduce a spatial dimension of a feature map by using a pooling operation; the pooling layer and the pooling operation are indispensable on the function and. the structure; and. the pooling' operation. is a core computational process of pooling' layer, and. the pooling layer is a specific layer type for implementing the pooling operation.

[35] Preferably, step 4 includes the following steps:

[36] step 41: training a teacher model:

[37] firstly, training a teacher model, where the model has numerous parameters and high complexity, and can achieve high performance for training the data; and the teacher model can be trained in a process of training an ordinary deep learning model, which means that the best model improved so far is defined as the teacher model; and

[38] step 42: designing' and training a student model: designing a student model by means of network structure design, where the student model is simplified in structure, but learns from the teacher model through a distillation process; in the training process, the student model not only learns a standard learning target, but also learns to imitate an output of the teacher model, which is implemented by adding a distillation loss function; the student model is smaller than the teacher model; and in the knowledge distillation, a KL divergence loss is typically used to measure the difference between the probability distributions of the student and teacher model outputs, the KL divergence loss is used for a soft label of the teacher model output and a cross entropy loss is used for a true label.

[39] The present invention has the following beneficial effects:

[40] According to the present invention, by introducing the infrared. image branch, environmental constraints of relying solely' on the visible light image are overcome effectively, such as the vision identification problem of shadow occlusion and texture similarity, and a high identification rate can still be maintained under the conditions of insufficient light or reflection.

[41] In the present invention, by adding the infrared imaging technology, the visualization of cracks can be enhanced by using spectral information of different wavelengths, thereby improving the detection accuracy and reliability.

[42] In the present invention, by applying the knowledge distillation technology in the model design, the number of parameters and computational requirements of the model can be reduced effectively, so that the model is more suitable for running on a device with limited resources; and this is particularly important for the onsite application, especially in a mobile monitoring device or a remote monitoring system, and the realtime and efficient crack detection can be realized. BRIEF DESCRIPTION OF THE DRAWINGS

[43] FIG. 1 is a structural diagram. of a road crack segmentation system fusing infrared and visible light images.

[44] FIG. 2 is an architecture diagram of an RTFormerF4 network.

[45] FIG. 3 is a diagram of an interaction fusion module.

[46] FIG. 3a is a schematic structural diagram. of an interaction fusion module.

[47] FIG. 3b is a schematic structural diagram. of an interaction feature fusion module.

[48] FIG. 4 is an improved structural diagram of a DAPPM.

[49] FIG. 4a is a schematic structural diagram of a DAPPM.

[50] FIG. 4b is a schematic structural diagram of a DDAPPM.

[51] FIG. 5 is a flowchart. of a distillation learning process. DETAILED DESCRIPTION OF THE EMBODIMENTS

[52] To make the purpose, technical solutions, and advantages of the present invention more clear and understandable, the present invention is described below through embodiments shown in the accompanying' drawings. However, it should be noted that these descriptions are only exemplary, but not intended to limit the scope of the present invention. In the following description, descriptions of wellknown structures and technologies are omitted to avoid unnecessarily confusing concepts of the present invention.

[53] Embodiment I: The present embodiment is described with reference to FIG. 1 to FIG. 5. A method for constructing a road crack segmentation system fusing infrared and visible light images in the present embodiment includes the following steps:

[54] Step 1: A RTFormer 14 is a deep learning model and is configured for an image segmentation task; based on a Transformer architecture, the RTFormer model is configured to process sequence data, especially in the field of natural language processing; in the image segmentation, the Transformer can be used to effectively capture longdistance dependence in the image, thereby improving the segmentation accuracy; however, in the present invention, inspired by the DenseFuse, the RTFormer model is improved, and applied to the road crack segmentation based on the RTFormerslim model fusing the infrared and visible light images;

[55] In step 1, an original single visible light input channel is copied to an identical channel to process the infrared images, which includes the following steps:

[56] Step 11: Introduce an infrared branch:

[57] A.branch specially for processing the infrared image is added into the RTFormer model; the infrared image can provide different information from the visible light image, such as temperature distribution, and is especially suitable for identifying cracks in low light or under other complex lighting conditions;

[58] Step 12: Optimize feature fusion:

[59] Infrared and visible light image features are fused at layer3 of the RTFormer model; layer3, as a deeper network layer, processes advanced feature information; and experiments demonstrate that fusing at this layer maximizes the synergy between the two image features, thereby improving crack detection accuracy;

[60] Step 13: Deepen a network structure: extra layers (compression3e and layer31e, compression3e is a component in the RTFormer model and is configured to quickly extract local information of the image at first three stages, layer 31e is another important component in the RTFormer model and is configured. to efficiently obtain global context information required for a semantic segmentation task, and layer31e is an identical module in the network structure and. is named. for being" distinguished. from. the original module with the same name) are added respectively into the infrared image branch and the visible light branch, and configured to enhance depth and feature extraction capacity of the network; and these new layers enable the network to better capture and fuse the complex features from different sources while maintaining relative complexity;

[61] By fusing the infrared image, environmental constraints of relying solely on the visible light images, such as shadow occlusion and texture similarity, can be overcome effectively. The infrared images can provide temperature distribution information, maintain high performance in low light or under other complex lighting conditions, and improve the identification rate.

[62] Step 2: Propose a visible light and infrared fusion module, i.e. an interaction feature fusion (IFF) module;

[63] In order to effectively extract the features, the feature fusion module, i.e. IFF based on an attention mechanism is proposed; the module is configured to fuse the features from the visible light branch and the infrared branch in a backbone network; the structure of IFF is similar to the interaction fusion module (IFM), and the difference between the two lies in that IFM performs Concat on the visible light and infrared features respectively after the enhancement by using an attention feature; the IFF enhances the fused feature of the visible light and infrared light by using the attention feature, and uses an add operation to replace a Concat operation, which keeps the number of channels unchanged, and facilitates the reduction of subsequent computation; and the IFF utilizes the following strategies, which specifically includes the following steps:

[64] Step 21: Perform feature enhancement:

[65] The IFF preprocesses and enhances the features from the visible light and infrared images by using an attention mechanism; the attention mechanism can automatically identify and emphasize the most important information in the two images; and for example, in the crack detection, the attention is concentrated on the edges of cracks or in an area with an obvious contrast with the environment;

[66] To be specific, the visible light image is preprocessed to improve the contrast, and operations such as denoising and enhancement are performed on the infrared image; and in the feature fusion process, a channel attention operation (such as CBAM) is used to amplify important features and suppress irrelevant features;

[67] Step 22: Perform fusion strategy:

[68] The enhanced. features are fused. by using an add operation (add) rather than simple mosaic, and a fusion mode is adding by elements, which facilitates the integration of multiscale features, enhancing overall detection performance; and this method effectively combines the local and global information, keeps the number of channels unchanged, reduces the complexity and computational burden of the model, and also avoids the problem of feature redundancy brought by the traditional fusion method; and by replacing the Concat method with the add operation, the effective integration of features can be achieved without increasing the computational complexity, and the identification capacity and segmentation accuracy of the model can be improved;

[69] Step 23: Integrate modules:

[70] The IFF is integrated into a backbone of the RTFormer network to ensure the effective fusion of features of the visible light and infrared data in the whole network, thereby improving the capacity of the model for identifying the cracks; and

[71] To be specific, a processed feature fusion result is used as a part of the IFF to further remove interference features and enhance the fused features; and an output of the IFF module is integrated. into the backbone of the RTFormer network to realize more efficient feature fusion and target detection;

[72] Step 3: Obtain a DDAPPM: an improved DAPPM module; and

[73] The DDAPPM module is configured to fuse the features of a lowresolution infrared branch, extract the context information, input feature maps of different resolutions to a next convolutional layer, and finally connect the outputs of all layers; and in order to enhance the feature transfer, and increase a feature reuse rate, the DDAPPM is connected in a dense connection mode, and step 3 specifically includes the following steps:

[74] Step 31: Improve the DAPPM structure:

[75] Based on the original DAPPM (as shown in FIG. 4a, a deep fusion pyramid pooling module), more dense connection and pooling operations are added to better aggregate the features from different network depths (connection of different submodules in the module represents the integration of information in different depths); and this design is helpful for the model to capture richer context information, especially when processing the large-scale image with complex backgrounds, thus obtaining the DDAPPM (as shown in FIG. 4b, DDAPPM is an improved DAPPM, and the connection of the DDAPPM is denser than that of the DAPPAM, which improves the complementarity of information in network circulation);

[76] Step 32: Reuse the features:

[77] Through an improved dense connection mode, each layer is connected not only with its immediate predecessor but also with all previous layers, ensuring full interconnectivity, for example, a third layer receiving the feature from the first layer is connected to the first layer, that is, the extracted information is retransmitted to the first layer, i.e. feature reuse, and in this way, the information stream and feature reuse can be maximized; this method is helpful to improve the characterization capacity of features, especially in the distinction between cracked areas and non-cracked areas in the images; and the layer refers to a layer inside the RTFormer; and

[78] Step 33: Perform multiscale information fusion:

[79] The DDAPPM adopts different scales of pooling layers, which can process the information of multiple scales; the multiscale fusion strategy' enables the model to better understand an overall scene structure, while retaining key details; the pooling layers reduce a spatial dimension of the feature map by using a pooling operation; the pooling layer and the pooling operation are indispensable on the function and the structure; and the pooling operation is a core computational process of pooling layer, and the pooling layer is a specific layer type for implementing the pooling operation;

[80] Step 4: Perform training by using a knowledge distillation method;

[81] Knowledge distillation (FIG. 5) is a model training technology, which improves the performance of small networks by transferring knowledge from a large and complex network (a teacher. model) to a small and. simplified. network (a student model); this method is particularly suitable for applications that need to be deployed in resource constrained environments, such as mobile devices or edge computing devices; in an image segmentation task of detecting road cracks after infrared. and visible light fusion, the knowledge distillation can be used to reduce the model size and computational requirements while maintaining high accuracy; and a study, conducted by Liu, puts forward a structured knowledge distillation method, which is used for semantic segmentation, and improves the performance of the student model effectively by transferring the knowledge from the teacher model to the student model in combination with other steps of the present invention. Step 4 specifically includes the following steps:

[82] Step 41: Train a teacher model:

[83] Firstly, a teacher model (a large RTFormer model, and the RTFormerslim.model is a model in the paper of RTFormer: Efficient design for realtime semantic segmentation with transformer) is trained, and the model has numerous parameters and high complexity, and can achieve high performance for training the data; and the teacher model can be processed. in a}orocess of training' an ordinary deep learning model, which means that the best model improved so far is defined as the teacher model; and

[84] Step 42: Design and train a student model: a student model (a smaller and. more efficient RTFormer model) is designed by means of network structure design, and the model is simplified in structure, but learns from the teacher model through a distillation process; in the training process, the student model not only learns a standard learning target (such as a cross entropy loss), but also learns to imitate an output of the teacher model, which is implemented.by adding a distillation loss function (KLLoss), and the student model is smaller than the teacher model; and in the knowledge distillation, a KL divergence loss (KLDivLoss) is typically used. to measure the difference between the probability distributions of the student and teacher model outputs, the KL divergence loss is used for a soft label of the teacher model output, and a cross entropy loss is used for a true label.

[85] According to the present invention, it is verified by extensive experiments that the performance of the improved RTFormerFlO model on selfbuilt datasets is better than that of the basic model; especially an mIoU evaluation index of the present model is improved by 1% compared with the basic model, and reached 0.8541; the performance improvement is attributed to a fact that the model can identify and segment crack areas more accurately, and can maintain a high identification rate even under conditions of insufficient light or reflection; and by introducing the infrared image branch, environmental constraints of relying solely on the visible light image are overcome effectively, such as the visual identification problem of shadow occlusion and texture similarity; by introducing the infrared. imaging technology, the system can use the spectral information of different wavelengths to enhance the visualization of cracks, thereby improving the detection accuracy and reliability; by applying the knowledge distillation technology in the model design, the number of parameters and computational requirements of the model are reduced effectively, so that the model is more suitable for running on a resourcelimited device; and this is very important in the onsite application, especially in Hwbile Hmnitoring devices or remote monitoring systems, and the realtime and efficient crack detection can be realized.

[86] It should be noted that in the above embodiments, as long as the technical solutions that are not contradictory can be arranged and combined, those skilled in the art can exhaust all possibilities according to the mathematical knowledge of arrangement and combination, so the combined and arranged technical solutions are not explained one by one in the present invention, but it should be understood that the combined and arranged technical solutions have been disclosed in the present invention.

[87] The above description is only preferred embodiments of the present invention and is not used to limit the present invention. For those skilled in the art, various changes and variations of the present invention can be made. Any modifications, equivalent substitution, and improvements made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.

Claims

1. A method for constructing a road crack segmentation system that produces infrared and visible light images merges, including the following steps: Step 1: Improving an RTFormer backbone network work; step 2: proposing a visible light infrared fusion module IFF; Step 3: Improving a Deep Aggregation Piracy mid-pooling module (DAPPM); and Step 4: Performing Training Using of a knowledge distillation method.

2. Method for constructing a road crack segmentation system that produces infrared and visible light images merges according to claim 1, wherein step 1 is the following step pen includes: Step 11: Introducing an infrared branch: adding an infrared image branch in an RTFormer model; Step 12: Optimizing Aspect Fusion: the merging of infrared and visible light image axis pecten at layer 3 of the model; and Step 13: Deepening a Network Structure: The adding extra layers in the infrared respectively image branch and the visible light branch.

3. Method for constructing a road crack segmentation system that produces infrared and visible light images merges according to claim 2, wherein step 2 is the following step pen includes: Step 21: Performing Aspect Improvement: the pre-processing and improvement, by the IFF, of aspects from the visible light and infrared images by ge to use an attention mechanism; Step 22: Executing the merger strategy: merging the improved aspects via an add-on editing; and Step 23: Integrating modules: integrating the IFF into an RTFor backbone mer network.

4. Method for constructing a road crack segmentation system that produces infrared and visible light images merges according to claim 3, wherein step 3 is the next step pen includes: Step 31: Improving DAPPM: adding a tight connection and poolingbe operation based on DAPPM to obtain a DDAPPM; step 32: reusing aspects: the interconnection of layers; and Step 33: Performing Multi-Scale Information sie: the adoption of different scales of pooling layers for DDAPPM.

5. Method for constructing a road crack segmentation system that produces infrared and visible light images merges according to claim 4, wherein step 4 is the following step pen includes: Step 41: Training a Teacher Model: training a teacher model, in which teaching ten model is trained in a process of training a normal deep learning model; and Step 42: Designing and Training a Student Model share: designing a student model, learning from the teacher model through a distillation process, in which in the training process, the student model a standard training goal learns, and learns to output the teacher model imitate, and where the training process is implemented tarred by adding a distillation loss function. 000 FIG. 1