An unmanned aerial vehicle tunnel infrared and visible light edge orthogonal fusion crack segmentation method

Abstract

The application provides a UAV tunnel infrared and visible light edge orthogonal fusion crack segmentation method, which synchronously collects infrared images and RGB images of a tunnel lining surface through a UAV, and performs registration, normalization and cropping processing; then a UTICS-Net network is constructed for crack segmentation, the network comprising an infrared coding branch, an RGB coding branch, an SCO bimodal fusion structure and an RGEA decoding structure: the infrared coding extracts the thermal response, multi-scale weak edge and slender structure features of cracks in a low-illumination tunnel environment; the RGB coding extracts crack texture, color difference, geometric boundary and channel, and spatial attention features; the SCO structure enhances the spatial continuity of cracks in different directions; and a crack probability map and a binary crack mask are generated through the decoding stage. The method can improve the segmentation integrity and boundary accuracy of slender, intermittent and low-contrast cracks in a complex tunnel environment, reduce missed detection, false detection and segmentation fracture phenomena, and provide a reliable basis for crack length, width, area and disease grade evaluation.

Description

Technical Field

[0001] This invention relates to the field of intelligent detection technology for tunnel defects, and in particular provides a method for segmenting cracks in tunnels using infrared and visible light edge orthogonal fusion from an unmanned aerial vehicle (UAV). Background Technology

[0002] In specific application scenarios, tunnels typically suffer from insufficient lighting, wall reflections, water stains, dirt, dust, and complex concrete textures. Cracks themselves are characterized by their elongated, discontinuous, blurred edges, irregular shapes, and low contrast with the background, making it difficult to effectively distinguish crack features from the background texture. Related research on tunnel crack detection also indicates that real-time crack detection is significantly affected by low contrast, irregular shapes, and complex background textures; crack signals are often coupled with background interference such as water stains, uneven lighting, and rough concrete textures.

[0003] This leads to situations where relying solely on ordinary RGB images can easily result in missed detections, fragmented segmentation, or misidentification of background textures as cracks. Related research on tunnel crack detection also indicates that low contrast, irregular shapes, and complex background textures significantly affect the reliable identification of tunnel cracks.

[0004] The technical solution of this invention is as follows: During the process of UAV inspection of tunnels, crack features are automatically extracted from infrared and RGB dual-modal images, cracks are accurately separated from non-crack backgrounds, and complete, continuous, and clearly defined tunnel crack segmentation results are generated, thereby improving the automation level and reliability of tunnel defect detection.

[0005] The UTICS-Net network is used to collaboratively process infrared and RGB information. The infrared branch is used to capture the thermal response and edge differences of tunnel cracks in low-light environments, while the RGB branch is used to extract the texture, color changes, and geometric contours of the cracks. The SCO fusion structure is used to integrate the effective information from the two types of images, making the cracks stand out more in complex backgrounds and reducing crack breakage, boundary offset, and missegmentation. Summary of the Invention

[0006] Based on this, the present invention provides a method for segmenting cracks in tunnels using infrared and visible light orthogonal fusion, which can improve the segmentation integrity and boundary accuracy of slender, discontinuous, and low-contrast cracks in complex tunnel environments, and reduce missed detections, false detections, and segmentation breakage phenomena.

[0007] To achieve a method for orthogonally fusing infrared and visible light edge crack segmentation in UAV tunnels, the steps include:

[0008] S100. Dual-modal tunnel aerial image acquisition and standardization processing: Tunnel lining images acquired simultaneously by the RGB camera and infrared thermal imaging camera carried by the UAV are registered, normalized and cropped.

[0009] S200. Based on the infrared crack feature extraction of three IRMCE structural blocks, infrared crack edge enhancement features at three scales are obtained;

[0010] S300. RGB crack geometric attention feature extraction based on three RGEA structural blocks;

[0011] S400. Based on SCO, infrared-RGB dual-mode orthogonal continuous fusion;

[0012] S500. Crack mask generation based on bottleneck RGEA and three-level decoding RGEA.

[0013] Furthermore, the input for S100 dual-modal tunnel aerial image acquisition and standardization processing is: tunnel lining images simultaneously acquired by the RGB camera and infrared thermal imaging camera mounted on the UAV, denoted as:

[0014]

[0015]

[0016] During the training phase, manually annotated crack maps were also input:

[0017] In Y, crack pixels are marked as 1, and non-crack background pixels are marked as 0;

[0018] The processing steps include:

[0019] S110. For RGB images and infrared images Time synchronization, camera calibration, distortion correction, spatial registration, and scale normalization are performed to ensure that the two images have a pixel-level or near-pixel-level correspondence within the same tunnel wall area.

[0020] S120. Perform brightness normalization, infrared grayscale normalization, and image block cropping on the registered image to obtain a fixed-size dual-modal input:

[0021]

[0022] S130. During the training phase, a crack edge supervision map is generated based on the crack annotation map Y:

[0023]

[0024] in:

[0025] Used to extract the first-order direction edge of the crack. This process is used to extract the second-order geometric contour of cracks. It is used to alleviate the problem of blurred boundaries of tunnel cracks under low illumination, low contrast and complex background texture. The uploaded tunnel crack segmentation literature also points out that low illumination and complex background can cause background noise to interfere with the identification of fine cracks and cause false detection.

[0026] The output is:

[0027] The standardized bimodal input image is obtained:

[0028] And the crack region label Y and crack edge label E used during the training phase.

[0029] Furthermore, the input for S200's infrared crack feature extraction based on three IRMCE structural blocks is:

[0030] Preprocessed infrared image:

[0031] And features derived from the infrared features of the previous layer:

[0032] The input to the first IRMCE structure block is:

[0033]

[0034] The processing steps include:

[0035] S210. Will Three IRMCE structure blocks are input sequentially; for the k-th IRMCE structure block, the infrared crack thermal response and weak boundary features are first extracted through multi-scale convolution:

[0036]

[0037] Where d=3,5,7 represent different expansion rates, used to adapt to the scale differences of fine cracks, wide cracks, discontinuous cracks and bifurcated cracks in UAV aerial images;

[0038] S220. Generating infrared crack edge response from infrared multi-scale features:

[0039]

[0040] And embed the edge response into spatial attention:

[0041]

[0042] in, Indicates spatial attention. This represents the Sigmoid activation function;

[0043] S230. Weighting infrared features using edge-enhanced attention:

[0044]

[0045] Finally, the k-th infrared coded feature is obtained:

[0046]

[0047] This step utilizes the characteristic that infrared images are insensitive to changes in illumination to enhance the expression of weak crack edges under conditions of low illumination, dust, water stains, reflection, and concrete texture interference in tunnels.

[0048] Explicitly embedding edge features into the attention mechanism helps improve boundary detection accuracy; the application of edge semantic collaborative network based on edge-aware attention mechanism in infrared small target detection.

[0049] The output is:

[0050] Infrared crack edge enhancement features at three scales were obtained:

[0051] Among them, shallow features Preservation of fine crack thermal edges, mid-layer features Indicates the local orientation and discontinuous structure of the crack, and its deep features. This indicates the high-level thermal semantic differences between the crack region and the tunnel background.

[0052] Furthermore, the input for S300's RGB crack geometric attention feature extraction based on three RGEA structural blocks is:

[0053] Preprocessed RGB image:

[0054] And the features obtained from the previous layer's RGB features:

[0055] The input to the first RGEA structure block is:

[0056]

[0057] The processing steps include:

[0058] S310. Will Three RGEA structural blocks are input sequentially. For the k-th RGEA structural block, the input features are first divided into geometric edge sub-features and attention sub-features along the channel dimension:

[0059]

[0060] in, Used to extract the geometric edges of RGB cracks. Used to extract attention features from RGB crack textures;

[0061] For geometric edge sub-features, first generate the crack foreground response:

[0062]

[0063] S320. Extracting RGB crack geometric boundaries using edge operators:

[0064]

[0065] For attention sub-features, first generate channel attention weights:

[0066]

[0067] CA Channel Attention: This feature filters crack-related characteristics from a channel perspective, enhancing channels associated with crack texture, color abrupt changes, and edge direction, while suppressing background interference channels such as water stains, dirt, shadows, reflections, and rough concrete textures. Its network structure is as follows:

[0068]

[0069]

[0070] in, Indicates global average pooling. Indicates global max pooling. This represents a two-layer perceptron. This represents the Sigmoid function. This indicates element-wise multiplication;

[0071] S330. Regenerating Spatial Attention Weights:

[0072]

[0073] SA Spatial Attention: Locates crack regions in a spatial dimension, enhancing the response at crack trunks, endpoints, bifurcation points, and boundaries, while reducing the influence of non-crack background areas; its network structure is as follows:

[0074]

[0075]

[0076] in, This indicates averaging along the channel dimension. This indicates taking the maximum value along the channel dimension. express convolution, Indicates RGB attention enhancement features;

[0077] And obtain RGB attention enhancement features:

[0078]

[0079] S340. Fuse RGB geometric edge features and RGB attention enhancement features:

[0080]

[0081] This step addresses the issue of high similarity between cracks and tunnel joints, water stains, shadows, dirt, and rough textures in RGB images. The RGB branch enhances crack contours through geometric edge constraints and suppresses background interference through channel attention and spatial attention. Literature on tunnel crack segmentation points out that low-level features are beneficial for preserving crack boundaries, while high-level features are beneficial for suppressing noise. Therefore, it is necessary to consider both details and semantics during feature extraction.

[0082] The output is:

[0083] Obtain RGB crack geometric attention features at three scales:

[0084]

[0085] in, Indicates crack texture and fine edges. This indicates the local structure and bifurcation features of the crack. This indicates a high-level semantic difference between the crack and the tunnel background.

[0086] Furthermore, the input for the S400's SCO-based infrared-RGB dual-mode orthogonal continuous fusion is:

[0087] Infrared and RGB crack features at the k-th scale:

[0088]

[0089] The processing steps include:

[0090] S410. Perform channel stitching and channel alignment on infrared and RGB features at the same scale:

[0091]

[0092] S420. Input SCO structure; SCO first... Divide into two sub-features along the channel dimension:

[0093]

[0094] S430. Perform horizontal and vertical displacement respectively:

[0095]

[0096]

[0097] S440. Integrate orthogonal features in the horizontal and vertical directions:

[0098]

[0099] The two orthogonal axial features obtained from the 1×1 branch and the 3×3 branch are further fused into:

[0100]

[0101] Finally, through normalization, MLP, and residual connections, the dual-modal fusion feature at the k-th scale is obtained:

[0102]

[0103] The role of SCO is to model infrared weak edge features and RGB geometric texture features in orthogonal directions, enabling the network to maintain the continuity of crack structure along horizontal, vertical and indirect oblique directions; the original SCO structure models cross-dimensional dependencies through horizontal and vertical Axial Shifts and uses dual-scale convolutional branches to enhance structural continuity.

[0104] The output is:

[0105] We obtained infrared-RGB orthogonal continuous fusion features at three scales:

[0106] in, Used to provide shallow crack edge details to the decoding end. Used to provide information on the orientation of mid-layer fractures and local structure. Used to provide semantic and background suppression information for deep cracks.

[0107] Furthermore, the input for the S500's crack mask generation based on bottleneck RGEA and three-level decoding RGEA is:

[0108] The third level of RGB deep features output by the RGB Encoder in the S300:

[0109] Three-scale infrared-RGB orthogonal continuous fusion features output by SCO in S400:

[0110] During the training phase, the crack region label is also input: Y

[0111] And the crack edge label: E

[0112] in, Used to generate semantic features of the bottleneck layer Used to provide information on deep dual-mode crack structures Used to provide information on the orientation and local continuity of mid-layer fractures. Used to provide information on the edges and details of shallow cracks;

[0113] The processing steps include:

[0114] S510. Convert the third-level output feature of the RGB Encoder. Input downsampling layer to obtain bottleneck layer input features:

[0115]

[0116] S520. Input the bottleneck layer RGEA structure block and extract deep geometric edge attention features of the tunnel crack:

[0117]

[0118] in, This represents the bottleneck layer RGEA structure block; this structure block is located between the encoding and decoding ends and is used to further model the global semantics of cracks, crack trunk direction, bifurcation relationship, and crack region discrimination information in complex backgrounds at the lowest spatial resolution.

[0119] S530. Upsample the bottleneck layer output feature B and fuse it with the dual-modal feature output of the 3rd level SCO. Element-by-element addition yields the third-level decoding input:

[0120]

[0121] Will Input the first decoded RGEA structure block to obtain the level 3 decoded features:

[0122]

[0123] S540. Upsampling is performed, and the features are fused with the dual-modal features output by the second-level SCO. Add:

[0124]

[0125] Will Input the second decoded RGEA structure block to obtain the second-level decoded features:

[0126]

[0127] S550. Upsampling is performed, and the features are fused with the dual-modal features output from the first-level SCO. Add:

[0128]

[0129] Will Input the third decoded RGEA structure block to obtain the first-level decoded features:

[0130]

[0131] S560. Perform another upsampling step to restore the spatial resolution corresponding to the input image:

[0132]

[0133] pass Convolution and the Sigmoid function generate a crack probability map:

[0134]

[0135] And based on the threshold Generate a binary segmentation mask for the crack:

[0136]

[0137] During the training phase, UTICS-Net is jointly optimized using crack region loss and edge supervision loss; the crack region loss is:

[0138]

[0139] The infrared edge monitoring loss is:

[0140]

[0141] The RGB edge supervision loss is:

[0142]

[0143] The total loss function is:

[0144]

[0145] in, and The loss weight coefficients are used to simultaneously constrain crack region prediction, infrared weak edge response, and RGB geometric boundary response, thereby mitigating the problem of missed detection caused by small pixel proportion, blurred edges, thin and discontinuous shape, low contrast, and background texture interference in tunnel cracks.

[0146] The output is:

[0147] Obtain the crack probability map corresponding to the drone aerial image of the tunnel:

[0148] And the final binary segmentation mask for the crack:

[0149] in, This is used to represent the probability that each pixel belongs to the crack region. Used to identify crack areas on the tunnel lining surface; the mask can be further used for crack connectivity screening, skeleton extraction, length estimation, width estimation, and tunnel defect level assessment.

[0150] Compared with existing technologies, the technical advantages of the UAV tunnel infrared-visible light edge orthogonal fusion crack segmentation method provided by this invention are as follows:

[0151] This invention leverages the complementary advantages of infrared and RGB images to improve the reliability of tunnel crack segmentation using UAV aerial photography. The infrared branch enhances the thermal response and weak edge features of cracks in low-light environments, while the RGB branch extracts texture, color differences, and geometric contour features. The SCO fusion structure maintains the spatial continuity of cracks across different directions and scales, and the decoding end further recovers crack boundaries and details. The goal is to automatically generate continuous, complete, and clearly defined crack segmentation results under conditions of low illumination, strong interference, and complex backgrounds in tunnels, providing a more accurate data foundation for subsequent assessments of crack length, width, area, and severity. Specifically, the surface is as follows:

[0152] Firstly, infrared-RGB dual-modal collaborative segmentation: infrared images are used to enhance the response of cracks in low-light and weak edge conditions, while RGB images are used to supplement crack texture, color differences and geometric contour information, thus solving the problem of missed detection and false detection of single visible light images in complex tunnel environments.

[0153] Secondly, the IRMCE and RGEA feature extraction are divided into two branches: the infrared branch extracts thermal anomalies, multi-scale weak cracks and edge information through three IRMCE structural blocks; the RGB branch extracts texture, geometric boundaries and attention features through three RGEA structural blocks, which enhances the ability to express slender, discontinuous and low-contrast cracks.

[0154] Thirdly, SCO orthogonal continuous fusion and RGEA decoding recovery: By fusing infrared and RGB features at multiple scales using SCO, the continuity of cracks in horizontal, vertical and irregular directions is enhanced; then, the crack boundaries are recovered step by step through bottleneck RGEA and three decoding RGEAs, outputting more complete, continuous and clear crack segmentation results.

[0155] Explanation of reference numerals in the attached diagram:

[0156] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The accompanying drawings, which are provided to further illustrate the invention and constitute a part of this application, do not constitute an undue limitation of the invention.

[0157] Figure 1 This is the overall network framework diagram of UTICS-Net provided by this invention;

[0158] Figure 2 This is the IRMCE network structure diagram provided by the present invention;

[0159] Figure 3 This is the RGEA network structure diagram provided by the present invention;

[0160] Figure 4 This is the RGEA network structure diagram provided by the present invention;

[0161] Figure 5 is a visual comparison of the crack segmentation results of the method of the present invention and existing segmentation algorithms. Detailed Implementation

[0162] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the following embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the present invention. Equivalent substitutions or modifications made by those skilled in the art without departing from the concept of the present invention should fall within the scope of protection of the present invention.

[0163] like Figure 1 As shown, this invention achieves the aforementioned design goals by constructing the UTICS-Net UAV tunnel infrared-RGB crack segmentation network. The scheme first involves the UAV simultaneously acquiring RGB and infrared images of the tunnel lining surface, and then registering, normalizing, and cropping the two types of images to ensure they correspond to the same tunnel region. During network processing, the infrared image input consists of an infrared encoding branch composed of three IRMCE structural blocks, used to extract crack thermal response, weak edges, and multi-scale fine crack features under low-light conditions; the RGB image input consists of an RGB encoding branch composed of three RGEA structural blocks, used to extract crack texture, color differences, geometric boundaries, and spatial attention features. Subsequently, the network uses SCO to fuse infrared and RGB features at three scales, allowing the weak edge information in the infrared image to complement the texture contour information in the RGB image, thereby enhancing the identifiability and continuity of cracks in complex backgrounds.

[0164] In the decoding phase, the network first extracts deeper crack semantic information through the bottleneck layer RGEA, then upsamples step-by-step through three decoding RGEA structural blocks, and combines the fusion features output by SCO at each scale to recover the spatial details, edge contours, and discontinuous structure of the cracks, ultimately generating a crack probability map and a binary crack mask. During training, the network is optimized through crack region loss and edge constraint loss, enabling the model not only to determine which pixels belong to cracks, but also to more accurately preserve crack boundaries, endpoints, and slender continuous shapes.

[0165] Therefore, the implementation path of this technical solution can be summarized as follows: UAVs acquire dual-modal images of the tunnel; infrared branches enhance weak edges; RGB branches enhance texture and geometric contours; SCO fuses dual-modal multi-scale features; and RGEA decoding restores crack details, ultimately outputting a complete, continuous, and clearly defined tunnel crack segmentation result. This implementation method is designed to address issues such as low contrast, complex backgrounds, large scale variations, blurred edges, and thin, discontinuous cracks in tunnels, improving the accuracy and stability of crack segmentation during UAV tunnel inspection.

[0166] S100 Dual-Modal Tunnel Aerial Image Acquisition and Standardization Processing

[0167] Input is:

[0168] The tunnel lining images simultaneously acquired by the RGB camera and infrared thermal imaging camera mounted on the drone are denoted as follows:

[0169]

[0170]

[0171] During the training phase, manually annotated crack maps were also input:

[0172] In Y, crack pixels are marked as 1, and non-crack background pixels are marked as 0.

[0173] The technical means of processing are as follows:

[0174] First, for RGB images and infrared images Time synchronization, camera calibration, distortion correction, spatial registration, and scale normalization are performed to ensure pixel-level or near-pixel-level correspondence between the two images within the same tunnel wall region. Subsequently, the registered image undergoes brightness normalization, infrared grayscale normalization, and image patch cropping to obtain a fixed-size dual-modal input.

[0175]

[0176] During the training phase, a crack edge supervision map is generated based on the crack annotation map Y:

[0177]

[0178] in:

[0179] Used to extract the first-order direction edge of the crack. This method is used to extract the second-order geometric contours of cracks. It aims to mitigate the problem of blurred crack boundaries in low-light, low-contrast, and complex background textures. Uploaded literature on tunnel crack segmentation also points out that low light and complex backgrounds can cause background noise to interfere with the identification of fine cracks, leading to false detections.

[0180] The output is:

[0181] The standardized bimodal input image is obtained:

[0182] And the crack region label Y and crack edge label E used during the training phase.

[0183] S200 Infrared Crack Feature Extraction Based on Three IRMCE Structure Blocks

[0184] like Figure 2 As shown, the input is:

[0185] Preprocessed infrared image:

[0186] And features derived from the infrared features of the previous layer:

[0187] The input to the first IRMCE structure block is:

[0188]

[0189] This represents the convolutional embedding layer for infrared images, preferably using a convolutional kernel size of [size missing]. A convolution operation with a stride of 2 and padding of 1 is used to transform the standardized infrared image. Mapped to first-level infrared features: , This represents the number of first-level feature channels.

[0190] The technical means of processing are as follows:

[0191] Will Three IRMCE structure blocks are input sequentially. For the k-th IRMCE structure block, the infrared crack thermal response and weak boundary features are first extracted through multi-scale convolution:

[0192]

[0193] Where d=3,5,7 represents different expansion rates, used to adapt to the scale differences of fine cracks, wide cracks, discontinuous cracks, and bifurcation cracks in UAV aerial images. Subsequently, infrared crack edge responses are generated from infrared multi-scale features:

[0194]

[0195] And embed the edge response into spatial attention:

[0196]

[0197] in, Indicates spatial attention. This represents the Sigmoid activation function. Then, edge enhancement attention is used to weight the infrared features:

[0198]

[0199] Finally, the k-th infrared coded feature is obtained:

[0200]

[0201] This step leverages the insensitivity of infrared images to changes in illumination to enhance the representation of weak crack edges under conditions of low illumination, dust, water stains, reflection, and concrete texture interference in tunnels. Literature on small infrared targets also emphasizes that explicitly embedding edge features into attention mechanisms helps improve boundary detection accuracy. The application of edge-semantic collaborative networks based on edge-aware attention mechanisms in infrared small target detection is discussed.

[0202] The output is:

[0203] Infrared crack edge enhancement features at three scales were obtained:

[0204] Among them, shallow features Preservation of fine crack thermal edges, mid-layer features Indicates the local orientation and discontinuous structure of the crack, and its deep features. This indicates the high-level thermal semantic differences between the crack region and the tunnel background.

[0205] S300 RGB Crack Geometric Attention Feature Extraction Based on Three RGEA Structure Blocks

[0206] like Figure 3 As shown, the input is:

[0207] Preprocessed RGB image:

[0208] And the features obtained from the previous layer's RGB features:

[0209] The input to the first RGEA structure block is:

[0210]

[0211] in, This represents a convolutional embedding layer for an RGB image, preferably using a convolutional kernel size of [size missing]. A convolution operation with a stride of 2 and padding of 1.

[0212] The technical means of processing are as follows:

[0213] Will Input three RGEA structure blocks sequentially. For the k-th RGEA structure block, first divide the input features along the channel dimension into geometric edge sub-features and attention sub-features:

[0214]

[0215] in, Used to extract the geometric edges of RGB cracks. Used to extract attention features from RGB crack textures.

[0216] For geometric edge sub-features, first generate the crack foreground response:

[0217]

[0218] Then, the edge operator is used to extract the geometric boundary of the RGB crack:

[0219]

[0220] For attention sub-features, first generate channel attention weights:

[0221]

[0222] CA Channel Attention: This feature filters crack-related characteristics from a channel perspective, enhancing channels associated with crack texture, color abrupt changes, and edge direction, while suppressing background interference channels such as water stains, dirt, shadows, reflections, and rough concrete textures. Its network structure is as follows:

[0223]

[0224]

[0225] in, Indicates global average pooling. Indicates global max pooling. This represents a two-layer perceptron. This represents the Sigmoid function. This indicates element-wise multiplication.

[0226] Regenerate spatial attention weights:

[0227]

[0228] SA Spatial Attention: Locates crack regions in a spatial dimension, enhancing the response at crack trunks, endpoints, bifurcations, and boundaries, while reducing the influence of non-crack background areas. Its network structure is as follows:

[0229]

[0230]

[0231] in, This indicates averaging along the channel dimension. This indicates taking the maximum value along the channel dimension. express convolution, This indicates RGB attention enhancement features.

[0232] And obtain RGB attention enhancement features:

[0233]

[0234] Finally, the RGB geometric edge features and RGB attention enhancement features are fused:

[0235]

[0236] This step addresses the issue of high similarity between cracks and tunnel joints, water stains, shadows, dirt, and rough textures in RGB images. The RGB branch enhances crack contours through geometric edge constraints and suppresses background interference through channel attention and spatial attention. Literature on tunnel crack segmentation indicates that low-level features are beneficial for preserving crack boundaries, while high-level features are beneficial for suppressing noise; therefore, it is necessary to consider both detail and semantics during feature extraction.

[0237] The output is:

[0238] Obtain RGB crack geometric attention features at three scales:

[0239]

[0240] in, Indicates crack texture and fine edges. This indicates the local structure and bifurcation features of the crack. This indicates a high-level semantic difference between the crack and the tunnel background.

[0241] S400 based on SCO infrared-RGB dual-mode orthogonal continuous fusion

[0242] like Figure 4 As shown, the input is:

[0243] Infrared and RGB crack features at the k-th scale:

[0244]

[0245] The technical means of processing are as follows:

[0246] First, channel stitching and channel alignment are performed on infrared and RGB features at the same scale:

[0247]

[0248] Then, Input the SCO structure. SCO is first... Divide into two sub-features along the channel dimension:

[0249]

[0250] Then, horizontal and vertical displacements are performed respectively:

[0251]

[0252]

[0253] Reintegrate orthogonal features in the horizontal and vertical directions:

[0254]

[0255] The two orthogonal axial features obtained from the 1×1 branch and the 3×3 branch are further fused into:

[0256]

[0257] Finally, through normalization, MLP, and residual connections, the dual-modal fusion feature at the k-th scale is obtained:

[0258]

[0259] The role of SCO is to model infrared weak edge features and RGB geometric texture features in orthogonal directions, enabling the network to maintain the continuity of the crack structure along horizontal, vertical, and indirect oblique directions. The original SCO structure models cross-dimensional dependencies through horizontal and vertical axial shifts and uses dual-scale convolutional branches to enhance structural continuity.

[0260] The output is:

[0261] We obtained infrared-RGB orthogonal continuous fusion features at three scales:

[0262] in, Used to provide shallow crack edge details to the decoding end. Used to provide information on the orientation of mid-layer fractures and local structure. Used to provide semantic and background suppression information for deep cracks.

[0263] S500 Crack Mask Generation Based on Bottleneck RGEA and Three-Level Decoding RGEA

[0264] Input is:

[0265] The third level of RGB deep features output by the RGB Encoder in S3:

[0266] Three-scale infrared-RGB orthogonal continuous fusion features of SCO output in S4:

[0267] During the training phase, the crack region label is also input: Y

[0268] And the crack edge label: E

[0269] in, Used to generate semantic features of the bottleneck layer Used to provide information on deep dual-mode crack structures Used to provide information on the orientation and local continuity of mid-layer fractures. Used to provide information on the edges and details of shallow cracks.

[0270] The technical means of processing are as follows:

[0271] First, the third-level output feature of the RGB Encoder... Input downsampling layer to obtain bottleneck layer input features:

[0272]

[0273] Then, Input the bottleneck layer RGEA structure block and extract deep geometric edge attention features of the tunnel crack:

[0274]

[0275] in, This represents the bottleneck layer RGEA structure block. Located between the encoder and decoder, this block is used to further model the global semantics of cracks, the direction of the crack trunk, the bifurcation relationship, and crack region discrimination information in complex backgrounds at the lowest spatial resolution.

[0276] Then, the bottleneck layer output feature B is upsampled and fused with the dual-modal feature output by the third-level SCO. Element-by-element addition yields the third-level decoding input:

[0277]

[0278] Will Input the first decoded RGEA structure block to obtain the level 3 decoded features:

[0279]

[0280] Next, regarding Upsampling is performed, and the features are fused with the dual-modal features output by the second-level SCO. Add:

[0281]

[0282] Will Input the second decoded RGEA structure block to obtain the second-level decoded features:

[0283]

[0284] Subsequently, Upsampling is performed, and the features are fused with the dual-modal features output from the first-level SCO. Add:

[0285]

[0286] Will Input the third decoded RGEA structure block to obtain the first-level decoded features:

[0287]

[0288] Finally, for Perform another upsampling step to restore the spatial resolution corresponding to the input image:

[0289]

[0290] pass Convolution and the Sigmoid function generate a crack probability map:

[0291]

[0292] And based on the threshold Generate a binary segmentation mask for the crack:

[0293]

[0294] During the training phase, UTICS-Net is jointly optimized using crack region loss and edge supervision loss. The crack region loss is:

[0295]

[0296] The infrared edge monitoring loss is:

[0297]

[0298] The RGB edge supervision loss is:

[0299]

[0300] The total loss function is:

[0301]

[0302] in, and The weighting coefficients are used for the loss. This joint loss is used to simultaneously constrain the crack region prediction, the weak infrared edge response, and the RGB geometric boundary response, thereby mitigating the problem of missed detection caused by the small pixel proportion of tunnel cracks, blurred edges, thin and discontinuous shapes, low contrast, and background texture interference.

[0303] The output is:

[0304] Obtain the crack probability map corresponding to the drone aerial image of the tunnel:

[0305] And the final binary segmentation mask for the crack:

[0306] in, This is used to represent the probability that each pixel belongs to the crack region. Used to identify cracked areas on the tunnel lining surface. This mask can be further used for crack connectivity screening, skeleton extraction, length estimation, width estimation, and tunnel distress level assessment.

[0307] Experimental verification

[0308] Experimental parameters and evaluation indicators

[0309] To verify the effectiveness of the "UAV tunnel infrared and visible light edge orthogonal fusion crack segmentation method" described in this invention, experiments were conducted on a self-built UAV tunnel dual-modal crack dataset. The experimental data was simultaneously collected from the tunnel lining surface by an RGB camera and an infrared thermal imaging camera mounted on a UAV. The collected scenes covered complex backgrounds such as low illumination, uneven lighting, wall reflections, water stains, dirt, dust, and rough concrete textures.

[0310] Following the dataset organization method in the tunnel crack segmentation experiment, the acquired images underwent time synchronization, spatial registration, distortion correction, size normalization, and manual pixel-level annotation. Crack pixels in the annotated images were marked as 1, and background pixels (excluding cracks) were marked as 0. Experimental samples were uniformly processed to 512×512 pixels. The dataset contains 982 images of tunnel lining cracks, including 794 images for training, 89 for validation, and 99 for testing. The training set was used for model parameter learning, the validation set for training process evaluation and model selection, and the test set for final performance evaluation.

[0311] The experimental comparison methods include typical semantic segmentation algorithms such as PSPNet, Swin Transformer, and DeepLabV3+. The method of this invention uses the aforementioned UTICS-Net as the crack segmentation model, and completes crack feature enhancement, dual-modal fusion, and crack mask output through an infrared branch IRMCE, an RGB branch RGEA, an SCO dual-modal orthogonal continuous fusion structure, and a bottleneck RGEA and a three-level decoding RGEA. To ensure the fairness of the experimental comparison, all comparison methods were trained and evaluated under the same training, validation, and test set partitioning conditions.

[0312] The experiment used IoU, F1 score, and AP50 as evaluation metrics. IoU represents the cross-union ratio between the predicted crack mask and the manually labeled mask, measuring the degree of overlap between crack regions; F1 score is the harmonic mean of precision and recall, used to comprehensively measure the completeness of crack pixel detection and the degree of missegmentation; AP50 represents the average precision at an cross-union threshold of 0.50, used to measure the model's ability to detect and segment crack regions. Higher values ​​for these metrics indicate better crack segmentation performance.

[0313] Experimental results

[0314] The method of this invention was compared with methods such as PSPNet, Swin Transformer, and DeepLabV3+, and the experimental results are shown in Table 1.

[0315] Table 1. Performance comparison of the method of this invention with existing segmentation algorithms on a self-built tunnel crack dataset.

[0316]

[0317] As shown in Table 1, the method of this invention achieves the best results in IoU, F1, and AP50, reaching 0.82, 0.92, and 0.75, respectively. Compared with PSPNet, the method of this invention improves IoU, F1, and AP50 by 0.12, 0.11, and 0.08, respectively, indicating that this invention can significantly improve the segmentation shortcomings of traditional pyramid scene parsing networks in the face of narrow cracks and complex texture backgrounds.

[0318] Compared to the Swin Transformer, the method of this invention improves IoU, F1, and AP50 by 0.07, 0.06, and 0.05, respectively, indicating that relying solely on global window attention is insufficient to adequately preserve the elongated edges and discontinuous structure of cracks. The infrared-RGB dual-modal feature fusion of this invention enhances the continuous representation of crack regions. Compared to DeepLabV3+, the method of this invention improves IoU, F1, and AP50 by 0.03, 0.04, and 0.03, respectively, demonstrating that while maintaining high region overlap, this invention further improves the pixel-level detection completeness and segmentation accuracy of cracks.

[0319] The results show that enhancing weak edges and thermal response in infrared images through IRMCE, extracting geometric boundaries, texture differences and spatial attention information in RGB images through RGEA, and performing infrared-RGB orthogonal continuous fusion at multiple scales through SCO can effectively reduce the interference of low illumination, reflection, water stains, dirt and rough concrete textures in tunnels on crack segmentation, thereby improving the integrity and boundary accuracy of crack masks in complex tunnel environments.

[0320] Visualized results analysis

[0321] To further verify the segmentation effect of the method of the present invention in actual tunnel crack scenarios, the crack segmentation results of the method of the present invention are visually compared with those of PSPNet, Swin Transformer, and DeepLabV3+, as shown in Figure 5. The first column of the figure is the original tunnel crack image, and the subsequent columns are the segmentation results of the method of the present invention, DeepLabV3+, SwinTransformer, and PSPNet, respectively. The white area represents the crack prediction area, and the black area represents the non-crack background.

[0322] As can be seen from Figure 5, in the three sets of tunnel crack examples, PSPNet is insufficient in maintaining the continuity of slender cracks, and local crack trunk breaks, missing endpoints, or omissions in small areas are prone to occur; Swin Transformer can segment the main crack area, but local breaks or boundary instability still exist at crack bends, edges of fine cracks, and discontinuous cracks; DeepLabV3+ has a good segmentation ability for the main crack structure, but boundary offset, local over-coarsening, or insufficient detail preservation still occur in some locations.

[0323] In contrast, the crack mask output by the method of this invention is more continuous overall, better preserving the crack trunk, bending areas, endpoints, and small branches, with less background noise response, and the segmentation result is more consistent with the crack direction and shape in the original image. This result demonstrates that the present invention, through infrared weak edge enhancement, RGB geometric attention feature extraction, and SCO orthogonal continuum fusion, can improve the segmentability of low-contrast, slender, curved, and discontinuous cracks in complex tunnel backgrounds.

[0324] In summary, the experimental results show that the UAV tunnel infrared-RGB edge orthogonal fusion crack segmentation method proposed in this invention achieves superior segmentation performance on the self-built tunnel crack dataset. Comparative experiments verify the comprehensive advantages of this invention compared with existing methods such as PSPNet, Swin Transformer, and DeepLabV3+. Visualization results further demonstrate that this invention has a more stable segmentation ability for slender, low-contrast, and discontinuous cracks in complex tunnel environments.

[0325] This invention provides a dual-modal crack segmentation method for tunnels using unmanned aerial vehicles (UAVs). The method involves simultaneously acquiring infrared and RGB images of the tunnel lining surface using a UAV, and then performing registration, normalization, and cropping on the two types of images. Subsequently, a UTICS-Net network is constructed for crack segmentation. The network includes an infrared encoding branch, an RGB encoding branch, an SCO dual-modal fusion structure, and an RGEA decoding structure. The infrared encoding branch consists of three IRMCE blocks, used to extract the thermal response, multi-scale weak edges, and slender structural features of cracks under low-light conditions in the tunnel. The RGB encoding branch consists of three RGEA blocks, used to extract crack texture, color differences, geometric boundaries and channels, and spatial attention features. The SCO structure is used to fuse infrared and RGB features at multiple scales, enhancing the spatial continuity of cracks in different directions. In the decoding stage, bottleneck RGEA and three-level decoding RGEA are used to progressively recover crack boundaries and spatial details, ultimately generating a crack probability map and a binary crack mask. This invention can improve the segmentation integrity and boundary accuracy of slender, discontinuous, and low-contrast cracks in complex tunnel environments, reduce missed detections, false detections, and segmentation fractures, and provide a reliable basis for assessing crack length, width, area, and disease severity.

[0326] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them; although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications can still be made to the specific implementation of the present invention or equivalent substitutions can be made to some technical features without departing from the spirit of the technical solutions of the present invention, and all such modifications and substitutions should be covered within the scope of the technical solutions claimed in the present invention.

Claims

1. A method for segmenting cracks in UAV tunnels using infrared and visible light edge orthogonal fusion, characterized by the following steps: include: S100. Perform dual-modal tunnel aerial image acquisition and standardization processing; S200. Perform infrared crack feature extraction based on three IRMCE structural blocks to obtain infrared crack edge enhancement features at three scales; S300. Perform RGB crack geometric attention feature extraction based on three RGEA structural blocks; S400 performs SCO-based infrared-RGB dual-mode orthogonal continuous fusion; S500 performs crack mask generation based on bottleneck RGEA and three-level decoding RGEA.

2. The method for segmenting cracks in UAV tunnels using infrared and visible light edge orthogonal fusion according to claim 1, characterized in that, The steps for aerial image acquisition and standardization processing of the S100 dual-modal tunnel include: S110. For RGB images and infrared images Time synchronization, camera calibration, distortion correction, spatial registration, and scale normalization are performed to ensure that the two images have a pixel-level or near-pixel-level correspondence within the same tunnel wall area. S120. Perform brightness normalization, infrared grayscale normalization, and image block cropping on the registered image to obtain a fixed-size dual-modal input: ； Where b represents the batch size, and H and W represent the height and width of the input image, respectively. This represents a standardized RGB image. Y represents the standardized infrared image, and Y represents the manually annotated crack map. S130. During the training phase, a crack edge supervision map is generated based on the crack annotation map Y: in: Used to extract the first-order direction edge of the crack. Used to extract the second-order geometric profile of cracks; , , For the standardized bimodal input image, the input during the training phase is a manually annotated crack image: In Y, crack pixels are marked as 1, and non-crack background pixels are marked as 0.

3. The method for segmenting cracks in UAV tunnels using infrared and visible light edge fusion according to claim 1, characterized in that, The input for S200 infrared crack feature extraction based on three IRMCE structural blocks is: Preprocessed infrared image: ; And features derived from the infrared features of the previous layer: ; The input to the first IRMCE structure block is: ； The output is: Infrared crack edge enhancement features at three scales were obtained: ; Among them, shallow features Preservation of fine crack thermal edges, mid-layer features Indicates the local orientation and discontinuous structure of the crack, and its deep features. This indicates the high-level thermal semantic differences between the crack region and the tunnel background.

4. The method for segmenting cracks in UAV tunnels using infrared and visible light edge orthogonal fusion according to claim 3, characterized in that, The processing steps for infrared crack feature extraction based on three IRMCE structural blocks in S200 include: S210. Will Three IRMCE structure blocks are input sequentially; for the k-th IRMCE structure block, the infrared crack thermal response and weak boundary features are first extracted through multi-scale convolution: ； Where d=3,5,7 represent different expansion rates; S220. Generating infrared crack edge response from infrared multi-scale features: ； And embed the edge response into spatial attention: ； in, Indicates spatial attention, This represents the Sigmoid activation function; S230. Weighting infrared features using edge-enhanced attention: ； Finally, the k-th infrared coded feature is obtained: 。 5. The method for segmenting cracks in UAV tunnels using infrared and visible light edge orthogonal fusion according to claim 1, characterized in that, The input for S300's RGB crack geometric attention feature extraction based on three RGEA structural blocks is: Preprocessed RGB image: ; And the features obtained from the previous layer's RGB features: ; The input to the first RGEA structure block is: ； The output is: Obtain RGB crack geometric attention features at three scales: ； in, Indicates crack texture and fine edges. This indicates the local structure and bifurcation features of the crack. This indicates a high-level semantic difference between the crack and the tunnel background.

6. The method for segmenting cracks in UAV tunnels using infrared and visible light edge orthogonal fusion according to claim 5, characterized in that, The processing steps for S300 RGB crack geometric attention feature extraction based on three RGEA structural blocks include: S310. Will Three RGEA structural blocks are input sequentially. For the k-th RGEA structural block, the input features are first divided into geometric edge sub-features and attention sub-features along the channel dimension: ； in, Represents the RGB geometric edge sub-features at the k-th scale. Represents the RGB attention sub-feature at the k-th scale; For geometric edge sub-features, first generate the crack foreground response: ； S320. Extracting RGB crack geometric boundaries using edge operators: ； For attention sub-features, first generate channel attention weights: ； Its network structure is ； ； in, Indicates global average pooling. Indicates global max pooling. This represents a two-layer perceptron. This represents the Sigmoid function. This indicates element-wise multiplication; S330. Regenerating Spatial Attention Weights: ； SA Spatial Attention: Locates crack regions in a spatial dimension, enhancing the response at crack trunks, endpoints, bifurcation points, and boundaries, while reducing the influence of non-crack background areas; its network structure is as follows: ； ； in, This indicates averaging along the channel dimension. This indicates taking the maximum value along the channel dimension. express convolution, Indicates RGB attention enhancement features; And obtain RGB attention enhancement features: ； S340. Fuse RGB geometric edge features and RGB attention enhancement features: 。 7. The method for segmenting cracks in UAV tunnels using infrared and visible light edge orthogonal fusion according to claim 1, characterized in that, The input for S400's SCO-based infrared-RGB dual-modal orthogonal continuous fusion is: Infrared and RGB crack features at the k-th scale: ； The output is: We obtained infrared-RGB orthogonal continuous fusion features at three scales: ; in, Used to provide shallow crack edge details to the decoding end. Used to provide information on the orientation of mid-layer fractures and local structure. Used to provide semantic and background suppression information for deep cracks.

8. The method for segmenting cracks in UAV tunnels using infrared and visible light edge orthogonal fusion according to claim 7, characterized in that, The S400-based SCO-based infrared-RGB dual-mode orthogonal continuous fusion processing steps include: S410. Perform channel stitching and channel alignment on infrared and RGB features at the same scale: ； S420. Input SCO structure; SCO first... Divide into two sub-features along the channel dimension: ； S430. Perform horizontal and vertical displacement respectively: ； ； in, This indicates a horizontal axial displacement along the width of the image. This indicates vertical axial displacement along the image height direction; S440. Integrate orthogonal features in the horizontal and vertical directions: ； The two orthogonal axial features obtained from the 1×1 branch and the 3×3 branch are further fused into: ； Finally, through normalization, MLP, and residual connections, the dual-modal fusion feature at the k-th scale is obtained: 。 9. The method for segmenting cracks in UAV tunnels using infrared and visible light edge orthogonal fusion according to claim 1, characterized in that, The input for the S500 crack mask generation based on bottleneck RGEA and three-level decoding RGEA is: The third level of RGB deep features output by the RGB Encoder in the S300: ; Three-scale infrared-RGB orthogonal continuous fusion features output by SCO in S400: ; During the training phase, the crack region label is also input: Y; And the crack edge label: E in, Used to generate semantic features of the bottleneck layer Used to provide information on deep dual-mode crack structures Used to provide information on the orientation and local continuity of mid-layer fractures. Used to provide information on the edges and details of shallow cracks; The output is: Obtain the crack probability map corresponding to the drone aerial image of the tunnel: ; And the final binary segmentation mask for the crack: ; in, This is used to represent the probability that each pixel belongs to the crack region. Used to identify cracked areas on the surface of tunnel lining.

10. The method for segmenting cracks in UAV tunnels using infrared and visible light edge orthogonal fusion according to claim 9, characterized in that, The S500 crack mask generation process based on bottleneck RGEA and three-level decoding RGEA includes the following steps: S510. Convert the third-level output feature of the RGB Encoder. Input downsampling layer to obtain bottleneck layer input features: ； S520. Input the bottleneck layer RGEA structure block and extract deep geometric edge attention features of the tunnel crack: ； in, This represents the bottleneck layer RGEA structure block; S530. Upsample the bottleneck layer output feature B and fuse it with the dual-modal feature output of the 3rd level SCO. Element-by-element addition yields the third-level decoding input: ； Will Input the first decoded RGEA structure block to obtain the level 3 decoded features: ； S540. Upsampling is performed, and the features are fused with the dual-modal features output by the second-level SCO. Add: ； Will Input the second decoded RGEA structure block to obtain the level 2 decoded features: ； S550. Upsampling is performed, and the features are fused with the dual-modal features output from the first-level SCO. Add: ； Will Input the third decoded RGEA structure block to obtain the first-level decoded features: ； S560. Perform another upsampling to restore the spatial resolution corresponding to the input image: ； pass Convolution and the Sigmoid function generate a crack probability map: ； And based on the threshold Generate a binary segmentation mask for the crack: ； in, The threshold for binarizing the crack probability map is defined as follows: Preferably, ; During the training phase, UTICS-Net is jointly optimized using crack region loss and edge supervision loss; the crack region loss is: ； The infrared edge monitoring loss is: ； The RGB edge supervision loss is: ； The total loss function is: ； in, and This is the loss weighting coefficient.

Images