A tunnel lining apparent defect recognition method based on query matching attention

Abstract

The present invention relates to the field of semantic segmentation in computer vision and is a method for identifying apparent defects in tunnel linings based on query matching attention. The method comprises the following steps: collecting apparent images of tunnel linings and constructing a defect image dataset; preprocessing the apparent defect images of tunnel linings and corresponding defect mask images; constructing a feature extraction model and calculating a multi-stage defect location attention feature map of the apparent defect images of tunnel linings; constructing a defect location attention feature decoding model and calculating the segmentation results of the segmentation feature map at each stage; binarizing and optimizing the segmentation results, and identifying the direction of crack defects based on the optimized binarization results of the tunnel crack defects. The present invention considers features at different levels, which helps to capture the manifestation of defects at various scales while ensuring a low computational complexity. When the scales of the defects and background are greatly unbalanced, the influence of background noise can be effectively suppressed, thereby improving the accuracy of defect detection.

Description

Technical Field

[0001] The present invention relates to the field of semantic segmentation in computer vision, and in particular to a method for identifying apparent defects in tunnel linings based on query matching attention. Background Art

[0002] Traditional methods for detecting apparent defects in tunnel linings rely primarily on manual inspections supplemented by simple measurement tools. This approach suffers from low efficiency, strong subjectivity, and difficulty in accurate quantification. In recent years, with the rapid development of image processing technology and deep learning, automated detection of apparent defects in tunnel linings based on computer vision models has made significant progress. These methods typically construct training datasets from collected tunnel lining surface images and apply algorithms to automatically identify and segment defects at the pixel level, greatly improving detection speed and accuracy while also facilitating subsequent quantitative assessment by professionals. However, existing technologies still face challenges, such as accurately identifying defects under complex lighting and background conditions, overcoming the relatively small proportion of defects in images, and achieving refined segmentation of defects. Therefore, developing a high-precision and efficient defect segmentation solution has become a key research direction.

[0003] In recent years, researchers in the field of tunnel defect detection have increasingly focused on improving the accuracy and efficiency of defect recognition by refining the model's attention mechanism. The introduction of the attention mechanism aims to enable the model to more effectively focus on defect features during training, avoiding interference from irrelevant background noise, thereby achieving more accurate defect localization and segmentation. Convolutional neural networks (CNNs) are widely used in defect detection due to their outstanding performance in image processing. However, CNNs tend to capture local information, which facilitates accurate local feature extraction. However, when dealing with defect segmentation, their lack of a global perspective can lead to discontinuities in defect regions. For example, in tunnel crack detection, cracks are often distributed in an elongated pattern, and the local receptive field of CNNs limits their performance in this area. To overcome these limitations of CNNs, researchers have begun exploring and applying different attention mechanisms to enhance the model's sensitivity to key defect features. Channel-wise and spatial-wise attention help the model enhance important feature channels and pixels. Channel-wise attention helps the model select features closely related to the defect, while spatial attention focuses on the specific location of the defect in the image, ensuring consistent and complete segmentation. The multi-scale dilated pyramid expands the model's receptive field by introducing dilated convolutions of different scales, enabling it to simultaneously capture local details and global contextual information, effectively improving the accuracy of disease detection. The self-attention mechanism originated in the field of natural language processing and was later transplanted to the field of computer vision. By constructing a triple of query vectors, key vectors, and value vectors, it enables the model to automatically identify areas in the image that are highly correlated with disease features, thereby highlighting disease signals in the complex tunnel background. The core of this mechanism is that it allows the model to form a self-referential attention distribution when processing image features, effectively enhancing the model's ability to capture long-range dependencies. Although the attention mechanism significantly improves the performance of disease detection, when the proportion of diseases in the image is extremely small, the full-image attention calculation, while capable of extracting long-range correlations in parallel, may also introduce unnecessary computational burden and may introduce excessive noise into model training.

[0004] Furthermore, decoders for disease segmentation models are also showing a trend of diversification. Each decoder architecture is designed to optimize for specific feature extraction methods and segmentation tasks, thereby improving disease detection capabilities. The SegNet model employs a direct upsampling strategy, utilizing pooling indices obtained during the encoder stage for pixel-level prediction. This mechanism simplifies the computational process to a certain extent, but may sacrifice some detail, especially when processing subtle disease features. The DeepLaV3+ model introduces intermediate layer features to provide information for decoding, reducing decoding complexity while maintaining high resolution. The UNet model effectively compensates for the information loss caused by increased model depth. Skip connections allow direct interaction between deep and shallow layer features, achieving finer boundary definition in disease segmentation tasks. This architecture consumes a lot of memory and may perform poorly when the object scale varies greatly. SegFormer, a representative lightweight decoder, processes features extracted by a self-attention mechanism through linear layers and then achieves disease segmentation through feature concatenation. This design not only reduces computational resources but also improves model flexibility. However, directly splicing multi-scale features may cause some details to be obscured, especially for diseases with complex shapes and varying scales. This limitation is particularly obvious. Summary of the Invention

[0005] To overcome the shortcomings of the aforementioned prior art, this paper proposes a method for identifying apparent defects in tunnel linings based on query matching attention. By constructing a feature extraction model and a defect localization attention feature decoding model, this method matches highly relevant key and value vectors to query vectors, reducing the model's computational burden and focusing its attention on defects with a relatively small proportion. A deep segmentation strategy and an adaptive feature fusion algorithm are employed to enhance the model's ability to detect defects of varying scales, including those at the very smallest. This method effectively suppresses the influence of background noise when the scale of the defect is significantly out of balance with the background, thereby improving the accuracy of defect detection.

[0006] A method for identifying apparent defects in tunnel linings based on query matching attention includes the following steps:

[0007] S1. Collect tunnel lining surface images, mark the defects in the tunnel lining surface images, and build a defect image dataset;

[0008] S2. Preprocess the tunnel lining surface defect images and the corresponding defect mask images, extract non-overlapping sub-image blocks containing the defects, and construct a training dataset and a test dataset for the defect images.

[0009] S3. Construct a feature extraction model to extract the detail feature map of the tunnel lining apparent disease image, and calculate the multi-stage disease location attention feature map of the tunnel lining apparent disease image based on the detail feature map combined with the self-attention mechanism;

[0010] S4. Construct a disease localization attention feature decoding model, use an adaptive feature fusion method to fuse shallow and deep features, and calculate the segmentation results of the segmentation feature map at each stage through a deep supervision strategy;

[0011] S5. Binarization and optimization are performed on the segmentation results to obtain an optimized binarization result of the tunnel crack disease, and the crack disease direction is identified according to the optimized binarization result of the tunnel crack disease.

[0012] Specifically, step S1 includes:

[0013] The inspection vehicle equipped with a linear array camera collects tunnel lining surface images twice to obtain two half-tunnel images, which are then aligned and spliced ​​to obtain a complete tunnel lining surface image.

[0014] The complete tunnel lining surface image is cut without overlap according to a preset resolution, and the cut tunnel lining surface image is classified to obtain a tunnel lining surface disease image and a background; the tunnel lining surface disease image is divided into a crack disease image and a tile peeling disease image;

[0015] The tunnel lining surface defect image is annotated by the category number mapping method to obtain the annotated defect mask image.

[0016] A damage image dataset is constructed, which includes tunnel lining apparent damage images and annotated damage mask images.

[0017] Specifically, step S2 includes:

[0018] The uneven illumination correction algorithm is used to correct the brightness difference of the tunnel lining surface defect image and obtain a uniformly illuminated grayscale image.

[0019] The uniformed grayscale image is converted into an RGB color image to obtain a corrected tunnel lining apparent disease image. The corrected tunnel lining apparent disease image and the corresponding disease mask image are cut into multiple sub-image blocks, and the disease-free sub-image blocks are eliminated. The remaining image blocks are divided into a tunnel image training dataset and a tunnel image test dataset according to several proportions.

[0020] Specifically, step S3 includes:

[0021] S31. Construct a feature extraction model based on a self-attention mechanism neural network. The feature extraction model includes a query matching disease attention extraction branch and a multi-scale content feature extraction branch.

[0022] S32. Extract key low-level visual features of tunnel lining surface damage images through convolutional neural networks to generate detail feature maps F.D ;

[0023] S33, detail feature map F D Input the query matching-based disease attention extraction branch and calculate the query vector V Q and the key vector V that matches the query QK ;

[0024] S34, detail feature map F D Input the multi-scale content feature extraction branch and calculate the value vector V V and the query matching value vector V QV ;

[0025] S35. Query vector V Q , query matching key vector V Q K and query matching value vector V QV They are used as the query, key and value inputs of the self-attention mechanism respectively to calculate the first-stage disease localization attention feature map F1;

[0026] S36. Based on the first-stage disease localization attention feature map F1, the second-stage disease localization attention feature map F2, the third-stage disease localization attention feature map F3, and the fourth-stage disease localization attention feature map F4 are calculated respectively.

[0027] Specifically, the step S33 includes: extracting the detail feature map F by querying the matched disease attention branch D The network is divided into S×S blocks in width and height directions, and the embedded features E are extracted for each block through the convolutional neural network. The query vector V is calculated from the embedded features E using the linear layer. Q and the key vector V K ;

[0028] Compute query vector V by channel Q The mean M Q , and then calculate the key vector V by channel K The mean M K , calculate the mean M Q and mean M K The cosine similarity of , as the correlation between the query vector and the key vector of each block;

[0029] Take the query vector V Q Calculate the dimension for the target and select the same vector V as the query vector Q The top K key vectors with the highest relevance constitute the key vector set V that matches the query QK .

[0030] Specifically, the step S34 includes extracting the detail feature map FD The channel dimension is divided into three groups of features, which are represented by F D1 、F D2 and F D3 ;

[0031] For three sets of features F D1 、F D2 、F D3 Use depth-separable convolution kernels of different sizes to extract multi-scale features and extract feature maps F D1 、F D2 and F D3 Multi-scale features F s 、F m and F l ;

[0032] Concatenate feature maps F by channel D1 、F D2 and F D3 Multi-scale features F s 、F m and F l , multi-scale features are fused through 1×1 convolution to obtain the value vector V V ;

[0033] According to the value vector V V Extract query matching value vector V according to relevance QV .

[0034] Specifically, step S4 includes:

[0035] Construct a disease localization attention feature decoding model, which includes an adaptive feature fusion module and a linear layer;

[0036] Extract the segmentation feature map of the disease localization attention feature map through a linear layer, extract the segmentation feature map S4 from the four-stage disease localization attention feature map F4 through a linear layer, input the segmentation feature map S4 and the three-stage feature map F3 into the adaptive feature fusion module, and extract the three-stage segmentation feature map S3;

[0037] Input the three-stage segmentation feature map S3 and the two-stage feature map F2 into the adaptive feature fusion module to extract the two-stage segmentation feature map S2; input the two-stage segmentation feature map S2 and the one-stage feature map F1 into the adaptive feature fusion module to extract the one-stage segmentation feature map S1;

[0038] The sub-image blocks of the disease mask image are bilinearly downsampled in stages to obtain the sampling annotations. The resolution and segmentation features of each stage of downsampling are Figure 1 One-to-one, the sampled annotations are input into the 1×1 convolution layer to calculate the segmentation results of the segmentation feature map at each stage;

[0039] Calculate the segmentation result T according to the loss function x With L x The edge-guided loss is used to extract the segmentation result T through the FFT algorithm. x and marked L x High-frequency information H y and H l ;

[0040] Specifically, step S5 includes:

[0041] After segmentation, the results are binarized according to the disease category, and the segmentation results are converted into diseased areas and backgrounds;

[0042] The segmentation results are optimized through the opening and closing operations in morphology, and segmentation edges are refined, holes in the segmented area are filled, and small area defects are removed.

[0043] The skeleton of the binary results of tunnel crack disease is extracted through the morphology algorithm, the crack propagation trend is visualized, and the direction of the crack disease is identified.

[0044] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0045] This paper proposes a method for identifying apparent defects in tunnel linings based on query matching attention. By constructing a feature extraction model, this method matches highly correlated key and value vectors to query vectors, reducing the model's computational burden and focusing its attention on defects with a relatively small proportion. By increasing the capture of local defect features, the influence of background noise can be effectively suppressed when there is a significant scale imbalance between the defect and the background, improving defect detection accuracy. Query matching attention helps the model learn universal defect characteristics, enabling it to maintain good performance across defects in diverse environments.

[0046] By building a disease localization attention feature decoding model, deep supervision allows the model to simultaneously consider features at different levels, helping to capture disease manifestations at various scales. The presence of supervisory signals at multiple levels allows for finer control of disease boundaries, resulting in more accurate segmentation results. The introduction of adaptive feature fusion methods and linear layers in the encoder structure ensures minimal computational complexity while incorporating shallow-level features. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0048] Figure 1 is a flow chart of a method for identifying apparent defects in tunnel linings based on query matching attention in an embodiment of the present invention;

[0049] Figure 2 Schematic diagram of a feature extraction method based on query matching attention in an embodiment of the present invention;

[0050] Figure 3 Schematic diagram of deep supervised disease segmentation integrating multi-scale features in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It is obvious that the embodiments described are only some embodiments of the present invention, not all embodiments, and the implementation of the present invention is not limited to these. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0052] Example 1:

[0053] The query-matching attention-based tunnel lining surface defect recognition method described in this invention comprises five steps: data acquisition, data processing, feature extraction, defect segmentation, and post-processing of the segmentation results. This method optimizes the segmentation results using image algorithms, extracts the skeleton of the binarized tunnel crack defect results, visualizes the crack propagation trend, and identifies the crack direction. This embodiment utilizes an improved self-attention mechanism and deep supervision strategy to enhance defect segmentation capabilities.

[0054] like Figure 1 As shown, a flow chart of a method for identifying apparent defects in tunnel linings based on query matching attention is provided. The method for identifying apparent defects in tunnel linings based on query matching attention of the present invention comprises:

[0055] S1. Data acquisition: Collect tunnel lining surface images, annotate the defects in the tunnel lining surface images, and construct a defect image dataset.

[0056] S11. Using an inspection vehicle equipped with a linear array camera, collect tunnel lining surface images twice to obtain two half-width tunnel images. Using existing image acquisition software, align and stitch the two half-width tunnel images to obtain a complete tunnel lining surface image.

[0057] The line array camera on the inspection vehicle is used to capture images of the tunnel lining surface, ensuring comprehensive image coverage and high quality. Image acquisition software is used to precisely align and stitch the two captured half images together to form a complete image of the tunnel lining surface.

[0058] S12. Cutting the complete tunnel lining surface image without overlap according to a preset resolution, classifying the cut tunnel lining surface image, and obtaining the tunnel lining surface defect image and background.

[0059] The complete image was cropped at high resolution, achieving a resolution of 8192 x 4000 to ensure clear details. Manual screening and classification of the cropped images by damage type yielded images of tunnel lining surface damage and background damage. These images were further categorized as images of cracks and images of tile peeling.

[0060] S13. Label the tunnel lining surface defect image using a category number mapping method to obtain a labeled defect mask image.

[0061] Specifically, different category numbers are assigned to the categories of background, crack disease image, and tile peeling disease image, which are 0, 1, and 2 respectively. The mask image is filled pixel by pixel according to the category number to generate a disease mask image, and a detailed disease annotation map is obtained. The resolution of the annotated disease mask image is consistent with the resolution of the tunnel lining apparent image before segmentation, ensuring that the annotation information completely corresponds to the original image. This implementation uses LabelMe software to perform pixel-level annotation on the tunnel lining apparent disease image. Through manual judgment, the software draws a polygonal area on the image to select the disease, and the built-in algorithm is used to fill the selected area to obtain the annotation, which can ensure the annotation accuracy. The annotation results are stored as JSON files for easy data management and post-processing.

[0062] S14. Construct a disease image dataset, where the disease image dataset includes tunnel lining apparent disease images and labeled disease mask images.

[0063] S2. Data processing: Preprocess the tunnel lining surface defect images and the corresponding defect mask images, extract non-overlapping sub-image blocks containing defects, and construct the defect image training dataset and test dataset.

[0064] S21. Correct the brightness difference of the tunnel lining apparent disease image through the illumination unevenness correction algorithm to obtain a grayscale image after uniform illumination, thereby balancing the brightness difference of the tunnel lining apparent disease image caused by light supplementation.

[0065] Specifically, the uneven illumination correction algorithm is used to correct brightness differences in tunnel lining surface defects, including:

[0066] The tunnel lining surface disease image is converted from RGB image to grayscale image G, and the average value of all pixels in the grayscale image G is calculated as the average brightness g of the grayscale image G. m .

[0067]

[0068] Among them, M and N represent the width and height of the grayscale image G respectively. i，j Represents a pixel in the grayscale image, i and j represent the coordinates of the pixel in width and height respectively, and the coordinate intervals are i∈[0, M-1] and j∈[0, N-1].

[0069] Set the local grayscale correction window size w l The horizontal and vertical step sizes and the number of windows required to traverse the image are calculated. A local window brightness map is created based on the number of windows, and the local window brightness B is calculated. A matrix with the same size as the grayscale image is created based on the number of windows as the local window brightness map B. Each correction window is traversed, and the average value of the pixels in the window is calculated as the window average brightness, which is recorded in the corresponding position of the local window brightness map B.

[0070]

[0071] Where a and b represent the pixel coordinates of the local window brightness map B.

[0072] Calculate the local window brightness B and the average brightness g m The ratio E is used to reflect the degree of local brightness abnormality:

[0073]

[0074] The local window brightness image is upsampled to the same size as the original image, and a Gaussian blur transition window is superimposed to smooth the brightness changes. The ratio of the original grayscale image G to the ratio E is calculated, the pixel values ​​that exceed the boundary are cut, and the integer part of the pixel value is retained to obtain the grayscale image G′ after uniform illumination:

[0075]

[0076] The uniformly lighted grayscale image G′ is converted back into an RGB color image, which helps to enhance the contrast and color saturation of the image, thereby improving the expression of image details.

[0077] S22. Convert the uniformed grayscale image into an RGB color image to obtain a corrected tunnel lining apparent disease image. Cut the corrected tunnel lining apparent disease image and the corresponding disease mask image into multiple sub-image blocks, remove the disease-free sub-image blocks, and divide the remaining image blocks into a tunnel image training data set and a tunnel image test data set according to a certain ratio.

[0078] In this embodiment, the grayscale image after uniform illumination is converted into an RGB color image to increase the contrast and color saturation of the image, further improve the image details, and make the disease features more prominent. The corrected tunnel lining apparent disease image and the corresponding disease mask image are cut into multiple sub-image blocks. The resolution of crack type disease segmentation is 512×512, and the resolution of tile peeling disease segmentation is 1024×1024. Image blocks containing disease features are screened out, and sub-image blocks without diseases are eliminated to reduce invalid data and improve model training efficiency. The image blocks are divided into a tunnel image training data set and a tunnel image test data set in a ratio of 8:2 to ensure the representativeness of the data set and provide sufficient data support for model training and verification.

[0079] S3. Feature extraction: construct a feature extraction model to extract the detail feature map of the tunnel lining apparent disease image. Based on the detail feature map and the self-attention mechanism, calculate the multi-stage disease location attention feature map of the tunnel lining apparent disease image.

[0080] S31. A feature extraction model is constructed based on a neural network with a self-attention mechanism. The feature extraction model includes a query matching disease attention extraction branch and a multi-scale content feature extraction branch. The query matching disease attention extraction branch extracts features based on a convolutional neural network and calculates the matching vector using cosine similarity. The multi-scale content feature extraction branch is based on a depth-separable convolutional network with multiple branches, and a convolution layer is added at the end to fuse the calculation results of multiple branches. Finally, after completing a matching operation, the output results of the two branches are used to calculate the attention features through matrix multiplication to obtain enhanced features of the diseased area. The model uses depth-separable convolution and attention mechanisms to extract and fuse content features of different scales, thereby achieving more accurate feature representation. Secondly, when calculating the self-attention features, the model eliminates the interference of irrelevant areas and reduces the computational complexity of matrix multiplication.

[0081] S32. Extract the key low-level visual features of the tunnel lining surface disease images in the training dataset through convolutional neural network (CNN) to generate detail feature map F D .

[0082] The key low-level visual features of tunnel lining apparent disease images include edges, textures, etc. of tunnel lining apparent disease images.

[0083] S33, detail feature map F D Input the query matching-based disease attention extraction branch and calculate the query vector V Q and the key vector V that matches the query QK .

[0084] like Figure 2 FIG. 1 is a schematic diagram of a feature extraction method based on query matching attention in an embodiment of the present invention; the detail feature map F D Input the query matching-based disease attention extraction branch, and convert the detail feature map F with width W, height H and number of channels C into D , divided into S×S blocks in width and height directions, the convolutional neural network is used to extract the embedded feature E for each block, and the linear layer is used to calculate the query vector V from the embedded feature E Q and the key vector V K .

[0085] Calculate and query vector V Q Matching key vector V K , specifically including: calculating the query vector V by channel Q The mean M Q , and then calculate the key vector V by channel K The mean M K , then calculate the mean M Q and mean M K The cosine similarity of is used as the correlation index between the query vector and the key vector between blocks. Q Calculate the dimension for the target and select the same vector V as the query vector Q The top K key vectors with the highest relevance constitute the key vector set V that matches the query QK .

[0086] S34, detail feature map F D Input the multi-scale content feature extraction branch and calculate the value vector V V and the query matching value vector V QV ;

[0087] The detail feature map F D Input multi-scale content feature extraction branch, through which the detail feature map F D In the channel dimension, it is divided into three groups of features, denoted as F D1 、F D2 and F D3 , the number of channels are c1, c2 and c3 respectively.

[0088] For three sets of features F D1 、F D2、F D3 The multi-scale features are extracted using depth-wise separable convolution kernels of different sizes, which are 3×3, 5×5, and 7×7, respectively. The multi-scale features F of three groups of features are extracted. s 、F m and F l .

[0089] The multi-scale features are spliced ​​by channel and fused through 1×1 convolution to obtain the fused features as the value vector V V , according to the value vector V V Extract query matching value vector V according to relevance QV . According to the value vector V V Extract query matching value vector V according to relevance QV Specifically include: calculating the query vector V by channel Q The mean M Q , and then calculate the value vector V by channel V The mean M V , then calculate the mean M Q and mean M V The cosine similarity of is used as the correlation index between the query vector and the value vector between blocks. Q Calculate the dimension for the target and select the same vector V as the query vector Q The top K value vectors with the highest relevance constitute the value vector set V that matches the query QV .

[0090] S35. Query vector V Q , query matching key vector V Q K and query matching value vector V QV The first-stage disease localization attention feature map F1 is calculated by using the following formula:

[0091]

[0092] Where C is the key vector V QK The dimension of . Formula (5) is Figure 2 It is represented as two matrix multiplication calculations.

[0093] S36. Based on the first-stage disease location attention feature map F1, repeat the above process to construct a multi-stage deep feature extraction model, and obtain the disease location attention feature maps of each stage, represented as F1, F2, F3 and F4 respectively.

[0094] The step S35 includes:

[0095] The first-stage disease localization attention feature map F1 is input into the query matching-based disease attention extraction branch to calculate the query vector and the key vector matching the query; the first-stage disease localization attention feature map F1 is input into the multi-scale content feature extraction branch to calculate the value vector and the value vector matching the query; the query vector, query matching key vector and query matching value vector are used as the query, key and value input of the self-attention mechanism respectively to calculate the second-stage disease localization attention feature map F2;

[0096] The two-stage disease localization attention feature map F2 is input into the query matching-based disease attention extraction branch to calculate the query vector and the key vector matching the query; the two-stage disease localization attention feature map F2 is input into the multi-scale content feature extraction branch to calculate the value vector and the value vector matching the query; the query vector, query matching key vector and query matching value vector are used as the query, key and value input of the self-attention mechanism respectively to calculate the three-stage disease localization attention feature map F3;

[0097] The three-stage disease localization attention feature map F3 is input into the query matching-based disease attention extraction branch to calculate the query vector and the key vector matching the query; the three-stage disease localization attention feature map F3 is input into the multi-scale content feature extraction branch to calculate the value vector and the value vector matching the query; the query vector, query matching key vector and query matching value vector are used as the Query, Key and Value inputs of the self-attention mechanism respectively to calculate the four-stage disease localization attention feature map F4.

[0098] S4. Disease segmentation: build a disease localization attention feature decoding model, use the adaptive feature fusion method to fuse shallow and deep features, and calculate the segmentation results of the segmentation feature map of each stage through the deep supervision strategy.

[0099] like Figure 3 FIG. 4 is a schematic diagram of deep supervised disease segmentation integrating multi-scale features in an embodiment of the present invention. Specifically, step S4 includes:

[0100] S41. Construct a disease localization attention feature decoding model, which includes an adaptive feature fusion module and a linear layer.

[0101] Specifically, the disease localization attention feature decoding model includes an adaptive feature fusion module and a linear layer. The adaptive feature fusion module fuses the disease localization attention feature maps generated at each stage, and the fused features are input into the segmentation layer based on the convolutional neural network to predict the disease segmentation results. The adaptive feature fusion module introduces a linear layer to adjust the dimension of the feature map. The features are fused with the features of the previous layer through the upsampling layer, pooling layer and linear layer to restore the detailed information lost due to downsampling. For each layer of recovered features, the model introduces a segmentation layer to obtain disease segmentation results of different scales. The segmentation loss of the segmentation results and segmentation annotations is calculated to guide model training.

[0102] S42, extracting a segmentation feature map of the disease localization attention feature map through a linear layer, extracting a segmentation feature map S4 from the four-stage disease localization attention feature map F4 through a linear layer, inputting the segmentation feature map S4 and the three-stage feature map F3 into an adaptive feature fusion module, and extracting a three-stage segmentation feature map S3;

[0103] The adaptive feature fusion module will gradually fuse the feature maps of different stages. The upsampled segmentation feature map S4 has the same resolution as the feature map F3 of the previous stage. Then the two are input into the adaptive feature fusion module to extract the segmentation feature map S3 of this stage.

[0104] The adaptive feature fusion module is used to splice the disease localization attention feature map F at the current stage by channel. x Compared with the segmentation feature map S of the previous stage x-1 , get the channel attention feature map C x , calculate the channel attention feature map C x The average value of each channel in the linear layer is input, and the fusion ratio A of the feature is obtained through the Softmax activation function x , using the fusion ratio A x Calculate the fused segmentation feature map S x .

[0105] S x =A x ·S x-1 +(1-A x )·F x (6)

[0106] S42, input the three-stage segmentation feature map S3 and the two-stage feature map F2 into the adaptive feature fusion module to extract the two-stage segmentation feature map S2; input the two-stage segmentation feature map S2 and the one-stage feature map F1 into the adaptive feature fusion module to extract the one-stage segmentation feature map S1, and obtain the segmentation feature maps S2 and S1.

[0107] S43, using a deep supervision strategy, bilinearly downsample the sub-image blocks of the disease mask image in stages to obtain the annotations after sampling. The resolution and segmentation features of each stage of downsampling are Figure 1 One by one, the sampled annotations are input into the 1×1 convolution layer to calculate the segmentation results of the segmentation feature map at each stage.

[0108] The use of deep supervision strategies can further improve the generalization ability and stability of the model, such as Figure 3 The dotted double arrow part. For annotation, bilinear interpolation downsampling is a commonly used image resizing algorithm. It estimates the value of the new position by weighted average of the four nearest known points. The implementation details are seldom explained. Bilinear downsampling is performed on the sub-image blocks of the disease mask image generated in step S2 to ensure that the annotation resolution of each stage matches the segmentation feature map. The annotations L1, L2, L3 and L4 are obtained and input into the 1×1 convolution layer to calculate the segmentation feature map S at each stage. x The segmentation result Y x .

[0109] S44. Calculate segmentation result Y x With L x The edge-guided loss guides the continuity of the disease segmentation

[0110] Extract the segmentation result T through FFT algorithm x and marked L x High-frequency information H y and H l , calculate the edge guidance loss according to the loss function. High frequency information H y and H l Used for edge detail alignment to improve the continuity and accuracy of disease segmentation.

[0111] The segmentation result T is extracted by Fourier transform (FFT) algorithm x and marked L x High-frequency information H y and H l , follow these steps:

[0112] Convert the image to the frequency domain, for the segmentation result Y x and marked L x For each pixel value in , a two-dimensional discrete Fourier transform (DFT) is performed. The two-dimensional discrete Fourier transform formula is as follows:

[0113]

[0114] Among them, f(x, y) is the pixel value of the original image, F(u, v) is the corresponding frequency domain value, M and N are the height and width of the image respectively, and j is the imaginary unit.

[0115] Extract high-frequency information. In the frequency domain, low-frequency information is located near the center, while high-frequency information is located at the edge. Therefore, we can only focus on the edge of the spectrum to extract high-frequency information. x and L x The spectrum F x and F l , the elements in the non-central area can be taken as high-frequency information H y and H l The edge-guided loss can be calculated from the obtained high-frequency information. High-frequency information usually contains edge and texture details in the image. By comparing the predicted results with the annotated high-frequency information, the model can focus on capturing edge and texture changes, while more accurately locating edges and contours, thereby improving the accuracy of disease recognition.

[0116] The edge-guided loss function is defined as follows:

[0117]

[0118] Among them, H yr 、H yi Represent high-frequency information H y The imaginary and real parts, H lr 、H li Represent high-frequency information H l The imaginary and real parts of , α and β are the balancing coefficients, which are used to adjust the relative importance of the two loss parts.

[0119] S5. Post-processing the segmentation results: binarizing and optimizing the segmentation results to obtain optimized binarization results of tunnel crack diseases, and identifying the direction of the crack diseases based on the optimized binarization results of the tunnel crack diseases.

[0120] Specifically, step S5 includes:

[0121] S51, performing binarization processing on the segmentation results according to the disease category, and converting the segmentation results into the disease area and background;

[0122] First, the segmentation results output by the model are binarized according to different disease categories. This process converts the segmentation results into clear foreground (disease area) and background, which is convenient for subsequent morphological operations and analysis.

[0123] S52. Optimize the segmentation results through morphological opening and closing operations, perform segmentation edge refinement, fill holes in the segmented area, remove small area defects, and other operations to further improve the accuracy and completeness of disease recognition;

[0124] Post-processing the segmentation results using morphological opening and closing operations removes burrs and noise from the defective area, smoothing the edges. Small holes within the defective area are filled to ensure the integrity of the defective area. Segmentation results with too small an area, which may be noise or non-defective areas, are removed, improving the accuracy of defect recognition.

[0125] S53. The skeleton of the binary results of tunnel crack disease is extracted through the morphology algorithm, the crack propagation trend is visualized, and the direction of the crack disease is identified.

[0126] Specifically, the steps for extracting the skeleton of tunnel crack defects using a morphology algorithm include: Binarizing the crack image. This involves converting the crack segmentation result into black and white, with black representing the background and white representing the crack. Thinning the object in the binary image to its skeleton, retaining the central core of the crack while removing the filler, making it appear as a thin line. Dilation and erosion are then performed on the skeleton. By alternating these dilation and erosion operations, pixels surrounding the crack are gradually removed until only a single-pixel-wide line remains, representing the crack skeleton. Branch points are removed from the skeleton. The thinning result may contain branch points, which do not represent the true path of the crack. Therefore, further processing is required to remove these points, leaving only a continuous skeleton. Connecting broken skeletons. During the thinning process, some cracks may be cut off. To fully represent the crack, these broken sections must be reconnected.

[0127] Using the Morphology algorithm, we focus on the binary results of crack damage and extract the skeleton of the binary results. This skeleton represents the main extension path of the cracks. This step can visually demonstrate the propagation trend of the cracks, helping analysts quickly understand their development and providing a basis for subsequent repair plans.

[0128] In summary, the present invention proposes a method for identifying apparent defects in tunnel linings based on query matching attention. By introducing a defect attention extraction mechanism based on query matching, the model can focus on the key local features of the defect. Even when there is a significant scale difference between the defect and the background, it can effectively suppress background noise interference, significantly improving the accuracy and robustness of defect detection. The use of a deep supervision strategy enables the model to simultaneously consider and fuse multi-scale features at different levels, which not only helps to fully capture the manifestations of defects at the micro and macro levels, but also, by introducing supervisory signals at multiple levels, can more finely control the depiction of defect boundaries, thereby producing more accurate segmentation results. The adaptive feature fusion method and linear layer introduced in the encoder structure achieve effective fusion of shallow features while ensuring low computational complexity. This design not only ensures that the model fully utilizes shallow information, but also avoids the possibility of affecting real-time performance or large-scale deployment due to excessive consumption of computing resources. This method comprehensively utilizes the query matching attention mechanism, deep supervision strategy and adaptive feature fusion technology, effectively overcoming the limitations of traditional disease recognition methods in complex environments, significantly improving the accuracy and efficiency of disease recognition, and providing a more reliable technical means for tunnel lining health monitoring and maintenance.

[0129] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A method for identifying apparent defects in tunnel linings based on query matching attention, characterized in that: The following steps are involved: S1. Collect tunnel lining surface images, mark the defects in the tunnel lining surface images, and build a defect image dataset; S2. Preprocess the tunnel lining surface defect images and the corresponding defect mask images, extract non-overlapping sub-image blocks containing the defects, and construct a defect image dataset; S3. Construct a feature extraction model to extract the detail feature map of the tunnel lining apparent disease image, and calculate the multi-stage disease location attention feature map of the tunnel lining apparent disease image based on the detail feature map combined with the self-attention mechanism; The step S3 comprises: S31. Construct a feature extraction model based on a self-attention mechanism neural network. The feature extraction model includes a query matching disease attention extraction branch and a multi-scale content feature extraction branch. S32. Extract key low-level visual features of tunnel lining surface damage images through convolutional neural networks to generate detail feature maps ; S33, detail feature map Input the query matching-based disease attention extraction branch and calculate the query vector and the key vector that matches the query ; S34, detail feature map Input multi-scale content feature extraction branch and calculate the value vector and the query matching value vector ; S35, query vector , query matching key vector and query matching value vector As the query, key and value input of the self-attention mechanism, calculate the first-stage disease location attention feature map ; S36, Attention Feature Map Based on One-stage Disease Localization , respectively calculate the two-stage disease location attention feature map , get the three-stage disease location attention feature map , Four-stage disease location attention feature map ; S4. Construct a disease localization attention feature decoding model, use an adaptive feature fusion method to fuse shallow and deep features, and calculate the segmentation results of the segmentation feature map at each stage through a deep supervision strategy; S5. Binarization and optimization are performed on the segmentation results to obtain an optimized binarization result of the tunnel crack disease, and the crack disease direction is identified according to the optimized binarization result of the tunnel crack disease.

2. The method for identifying apparent defects in tunnel linings based on query matching attention according to claim 1 is characterized in that: The step S1 comprises: The inspection vehicle equipped with a linear array camera collects tunnel lining surface images twice to obtain two half-tunnel images, which are then aligned and spliced ​​to obtain a complete tunnel lining surface image. The complete tunnel lining surface image is cut without overlap according to a preset resolution, and the cut tunnel lining surface image is classified to obtain a tunnel lining surface disease image and a background; the tunnel lining surface disease image is divided into a crack disease image and a tile peeling disease image; The tunnel lining surface defect image is annotated by the category number mapping method to obtain the annotated defect mask image. A damage image dataset is constructed, which includes tunnel lining apparent damage images and annotated damage mask images.

3. The method for identifying apparent defects in tunnel linings based on query matching attention according to claim 2 is characterized in that: The step S2 comprises: The uneven illumination correction algorithm is used to correct the brightness difference of the tunnel lining surface defect image and obtain a uniformly illuminated grayscale image. The uniformed grayscale image is converted into an RGB color image to obtain a corrected tunnel lining apparent disease image. The corrected tunnel lining apparent disease image and the corresponding disease mask image are cut into multiple sub-image blocks, and the disease-free sub-image blocks are eliminated. The remaining image blocks are divided into a tunnel image training dataset and a tunnel image test dataset according to several proportions.

4. The method for identifying apparent defects in tunnel linings based on query matching attention according to claim 1 is characterized in that: The step S33 includes: extracting the detail feature map by querying the matched disease attention branch Split into width and height Blocks, and embedding features are extracted for each block through convolutional neural network , using linear layers to embed features Calculate the query vector With key vector ; Compute query vectors channel by channel The mean , and then calculate the key vector by channel The mean , calculate the mean and mean The cosine similarity of , as the correlation between the query vector and the key vector of each block; Query vector Calculate the dimension for the target, select and query vector The most relevant key vectors, forming a set of key vectors that match the query .

5. The method for identifying apparent defects in tunnel linings based on query matching attention according to claim 1 is characterized in that: The step S34 includes extracting the detail feature map by a multi-scale content feature extraction branch. It is divided into three groups of features in the channel dimension, and the three groups of features are represented as 、 and ; For three groups of features 、 、 Use depth-separable convolution kernels of different sizes to extract multi-scale features and extract feature maps 、 and Multi-scale features 、 and ; Concatenate feature maps by channel 、 and Multi-scale features 、 and , multi-scale features are fused through 1×1 convolution to obtain the value vector ; According to the value vector Extract query matching value vectors based on relevance .

6. The method for identifying apparent defects in tunnel linings based on query matching attention according to claim 1 is characterized in that: The step S35 includes: Positioning the first-stage disease on the attention feature map Input the query matching-based disease attention extraction branch, calculate the query vector and the key vector matching the query; locate the first-stage disease attention feature map Input the multi-scale content feature extraction branch and calculate the value vector and the query matching value vector; use the query vector, query matching key vector and query matching value vector as the query, key and value input of the self-attention mechanism respectively to calculate the second-stage disease location attention feature map ; Positioning the second-stage disease on the attention feature map Input the query matching-based disease attention extraction branch, calculate the query vector and the key vector matching the query; locate the second-stage disease attention feature map Input the multi-scale content feature extraction branch and calculate the value vector and the query matching value vector; use the query vector, query matching key vector and query matching value vector as the query, key and value input of the self-attention mechanism respectively to calculate the three-stage disease location attention feature map ; Positioning the three-stage disease attention feature map Input the query matching-based disease attention extraction branch, calculate the query vector and the key vector matching the query; locate the three-stage disease attention feature map Input the multi-scale content feature extraction branch and calculate the value vector and the query matching value vector; use the query vector, query matching key vector and query matching value vector as the query, key and value input of the self-attention mechanism respectively, and calculate the four-stage disease location attention feature map .

7. The method for identifying apparent defects in tunnel linings based on query matching attention according to claim 1 is characterized in that: The step S4 comprises: Construct a disease localization attention feature decoding model, which includes an adaptive feature fusion module and a linear layer; The segmentation feature map of the disease localization attention feature map is extracted through the linear layer, and the segmentation feature map of the disease localization attention feature map is extracted through the linear layer from the four-stage disease localization attention feature map Extract segmentation feature maps , segment the feature map and three-stage feature maps Input the adaptive feature fusion module to extract the three-stage segmentation feature map ; The three-stage segmentation feature map and the second-stage feature map Input the adaptive feature fusion module to extract the two-stage segmentation feature map ; Segment the feature map of the second stage With the one-stage feature map Input the adaptive feature fusion module to extract the first-stage segmentation feature map ; Perform bilinear downsampling on the sub-image blocks of the disease mask image in stages to obtain the sampled annotations. The resolution of the downsampling at each stage corresponds one-to-one to the segmentation feature map. The sampled annotations are input into a 1×1 convolutional layer to calculate the segmentation results of the segmentation feature map at each stage. Calculate the segmentation result based on the loss function and annotation The edge-guided loss is used to extract the segmentation results through the FFT algorithm. and annotations High-frequency information and .

8. The method for identifying apparent defects in tunnel linings based on query matching attention according to claim 7 is characterized in that: The adaptive feature fusion module is used to splice the disease localization attention feature map of the current stage with the segmentation feature map of the previous stage by channel to obtain a channel attention feature map; calculate the average value of each channel in the channel attention feature map and input it into the linear layer, obtain the feature fusion ratio through the Softmax activation function, and use the fusion ratio to calculate the fused segmentation feature map.

9. The method for identifying apparent defects in tunnel linings based on query matching attention according to claim 7 is characterized in that: The step S5 comprises: After segmentation, the results are binarized according to the disease category, and the segmentation results are converted into diseased areas and backgrounds; The segmentation results are optimized through the opening and closing operations in morphology, and segmentation edges are refined, holes in the segmented area are filled, and small area defects are removed. The skeleton of the binary results of tunnel crack disease is extracted through the morphology algorithm, the crack propagation trend is visualized, and the direction of the crack disease is identified.

Images