Airport runway underground disease automatic detection method based on image text guidance

Abstract

The invention discloses an automatic detection method for underground diseases of an airport runway based on image text guidance. The method comprises the following steps: constructing an airport runway underground data set; making text description on each type of underground diseases by professionals according to experience; constructing an original text-guided multi-modal detection model, and training the original text-guided multi-modal detection model; and inputting a B-scan GPR image of the airport runway, which is acquired in real time, into the text-guided multi-modal detection model, and finally outputting a corresponding bounding box and a category prediction result. According to the method, the text mode is introduced into the model, and the recognition capability of the model on intra-class feature diversity is enhanced by utilizing the high generality of the text mode when key features of the disease morphology are expressed. And a learnable text embedding mechanism is adopted, so that the model can dynamically optimize text embedding according to data, and the adaptability and detection performance of the model to various characteristics are more efficiently improved. Horizon distribution prior information is introduced, and different types of targets with similar forms can be effectively distinguished by using spatial distribution characteristics of underground targets.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of airport runway performance detection, and particularly relates to an airport runway underground disease automatic detection method based on image text guidance. BACKGROUND

[0002] The runway is the core part of airport infrastructure, bearing the key task of aircraft take-off and landing. With the increase of aircraft flow and the long-term high-load use of the runway, the underground structure below it is prone to various diseases. These diseases not only affect the carrying capacity of the runway, but also may cause serious safety accidents. Therefore, being able to accurately detect the underground diseases of the airport runway is of great importance to ensure the safety of the airport runway and prolong its service life.

[0003] Currently, ground penetrating radar (GPR) is widely used in airport runway underground disease detection due to its advantages of non-destructive, fast and comprehensive coverage. This technology transmits high-frequency electromagnetic waves and receives their reflected signals to detect underground targets according to the propagation characteristics and time delay.

[0004] Traditionally, GPR images are usually interpreted by experts according to experience to analyze underground disease information. However, the manual analysis method is inefficient and highly dependent on expert experience. As a more efficient and objective means, deep learning gradually replaces the traditional manual analysis method. However, due to the significant differences in the geological environment of different airports, including the composition, structure and humidity of underground media. At the same time, underground diseases have irregularity and diversity in shape and size. These factors together increase the complexity of radar data, which is specifically manifested in: (1) large intra-class feature difference. The same type of underground disease may exhibit significantly different reflection characteristics in different environments, Figure 1 (a) is an example of large intra-class shape difference of two types of underground diseases, which makes it difficult for the model to extract stable and discriminative features, and is prone to missed detection. (2) high inter-class feature similarity. As shown in Figure 1 (b), underground diseases of different categories may exhibit similar image features in some cases, which is prone to model misclassification.

[0005] In addition, underground diseases have obvious distribution density differences at different depths, Figure 2 shows the distribution density of four types of underground diseases, Gap, Crack, Subsidence, and Rebar, or at different depths in the data collected from multiple airports.

[0006] For the problem of underground disease detection based on GPR images, researchers in the field have proposed various methods to improve the accuracy and robustness of detection. Existing research can be divided into three categories: methods based on traditional image processing and signal analysis, methods based on machine learning, and methods based on deep learning. Each method has significant differences in feature extraction, adaptability to environmental changes, and ability to handle complex targets.

[0007] 1) Traditional image processing and signal analysis: Early research relies heavily on the geometric features or frequency domain features of GPR images. This type of method focuses on locating and identifying targets by extracting geometric features. For example, using least squares method to analyze GPR image differences to achieve real-time underground target detection. However, these traditional methods are sensitive to environmental changes and have limitations in handling complex, noisy data.

[0008] 2) Machine learning: To overcome the limitations of traditional methods in feature expression and adaptability to complex scenarios, machine learning methods are introduced to improve the accuracy and generalization ability of underground target detection. This type of method usually relies on manually designed features to enhance the model's ability to identify targets. For example, by combining the Viola-Jones algorithm and the Hough transform method, automatic positioning of hyperbolic curves in GPR images is achieved to detect the location of underground objects. However, these machine learning methods are highly dependent on feature engineering, and due to the complexity of GPR images, this process is highly challenging, which affects the performance and reliability of the model. 3) Deep learning: In recent years, the introduction of deep learning technology has greatly promoted the development of target detection technology based on GPR images. Compared with the previous methods, deep learning methods can automatically learn complex features from a large amount of data, reducing the dependence on manual feature engineering, thereby significantly improving the performance of the model. These methods are usually based on classic target detection algorithms, using deep network structures such as convolutional neural networks (CNN) to effectively handle complex visual tasks, providing higher accuracy and robustness. For example, by improving the Faster R-CNN and Mask R-CNN methods, targets in GPR images are detected. In recent years, some research has begun to explore data fusion methods, aiming to combine GPR images from different perspectives or different types to more comprehensively and accurately analyze underground targets, thereby improving the reliability and robustness of detection.

[0009] Although deep learning methods can significantly improve target detection accuracy, all current underground disease detection methods based on GPR images are still limited to using only GPR images. Single modal data often fails to provide sufficient contextual information, to some extent, limiting the performance and applicability of the method, leading to limitations in detection performance, prone to false positives and false negatives. SUMMARY

[0010] To solve the above problems, the purpose of the present application is to provide an airport runway underground disease automatic detection method based on image text guidance.

[0011] To achieve the above purpose, the airport runway underground disease automatic detection method based on image text guidance provided by the present application comprises the following steps in sequence:

[0012] 1) Using a robot equipped with GPR to drive on multiple airport runways according to a pre-planned path, using GPR to perform full-coverage path scanning on the airport runway, collecting a large number of B-scan GPR images, then transmitting the B-scan GPR images in real time to a mobile data analysis center through a 4G network or Wi-Fi for storage, and then performing preprocessing including removing part of the noise and background interference and unifying the size; the airport runway underground data set is composed of all the preprocessed GPR images;

[0013] 2) A professional makes a text description of each type of underground disease according to experience;

[0014] 3) Construct an original text-guided multi-modal detection model; the original text-guided multi-modal detection model includes a learnable text embedding extraction module, an image feature extraction module, a text-image-layer information fusion module, and a detection head;

[0015] 4) Input the preprocessed GPR images in the airport runway underground data set obtained in step 1) and the text description obtained in step 2) into the original text-guided multi-modal detection model obtained in step 3) for training, to obtain a trained text-guided multi-modal detection model;

[0016] 5) After preprocessing the real-time collected B-scan GPR images of the airport runway to remove part of the noise and background interference and unify the size, input them into the above trained text-guided multi-modal detection model, and finally output the corresponding bounding box and class prediction results.

[0017] In step 1), the GPR uses equally spaced 14-channel Raptor tM GPR, each channel is 0.7 meters apart, is equipped with a 900MHz antenna, and is set to a distance trigger mode to ensure synchronous sampling; the airport runway underground data set includes four typical underground diseases of void, crack, settlement, and reinforcement.

[0018] In step 2), the text description is represented as:

[0019] T={t1,t2,…,t N} (1)

[0020] Where N represents the number of categories, t k This represents a textual description of the kth type of underground disease.

[0021] In step 3), the learnable text embedding extraction module includes a CLIP text encoder for extracting key information from the text and generating text embeddings; the image feature extraction module includes a convolutional neural network-based image encoder for extracting feature maps from the preprocessed GPR image; the text-image-layer information fusion module includes a Dark Bottleneck module, a fully connected layer, and a multimodal fusion attention module for dynamically integrating text embeddings and layer prior knowledge information into the feature maps; the detection head includes a bounding box branch and a class branch, which complete the final bounding box and class prediction based on the fused feature maps.

[0022] In step 4), the preprocessed GPR image from the airport runway underground dataset obtained in step 1) and the text description obtained in step 2) are input into the original text-guided multimodal detection model obtained in step 3) for training, and the method for obtaining the trained text-guided multimodal detection model is as follows:

[0023] 4.1) Extract text embeddings from text descriptions using a learnable text embedding extraction module;

[0024] Inputting N category-related text descriptions T into a learnable text embedding extraction module, and generating text embeddings using a Transformer-based CLIP text encoder, is represented as follows:

[0025] E = [e1; e2; ...; e N ]∈R N×D (2)

[0026] Among them, e k =f text (t k )∈R D f represents the embedding of the Kth class of text description. text This indicates the CLIP text encoder, where D represents the dimension of the text embedding.

[0027] 4.2) The image feature extraction module is used to extract three feature maps of different scales from the input preprocessed GPR image;

[0028] A B-scan GPR image of size h*w*c is input into the image feature extraction module, where h represents the image height, w represents the image width, and c represents the number of channels. Then, a convolutional neural network-based image encoder performs preliminary feature extraction, outputting three feature maps P1, P2, and P3 at different scales. Feature map P1 is a low-level feature map obtained by downsampling the B-scan GPR image by 8 times, with 256 channels; feature map P2 is a mid-level feature map obtained by downsampling the B-scan GPR image by 16 times, with 512 channels; and feature map P3 is a high-level feature map obtained by downsampling the B-scan GPR image by 32 times, with 512 channels.

[0029] 4.3) The text embedding and multi-scale feature maps are fused using the text-image-layer information fusion module;

[0030] 4.3.1) Upsample feature map P3 progressively and concatenate it with feature maps P1 and P2 along the channel dimension to generate a concatenated feature map P. i，j After each stitch, the Dark Bottleneck module is used to analyze the stitched feature map P. i，j The processing is represented as follows:

[0031] P′ i,j =f dark (P i,j )∈R h′×w′×c′ ,i,j∈{1,2,3} (3)

[0032] Among them, P′ i,j This represents the processed feature map, where h′, w′, and c′ represent the height, width, and channel dimension of the feature map, respectively.

[0033] 4.3.2) Using fully connected layers to embed text E∈R N×D The embedding dimension is processed to match the channel dimension c′ of the feature map:

[0034]

[0035] 4.3.3) The processed feature map P′ i，j ∈R h′×w′×c′ Flattened into a two-dimensional matrix according to spatial location Then, the similarity score S between each feature point in the two-dimensional matrix and each type of text embedding after dimension processing is calculated by dot product. x，y，k Based on all similarity scores S x，y，k Obtain the image-text similarity score map S, and finally reconstruct the feature map back into its spatial structure:

[0036]

[0037] Among them, S x,y,k This represents the similarity score between a feature point in the feature map and the text embedding of category k at the spatial location (x, y) in the feature map.

[0038] 4.3.4) After obtaining the similarity score for each category, the weights W∈R are distributed hierarchically. h′×N The similarity scores are dynamically modulated, where h′ represents the height of the feature map and N represents the number of categories. The method involves first expanding the layer distribution weights W along the width dimension, and then adjusting the similarity score S for each spatial location. x，y ∈R N Its corresponding stratum distribution weight Performing element-wise multiplication and extending the above operation to all spatial locations of the entire feature map, it can be represented as:

[0039]

[0040] 4.3.5) The multimodal fusion attention module selects the maximum similarity score of each feature point as the final similarity score for that feature point:

[0041]

[0042] Among them, M x，y This represents the similarity score of the most relevant category at spatial location (x, y); a two-dimensional response map M∈R is constructed from the similarity scores of all feature points. h′×w′ ;

[0043] 4.3.6) The multimodal fusion attention module applies the Sigmoid activation function to the similarity score of each feature point to generate the attention weights corresponding to that feature point:

[0044] A=σ(M) (9)

[0045] Where σ(·) represents the Sigmoid function, which maps the similarity score to the range [0,1], indicating the importance of the feature point in the final feature map;

[0046] After processing all feature points through the above steps, a complete weight matrix is ​​generated. This weight matrix is ​​then expanded along the channel dimension and multiplied element-wise with the feature map P from the input text-image-layer information fusion module to obtain the updated feature map P″.

[0047] P″=P′ i,j ⊙A (10)

[0048] Among them, P′ i，jP″ represents the feature map of the input text-image-layer information fusion module, and P″ represents the feature map updated by the text-image-layer information fusion module.

[0049] 4.3.7) First, the updated feature map P′1 is obtained. Then, the updated feature map P′1 is downsampled layer by layer and fused with the corresponding scale features to generate the updated feature maps P′2 and P′3 respectively.

[0050] 4.4) The updated feature maps P′1, P′2, and P′3 from the above multi-scale model are input into three independent detection heads to predict bounding boxes and categories. The bounding box branch processes the updated feature maps through convolutional layers to generate an output with 4 × reg_max channels, which is used to predict the distributed regression values ​​of the bounding boxes. Here, reg_max represents the number of discretized intervals for each bounding box coordinate. The model obtains more accurate regression results by predicting the probability distribution of these discretized intervals. The category branch processes the updated feature maps through another set of convolutional operations to generate the probability distribution of the target category by outputting the number of channels equal to the number of categories. This yields the trained text-guided multimodal detection model.

[0051] The automatic detection method for underground defects in airport runways based on image and text guidance provided by this invention has the following beneficial effects:

[0052] Compared to image modalities, text modalities can leverage the experience of human experts to extract potentially important information from radar images. In this way, text can provide targeted guidance to the model, helping it to more comprehensively understand the manifestations of various underground targets. Therefore, this invention introduces a text modal into the model, utilizing its high generalization in expressing key features of disease morphology to enhance the model's ability to recognize intra-class feature diversity. Furthermore, to avoid the subjectivity and limitations of manually designed text description templates, the model employs a learnable text embedding mechanism. This design allows the model to dynamically optimize text embeddings based on data, thereby more efficiently improving the model's adaptability to various features and its detection performance. In addition, prior information on stratigraphic distribution is introduced into the model. By utilizing the spatial distribution characteristics of underground targets, the model can effectively distinguish between different categories of targets with similar morphologies. By fusing radar image and text modal information, the model's ability to learn complex target features is significantly enhanced. Simultaneously, the introduction of stratigraphic information, utilizing the spatial distribution characteristics of underground targets, further optimizes the model's understanding of the spatial distribution of targets, effectively improving the model's ability to distinguish between targets with similar morphologies. Attached Figure Description

[0053] Figure 1The images are GPR-based images of underground lesions; (a) is an example of significant intra-class morphological variation, with each group containing two images arranged vertically; (b) is an example of inter-class similar underground lesions.

[0054] Figure 2 Data collected from multiple airport runways showed differences in the distribution density of four types of underground defects—vacuuming, cracks, settlement, and steel reinforcement—at different depths.

[0055] Figure 3 This is a schematic diagram of the automatic detection method for underground defects in airport runways based on image and text guidance provided by the present invention.

[0056] Figure 4 This is a framework diagram of the original text-guided multimodal detection model in this invention.

[0057] Figure 5 This is a schematic diagram of the multimodal fusion attention module in this invention.

[0058] Figure 6 For comparison of experimental bar charts.

[0059] Figure 7 Visualize and compare the detection results of different models on the test set.

[0060] Figure 8 Visualize and compare the detection results of different models on cross-domain test sets. Detailed Implementation

[0061] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0062] like Figure 3 As shown, the automatic detection method for underground defects in airport runways based on image and text guidance provided by the present invention includes the following steps performed in sequence:

[0063] 1) Robots equipped with GPR (Geometry Preview) travel along pre-planned paths on multiple airport runways, performing full-coverage path scanning to ensure that every area of ​​the runways is detected without omission. This process collects a large number of B-scan GPR images, which are then transmitted in real-time to a mobile data analysis center via 4G network or Wi-Fi for storage. Preprocessing is then performed, including noise and background interference removal and size standardization, to enhance the signal of the target subsurface defects, thereby improving data quality and the accuracy of subsequent analysis. All preprocessed GPR images constitute the Airport Runway Subsurface Dataset (AUD).

[0064] The GPR uses a 14-channel Raptor with equal spacing. TNGPR, with each channel spaced 0.7 meters apart, is equipped with a 900MHz antenna and set to distance-triggered mode to ensure synchronous sampling; the airport runway underground data collection includes four typical underground defects: voids, cracks, settlement, and steel reinforcement.

[0065] 2) Professionals will provide textual descriptions of each type of underground disease based on their experience;

[0066] The text description is expressed as follows:

[0067] T = {t1, t2, ..., t} N} (1)

[0068] Where N represents the number of categories, t k This represents a textual description of the kth type of underground disease.

[0069] 3) Constructing such Figure 4 The original text-guided multimodal detection model (TIGNet) shown is included; the original text-guided multimodal detection model includes a learnable text embedding extraction module, an image feature extraction module, a text-image-layer information fusion module, and a detection head;

[0070] The learnable text embedding extraction module includes a CLIP text encoder for extracting key information from text and generating text embeddings; the image feature extraction module includes a convolutional neural network-based image encoder for extracting feature maps from the preprocessed GPR image; the text-image-layer information fusion module includes a DarkBottleneck module, a fully connected layer, and a multimodal fusion attention module (MFAM) for dynamically integrating text embeddings and layer prior knowledge into the feature maps; the detection head includes bounding box branches and class branches, which complete the final bounding box and class prediction based on the fused feature maps.

[0071] 4) Input the preprocessed GPR image from the airport runway underground dataset obtained in step 1) and the text description obtained in step 2) into the original text-guided multimodal detection model obtained in step 3) for training, and obtain the trained text-guided multimodal detection model.

[0072] The method is as follows:

[0073] 4.1) Extract text embeddings from text descriptions using a learnable text embedding extraction module;

[0074] Inputting N category-related text descriptions T into a learnable text embedding extraction module, and generating text embeddings using a Transformer-based CLIP text encoder, is represented as follows:

[0075] E = [e1; e2; ...; eN ]∈R N×D (2)

[0076] Among them, e k =f text (t k )∈R D f represents the embedding of the Kth class of text description. text This represents the CLIP text encoder, where D represents the dimension of the text embedding.

[0077] The generated text embedding E provides basic semantic features. Since different text descriptions generate different text embeddings, thus affecting the model's detection performance, finding a suitable text description is crucial. However, manually adjusting the text description to achieve the best results is time-consuming and laborious; allowing the model to adaptively optimize and adjust during training is clearly a more efficient strategy. Furthermore, this invention does not choose to directly tune the text encoder because the CLIP text encoder's parameters are trained on a large amount of data, and it already possesses excellent semantic representation and generalization capabilities—its strengths. Therefore, this invention first generates text embeddings using the CLIP text encoder, providing a good initialization, and then freezes the CLIP text encoder during model training, optimizing only the text embeddings.

[0078] 4.2) The image feature extraction module is used to extract three feature maps of different scales from the input preprocessed GPR image;

[0079] A B-scan GPR image of size h*w*c is input into the image feature extraction module, where h represents the image height, w represents the image width, and c represents the number of channels. Then, a convolutional neural network-based image encoder performs preliminary feature extraction, outputting three feature maps P1, P2, and P3 at different scales. Feature map P1 is a low-level feature map obtained by downsampling the B-scan GPR image by 8 times, with 256 channels; feature map P2 is a mid-level feature map obtained by downsampling the B-scan GPR image by 16 times, with 512 channels; and feature map P3 is a high-level feature map obtained by downsampling the B-scan GPR image by 32 times, with 512 channels.

[0080] 4.3) Utilizing, for example Figure 4 The text-image-layer information fusion module shown fuses the aforementioned text embeddings and multi-scale feature maps;

[0081] 4.3.1) Upsample feature map P3 progressively and concatenate it with feature maps P1 and P2 along the channel dimension to generate a concatenated feature map P. i，jAfter each stitch, the Dark Bottleneck module is used to analyze the stitched feature map P. i，j The processing is represented as follows:

[0082] P′ i,j =f dark (P i,j )∈R h′×w′×c′ ,i,j∈{1,2,3} (3)

[0083] Among them, P′ i,j This represents the processed feature map, where h′, w′, and c′ represent the height, width, and channel dimension of the feature map, respectively.

[0084] 4.3.2) Using fully connected layers to embed text E∈R N×D The embedding dimension is processed to match the channel dimension c′ of the feature map:

[0085]

[0086] 4.3.3) The processed feature map P′ i，j ∈R h′×w′×c′ Flattened into a two-dimensional matrix according to spatial location Then, the similarity score S between each feature point in the two-dimensional matrix and each type of text embedding after dimension processing is calculated by dot product. x，y，k Based on all similarity scores S x，y，k Obtain the image-text similarity score map S, and finally reconstruct the feature map back into its spatial structure:

[0087]

[0088] Among them, S x,y,k This represents the similarity score between a feature point in the feature map and the text embedding of category k at the spatial location (x, y) in the feature map.

[0089] 4.3.4) After obtaining the similarity score for each category, the weights W∈R are distributed hierarchically. h′×NThe similarity score is dynamically modulated, and the layer distribution weights reflect the distribution preferences of different semantic categories at different layers. Here, h′ represents the height of the feature map, N represents the number of categories, and N represents the model generating an independent weight distribution for each category. Each column in the layer distribution weights W corresponds to the weighted value of the same category at different layers, and each row represents the weighted value of different categories at a specific layer. The multimodal fusion attention module multiplies the similarity score of each category with its weighted value at different layers, thereby dynamically adjusting the similarity score, strengthening the response of regions consistent with semantics and layer, and suppressing high-similarity regions that do not conform to the layer distribution rules. The method is to first expand the layer distribution weights W along the width dimension, and then... x，y ∈R N Its corresponding stratum distribution weight Performing element-wise multiplication and extending the above operation to all spatial locations of the entire feature map can be represented as:

[0090]

[0091] In this way, the model can dynamically adjust the influence of different categories of text information on different spatial levels of the image based on the hierarchical distribution characteristics of the categories, ensuring that the model gives appropriate attention to each category of target at different levels.

[0092] 4.3.5) The multimodal fusion attention module selects the maximum similarity score for each feature point as the final similarity score for that feature point. This step, based on the needs of the feature extraction stage, aims to determine the most relevant category for each feature point and ignore the interference of irrelevant categories.

[0093]

[0094] Among them, M x，y This represents the similarity score of the most relevant category at spatial location (x, y); a two-dimensional response map M∈R is constructed from the similarity scores of all feature points. h′×w′ ;

[0095] 4.3.6) The multimodal fusion attention module applies the Sigmoid activation function to the similarity score of each feature point to generate the attention weights corresponding to that feature point:

[0096] A=σ(M) (9)

[0097] Where σ(·) represents the Sigmoid function, which maps the similarity score to the range [0,1], indicating the importance of the feature point in the final feature map.

[0098] After processing all feature points through the above steps, a complete weight matrix is ​​generated. This weight matrix is ​​then expanded along the channel dimension and multiplied element-wise with the feature map P from the input text-image-layer information fusion module to obtain the updated feature map P″.

[0099] P″=P′ i,j ⊙A (10)

[0100] Among them, P′ i,j P″ represents the feature map of the input text-image-layer information fusion module, and P″ represents the feature map updated by the text-image-layer information fusion module.

[0101] 4.3.7) First, the updated feature map P′1 is obtained. Then, the updated feature map P′1 is downsampled layer by layer and fused with the corresponding scale features to generate the updated feature maps P′2 and P′3 respectively.

[0102] 4.4) The updated feature maps P′1, P′2, and P′3 from the above multi-scale model are input into three independent detection heads to predict bounding boxes and categories. The bounding box branch processes the updated feature maps through convolutional layers to generate an output with 4 × reg_max channels, which is used to predict the distributed regression values ​​of the bounding boxes. Here, reg_max represents the number of discretized intervals for each bounding box coordinate. The model obtains more accurate regression results by predicting the probability distribution of these discretized intervals. The category branch processes the updated feature maps through another set of convolutional operations to generate the probability distribution of the target category by outputting the number of channels equal to the number of categories. This yields the trained text-guided multimodal detection model.

[0103] By incorporating textual information and hierarchical characteristics into the feature map, the model can more fully capture the semantic relationships and hierarchical distribution patterns of the target category, providing more targeted feature representations for subsequent recognition and localization.

[0104] 5) After preprocessing the real-time acquired B-scan GPR images of the airport runway, including removing some noise and background interference and unifying the size, the images are input into the text-guided multimodal detection model trained above, and finally output the corresponding bounding boxes and category prediction results.

[0105] To verify the effectiveness of the method of the present invention, the inventors conducted the following experiments:

[0106] Robots equipped with GPR (Gas Probe) navigate along pre-planned paths on 11 airport runways, performing full-coverage path scanning to collect a large number of B-scan GPR images. These images are then transmitted in real-time via 4G network or Wi-Fi to a mobile data analysis center for storage. The GPR utilizes 14 equally spaced raptors. TM The GPR (Gross Runway Image Processing) dataset, with each channel spaced 0.7 meters apart, is equipped with a 900MHz antenna and set to distance-triggered mode to ensure synchronous sampling. After preprocessing, including noise and background interference removal and size standardization, the preprocessed GPR images constitute the Airport Runway Underground Dataset (AUD). The preprocessed GPR images from seven airports are merged and divided into a training set (AUD-T) and a test set (AUD-Z) in an 8:2 ratio. The remaining four airports' preprocessed GPR images are used as the cross-domain test set (AUD-E) to evaluate the model's generalization performance. The training set (AUD-T), test set (AUD-Z), and cross-domain test set (AUD-E) all include four typical airport runway underground defects: voids, cracks, settlement, and reinforcement defects.

[0107] (1) Parameter settings: The size of the input B-scan GPR image was set to 448×448 pixels, and the text embedding dimension D = 512; the reg_max of the detector head represents the number of discretization intervals for each bounding box coordinate, which was set to 16 in the experiment. For model optimization, the stochastic gradient descent (SGD) optimizer was used, with the initial learning rate set to 0.001, momentum set to 0.937, weight decay set to 0.0005, and batch size set to 8.

[0108] (2) Evaluation Metrics: This invention employs multiple evaluation metrics commonly used in object detection to comprehensively assess the performance of the text-guided multimodal detection model, including precision, recall, F1 score, average precision (AP), and mean average precision (mAP). These metrics are widely used in object detection tasks and can measure the model's performance from different perspectives. Furthermore, this invention proposes the Normalized Performance Degradation Index (NPDI), which measures the degree of performance degradation of the model across different data scenarios relative to the source domain scenario, and is used to evaluate the model's generalization ability.

[0109] Accuracy calculation relies on true positives (TP) and false positives (FP). TP indicates that the model correctly predicts the target, and the IoU (Intersection over Union) between the bounding box and the ground truth bounding box is greater than a set threshold. FP, on the other hand, indicates that the model incorrectly predicts a non-target region as a target. In this invention, the IoU threshold is set to 0.5. The accuracy calculation formula is as follows:

[0110]

[0111] Recall measures how many true positive samples a model can detect. A higher recall means the model can capture all targets well, i.e., reduce false negatives (FN). The formula for recall is:

[0112]

[0113] To balance precision and recall, the F1 score was introduced as a comprehensive metric. The F1 score is the harmonic mean of precision and recall, providing a holistic assessment that considers the trade-offs between these two metrics. The formula for calculating the F1 score is:

[0114]

[0115] AP measures a model's detection capability at different confidence thresholds by calculating the area under the precision-recall (PR) curve. mAP, on the other hand, averages the AP values ​​across all classes, providing an overall performance metric that reflects the model's comprehensive detection capability across all classes. The calculation formula is:

[0116]

[0117] Where r is the recall rate, p(r) is the precision value at the recall rate r, N is the number of classes, and AP_i is the average precision of the i-th class.

[0118] NPDI measures the performance variation of a model across different data distributions relative to the source domain. By normalizing the performance degradation, the impact of performance benchmark differences between different models can be reduced, thus more accurately assessing the model's generalization ability across different data distributions. The formula for calculating NPDI is:

[0119]

[0120] Among them, P source P represents the model's performance on a test set that is identically distributed to the training data. targetThis indicates the model's performance on test sets with different data distributions. The NPDI (NP-Difference Performance Index) allows us to evaluate a model's adaptability to different data distributions. A higher NPDI value indicates a significant performance drop under new data distributions, suggesting poor generalization ability.

[0121] (3) Comparison method: The method of the present invention is compared with four mainstream target detection algorithms.

[0122] ①Centernet achieves efficient dense target detection through a center-point-based prediction method. See the reference: K. Duan, S. Bai, L. Xie, H. Qi, Q. Huang, and Q. Tian, ​​“Centernet: Keypoint triplets for object detection,” in Proceedings of the IEEE / CVF international conference on computer vision, 2019, pp. 6569–6578.

[0123] ②Deformable DETR introduces deformable convolution, enhancing the model's flexibility in handling different scales and complex scenes. See the reference: X. Zhu, W. Su, L. Lu, B. Li, X. Wang, and J. Dai, "Deformable DETR: Deformable transformers for end-to-end object detection," arXiv preprint arXiv:2010.04159, 2020.

[0124] ③DINO employs a self-supervised learning method, enabling effective object detection without labeled data and demonstrating strong transfer capabilities. See the reference: H. Zhang, F. Li, S. Liu, L. Zhang, H. Su, J. Zhu, LM Ni, and H.-Y. Shum, “Dino: Detr with improved denoising anchor boxes for end-to-end object detection,” arXiv preprint arXiv:2203.03605, 2022.

[0125] ④ YOLOv11, as the latest version in the YOLO series, further optimizes the network structure and training strategy, significantly improving detection accuracy and robustness while maintaining fast inference speed, and performing excellently in various detection tasks. See reference: J. Glenn and Q. Jing, “Ultralytics yolo (version 11),” 2024. [Online]. Available: https: / / github.com / ultralytics / ultralytics

[0126] First, the inventors tested and evaluated the data on the AUD-Z test set, which has the same distribution as the training data. The experimental results of different contrastive models on the AUD-Z test set are shown in Table 1. Figure 6 The bar charts further visually illustrate the F1-scores of each model across different categories, and also include the overall mAP metric. The experimental results show that TIGNet, provided by this invention, exhibits superior performance across all detection tasks, achieving an mAP of 89.99%. This result demonstrates that TIGNet outperforms existing mainstream methods in both overall detection performance and category balance.

[0127] Table 1. Performance comparison of different algorithms on the AUD-Z test set

[0128]

[0129] Figure 7 The visualizations show the inference results of different models on the AUD-Z test set. The Ground Truth column displays the actual location and category information of the labeled target, while the other columns show the detection results of each model. The first two sets of results mainly show cases where voids are misidentified as rebar. Notably, the second row of results shows that some models also misidentify voids with a certain tilt angle as cracks. This may be because cracks in the dataset usually exhibit a tilting trend, leading to misguidance during feature learning. TIGNet did not exhibit similar false detections in this scenario, indicating its strong ability to distinguish voids. The third set shows the detection of cracks. Since both cracks and settlement have a certain tilting feature, some models failed to correctly distinguish between the two, resulting in false detections. However, the discrimination of these two types of underground defects depends not only on the tilt angle but also on contextual information. From the visualization results, TIGNet performs relatively stably in this type of task and can accurately detect cracks. The fourth and fifth sets of results show that some comparative models are prone to misdetecting large voids with a tilting trend as settlement, or failing to detect the target correctly. This indicates that this part of the model still has limitations in feature learning, failing to truly grasp the subtle differences between underground disease categories. In contrast, TIGNet has stronger capabilities in feature extraction and category differentiation.

[0130] To further verify TIGNet's cross-domain generalization ability, the inventors designed a specific experiment based on the cross-domain test set AUD-E. The cross-domain test set AUD-E differs significantly from the training data in terms of airport runway material aging and environmental noise, thus constituting a representative cross-domain testing environment for evaluating the model's adaptability and generalization ability under different data distributions.

[0131] Table 2 shows the detection performance of each model on the cross-domain test set AUD-E. Experimental results show that although all models experience varying degrees of performance degradation in cross-domain scenarios, TIGNet still maintains high detection performance across multiple categories, demonstrating strong robustness. Table 3 compares the generalization ability of each model from the perspective of cross-domain performance degradation, using the NPDI-mAP metric to measure the extent of performance degradation on the cross-domain test set. TIGNet has the lowest value on this metric, indicating that it experiences the least performance degradation in cross-domain scenarios and possesses the strongest generalization ability. This stability may be attributed to its multimodal fusion mechanism and semantically guided representation design, enabling the model to effectively capture discriminative information common across domains, rather than overfitting to the specific feature distribution of the training domain. In contrast, other methods such as YOLOv11 have significantly higher NPDI-mAP metrics, indicating limited transfer capabilities and strong domain dependence.

[0132] Table 2. Performance comparison of different algorithms on the cross-domain test set AUD-E

[0133]

[0134] Table 3. NPDI-mAP evaluation results of different models on the cross-domain test set AUD-E

[0135]

[0136] Figure 8The results showcase the visual comparison of each model on the cross-domain test set AUD-E. The first two sets of results illustrate the misidentification problem between voids and rebar. In the first set of experiments, Centernet misidentified the rebar located in the upper layer as a void, while DINO and Yolov11 failed to identify the target. The second set of experiments demonstrates a short, slightly tilted void. In this scenario, the other four models exhibited false positives or false negatives, while TIGNet still detected it accurately, indicating its stronger semantic discrimination ability. The third set of experiments shows a false detection example of a void with a certain tilt angle. Centernet, Deformable Detr, and Dino all misidentified this target as settlement, while only Yolov11 and TIGNet correctly detected it as a void. The fourth and fifth sets show examples of mutual false detection between cracks and settlement, both of which have tilted features. These two types of underground defects exhibit similar local texture features in radar images, making them highly susceptible to mutual false detection. In samples with a high risk of confusion, all models except TIGNet exhibited false positives or false negatives, further illustrating the limitations of traditional detection models in handling fine-grained differences between categories across domains. TIGNet, by introducing a text-guided category awareness mechanism and a hierarchical information fusion module, not only enhances the model's ability to accurately depict semantic boundaries but also improves its ability to model the spatial distribution features of targets, thereby enabling it to more effectively distinguish between highly similar categories.

Claims

1. An automatic detection method for underground defects in airport runways based on image-text guidance, characterized in that: The image-text-guided automatic detection method for underground defects in airport runways includes the following steps performed in sequence: 1) Use robots equipped with GPR to travel on multiple airport runways along pre-planned paths. Use GPR to perform full-coverage path scanning on the airport runways to collect a large number of B-scan GPR images. Then, transmit the B-scan GPR images to the mobile data analysis center in real time via 4G network or Wi-Fi for storage. After that, perform preprocessing including removing some noise and background interference and unifying the size. The airport runway underground dataset is composed of all preprocessed GPR images. 2) Professionals will provide textual descriptions of each type of underground disease based on their experience; 3) Construct a raw text-guided multimodal detection model; the raw text-guided multimodal detection model includes a learnable text embedding extraction module, an image feature extraction module, a text-image-layer information fusion module, and a detection head; 4) Input the preprocessed GPR image from the airport runway underground dataset obtained in step 1) and the text description obtained in step 2) into the original text-guided multimodal detection model obtained in step 3) for training, and obtain the trained text-guided multimodal detection model. 5) After preprocessing the real-time acquired B-scan GPR images of the airport runway, including removing some noise and background interference and unifying the size, the images are input into the text-guided multimodal detection model trained above, and finally output the corresponding bounding boxes and category prediction results.

2. The automatic detection method for underground defects of airport runways based on image and text guidance according to claim 1, characterized in that: In step 1), the GPR uses 14 equally spaced Raptors. TM GPR, with each channel spaced 0.7 meters apart, is equipped with a 900MHz antenna and is set to distance-triggered mode to ensure synchronous sampling; The underground data of airport runways includes four typical underground defects: voids, cracks, settlement, and steel reinforcement.

3. The automatic detection method for underground defects of airport runways based on image and text guidance according to claim 1, characterized in that: In step 2), the text description is represented as: T={t1,t2,…,t N } (1) Where N represents the number of categories, t k This represents a textual description of the kth type of underground disease.

4. The automatic detection method for underground defects of airport runways based on image and text guidance according to claim 1, characterized in that: In step 3), the learnable text embedding extraction module includes a CLIP text encoder for extracting key information from the text and generating text embeddings; the image feature extraction module includes a convolutional neural network-based image encoder for extracting feature maps from the preprocessed GPR image; the text-image-layer information fusion module includes a Dark Bottleneck module, a fully connected layer, and a multimodal fusion attention module for dynamically integrating text embeddings and layer prior knowledge information into the feature maps; the detection head includes a bounding box branch and a class branch, which complete the final bounding box and class prediction based on the fused feature maps.

5. The automatic detection method for underground defects of airport runways based on image and text guidance according to claim 1, characterized in that: In step 4), the preprocessed GPR image from the airport runway underground dataset obtained in step 1) and the text description obtained in step 2) are input into the original text-guided multimodal detection model obtained in step 3) for training, and the method for obtaining the trained text-guided multimodal detection model is as follows: 4.1) Extract text embeddings from text descriptions using a learnable text embedding extraction module; Inputting N category-related text descriptions T into a learnable text embedding extraction module, and generating text embeddings using a Transformer-based CLIP text encoder, is represented as follows: E=[e1;e2;…;e N ]∈R N×D (2) in, e k =f text (t k )∈R D f represents the embedding of the Kth class of text description. text This indicates the CLIP text encoder, where D represents the dimension of the text embedding. 4.2) The image feature extraction module is used to extract three feature maps of different scales from the input preprocessed GPR image; A B-scan GPR image of size h*w*c is input into the image feature extraction module, where h represents the image height, w represents the image width, and c represents the number of channels. Then, a convolutional neural network-based image encoder performs preliminary feature extraction, outputting three feature maps P1, P2, and P3 at different scales. Feature map P1 is a low-level feature map obtained by downsampling the B-scan GPR image by 8 times, with 256 channels; feature map P2 is a mid-level feature map obtained by downsampling the B-scan GPR image by 16 times, with 512 channels; and feature map P3 is a high-level feature map obtained by downsampling the B-scan GPR image by 32 times, with 512 channels. 4.3) The text embedding and multi-scale feature maps are fused using the text-image-layer information fusion module; 4.3.1) Upsample feature map P3 progressively and concatenate it with feature maps P1 and P2 along the channel dimension to generate a concatenated feature map P. i，j After each stitch, the Dark Bottleneck module is used to analyze the stitched feature map P. i，j The processing is represented as follows: P′ i,j =f dark (P i,j )∈R h′×w′×c′ ,i,j∈{1,2,3} (3) Among them, P′ i，j This represents the processed feature map, where h′, w′, and c′ represent the height, width, and channel dimension of the feature map, respectively. 4.3.2) Using fully connected layers to embed text E∈R N×D The embedding dimension is processed to match the channel dimension c′ of the feature map: 4.3.3) The processed feature map P′ i，j ∈R h′×w′×c′ Flattened into a two-dimensional matrix according to spatial location Then, the similarity score S between each feature point in the two-dimensional matrix and each type of text embedding after dimension processing is calculated by dot product. x，y，k Based on all similarity scores S x，y，k Obtain the image-text similarity score map S, and finally reconstruct the feature map back into its spatial structure: in, S x，y，k This represents the similarity score between a feature point in the feature map and the text embedding of category k at the spatial location (x, y) in the feature map. 4.3.4) After obtaining the similarity score for each category, the weights W∈R are distributed hierarchically. h′×N The similarity scores are dynamically modulated, where h′ represents the height of the feature map and N represents the number of categories. The method involves first expanding the layer distribution weights W along the width dimension, and then adjusting the similarity score S for each spatial location. x，y ∈R N Its corresponding stratum distribution weight Performing element-wise multiplication and extending the above operation to all spatial locations of the entire feature map, it can be represented as: 4.3.5) The multimodal fusion attention module selects the maximum similarity score of each feature point as the final similarity score for that feature point: in, M x，y This represents the similarity score of the most relevant category at spatial location (x, y); a two-dimensional response map M∈R is constructed from the similarity scores of all feature points. h′×w′ ; 4.3.6) The multimodal fusion attention module applies the Sigmoid activation function to the similarity score of each feature point to generate the attention weights corresponding to that feature point: A=σ(M) (9) Where σ(·) represents the Sigmoid function, which maps the similarity score to the range [0,1], indicating the importance of the feature point in the final feature map; After processing all feature points through the above steps, a complete weight matrix is ​​generated. This weight matrix is ​​then expanded along the channel dimension and multiplied element-wise with the feature map P from the input text-image-layer information fusion module to obtain the updated feature map P″. P″=P′ i,j ⊙A (10) Among them, P′ i,j P″ represents the feature map of the input text-image-layer information fusion module, and P″ represents the feature map updated by the text-image-layer information fusion module. 4.3.7) First, the updated feature map P′1 is obtained. Then, the updated feature map P′1 is downsampled layer by layer and fused with the corresponding scale features to generate the updated feature maps P′2 and P′3 respectively. 4.4) The updated feature maps P′1, P′2, and P′3 from the above multi-scale model are input into three independent detection heads to predict bounding boxes and categories. The bounding box branch processes the updated feature maps through convolutional layers to generate an output with 4 × reg_max channels, which is used to predict the distributed regression values ​​of the bounding boxes. Here, reg_max represents the number of discretized intervals for each bounding box coordinate. The model obtains more accurate regression results by predicting the probability distribution of these discretized intervals. The category branch processes the updated feature maps through another set of convolutional operations to generate the probability distribution of the target category by outputting the number of channels equal to the number of categories. This yields the trained text-guided multimodal detection model.

Images