Abnormity detection method and system for traffic infrastructure, medium and equipment

Abstract

The invention provides a traffic infrastructure anomaly detection method and system, a medium and equipment, and relates to the technical field of image processing, and the method comprises the steps: obtaining the image data of a traffic infrastructure; visual features of the image data and semantic vector features corresponding to the abnormal description text are extracted, the visual features and the semantic vector features are mapped to the same embedding space through a multi-modal model, and the feature matching degree of the visual features and the semantic vector features is calculated in the embedding space; and determining a text description matched with the abnormal information corresponding to the image data based on the feature matching degree, and converting the text description into an abnormal detection report of the traffic infrastructure by using a natural language generation model. According to the invention, the disease anomaly detection precision of the traffic infrastructure can be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, system, medium and device for detecting anomalies in transportation infrastructure. Background Art

[0002] Currently, anomaly detection in transportation infrastructure still mainly relies on manual inspections and traditional computer vision methods. It has problems such as low detection efficiency, strong data dependence, insufficient generalization ability, lack of intelligent analysis and report generation, and it is difficult to meet the intelligent needs of modern infrastructure operation and maintenance.

[0003] Manual inspections are costly and inefficient: Traditional infrastructure inspections rely on manual inspections by engineers, requiring close observation of defects such as cracks and deformations. This is not only time-consuming and labor-intensive, but also poses safety risks in scenarios like high bridges and tunnels. Manual inspections are limited by personnel experience and lack objective, standardized, and quantitative assessments, which can easily lead to inconsistent inspection results.

[0004] Existing computer vision detection methods rely on large amounts of labeled data and lack generalization capabilities: Existing AI detection methods are primarily based on supervised learning and require large amounts of high-quality labeled data to train models. However, labeling data on transportation infrastructure damage is expensive and difficult to obtain. Traditional computer vision methods primarily rely on pixel-level features (such as edges and textures), but their generalization capabilities are weak in complex environments (such as those with varying lighting, occlusion, and noise), making them prone to false detections or missed detections. Summary of the Invention

[0005] The purpose of this application is to provide a method, system, medium and equipment for detecting anomalies in transportation infrastructure, which can improve the accuracy of detecting defects in transportation infrastructure.

[0006] To solve the above technical problems, this application provides a method for detecting anomalies in traffic infrastructure. The specific technical solution is as follows:

[0007] Acquiring image data of the transportation infrastructure;

[0008] Extracting visual features of the image data using a visual encoder in a multimodal model;

[0009] Extracting semantic vector features of the abnormality description text corresponding to the image data using a text encoder in the multimodal model;

[0010] Introducing a domain knowledge base into the multimodal model, and using the multi-tuple relationship in the domain knowledge base to calculate the correlation between the semantic vector features and the corresponding vocabulary; the correlation is used to enhance the recognition strength of the multimodal model for the text corresponding to the disease category of transportation infrastructure;

[0011] Performing self-distillation on the image data and the abnormality description text, and adjusting the loss weight of the multimodal model according to the self-distillation result;

[0012] Mapping the visual features and the semantic vector features to the same embedding space using the multimodal model, and calculating a feature matching degree between the visual features and the semantic vector features in the embedding space;

[0013] A text description that matches the abnormality information corresponding to the image data is determined based on the feature matching degree, and the text description is converted into an abnormality detection report of the traffic infrastructure using a natural language generation model.

[0014] Optionally, before calculating the feature matching degree between the visual feature and the semantic vector feature in the embedding space, the method further includes:

[0015] Extracting attention weights in the attention layer of the multimodal model and mapping the attention weights to the pixel level of the image data to generate an attention heat map;

[0016] Extracting abnormal regions in the image data according to the attention heat map, and obtaining preliminary pseudo-annotation masks of the abnormal regions;

[0017] The preliminary pseudo-annotation mask is optimized to generate annotation data corresponding to the abnormal area; the annotation data is used to guide the multimodal model to perform abnormality recognition and detection.

[0018] Optionally, after generating the attention heat map, the following is also included:

[0019] Optimize the attention heat map using at least one of the following optimization methods:

[0020] Introducing a disease structure information database, screening abnormal areas based on the disease structure characteristics in the disease structure information database, so as to optimize the attention area corresponding to the abnormal area in the attention heat map;

[0021] Alternatively, a feature optimization method adapted to the attention heat map is used according to the size of the disease;

[0022] Alternatively, a feature pyramid including a deep feature layer is used to refine the detailed features of the attention heat map layer by layer.

[0023] Optionally, optimizing the preliminary pseudo-labeled mask includes at least one of the following optimization methods:

[0024] Introducing learnable semantic embedding to optimize the preliminary pseudo-annotation mask;

[0025] Performing adversarial training using the true annotation mask and the preliminary pseudo-annotation mask;

[0026] The attention heatmap is optimized for pixel accuracy to optimize the preliminary pseudo-annotation mask.

[0027] Optionally, introducing a learnable semantic embedding to optimize the preliminary pseudo-annotation mask includes:

[0028] Introducing learnable prompt information into the multimodal model to guide the multimodal model to identify the type of damage to the transportation infrastructure in the image data; wherein the prompt information includes soft prompt information and chained thinking prompt information, wherein the soft prompt information is a set of learnable semantic embeddings for input into the multimodal model; and the chained thinking prompt information is used to guide the multimodal model to focus on key features of the damage to the transportation infrastructure;

[0029] Different confidence regions are set using high-confidence true annotations, and the preliminary pseudo-annotation mask is screened using the different confidence regions to optimize the preliminary pseudo-annotation mask.

[0030] Optionally, using high-confidence true annotations to set different confidence regions, and screening the preliminary pseudo-annotation mask through the different confidence regions to optimize the preliminary pseudo-annotation mask includes:

[0031] Determining classification types using high-confidence true annotations; the classification types include high-confidence areas, medium-confidence areas, and low-confidence areas;

[0032] Calculating a joint distribution between the preliminary pseudo-annotation mask and the high-confidence true annotation using a confidence learning method, and determining suspected erroneous annotations based on the joint distribution;

[0033] The preliminary pseudo-annotation mask is classified according to the suspected erroneous labeling and the classification type, the preliminary pseudo-annotation mask belonging to the high confidence area is retained, the preliminary pseudo-annotation mask belonging to the medium confidence area is subjected to data enhancement processing and then participates in training, and the preliminary pseudo-annotation mask containing the suspected erroneous labeling is discarded; the data enhancement processing method includes at least one of data rotation and contrast adjustment.

[0034] The present application also provides a traffic infrastructure anomaly detection system, comprising:

[0035] An image data acquisition module, configured to acquire image data of the traffic infrastructure;

[0036] A visual feature extraction module, configured to extract visual features of the image data using a visual encoder in a multimodal model;

[0037] A semantic feature extraction module, configured to extract semantic vector features of the abnormality description text corresponding to the image data using a text encoder in the multimodal model;

[0038] A model optimization module is configured to introduce a domain knowledge base into the multimodal model, calculate correlations between vocabulary corresponding to the semantic vector features using multi-tuple relationships in the domain knowledge base, and use the correlations to enhance the multimodal model's recognition strength for text corresponding to traffic infrastructure disease categories; perform self-distillation on the image data and the anomaly description text, and adjust the loss weight of the multimodal model based on the self-distillation results;

[0039] a feature matching module, configured to map the visual features and the semantic vector features to the same embedding space using the multimodal model, and calculate a feature matching degree between the visual features and the semantic vector features in the embedding space;

[0040] A detection report generation module is used to determine a text description that matches the abnormal information corresponding to the image data based on the feature matching degree, and convert the text description into an abnormality detection report of the traffic infrastructure using a natural language generation model.

[0041] The present application also provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above-described method when executed by a processor.

[0042] The present application also provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the above-mentioned method when calling the computer program in the memory.

[0043] The present application provides a method for detecting anomalies in transportation infrastructure, comprising: acquiring image data of the transportation infrastructure; extracting visual features of the image data using a visual encoder in a multimodal model; extracting semantic vector features of an anomaly description text corresponding to the image data using a text encoder in the multimodal model; introducing a domain knowledge base into the multimodal model, and calculating the correlation between words corresponding to the semantic vector features using multi-tuple relationships in the domain knowledge base; the correlation is used to enhance the recognition strength of the multimodal model for texts of disease categories corresponding to transportation infrastructure; performing self-distillation on the image data and the anomaly description text, and adjusting the loss weight of the multimodal model based on the self-distillation result; mapping the visual features and the semantic vector features to the same embedding space using the multimodal model, and calculating the feature matching degree of the visual features and the semantic vector features in the embedding space; determining a text description that matches the anomaly information corresponding to the image data based on the feature matching degree, and converting the text description into an anomaly detection report for the transportation infrastructure using a natural language generation model.

[0044] When performing anomaly detection on transportation infrastructure, this application obtains its image data and annotates the text information of the corresponding features, and then uses a multimodal model to output a text description and a corresponding anomaly detection report that matches the anomaly information. The visual features corresponding to the image data and the text features corresponding to the text information are integrated to achieve accurate positioning and category determination of the defects in the image data, which can effectively improve the accuracy of anomaly detection of defects in transportation infrastructure. At the same time, the natural language generation model can automatically output anomaly detection reports, which helps engineers quickly locate the location of the defects, facilitate the provision of targeted maintenance suggestions, and improve maintenance efficiency. In addition, by introducing a domain knowledge base, this application can further enrich the knowledge graph related to transportation infrastructure defects, facilitate the multimodal model to better understand image features and text features, and at the same time, with the help of the multi-tuple relationship in the domain knowledge base to calculate the correlation between semantic vector features, it can further enhance the multimodal model's semantic understanding of the categories of transportation infrastructure defects. At the same time, by performing self-distillation on the image data and the anomaly description text and dynamically adjusting the loss weight, the model pays more attention to the difficult-to-identify defect categories during the training process, thereby improving the overall recognition accuracy.

[0045] The present application also provides a traffic infrastructure anomaly detection system, a computer-readable storage medium, and an electronic device, which have the above-mentioned beneficial effects and are not described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0047] Figure 1 A flowchart of a method for detecting anomalies in transportation infrastructure provided in an embodiment of the present application;

[0048] Figure 2 A schematic diagram of the multimodal model feature interaction with the cross-attention module provided in an embodiment of the present application;

[0049] Figure 3 A schematic diagram of the structure of an anomaly detection system for transportation infrastructure provided in an embodiment of the present application;

[0050] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0052] Please refer to Figure 1 , Figure 1 A flowchart of a method for detecting anomalies in transportation infrastructure provided in an embodiment of the present application, the method comprising:

[0053] S101: Acquire image data of the transportation infrastructure;

[0054] S102: extracting visual features of the image data using a visual encoder in a multimodal model;

[0055] S103: extracting semantic vector features of the abnormality description text corresponding to the image data using a text encoder in the multimodal model;

[0056] S104: Introducing a domain knowledge base into the multimodal model, and calculating the correlation between the semantic vector features and corresponding words using the multi-tuple relationships in the domain knowledge base; the correlation is used to enhance the recognition strength of the multimodal model for the text corresponding to the disease category of the transportation infrastructure;

[0057] S105: performing self-distillation on the image data and the abnormality description text, and adjusting the loss weight of the multimodal model according to the self-distillation result;

[0058] S106: Mapping the visual features and the semantic vector features to the same embedding space using the multimodal model, and calculating a feature matching degree between the visual features and the semantic vector features in the embedding space;

[0059] S107: Determine a text description that matches the abnormality information corresponding to the image data based on the feature matching degree, and convert the text description into an abnormality detection report of the traffic infrastructure using a natural language generation model.

[0060] There is no limitation on how to acquire image data of transportation infrastructure; high-resolution image data of transportation infrastructure can be acquired using drones, inspection robots, or fixed cameras. The transportation infrastructure includes, but is not limited to, bridges, roads, tunnels, and building exteriors. The embodiments of this application are intended to detect defects in structures such as roads, bridges, and tunnels, such as cracks, subsidence, and exposed rebar.

[0061] The collected image data can include general image data and infrared thermal imaging data. Drones equipped with sensors can be used to enter tunnels to automate disease detection tasks, reducing the workload and safety risks of manual inspections and significantly improving the efficiency of image data acquisition, especially when inspecting large or complex structures (such as bridges and tunnels).

[0062] Furthermore, the collected image data must cover diverse scenes, lighting conditions, and multiple types of damage. Data sources can also include: transportation infrastructure inspection images (e.g., bridges, roads, and tunnels); public datasets (e.g., SDNET2018 and Crack500); manually collected high-resolution crack and damage images; and simulated crack images (based on diffusion models).

[0063] Image data collection can cover different scales, lighting conditions, and defect types. Different scales encompass both long-range panoramic images and close-up crack detail images, ensuring the model accounts for both global structure and defect details. Different lighting conditions encompass sunny, cloudy, and nighttime conditions, improving model adaptability. Different defect types encompass a variety of defects, including cracks, spalling, and deformation, ensuring comprehensive anomaly detection.

[0064] A multimodal model can include a visual encoder and a text encoder. The visual encoder is used to extract visual features from image data. The specific type used is not limited here, and includes but is not limited to ViT-B / 16, ViT-L / 14, or ResNet50. To improve the expressiveness of visual features, a larger-scale visual transformer can be used to enhance global perception of defect areas. Combined with the Swin Transformer, it optimizes local feature extraction and improves the detection accuracy of small-scale defects (such as fine cracks).

[0065] In a feasible implementation, after acquiring the image data, a structured disease hierarchical labeling system corresponding to the image data may be constructed. The following is an exemplary structured disease hierarchical labeling system:

[0066] Defect categories: cracks, settlement, deformation, corrosion, exposed steel bars, etc.

[0067] Defect location: bottom of bridge piers, main beams, bridge decks, tunnel walls, etc.

[0068] Disease severity: mild (no impact), moderate (regular monitoring required), severe (immediate repair required).

[0069] Recommended maintenance measures: crack repair (grouting, sealing), structural reinforcement (steel reinforcement, carbon fiber reinforcement), coating protection (waterproof coating, corrosion-resistant coating).

[0070] The structured disease hierarchical labeling system can be used to train models and can also be directly mapped to the final anomaly detection report, improving the standardization and readability of the anomaly detection report.

[0071] In a feasible implementation, in order to improve the robustness of the model, the image data set may be expanded using at least one of the following data augmentation techniques:

[0072] Geometric transformation (random rotation, scaling, flipping): Increase the diversity of disease morphology.

[0073] Brightness / contrast adjustment (simulating different lighting conditions): Improves adaptability to scenes with uneven lighting.

[0074] Adding Gaussian noise (simulating the noise in real inspection images): Enhances the model's anti-interference ability.

[0075] Mixup / CutMix (target splicing): Enhance the model's ability to recognize targets in complex backgrounds.

[0076] In a feasible implementation, abnormal areas in the image data can be extracted based on the attention heat map to obtain a preliminary pseudo-annotation mask, and then the preliminary pseudo-annotation mask can be optimized to obtain text information corresponding to the features of the image data.

[0077] To improve detection accuracy in small sample environments, attention mechanisms, structural feature library optimization, and adaptive learning strategies can be combined to enhance the ability to identify diseased areas and reduce missed and false detections. Traditional object detection and segmentation methods rely on large amounts of labeled data, while this embodiment uses an attention heat map to automatically highlight diseased areas.

[0078] Specifically, the image data and the corresponding text information can be input into the multimodal model for forward propagation to obtain the output of the multimodal model. The attention weights are then extracted in the attention layer of the multimodal model and mapped to the pixel level of the image to generate an attention heatmap.

[0079] The image data is then input into the multimodal model, and an attention heat map is generated, which represents the areas that the model considers to be most relevant.

[0080] At the same time, in order to improve the annotation quality of the preliminary pseudo-annotation mask, the heat map can be optimized. The attention heat map can be optimized using at least one of the following optimization methods:

[0081] The first method is to use the disease structure information library to optimize the attention area of ​​the attention heat map;

[0082] There is no limitation on the type of structural damage information database used. In one feasible implementation, the Structural Health Knowledge Graph (SHKG) can be used, which contains structural information of damages such as cracks, spalling, and corrosion to enhance the model's understanding of damage morphology.

[0083] For example, bridge cracks mostly develop along the direction of force, have a large aspect ratio, and have less background texture noise. Road subsidence has a smooth shape and is easily confused with ordinary shadows, requiring the introduction of geometric constraints to optimize the heatmap.

[0084] The second method is to use a feature optimization method that is adapted to the size of the defect on the attention heat map. For cracks smaller than 0.5 mm, multi-scale feature extraction is used. For cracks larger than 1 mm, the attention weight of the attention heat map is optimized.

[0085] In practical applications, for small cracks of 0.1-0.5mm, focus on slender, low-contrast areas to avoid missed detection. Multi-scale feature extraction is used to enhance the perception of small cracks.

[0086] For wider cracks larger than 1 mm, the attention weight can be optimized to reduce the risk of misdetecting background textures as cracks, thereby integrating global contextual information and improving the ability to identify crack boundaries.

[0087] The third method is to use a feature pyramid containing deep feature layers to refine the attention heat map layer by layer.

[0088] Because attention heatmaps often have low resolution (e.g., 7x7), it's difficult to accurately label crack areas. Therefore, attention heatmaps can be combined with deeper feature layers (such as the features in the last few layers of ViT) to enhance detailed features. Specifically, a U-Net-style feature pyramid structure can be employed to refine the heatmap layer by layer, improving pixel-level segmentation capabilities.

[0089] The above describes how to optimize the attention heatmap. Furthermore, the preliminary pseudo-annotation mask can also be optimized. The preliminary pseudo-annotation mask generated by the attention heatmap may contain errors, such as blurred boundaries, background noise interference, and category confusion. Therefore, this embodiment can further improve the annotation quality of the preliminary pseudo-annotation mask and enhance the model's adaptability in small sample environments.

[0090] Optimizing the preliminary pseudo-annotation mask includes at least one of the following optimization methods:

[0091] The first method is to optimize the preliminary pseudo-annotation mask based on the multimodal model.

[0092] Specifically, learnable cues can be introduced into the multimodal model to guide it in identifying the types of damage to transportation infrastructure in image data. High-confidence ground-truth annotations can also be used to set different confidence regions, which can then be used to filter the initial pseudo-annotation masks.

[0093] The prompt information includes soft prompt information and chain thinking prompt information. The soft prompt information is a set of learnable semantic embeddings used to input into the multimodal model; the chain thinking prompt information is used to guide the multimodal model to gradually focus on the key characteristics of traffic infrastructure defects.

[0094] Learnable prompt tuning is used to optimize visual-semantic alignment, that is, the alignment between image data and text information.

[0095] The multimodal model is used for image-text matching, and its pixel-level mapping capability is relatively weak. Therefore, a learnable prompt optimization strategy is introduced to enhance the multimodal model's attention weight on the diseased area. The specific prompt information is as follows:

[0096] Soft Prompt: By training and optimizing a set of learnable semantic embeddings, the multimodal model can better understand disease categories and optimize visual-semantic mapping.

[0097] Chain-of-Thought Prompting: Guides the multimodal model to gradually focus on key features of the disease (such as crack width, morphology, color change, etc.), improving the quality of pseudo-annotation.

[0098] Instead, we use high-confidence true annotations to set different confidence regions to form a confidence scoring mechanism and improve the accuracy of false annotation screening. A feasible setting method can be as follows:

[0099] High confidence region (Confidence>0.8): directly used for supervised training to ensure that the model learns high-quality features.

[0100] Medium confidence area (Confidence 0.5-0.8): There may be some noise, so data enhancement strategies (such as rotation and contrast adjustment) are introduced to enable the multimodal model to learn more robust features in different environments.

[0101] Low confidence regions (Confidence < 0.5): may contain mislabeled regions and are not used for training to prevent the multimodal model from incorrectly learning background features.

[0102] The second method is to use the real annotation mask and the preliminary pseudo-annotation mask for adversarial training.

[0103] In the early stages of training, relying entirely on pseudo-annotations generated by CLIP may lead to unstable model convergence. Therefore, a small amount of real annotation optimization and adversarial training strategies are introduced to improve the model's cross-scenario adaptability.

[0104] Improve the ability to identify specific disease categories through optimization using a small amount of real-world annotations. Initially, a small number of high-confidence diseased areas are selected and carefully annotated manually to serve as high-quality training samples. Subsequently, a hybrid training strategy is employed, for example, with 80% pseudo-annotated data and 20% real-world annotated data, to ensure the model maintains stable classification boundaries early on. As training progresses, the proportion of manual annotations is gradually reduced to improve the model's adaptability.

[0105] Adversarial training can also be used to improve cross-environmental robustness. Because the lighting, materials, and background textures of defects vary significantly across different scenarios, such as bridges, roads, and tunnels, targeted adversarial perturbations are employed: slight perturbations, such as noise and contrast variations, are artificially introduced during training to ensure the model maintains stable detection capabilities in complex environments.

[0106] One feasible adversarial learning method is to use PGD (Projected Gradient Descent) perturbation optimization to enhance the model's robustness to illumination changes. Using the MixUp strategy, multiple samples are mixed to enable the model to learn more generalized feature representations.

[0107] The third method is to optimize the pixel accuracy of the attention heat map to optimize the preliminary pseudo-annotation mask.

[0108] Although the attention heat map generated by the multimodal model can provide a rough idea of ​​the diseased area, its pixel-level accuracy is limited. Therefore, a diffusion model is introduced to further optimize the crack area and make the boundary clearer.

[0109] Using the multimodal model attention heat map as the initial mask, the diffusion model is introduced to gradually optimize the crack area, making the marked area more consistent with the actual disease morphology. Training method:

[0110] By fine-tuning the diffusion model using crack databases (e.g., CFD and Crack500), we enhanced its morphological modeling capabilities for disease detection. Using latent variable optimization, we ensured that the diffusion model retained crack morphology information during image reconstruction, minimizing background noise.

[0111] Furthermore, cross-layer feature fusion, combined with deep features from multimodal models, can improve the spatial distribution of pseudo-annotations in crack regions. Specifically, a feature pyramid network (FPN) can be used to fuse attention information from different layers to generate higher-resolution crack annotation masks. Alternatively, a U-Net-style two-stream network can be used to extract global and local crack features, improving the detection of small cracks.

[0112] Steps S102 and S103 are used to extract visual features of the image data and semantic vector features of the corresponding abnormality description text, respectively.

[0113] In step S104, a domain knowledge base is introduced into the multimodal model. The multi-tuple relationships in the domain knowledge base are used to calculate the correlation between corresponding words in the text information. This correlation is used to enhance the multimodal model's recognition of text related to traffic infrastructure disease classifications. The introduction of the domain knowledge base (Structural Health Knowledge Graph, SHKG) enhances the professionalism of disease classification text and ensures the accuracy of semantic matching. Prompt Tuning can be learned to fine-tune text descriptions through task-adaptive fine-tuning, making them more suitable for disease detection. For example:

[0114] Original prompt: "This is a photo of a crack."

[0115] Optimization tips:

[0116] "This is a {direction} crack in a {component} of {material}, with width of {width} mm."

[0117] This method can more accurately align disease types and improve the detection robustness of the CLIP model.

[0118] In step S105, the image data and the anomaly description text are self-distilled, and the loss weight of the multimodal model is adjusted based on the self-distillation results. This step aims to dynamically adjust the loss weight by self-distilling the image data and the anomaly description text, so that the model pays more attention to difficult-to-identify disease categories during training, thereby improving overall recognition accuracy.

[0119] In step S106 , the visual features and the semantic vector features are mapped to the same embedding space using the multimodal model, and the feature matching degree between the visual features and the semantic vector features is calculated in the embedding space.

[0120] Specifically, by extracting image features and text features and calculating the semantic similarity between the two, the text description that best matches the disease category of the transportation infrastructure in the image data can be automatically matched. There is no limitation on how to calculate the semantic similarity between image features and text features. The cosine similarity between the two can be calculated after normalizing the features, or the semantic similarity between the two can be calculated using Euclidean distance. This allows the multimodal model to improve detection accuracy based on semantic information even in unseen disease categories or in different environments. For example, the system can match a crack image to text categories such as "longitudinal crack" and "width 5mm" to achieve intelligent labeling and classification.

[0121] In step S107, based on the feature matching degree, a text description matching the abnormality information corresponding to the image data is determined. This text description is then converted into an abnormality detection report for the transportation infrastructure using a natural language generation model. The specific content of the abnormality detection report is not limited and may include, but is not limited to, the type of defect, location, severity, and repair recommendations. It may also include visual annotations. This reduces manual analysis time, eliminating the need for engineers to manually organize test data. The system automatically generates test results, improving work efficiency.

[0122] When performing anomaly detection on transportation infrastructure, the embodiment of the present application obtains its image data and annotates text information of corresponding features, and then uses a multimodal model to output a text description matching the anomaly information and a corresponding anomaly detection report. The visual features corresponding to the image data and the text features corresponding to the text information are integrated to achieve accurate positioning and category determination of defects in the image data, which can effectively improve the accuracy of anomaly detection for transportation infrastructure.

[0123] In one feasible implementation, the system automatically generates reports covering basic disease information, severity assessments, and maintenance recommendations, which can be used for inspections of infrastructure such as bridges, roads, and tunnels. See Table 1 for an exemplary anomaly detection report.

[0124] Table 1 An exemplary anomaly detection report

[0125]

[0126] Since traditional text reports often lack intuitiveness, this embodiment introduces a visualization enhancement module based on this, combining disease detection heat maps and structural annotation information to help engineers intuitively understand the detection results.

[0127] For example, the Visualization Enhancement Module can be used to implement multimodal visualization: adding heat maps of detected cracks to reports and annotating damaged areas to improve readability. In combination with point cloud data, damage information can be annotated within 3D structural models of transportation infrastructure (e.g., BIM and GIS), enabling 3D visual assessment.

[0128] Furthermore, anomaly detection reports can be used to predict disease development trends. For example, time series prediction models (LSTM, Transformer Time-Series) can be used in conjunction with historical inspection data to analyze disease expansion trends and provide long-term maintenance recommendations. For example, "A crack width of 5mm was detected and is expected to expand to 12mm within the next 12 months. Reinforcement repair is recommended within 6 months."

[0129] This embodiment aims to automatically generate structured inspection reports, enhancing the value of engineering applications. By employing natural language generation (NLG) technology, text descriptions matching anomaly information are automatically converted into structured, highly readable anomaly inspection reports, helping engineers quickly understand the health of infrastructure. Combined with defect detection results, repair recommendations and maintenance deadline predictions can also be provided, improving the scientific and timely nature of maintenance work.

[0130] The following describes the traffic infrastructure anomaly detection method provided in this application, using CLIP (Contrastive Language-Image Pretraining) as an exemplary multimodal model. This embodiment uses CLIP as a multimodal model, leveraging image-text alignment to extract semantic information from defect images and matching defect categories through cross-modal similarity calculation, thereby enabling unsupervised or weakly supervised anomaly detection.

[0131] The basic CLIP network structure can include:

[0132] Visual encoder (ViT-B / 16, ViT-L / 14 or ResNet50): used to extract visual features of disease images.

[0133] Text encoder: used to extract semantic features of disease category text (such as "cracks", "settlement", and "corrosion").

[0134] Cross-modal feature alignment module: Both image and text features are mapped to the same embedding space. The cosine similarity between the disease image and text descriptions of different categories is calculated to achieve matching calculations and complete anomaly detection and classification.

[0135] Based on the above CLIP network structure, the visual encoder can be further optimized to improve feature expression capabilities:

[0136] Standard CLIP uses ViT-B / 16 or ResNet50 as visual encoders. However, in structural defect detection tasks, ViT-B / 16 performs well on large-scale images but has limited ability to capture small objects (such as tiny cracks). ResNet50, on the other hand, performs well in extracting detailed features but lacks the ability to model long-range dependencies.

[0137] Therefore, one feasible implementation involves using ViT-L / 14 (a larger-scale visual Transformer) to enhance global perception of defect areas. This, combined with the Swin Transformer, optimizes local feature extraction and improves detection accuracy for small-scale defects (such as fine cracks).

[0138] Another improvement is to optimize the text encoder to enhance the expression of semantic information:

[0139] CLIP's text encoder uses a standard Transformer, but its descriptions of structural defects are relatively general and cannot accurately match professional terms.

[0140] Among other improvements, cross-modal feature alignment can be optimized to improve class matching accuracy. Because defect detection involves fine-grained categories (e.g., “bridge longitudinal cracks” vs. “tunnel surface spalling”), CLIP’s standard cross-modal alignment mechanism can lead to reduced class distinction.

[0141] In one feasible implementation, a Cross-Attention Module (CAM) is introduced to enhance fine-grained feature interactions between text and images, improving the accuracy of category matching. A self-distillation strategy is used to optimize the embedding of disease images, making them more stable under different angles and lighting conditions.

[0142] See also Figure 2 , Figure 2 Schematic diagram of the multimodal model feature interaction introducing a cross-attention module provided in an embodiment of the present application. Figure 2 In this paper, CLIP is used as an exemplary multimodal model to achieve comparative learning of image features and text features with the help of image encoder and text encoder.

[0143] The specific process is as follows:

[0144] enter:

[0145] Defect images (e.g., bridge cracks, tunnel spalling, road subsidence).

[0146] Text category description (e.g., "cracks," "corrosion," "subsidence").

[0147] Calculation process:

[0148] The visual encoder (ViT-L / 14 combined with Swin Transformer) extracts disease image features and generates a series of hierarchical feature maps.

[0149] The text encoder (Transformer) processes the disease category text and obtains semantic embeddings (TextEmbeddings).

[0150] The cross-modal attention module (Cross-Attention) is used to calculate the matching degree between image features and text features:

[0151] Attention weights highlight the diseased area and form a preliminary attention heat map.

[0152] Region Aggregation improves the responsiveness of diseased areas.

[0153] The CLIP model can generate attention heat maps and perform threshold screening to determine the preliminary pseudo-annotation mask of the diseased area.

[0154] Based on the above embodiment, the following is a feasible self-distillation implementation process corresponding to step S105:

[0155] The first step was to collect a multimodal dataset containing image data and text information about transportation infrastructure. The image data was standardized, and the text information was segmented and encoded. The dataset was divided into training, validation, and test sets to ensure a balanced data distribution.

[0156] The second step is to perform preliminary training on the multimodal model using the training set to ensure that the model can learn the basic features of images and text. During the training process, the model output (such as prediction results and intermediate features) is recorded and used as soft labels for self-distillation.

[0157] Step 3: Use the trained model to predict the training set and generate soft labels. Combine the soft labels with the original labels to construct a new training target. Reinitialize the model and perform self-distillation training using the new training target to optimize model performance.

[0158] Step 4: During the self-distillation process, dynamically adjust the loss weights of the image and text modalities. Based on the performance indicators of the validation set (such as accuracy, loss value, etc.), gradually adjust the weights to make the model's performance on different modalities more balanced. In one feasible implementation, the optimal weight distribution can be determined by using cross-validation or grid search.

[0159] Step 5: Use the validation set to evaluate the model's performance and ensure that the adjusted loss weights improve the model's generalization ability. If the performance does not meet expectations, further adjust the weights or optimize the model structure.

[0160] Step 6: Test the final multimodal model on the test set to verify its performance in real-world scenarios. Based on the test results, further weight adjustments can be made.

[0161] The CLIP model uses the attention heatmap included in the model to extract the diseased area and generate a preliminary pseudo-annotation mask. This is then optimized with a small amount of high-confidence ground-truth annotations to improve the annotation quality, enabling the CLIP model to maintain stable learning even with unsupervised or small-scale annotations.

[0162] Through the confidence scoring mechanism, low-confidence data is optimized through data enhancement (rotation, contrast adjustment, etc.) before participating in training, thereby improving the generalization ability of the CLIP model and making it adaptable to different detection scenarios.

[0163] Because the initial pseudo-annotation masks generated by CLIP's attention heatmaps may contain errors, such as blurred boundaries, background noise interference, and category confusion, this method combines semantically guided weakly supervised learning with learnable hint optimization, a confidence scoring mechanism, data augmentation, and optimization of a small amount of real annotations to improve pseudo-annotation quality and enhance the model's adaptability in small sample environments. The semantic guidance process will not be repeated here.

[0164] As can be seen, this embodiment uses the CLIP model for visual-semantic alignment, leveraging multimodal semantic features to assist in disease recognition and improve object detection accuracy. CLIP's attention heatmap automatically generates preliminary pseudo-annotation masks as text information, reducing the workload of manual annotation and improving data consistency. Combined with a confidence scoring mechanism, low-confidence regions are manually optimized to further improve pseudo-annotation quality. This reduces reliance on large-scale manually annotated data, significantly saving data construction time and costs.

[0165] On this basis, semantic guidance is used to enhance the adaptability of the CLIP model to different environments (such as changes in lighting and structural materials), thereby improving cross-scene detection capabilities.

[0166] One feasible implementation method, to improve disease detection accuracy, can further employ object detection algorithms (such as YOLO and Faster R-CNN) combined with visual-semantic feature fusion composed of image and text features, simultaneously achieving precise disease location and classification. Semantic label matching can be introduced alongside object detection to improve disease classification accuracy. Combining the spatial information from object detection with semantically guided category matching ensures more stable and accurate detection results. Furthermore, after a disease is detected, the most appropriate semantic label can be automatically matched, enhancing the interpretability of anomaly detection reports.

[0167] By integrating CLIP semantic guidance, object detection, and multi-view image data acquisition, the CLIP model's adaptability to diverse environments, including complex lighting, varying materials, and occlusions, is optimized. Combined with data augmentation and cross-domain feature alignment, the CLIP model maintains stable performance in diverse scenarios (such as urban roads, highways, mountain bridges, and underground tunnels), maintaining high detection accuracy across diverse environments, lighting conditions, and infrastructure types.

[0168] See also Figure 3 , Figure 3A schematic diagram of the structure of a traffic infrastructure anomaly detection system provided in an embodiment of the present application is provided below. The following describes a traffic infrastructure anomaly detection system provided in an embodiment of the present application. The anomaly detection system described below and the traffic infrastructure anomaly detection method described above can be used in conjunction with each other.

[0169] An image data acquisition module, configured to acquire image data of the traffic infrastructure;

[0170] A visual feature extraction module, configured to extract visual features of the image data using a visual encoder in a multimodal model;

[0171] A semantic feature extraction module, configured to extract semantic vector features of the abnormality description text corresponding to the image data using a text encoder in the multimodal model;

[0172] A model optimization module is configured to introduce a domain knowledge base into the multimodal model, calculate correlations between vocabulary corresponding to the semantic vector features using multi-tuple relationships in the domain knowledge base, and use the correlations to enhance the multimodal model's recognition strength for text corresponding to traffic infrastructure disease categories; perform self-distillation on the image data and the anomaly description text, and adjust the loss weight of the multimodal model based on the self-distillation results;

[0173] a feature matching module, configured to map the visual features and the semantic vector features to the same embedding space using the multimodal model, and calculate a feature matching degree between the visual features and the semantic vector features in the embedding space;

[0174] A detection report generation module is used to determine a text description that matches the abnormal information corresponding to the image data based on the feature matching degree, and convert the text description into an abnormality detection report of the traffic infrastructure using a natural language generation model.

[0175] Based on the above embodiment, as a preferred embodiment, the text information annotation module includes:

[0176] a preliminary annotation unit, configured to extract abnormal regions in the image data according to the attention heat map to obtain a preliminary pseudo-annotation mask;

[0177] The annotation optimization unit is used to optimize the preliminary pseudo-annotation mask to obtain text information corresponding to the features of the image data.

[0178] Based on the above embodiment, as a preferred embodiment, the text information annotation module further includes:

[0179] The attention heat map generation module is used to input the image data and corresponding text information into the multimodal model for forward propagation to obtain the output of the multimodal model; extract the attention weight in the attention layer of the multimodal model, and map the attention weight to the pixel level of the image to generate the attention heat map.

[0180] Based on the above embodiment, as a preferred embodiment, the text information annotation module further includes:

[0181] The attention heat map optimization module optimizes the attention heat map using at least one of the following optimization methods:

[0182] Introducing a disease structure information database, screening abnormal areas based on the disease structure characteristics in the disease structure information database, so as to optimize the attention area corresponding to the abnormal area in the attention heat map;

[0183] Alternatively, a feature optimization method adapted to the attention heat map is used according to the size of the disease;

[0184] Alternatively, a feature pyramid including a deep feature layer is used to refine the detailed features of the attention heat map layer by layer.

[0185] Based on the above embodiment, as a preferred embodiment, the annotation optimization unit includes:

[0186] A first optimization unit is configured to introduce a learnable semantic embedding to optimize the preliminary pseudo-annotation mask;

[0187] A second optimization unit is configured to perform adversarial training using the real annotation mask and the preliminary pseudo-annotation mask;

[0188] The third optimization unit is used to perform pixel accuracy optimization on the attention heat map to optimize the preliminary pseudo-annotation mask.

[0189] Based on the above embodiment, as a preferred embodiment, the first optimization unit is a unit configured to perform the following steps:

[0190] Learnable prompt information is introduced into the multimodal model to guide the multimodal model to identify the type of damage to the transportation infrastructure in the image data; wherein the prompt information includes soft prompt information and chain thinking prompt information, and the soft prompt information is a set of learnable semantic embeddings for input into the multimodal model; the chain thinking prompt information is used to guide the multimodal model to focus on the key features of the damage to the transportation infrastructure; different confidence areas are set using high-confidence real annotations, and the preliminary pseudo-annotation mask is screened through the different confidence areas to optimize the preliminary pseudo-annotation mask.

[0191] The present application also provides an embodiment corresponding to a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method described in the above method embodiment.

[0192] It is understandable that if the method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and executes all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program code.

[0193] The computer-readable storage medium provided in this embodiment includes the above-mentioned method, and the effect is the same as above.

[0194] This application also provides an electronic device, see Figure 4 , a structural diagram of an electronic device provided in an embodiment of the present application, such as Figure 4 As shown, a processor 1410 and a memory 1420 may be included.

[0195] The processor 1410 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1410 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor 1410 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1410 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1410 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0196] The memory 1420 may include one or more computer-readable storage media, which may be non-transitory. The memory 1420 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 1420 is at least used to store the following computer program 1421, wherein, after the computer program is loaded and executed by the processor 1410, it can implement the relevant steps in the method performed by the electronic device side disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 1420 may also include an operating system 1422 and data 1423, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 1422 may include Windows, Linux, Android, etc.

[0197] In some embodiments, the electronic device may further include a display screen 1430 , an input / output interface 1440 , a communication interface 1450 , a sensor 1460 , a power supply 1470 , and a communication bus 1480 .

[0198] certainly, Figure 4 The structure of the terminal shown does not constitute a limitation on the terminal in the embodiment of the present application. In actual applications, the terminal may include Figure 4 More or fewer components than shown, or combinations of certain components.

[0199] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems provided in the embodiments, since they correspond to the methods provided in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0200] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core ideas of this application. It should be noted that for those skilled in the art, without departing from the principles of this application, various improvements and modifications can be made to this application, and such improvements and modifications also fall within the scope of protection of this application.

[0201] It should also be noted that, in this specification, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

Claims

1. A method for detecting anomalies in traffic infrastructure, characterized in that: include: Acquiring image data of the transportation infrastructure; Extracting visual features of the image data using a visual encoder in a multimodal model; Extracting semantic vector features of the abnormality description text corresponding to the image data using a text encoder in the multimodal model; Introducing a domain knowledge base into the multimodal model, and using the multi-tuple relationship in the domain knowledge base to calculate the correlation between the semantic vector features and the corresponding vocabulary; the correlation is used to enhance the recognition strength of the multimodal model for the text corresponding to the disease category of transportation infrastructure; Performing self-distillation on the image data and the abnormality description text, and adjusting the loss weight of the multimodal model according to the self-distillation result; Mapping the visual features and the semantic vector features to the same embedding space using the multimodal model, and calculating a feature matching degree between the visual features and the semantic vector features in the embedding space; A text description that matches the abnormality information corresponding to the image data is determined based on the feature matching degree, and the text description is converted into an abnormality detection report of the traffic infrastructure using a natural language generation model.

2. The anomaly detection method according to claim 1, wherein: Before calculating the feature matching degree between the visual feature and the semantic vector feature in the embedding space, the method further includes: Extracting attention weights in the attention layer of the multimodal model and mapping the attention weights to the pixel level of the image data to generate an attention heat map; Extracting abnormal regions in the image data according to the attention heat map, and obtaining preliminary pseudo-annotation masks of the abnormal regions; The preliminary pseudo-annotation mask is optimized to generate annotation data corresponding to the abnormal area; the annotation data is used to guide the multimodal model to perform abnormality recognition and detection.

3. The anomaly detection method according to claim 2, wherein: After generating the attention heat map, it also includes: The attention heat map is optimized using at least one of the following optimization methods: Introducing a disease structure information database, screening abnormal areas based on the disease structure characteristics in the disease structure information database, so as to optimize the attention area corresponding to the abnormal area in the attention heat map; Alternatively, a feature optimization method adapted to the attention heat map is used according to the size of the disease; Alternatively, a feature pyramid including a deep feature layer is used to refine the detailed features of the attention heat map layer by layer.

4. The anomaly detection method according to claim 2, wherein: Optimizing the preliminary pseudo-annotation mask includes at least one of the following optimization methods: Introducing learnable semantic embedding to optimize the preliminary pseudo-annotation mask; Performing adversarial training using the true annotation mask and the preliminary pseudo-annotation mask; The attention heatmap is optimized for pixel accuracy to optimize the preliminary pseudo-annotation mask.

5. The anomaly detection method according to claim 4, characterized in that: Introducing learnable semantic embedding to optimize the preliminary pseudo-annotation mask includes: Introducing learnable prompt information into the multimodal model to guide the multimodal model to identify the type of damage to the transportation infrastructure in the image data; wherein the prompt information includes soft prompt information and chained thinking prompt information, wherein the soft prompt information is a set of learnable semantic embeddings for input into the multimodal model; and the chained thinking prompt information is used to guide the multimodal model to focus on key features of the damage to the transportation infrastructure; Different confidence regions are set using high-confidence true annotations, and the preliminary pseudo-annotation mask is screened using the different confidence regions to optimize the preliminary pseudo-annotation mask.

6. The anomaly detection method according to claim 5, characterized in that: Using high-confidence true annotations to set different confidence regions, and screening the preliminary pseudo-annotation mask through the different confidence regions to optimize the preliminary pseudo-annotation mask includes: Determining classification types using high-confidence true annotations; the classification types include high-confidence areas, medium-confidence areas, and low-confidence areas; Calculating a joint distribution between the preliminary pseudo-annotation mask and the high-confidence true annotation using a confidence learning method, and determining suspected erroneous annotations based on the joint distribution; The preliminary pseudo-annotation mask is classified according to the suspected erroneous labeling and the classification type, the preliminary pseudo-annotation mask belonging to the high confidence area is retained, the preliminary pseudo-annotation mask belonging to the medium confidence area is subjected to data enhancement processing and then participates in training, and the preliminary pseudo-annotation mask containing the suspected erroneous labeling is discarded; the data enhancement processing method includes at least one of data rotation and contrast adjustment.

7. A traffic infrastructure anomaly detection system, characterized in that: include: An image data acquisition module, configured to acquire image data of the traffic infrastructure; A visual feature extraction module, configured to extract visual features of the image data using a visual encoder in a multimodal model; A semantic feature extraction module, configured to extract semantic vector features of the abnormality description text corresponding to the image data using a text encoder in the multimodal model; A model optimization module is configured to introduce a domain knowledge base into the multimodal model, calculate correlations between vocabulary corresponding to the semantic vector features using multi-tuple relationships in the domain knowledge base, and use the correlations to enhance the multimodal model's recognition strength for text corresponding to traffic infrastructure disease categories; perform self-distillation on the image data and the anomaly description text, and adjust the loss weight of the multimodal model based on the self-distillation results; a feature matching module, configured to map the visual features and the semantic vector features to the same embedding space using the multimodal model, and calculate a feature matching degree between the visual features and the semantic vector features in the embedding space; A detection report generation module is used to determine a text description that matches the abnormal information corresponding to the image data based on the feature matching degree, and convert the text description into an abnormality detection report of the traffic infrastructure using a natural language generation model.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for detecting anomalies in traffic infrastructure according to any one of claims 1 to 6 are implemented.

9. An electronic device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the method implements the steps of the abnormality detection method for traffic infrastructure according to any one of claims 1 to 6.

Images