A method, system, medium and device for detecting anomalies in transportation infrastructure

Abstract

This application provides a method, system, medium, and device for anomaly detection of transportation infrastructure, relating to the field of image processing technology. The method includes: acquiring image data of the transportation infrastructure; extracting visual features from the image data and semantic vector features of corresponding anomaly description text; mapping the visual features and semantic vector features to the same embedding space using a multimodal model, and calculating the feature matching degree between the visual features and semantic vector features in the embedding space; determining a text description that matches the anomaly information corresponding to the image data based on the feature matching degree, and converting the text description into an anomaly detection report of the transportation infrastructure using a natural language generation model. This application can effectively improve the accuracy of anomaly detection for transportation infrastructure.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, system, medium, and device for anomaly detection of transportation infrastructure. Background Technology

[0002] Currently, anomaly detection in transportation infrastructure still mainly relies on manual inspections and traditional computer vision methods, which suffer from problems such as low detection efficiency, strong data dependence, insufficient generalization ability, and lack of intelligent analysis and report generation, making it difficult to meet the intelligent needs of modern infrastructure operation and maintenance.

[0003] Manual inspections are costly and inefficient: Traditional infrastructure inspections rely on manual checks by engineers, requiring close observation of cracks, deformations, and other defects. This is not only time-consuming and labor-intensive, but also poses safety risks in scenarios such as high bridges and tunnels. Manual inspections are limited by personnel experience and lack objective, standardized, and quantitative assessments, which can easily lead to inconsistent inspection results.

[0004] Existing computer vision detection methods rely on large amounts of labeled data and have insufficient generalization ability: Current AI detection methods are mainly based on supervised learning, requiring a large amount of high-quality labeled data to train the model. However, labeling data for traffic infrastructure defects is costly and difficult to obtain. Traditional computer vision methods mainly rely on pixel-level features (such as edges and textures), which have weak generalization ability in complex environments (such as changes in lighting, occlusion, and noise interference), and are prone to false positives or false negatives. Summary of the Invention

[0005] The purpose of this application is to provide a method, system, medium, and equipment for detecting anomalies in transportation infrastructure, which can improve the accuracy of detecting defects in transportation infrastructure.

[0006] To address the aforementioned technical problems, this application provides a method for anomaly detection in transportation infrastructure, the specific technical solution of which is as follows:

[0007] Acquire image data of the transportation infrastructure;

[0008] Visual features of the image data are extracted using a visual encoder in a multimodal model;

[0009] The semantic vector features of the anomaly description text corresponding to the image data are extracted using the text encoder in the multimodal model.

[0010] A domain knowledge base is introduced into the multimodal model, and the correlation between words corresponding to the semantic vector features is calculated using the tuple relationships in the domain knowledge base; the correlation is used to enhance the recognition strength of the multimodal model for texts on the corresponding disease categories of transportation infrastructure.

[0011] The image data and the anomaly description text are self-distilled, and the loss weights of the multimodal model are adjusted based on the self-distillation results.

[0012] The visual features and semantic vector features are mapped to the same embedding space using the multimodal model, and the feature matching degree between the visual features and the semantic vector features is calculated in the embedding space.

[0013] Based on the feature matching degree, a text description matching the anomaly information corresponding to the image data is determined, and a natural language generation model is used to convert the text description into an anomaly detection report of the traffic infrastructure.

[0014] Optionally, before calculating the feature matching degree between the visual features and the semantic vector features in the embedding space, the method further includes:

[0015] Attention weights are extracted from the attention layer of the multimodal model, and the attention weights are mapped to the pixel level of the image data to generate an attention heatmap;

[0016] Based on the attention heatmap, abnormal regions in the image data are extracted to obtain preliminary pseudo-label masks for the abnormal regions.

[0017] The initial pseudo-label mask is optimized to generate labeled data corresponding to the abnormal region; the labeled data is used to guide the multimodal model to perform anomaly recognition and detection.

[0018] Optionally, after generating the attention heatmap, the following may also be included:

[0019] The attention heatmap is optimized using at least one of the following optimization methods:

[0020] A disease structure information database is introduced, and abnormal regions are screened based on the disease structure features in the disease structure information database in order to optimize the attention regions corresponding to the abnormal regions in the attention heatmap.

[0021] Alternatively, a feature optimization method can be adopted for the attention heatmap based on the scale of the disease.

[0022] Alternatively, the attention heatmap can be refined layer by layer using a feature pyramid containing deep feature layers.

[0023] Optionally, optimizing the initial pseudo-label mask includes at least one of the following optimization methods:

[0024] Learnable semantic embeddings are introduced to optimize the initial pseudo-label mask;

[0025] Adversarial training is conducted using the real labeled mask and the initial pseudo-labeled mask;

[0026] The attention heatmap is optimized in terms of pixel precision to improve the initial pseudo-label mask.

[0027] Optionally, optimizing the initial pseudo-annotation mask by introducing learnable semantic embeddings includes:

[0028] Learnable cue information is introduced into the multimodal model to guide it in identifying the types of defects in the transportation infrastructure in the image data; wherein, the cue information includes soft cue information and chain-thinking cue information, the soft cue information being a set of learnable semantic embeddings used as input to the multimodal model; the chain-thinking cue information is used to guide the multimodal model to focus on the key features of defects in the transportation infrastructure;

[0029] Different confidence regions are set using high-confidence real labels, and the preliminary pseudo-label mask is filtered through different confidence regions to optimize the preliminary pseudo-label mask.

[0030] Optionally, different confidence regions are set using high-confidence real labels, and the preliminary pseudo-label mask is filtered through different confidence regions to optimize the preliminary pseudo-label mask, including:

[0031] The classification type is determined using high-confidence ground truth labels; the classification type includes high-confidence regions, medium-confidence regions, and low-confidence regions.

[0032] The joint distribution between the initial pseudo-label mask and the high-confidence real label is calculated using a confidence learning method, and suspected erroneous labels are determined based on the joint distribution.

[0033] The preliminary pseudo-label masks are classified according to the suspected erroneous labels and the classification type. The preliminary pseudo-label masks belonging to the high-confidence region are retained, and the preliminary pseudo-label masks belonging to the medium-confidence region are subjected to data augmentation processing before being used in training. The preliminary pseudo-label masks containing the suspected erroneous labels are discarded. The data augmentation processing method includes at least one of data rotation and contrast adjustment.

[0034] This application also provides an anomaly detection system for transportation infrastructure, including:

[0035] An image data acquisition module is used to acquire image data of the traffic infrastructure;

[0036] A visual feature extraction module is used to extract visual features from the image data using a visual encoder in a multimodal model;

[0037] The semantic feature extraction module is used to extract the semantic vector features of the abnormal description text corresponding to the image data using the text encoder in the multimodal model;

[0038] The model optimization module is used to introduce a domain knowledge base into the multimodal model, calculate the correlation between words corresponding to the semantic vector features using the tuple relationships in the domain knowledge base, and enhance the recognition strength of the multimodal model for the text of the corresponding disease category of transportation infrastructure; perform self-distillation on the image data and the abnormal description text, and adjust the loss weight of the multimodal model according to the self-distillation results;

[0039] The feature matching module is used to map the visual features and the semantic vector features to the same embedding space using the multimodal model, and to calculate the feature matching degree between the visual features and the semantic vector features in the embedding space;

[0040] The detection report generation module is used to determine a text description that matches the abnormal information corresponding to the image data based on the feature matching degree, and to convert the text description into an anomaly detection report of the traffic infrastructure using a natural language generation model.

[0041] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0042] This application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described above when it invokes the computer program in the memory.

[0043] This application provides an anomaly detection method for transportation infrastructure, comprising: acquiring image data of the transportation infrastructure; extracting visual features of the image data using a visual encoder in a multimodal model; extracting semantic vector features of anomaly description text corresponding to the image data using a text encoder in the multimodal model; introducing a domain knowledge base into the multimodal model and calculating the correlation between words corresponding to the semantic vector features using tuple relationships in the domain knowledge base; the correlation being used to enhance the recognition strength of the multimodal model for text describing the defects of transportation infrastructure; performing self-distillation on the image data and the anomaly description text, and adjusting the loss weight of the multimodal model based on the self-distillation result; mapping the visual features and the semantic vector features to the same embedding space using the multimodal model, and calculating the feature matching degree of the visual features and the semantic vector features in the embedding space; determining a text description that matches the anomaly information corresponding to the image data based on the feature matching degree, and converting the text description into an anomaly detection report of the transportation infrastructure using a natural language generation model.

[0044] This application, when detecting anomalies in transportation infrastructure, acquires image data, annotates corresponding textual information, and then utilizes a multimodal model to output textual descriptions matching the anomaly information and corresponding anomaly detection reports. By fusing the visual features corresponding to the image data and the textual features corresponding to the textual information, it achieves precise location and category determination of defects in the image data, effectively improving the accuracy of anomaly detection for transportation infrastructure. Simultaneously, the use of a natural language generation model can automatically output anomaly detection reports, helping engineers quickly locate defects, facilitating targeted maintenance suggestions, and improving maintenance efficiency. Furthermore, by introducing a domain knowledge base, this application further enriches the knowledge graph related to transportation infrastructure defects, enabling the multimodal model to better understand image and textual features. Simultaneously, by using tuple relationships in the domain knowledge base to calculate the correlation between semantic vector features, it can further enhance the multimodal model's semantic understanding of transportation infrastructure defect categories. Moreover, by self-distilling the image data and anomaly description text, dynamically adjusting the loss weights, the model focuses more on difficult-to-identify defect categories during training, thereby improving overall recognition accuracy.

[0045] This application also provides an anomaly detection system for transportation infrastructure, a computer-readable storage medium, and an electronic device, which have the aforementioned beneficial effects, and will not be elaborated here. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0047] Figure 1 A flowchart illustrating an anomaly detection method for transportation infrastructure provided in this application embodiment;

[0048] Figure 2 This is a schematic diagram illustrating the feature interaction of a multimodal model that incorporates cross-attention modules, provided in an embodiment of this application.

[0049] Figure 3 A schematic diagram of an anomaly detection system for transportation infrastructure provided in an embodiment of this application;

[0050] Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0052] Please refer to Figure 1 , Figure 1 A flowchart illustrating an anomaly detection method for traffic infrastructure provided in this application embodiment, the method comprising:

[0053] S101: Acquire image data of the transportation infrastructure;

[0054] S102: Extract visual features from the image data using a visual encoder in a multimodal model;

[0055] S103: Extract the semantic vector features of the anomaly description text corresponding to the image data using the text encoder in the multimodal model;

[0056] S104: Introduce a domain knowledge base into the multimodal model, and use the tuple relationships in the domain knowledge base to calculate the correlation between words corresponding to the semantic vector features; the correlation is used to enhance the recognition strength of the multimodal model for texts on the corresponding disease categories of transportation infrastructure.

[0057] S105: Perform self-distillation on the image data and the anomaly description text, and adjust the loss weights of the multimodal model based on the self-distillation results;

[0058] S106: Using the multimodal model, the visual features and the semantic vector features are mapped to the same embedding space, and the feature matching degree between the visual features and the semantic vector features is calculated in the embedding space;

[0059] S107: Based on the feature matching degree, determine the text description that matches the abnormal information corresponding to the image data, and use a natural language generation model to convert the text description into an anomaly detection report of the traffic infrastructure.

[0060] The method for acquiring image data of transportation infrastructure is not limited here; high-resolution image data of transportation infrastructure can be acquired using drones, inspection robots, or fixed cameras. The transportation infrastructure includes, but is not limited to, bridges, roads, tunnels, and building facades. The purpose of this application is to detect defects in structures such as roads, bridges, and tunnels, such as cracks, settlement, and exposed rebar.

[0061] The acquired image data can include general image data as well as infrared thermal imaging data. Drones equipped with sensors can be deployed into tunnels to automate defect detection tasks, reducing the workload and safety risks of manual inspections and significantly improving the efficiency of image data acquisition, especially in the inspection of large or complex structures (such as bridges and tunnels).

[0062] In addition, the collected image data needs to cover different scenes, different lighting conditions, and various types of damage. The data sources may also include: inspection images of traffic infrastructure (such as bridges, roads, and tunnels); public datasets (such as SDNET2018 and Crack500); manually collected high-resolution crack and damage images; and simulated crack images (based on diffusion models).

[0063] When performing image data acquisition, it can cover different scales, different lighting conditions, and different types of defects. Different scales refer to encompassing long-distance panoramic images and close-up crack detail images, ensuring that the model takes into account both the overall structure and the details of defects. Different lighting conditions refer to including lighting conditions such as sunny days, cloudy days, and nighttime, improving the model's adaptability. Different types of defects refer to including the acquisition of various defects such as cracks, spalling, and deformation, ensuring the comprehensiveness of anomaly detection.

[0064] The multimodal model can include a visual encoder and a text encoder, where the visual encoder is used to extract visual features from the image data. The specific type used is not limited here, and includes, but is not limited to, ViT-B / 16, ViT-L / 14, or ResNet50. To improve the expressive power of visual features, a larger-scale visual Transformer can be used to enhance the global perception of diseased areas. Combining this with the Swing Transformer optimizes local feature extraction and improves the detection accuracy of small-scale diseases (such as fine cracks).

[0065] In one feasible implementation, after acquiring image data, a structured disease hierarchical labeling system corresponding to the image data can be constructed. The following is an exemplary structured disease hierarchical labeling system:

[0066] Types of defects: cracks, settlement, deformation, corrosion, exposed rebar, etc.

[0067] Location of defects: bottom of bridge piers, main beams, bridge deck, tunnel walls, etc.

[0068] Disease severity: minor (no impact), moderate (requires regular monitoring), severe (requires immediate repair).

[0069] Recommended repair measures: crack repair (grouting, sealing), structural reinforcement (steel reinforcement, carbon fiber reinforcement), coating protection (waterproof coating, corrosion-resistant coating).

[0070] The structured disease hierarchical labeling system can be used to train models and can also be directly mapped to the final generated anomaly detection report, improving the standardization and readability of the anomaly detection report.

[0071] In one feasible implementation, to improve the robustness of the model, at least one of the following data augmentation techniques can be used to augment the image data dataset:

[0072] Geometric transformations (random rotation, scaling, flipping): increase the diversity of disease morphology.

[0073] Brightness / contrast adjustment (simulating different lighting conditions): Improves adaptability to uneven lighting scenarios.

[0074] Adding Gaussian noise (simulating noise in real inspection images): enhances the model's resistance to interference.

[0075] Mixup / CutMix (Target Stitching): Enhances the model's ability to identify targets in complex backgrounds.

[0076] In one feasible implementation, abnormal regions in image data can be extracted based on attention heatmaps to obtain preliminary pseudo-label masks. Then, the preliminary pseudo-label masks can be optimized to obtain text information of the corresponding features of the image data.

[0077] To improve detection accuracy in small sample environments, attention mechanisms, structural feature library optimization, and adaptive learning strategies can be combined to enhance the identification ability of diseased areas and reduce missed detections and false detections. Traditional target detection and segmentation methods rely on large amounts of labeled data, while this embodiment uses an attention heatmap to automatically highlight diseased areas.

[0078] Specifically, image data and corresponding text information can be input into a multimodal model for forward propagation to obtain the model's output. Attention weights are then extracted from the attention layer of the multimodal model and mapped to the pixel level of the image to generate an attention heatmap.

[0079] Then, input image data into the multimodal model to generate an attention heatmap, which represents the region that the model considers most relevant.

[0080] Meanwhile, to improve the annotation quality of the initial pseudo-annotation mask, the heatmap can be optimized. At least one of the following optimization methods can be used to optimize the attention heatmap:

[0081] The first method involves optimizing the attention region of the attention heatmap using a disease structure information database.

[0082] There is no limitation on which type of disease structure information database to use. In one feasible implementation, a Structural Health Knowledge Graph (SHKG) can be used, which includes structural information of diseases such as cracks, spalling, and corrosion, to enhance the model's understanding of disease morphology.

[0083] For example: Bridge cracks: Most develop along the direction of stress, have a large length-to-width ratio, and have relatively little background texture noise. Road settlement: The shape is relatively smooth and easily confused with ordinary shadows, requiring the introduction of geometric constraints to optimize the heatmap.

[0084] The second approach is to use a feature optimization method that is appropriate to the size of the disease in the attention heatmap; for cracks smaller than 0.5 mm, multi-scale feature extraction is used; for cracks larger than 1 mm, the attention weight of the attention heatmap is optimized.

[0085] In practical applications, for fine cracks ranging from 0.1 to 0.5 mm, the focus should be on elongated, low-contrast areas to avoid missed detections. Multi-scale feature extraction is employed to enhance the ability to detect small cracks.

[0086] For wider cracks (greater than 1 mm), their attention weights can be optimized to reduce the risk of misdetecting background textures as cracks, thereby integrating global contextual information and improving crack boundary recognition capabilities.

[0087] The third method involves refining the attention heatmap layer by layer using a feature pyramid containing deep feature layers.

[0088] Since attention heatmaps often have low resolution (e.g., 7x7), it is difficult to accurately label crack areas. Therefore, attention heatmaps can be combined with deep feature layers (such as the features of the last few layers of ViT) to improve detailed features. Specifically, a U-Net-style feature pyramid structure can be used to refine the heatmap layer by layer, improving pixel-level segmentation capabilities.

[0089] The above describes the optimization method for the attention heatmap. Additionally, the initial pseudo-label mask can also be optimized. The initial pseudo-label mask generated from the attention heatmap may contain errors, such as blurred boundaries, background noise interference, and category confusion. Therefore, in this embodiment, the labeling quality of the initial pseudo-label mask can be further improved to enhance the model's adaptability in small sample environments.

[0090] Optimizing the initial pseudo-label mask includes at least one of the following optimization methods:

[0091] The first method involves optimizing the initial pseudo-label mask based on the multimodal model.

[0092] Specifically, learnable prompts can be introduced into multimodal models to guide them in identifying the types of defects in traffic infrastructure within image data. Furthermore, high-confidence ground truth annotations can be used to define different confidence levels, allowing for the filtering of initial pseudo-annotation masks.

[0093] The prompts include soft prompts and chained prompts. Soft prompts are a set of learnable semantic embeddings used as input to the multimodal model. Chained prompts are used to guide the multimodal model to gradually focus on the key features of traffic infrastructure defects.

[0094] Learnable prompts are used to optimize visual-semantic alignment, that is, the alignment between image data and text information.

[0095] Multimodal models are intended for image-text matching, but their pixel-level mapping capabilities are weak. Therefore, a learnable prompt optimization strategy is introduced to enhance the attention weights of the multimodal model to diseased areas. Specific prompt information is as follows:

[0096] Soft Prompt: By training and optimizing a set of learnable semantic embeddings, multimodal models can better understand disease categories and optimize visual-semantic mapping.

[0097] Chain-of-Thought Prompting: Guides multimodal models to gradually focus on key features of the disease (such as crack width, shape, color changes, etc.) and improves the quality of pseudo-annotations.

[0098] By using high-confidence real-valued annotations to define different confidence levels, a confidence scoring mechanism can be formed, thereby improving the accuracy of pseudo-annotation filtering. A feasible approach is shown below:

[0099] High confidence regions (Confidence>0.8): These are directly used for supervised training to ensure that the model learns high-quality features.

[0100] Medium confidence region (Confidence 0.5-0.8): There may be some noise, so data augmentation strategies (such as rotation and contrast adjustment) are introduced to enable the multimodal model to learn more robust features in different environments.

[0101] Low confidence regions (Confidence < 0.5): These may contain mislabeled regions and should not be used for training to prevent the multimodal model from incorrectly learning background features.

[0102] The second method involves adversarial training using the real labeled mask against the initial pseudo-labeled mask.

[0103] In the early stages of training, relying entirely on pseudo-labels generated by CLIP may lead to unstable model convergence. Therefore, a small number of real labels are introduced to optimize and adversarial training strategies to improve the model's cross-scene adaptability.

[0104] Optimization with a small amount of real-world annotations enhances the ability to identify specific disease categories. In the initial training phase, a small number of high-confidence disease regions are selected and meticulously annotated manually as high-quality training samples. Subsequently, a hybrid training strategy is employed, such as 80% pseudo-annotated data + 20% real-world labeled data, to help the model achieve stable classification boundaries in the early stages. As training progresses, the proportion of manual annotations is gradually reduced to improve the model's adaptability.

[0105] Adversarial training can also be used to improve robustness across environments. Since the lighting, materials, and background textures of defects vary significantly in different scenarios such as bridges, roads, and tunnels, adversarial perturbation can be employed: slight perturbations, such as noise and contrast changes, are artificially introduced during training to ensure the model maintains stable detection capabilities even in complex environments.

[0106] One feasible adversarial learning approach is to enhance the model's robustness to illumination changes by optimizing based on PGD (Projected Gradient Descent) perturbations. Using a MixUp strategy, multiple samples are mixed, allowing the model to learn more generalized feature representations.

[0107] The third approach is to optimize the pixel precision of the attention heatmap to improve the initial pseudo-annotation mask.

[0108] While the attention heatmap generated by the multimodal model can provide a general area of ​​damage, its pixel-level accuracy is limited. Therefore, a diffusion model is introduced to further optimize the crack area and make the boundaries clearer.

[0109] Using a multimodal model attention heatmap as the initial mask, a diffusion model is introduced to progressively optimize the crack region, making the labeled area more closely resemble the actual morphology of the disease. Training method:

[0110] The diffusion model is fine-tuned by combining crack databases (such as CFD and Crack500) to enhance its morphological modeling capabilities in disease detection tasks. Latent variable optimization is employed to ensure that the diffusion model retains crack morphology information and reduces background noise interference during image reconstruction.

[0111] Furthermore, cross-layer feature fusion, combined with deep features from multimodal models, can make the spatial distribution of pseudo-labels in crack regions more reasonable. Specifically, FPN (Feature Pyramid Network) can be used to fuse attention information from different layers to generate higher-resolution crack label masks. Alternatively, a U-Net-like two-stream network can be used to extract global and local crack features separately, improving the detection capability of small cracks.

[0112] Steps S102 and S103 are used to extract the visual features of the image data and the semantic vector features of the corresponding anomaly description text, respectively.

[0113] In step S104, a domain knowledge base is introduced into the multimodal model. The relevance between corresponding words in the text information is calculated using the tuple relationships in the domain knowledge base. This relevance is used to enhance the recognition strength of the multimodal model for texts related to traffic infrastructure disease categories. Introducing the domain knowledge base (Structural Health Knowledge Graph, SHKG) enhances the professionalism of disease category texts and ensures the accuracy of semantic matching. Learnable prompts (PromptTuning) are used for task-adaptive fine-tuning to make the text descriptions more closely match the needs of disease detection. For example:

[0114] Original caption: "This is a photo of a crack."

[0115] Optimization tips:

[0116] "This is a {direction} crack in a {component} of {material}, with width of {width} mm."

[0117] This method can more accurately align disease types and improve the detection robustness of the CLIP model.

[0118] In step S105, the image data and the anomaly description text are self-distilled, and the loss weights of the multimodal model are adjusted based on the self-distillation results. This step aims to dynamically adjust the loss weights by self-distilling the image data and the anomaly description text, so that the model pays more attention to difficult-to-identify disease categories during training, thereby improving the overall recognition accuracy.

[0119] In step S106, the visual features and the semantic vector features are mapped to the same embedding space using the multimodal model, and the feature matching degree of the visual features and the semantic vector features is calculated in the embedding space.

[0120] Specifically, by extracting image and text features and calculating their semantic similarity, the system can automatically match the text description that best matches the corresponding damage category of transportation infrastructure in the image data. The method for calculating the semantic similarity between image and text features is not limited here; it can be calculated by normalizing the features and then calculating the cosine similarity, or by using Euclidean distance, etc. This allows the multimodal model to improve detection accuracy based on semantic information even in unfamiliar damage categories or different environments. For example, the system can match a crack image to text categories such as "vertical crack" and "width 5mm," achieving intelligent annotation and classification.

[0121] In step S107, a text description matching the anomaly information in the image data is determined based on the feature matching degree, and the text description is converted into an anomaly detection report for the traffic infrastructure using a natural language generation model. The specific content of the anomaly detection report is not limited here, and may include, but is not limited to, the type, location, severity, and repair recommendations of the defect, and may also include visual annotations. This reduces manual analysis time, eliminating the need for engineers to manually organize detection data; the system can automatically generate detection results, improving work efficiency.

[0122] In this embodiment of the application, when performing anomaly detection on transportation infrastructure, image data is acquired and corresponding text information is labeled with features. Then, a multimodal model is used to output a text description that matches the anomaly information and a corresponding anomaly detection report. By fusing the visual features corresponding to the image data and the text features corresponding to the text information, the accurate location and category determination of defects in the image data can be achieved, which can effectively improve the accuracy of anomaly detection for transportation infrastructure.

[0123] In one feasible implementation, the automatically generated report covers basic information about the defects, severity assessment, and maintenance recommendations, and can be used for the inspection of infrastructure such as bridges, roads, and tunnels. See Table 1, which is an exemplary anomaly detection report.

[0124] Table 1. An exemplary anomaly detection report

[0125]

[0126] Since traditional text reports often lack intuitiveness, this embodiment introduces a visualization enhancement module that combines disease detection heatmaps and structural annotation information to help engineers intuitively understand the detection results.

[0127] For example, the visualization enhancement module can be used to achieve multimodal visualization: adding heat maps of detected cracks to reports, marking affected areas, and improving readability. By combining point cloud data, it can also mark defect information in 3D structural models of transportation infrastructure (such as BIM and GIS), enabling 3D stereoscopic visualization assessment.

[0128] Furthermore, anomaly detection reports can also be used to predict disease development trends. For example, by using time series prediction models (LSTM, Transformer Time-Series) combined with historical inspection data, the spread of disease can be analyzed to provide long-term maintenance recommendations. For example: "A crack width of 5mm was detected, and it is expected to expand to 12mm within the next 12 months. Reinforcement and repair are recommended within 6 months."

[0129] This embodiment aims to automatically generate structured inspection reports, enhancing their value in engineering applications. By employing Natural Language Generation (NLG) technology, text descriptions matching anomaly information are automatically converted into structured, highly readable anomaly inspection reports, helping engineers quickly grasp the health status of infrastructure. Combined with the damage detection results, repair suggestions and maintenance timeline predictions can also be provided, improving the scientific rigor and timeliness of maintenance work.

[0130] The following section uses CLIP (Contrastive Language-Image Pretraining) as an exemplary multimodal model to illustrate the anomaly detection method for traffic infrastructure provided in this application. This embodiment employs CLIP as a multimodal model, utilizing image-text alignment capabilities to extract semantic information from defect images, and achieving defect category matching through cross-modal similarity calculation, thereby realizing anomaly detection under unsupervised or weakly supervised conditions.

[0131] A basic CLIP network architecture can include:

[0132] Visual encoder (ViT-B / 16, ViT-L / 14 or ResNet50): Used to extract visual features from disease images.

[0133] Text encoder: Used to extract semantic features from disease category text (such as "cracks", "settlement", "corrosion").

[0134] Cross-modal feature alignment module: Image and text features are mapped to the same embedding space. Cosine similarity is calculated between the lesion image and text descriptions of different categories to achieve matching degree calculation, thus completing anomaly detection and classification.

[0135] Based on the CLIP network structure described above, the visual encoder can be further optimized to improve its feature representation capability:

[0136] The standard CLIP uses ViT-B / 16 or ResNet50 as the visual encoder, but in structural defect detection tasks: ViT-B / 16 performs better on large-scale images, but is insufficient in capturing small targets (such as tiny cracks). ResNet50, on the other hand, performs strongly in extracting detailed features, but lacks the ability to model long-range dependencies.

[0137] Therefore, in one feasible implementation, ViT-L / 14 (a larger-scale visual Transformer) can be used to enhance the global perception of diseased areas. Combined with the Swing Transformer, local feature extraction can be optimized to improve the detection accuracy of small-scale diseases (such as fine cracks).

[0138] Another improvement approach is to optimize the text encoder to enhance the expression of semantic information:

[0139] CLIP's text encoder uses the standard Transformer, but its description of structural defects is too generalized and cannot accurately match technical terms.

[0140] Among other improvements, cross-modal feature alignment can be optimized to improve category matching accuracy. However, because defect detection involves fine-grained categories (e.g., “longitudinal cracks in bridges” vs. “surface spalling in tunnels”), CLIP’s standard cross-modal alignment mechanism may lead to decreased category discrimination.

[0141] In one feasible implementation, a Cross-Attention Module (CAM) is introduced to enhance fine-grained feature interactions between text and images, thereby improving the accuracy of category matching. A self-distillation strategy is used to optimize the embedding of disease images, making them more stable for detection under different angles and lighting conditions.

[0142] See Figure 2 , Figure 2 This is a schematic diagram of feature interaction in a multimodal model that introduces a cross-attention module, as provided in an embodiment of this application. Figure 2 In this study, CLIP is used as an example of a multimodal model, which uses an image encoder and a text encoder to achieve comparative learning of image features and text features.

[0143] The specific process is as follows:

[0144] enter:

[0145] Images of defects (such as bridge cracks, tunnel spalling, and road subsidence).

[0146] Text category descriptions (such as "cracks", "corrosion", "settlement").

[0147] Calculation process:

[0148] The visual encoder (ViT-L / 14 combined with Swin Transformer) extracts disease image features and generates a series of hierarchical feature maps.

[0149] The text encoder (Transformer) processes disease category text to obtain semantic embeddings (TextEmbeddings).

[0150] The cross-modal attention module is used to calculate the matching degree between image features and text features:

[0151] Attention weights highlight the diseased areas, forming a preliminary attention heatmap.

[0152] Region aggregation improves the responsiveness of diseased areas.

[0153] The CLIP model can generate attention heatmaps and perform threshold filtering to determine the preliminary pseudo-label mask for diseased areas.

[0154] Based on the above embodiments, the following is a feasible self-distillation implementation process corresponding to step S105:

[0155] The first step is to collect a multimodal dataset containing image data and text information related to transportation infrastructure. The image data is then standardized, and the text information is segmented and encoded. The dataset is divided into training, validation, and test sets to ensure a balanced data distribution.

[0156] The second step involves initial training the multimodal model using the training set to ensure it can learn the basic features of images and text. During training, the model's output (such as predictions and intermediate features) is recorded and used as self-distilled soft labels.

[0157] The third step is to use the trained model to predict on the training set and generate soft labels. These soft labels are then combined with the original labels to construct a new training objective. The model is then reinitialized and trained using the new objective through self-distillation to optimize its performance.

[0158] Fourth, during the self-distillation process, dynamically adjust the loss weights for the image and text modalities. Based on the performance metrics of the validation set (such as accuracy and loss value), gradually adjust the weights to achieve a more balanced model performance across different modalities. One feasible implementation is to determine the optimal weight allocation using cross-validation or grid search.

[0159] Step 5: Evaluate the model's performance using the validation set to ensure that the adjusted loss weights improve the model's generalization ability. If the performance does not meet expectations, further adjust the weights or optimize the model structure.

[0160] Step 6: Test the final multimodal model on the test set to verify its performance in real-world scenarios. Based on the test results, the weights can be further adjusted.

[0161] The CLIP model's attention heatmap is used to extract diseased regions and generate preliminary pseudo-annotation masks. These masks are then optimized by incorporating a small number of high-confidence real-world annotations, improving annotation quality and enabling the CLIP model to learn stably even under unsupervised or minimally annotated conditions.

[0162] By using a confidence scoring mechanism, low-confidence data is optimized through data augmentation (rotation, contrast adjustment, etc.) before being used in training, thereby improving the generalization ability of the CLIP model and enabling it to adapt to different detection scenarios.

[0163] Since the initial pseudo-label mask generated by the CLIP attention heatmap may contain errors, such as blurred boundaries, background noise interference, and category confusion, this method combines semantically guided weakly supervised learning. Through learnable prompts, confidence scoring mechanisms, data augmentation, and a small amount of real label optimization, the quality of pseudo-labels is improved, enhancing the model's adaptability in small-sample environments. The semantic guidance process will not be repeated here.

[0164] As can be seen, in this embodiment, the CLIP model is used for visual-semantic alignment, and multimodal semantic features are used to assist in disease identification, thereby improving the accuracy of target detection. Preliminary pseudo-annotation masks are automatically generated as text information using CLIP's attention heatmap, reducing the workload of manual annotation and improving data consistency. Combined with a confidence scoring mechanism, low-confidence regions are manually optimized to further improve the quality of pseudo-annotations. This reduces reliance on large-scale manually annotated data, significantly saving data construction time and costs.

[0165] Based on this, semantic guidance is used to enhance the adaptability of the CLIP model to different environments (such as changes in lighting and structural materials), thereby improving its cross-scene detection capabilities.

[0166] In one feasible implementation, to improve the detection accuracy of diseases, object detection algorithms (such as YOLO / Faster R-CNN) can be further employed, combined with visual-semantic feature fusion consisting of image and text features, to simultaneously achieve precise disease localization and category identification. Based on object detection, semantic label matching can be introduced to improve the classification accuracy of disease types. Combining spatial information from object detection with semantically guided category matching ensures more stable and accurate detection results. Furthermore, after detecting a disease, its most suitable semantic label can be automatically matched, improving the interpretability of anomaly detection reports.

[0167] By employing a fusion of CLIP semantic guidance, object detection, and multi-view image data acquisition, the CLIP model's adaptability to various environments, including complex lighting, different materials, and occlusion interference, is optimized. Combined with data augmentation and cross-domain feature alignment, the CLIP model ensures stable operation in different scenarios (such as urban roads, highways, mountain bridges, and underground tunnels) and maintains high detection accuracy under varying environments, lighting conditions, and infrastructure types.

[0168] See Figure 3 , Figure 3This is a schematic diagram of an anomaly detection system for transportation infrastructure provided in an embodiment of this application. The anomaly detection system for transportation infrastructure provided in this application will be described below. The anomaly detection system described below can be referred to in conjunction with the anomaly detection method for transportation infrastructure described above.

[0169] An image data acquisition module is used to acquire image data of the traffic infrastructure;

[0170] A visual feature extraction module is used to extract visual features from the image data using a visual encoder in a multimodal model;

[0171] The semantic feature extraction module is used to extract the semantic vector features of the abnormal description text corresponding to the image data using the text encoder in the multimodal model;

[0172] The model optimization module is used to introduce a domain knowledge base into the multimodal model, calculate the correlation between words corresponding to the semantic vector features using the tuple relationships in the domain knowledge base, and enhance the recognition strength of the multimodal model for the text of the corresponding disease category of transportation infrastructure; perform self-distillation on the image data and the abnormal description text, and adjust the loss weight of the multimodal model according to the self-distillation results;

[0173] The feature matching module is used to map the visual features and the semantic vector features to the same embedding space using the multimodal model, and to calculate the feature matching degree between the visual features and the semantic vector features in the embedding space;

[0174] The detection report generation module is used to determine a text description that matches the abnormal information corresponding to the image data based on the feature matching degree, and to convert the text description into an anomaly detection report of the traffic infrastructure using a natural language generation model.

[0175] Based on the above embodiments, as a preferred embodiment, the text information annotation module includes:

[0176] The preliminary annotation unit is used to extract abnormal regions in the image data based on the attention heatmap and obtain a preliminary pseudo-annotation mask.

[0177] The annotation optimization unit is used to optimize the preliminary pseudo-annotation mask to obtain the text information of the features corresponding to the image data.

[0178] Based on the above embodiments, as a preferred embodiment, the text information annotation module further includes:

[0179] The attention heatmap generation module is used to input the image data and corresponding text information into the multimodal model for forward propagation, obtain the output of the multimodal model, extract attention weights in the attention layer of the multimodal model, and map the attention weights to the pixel level of the image to generate the attention heatmap.

[0180] Based on the above embodiments, as a preferred embodiment, the text information annotation module further includes:

[0181] The attention heatmap optimization module optimizes the attention heatmap using at least one of the following methods:

[0182] A disease structure information database is introduced, and abnormal regions are screened based on the disease structure features in the disease structure information database in order to optimize the attention regions corresponding to the abnormal regions in the attention heatmap.

[0183] Alternatively, a feature optimization method can be adopted for the attention heatmap based on the scale of the disease.

[0184] Alternatively, the attention heatmap can be refined layer by layer using a feature pyramid containing deep feature layers.

[0185] Based on the above embodiments, as a preferred embodiment, the annotation optimization unit includes:

[0186] The first optimization unit is used to introduce learnable semantic embeddings to optimize the initial pseudo-label mask;

[0187] The second optimization unit is used to perform adversarial training using the real labeled mask and the initial pseudo-labeled mask;

[0188] The third optimization unit is used to optimize the pixel accuracy of the attention heatmap in order to optimize the initial pseudo-label mask.

[0189] Based on the above embodiments, as a preferred embodiment, the first optimization unit is a unit for performing the following steps:

[0190] Learnable cue information is introduced into the multimodal model to guide it in identifying the types of defects in the transportation infrastructure in the image data. The cue information includes soft cue information and chain-thinking cue information. The soft cue information is a set of learnable semantic embeddings input into the multimodal model. The chain-thinking cue information guides the multimodal model to focus on the key features of the transportation infrastructure defects. Different confidence regions are set using high-confidence ground truth annotations, and the initial pseudo-annotation mask is optimized by filtering through these different confidence regions.

[0191] This application also provides an embodiment corresponding to a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method described in the above method embodiments.

[0192] It is understood that if the methods in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0193] The computer-readable storage medium provided in this embodiment includes the method mentioned above, and has the same effect.

[0194] This application also provides an electronic device, see [link to document]. Figure 4 The present application provides a structural diagram of an electronic device, such as... Figure 4 As shown, it may include a processor 1410 and a memory 1420.

[0195] The processor 1410 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 1410 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 1410 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 1410 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 1410 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0196] The memory 1420 may include one or more computer-readable storage media, which may be non-transitory. The memory 1420 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 1420 is used to store at least the following computer program 1421, which, after being loaded and executed by the processor 1410, is capable of implementing the relevant steps in the methods executed by the electronic device side as disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 1420 may also include an operating system 1422 and data 1423, etc., and the storage method may be temporary storage or permanent storage. The operating system 1422 may include Windows, Linux, Android, etc.

[0197] In some embodiments, the electronic device may further include a display screen 1430, an input / output interface 1440, a communication interface 1450, a sensor 1460, a power supply 1470, and a communication bus 1480.

[0198] certainly, Figure 4 The structure of the terminal shown does not constitute a limitation on the terminal in the embodiments of this application. In practical applications, the terminal may include more than [other components]. Figure 4 More or fewer components as shown, or combinations of certain components.

[0199] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. As the system provided in the embodiments corresponds to the method provided in the embodiments, the description is relatively simple; relevant parts can be found in the method section.

[0200] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of this application.

[0201] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method for detecting anomalies in transportation infrastructure, characterized in that, include: Acquire image data of the transportation infrastructure; Visual features of the image data are extracted using a visual encoder in a multimodal model; The semantic vector features of the anomaly description text corresponding to the image data are extracted using the text encoder in the multimodal model. A domain knowledge base is introduced into the multimodal model, and the correlation between words corresponding to the semantic vector features is calculated using the tuple relationships in the domain knowledge base; the correlation is used to enhance the recognition strength of the multimodal model for texts on the corresponding disease categories of transportation infrastructure. The image data and the anomaly description text are self-distilled, and the loss weights of the multimodal model are adjusted based on the self-distillation results. The visual features and semantic vector features are mapped to the same embedding space using the multimodal model, and the feature matching degree between the visual features and the semantic vector features is calculated in the embedding space. Based on the feature matching degree, a text description matching the anomaly information corresponding to the image data is determined, and a natural language generation model is used to convert the text description into an anomaly detection report of the traffic infrastructure. Before calculating the feature matching degree between the visual features and the semantic vector features in the embedding space, the method further includes: Attention weights are extracted from the attention layer of the multimodal model, and the attention weights are mapped to the pixel level of the image data to generate an attention heatmap; Based on the attention heatmap, abnormal regions in the image data are extracted to obtain preliminary pseudo-label masks for the abnormal regions. The initial pseudo-label mask is optimized to generate labeled data corresponding to the abnormal region; the labeled data is used to guide the multimodal model to perform anomaly recognition and detection. After generating the attention heatmap, at least one of the following optimization methods is used to optimize the attention heatmap: A disease structure information database is introduced, and abnormal regions are screened based on the disease structure features in the disease structure information database in order to optimize the attention regions corresponding to the abnormal regions in the attention heatmap. Alternatively, a feature optimization method can be adopted for the attention heatmap based on the scale of the disease. Alternatively, the attention heatmap can be refined layer by layer using a feature pyramid containing deep feature layers.

2. The anomaly detection method according to claim 1, characterized in that, Optimizing the initial pseudo-label mask includes at least one of the following optimization methods: Learnable semantic embeddings are introduced to optimize the initial pseudo-label mask; Adversarial training is conducted using the real labeled mask and the initial pseudo-labeled mask; The attention heatmap is optimized in terms of pixel precision to improve the initial pseudo-label mask.

3. The anomaly detection method according to claim 2, characterized in that, Optimizing the initial pseudo-label mask by introducing learnable semantic embeddings includes: Learnable cue information is introduced into the multimodal model to guide it in identifying the types of defects in the transportation infrastructure in the image data; wherein, the cue information includes soft cue information and chain-thinking cue information, the soft cue information being a set of learnable semantic embeddings used as input to the multimodal model; the chain-thinking cue information is used to guide the multimodal model to focus on the key features of defects in the transportation infrastructure; Different confidence regions are set using high-confidence real labels, and the preliminary pseudo-label mask is filtered through different confidence regions to optimize the preliminary pseudo-label mask.

4. The anomaly detection method according to claim 3, characterized in that, The initial pseudo-label mask is optimized by using high-confidence real labels to define different confidence regions and filtering the initial pseudo-label mask through these different confidence regions. The classification type is determined using high-confidence ground truth labels; the classification type includes high-confidence regions, medium-confidence regions, and low-confidence regions. The joint distribution between the initial pseudo-label mask and the high-confidence real label is calculated using a confidence learning method, and suspected erroneous labels are determined based on the joint distribution. The preliminary pseudo-label masks are classified according to the suspected erroneous labels and the classification type. The preliminary pseudo-label masks belonging to the high-confidence region are retained, and the preliminary pseudo-label masks belonging to the medium-confidence region are subjected to data augmentation processing before being used in training. The preliminary pseudo-label masks containing the suspected erroneous labels are discarded. The data augmentation processing method includes at least one of data rotation and contrast adjustment.

5. An anomaly detection system for transportation infrastructure, characterized in that, include: An image data acquisition module is used to acquire image data of the traffic infrastructure; A visual feature extraction module is used to extract visual features from the image data using a visual encoder in a multimodal model; The semantic feature extraction module is used to extract the semantic vector features of the abnormal description text corresponding to the image data using the text encoder in the multimodal model; The model optimization module is used to introduce a domain knowledge base into the multimodal model, calculate the correlation between words corresponding to the semantic vector features using the tuple relationships in the domain knowledge base, and enhance the recognition strength of the multimodal model for the text of the corresponding disease category of transportation infrastructure; perform self-distillation on the image data and the abnormal description text, and adjust the loss weight of the multimodal model according to the self-distillation results; The feature matching module is used to map the visual features and the semantic vector features to the same embedding space using the multimodal model, and to calculate the feature matching degree between the visual features and the semantic vector features in the embedding space; The detection report generation module is used to determine a text description that matches the abnormal information corresponding to the image data based on the feature matching degree, and to convert the text description into an abnormal detection report of the traffic infrastructure using a natural language generation model. The text information annotation module also includes: The attention heatmap generation module is used to input the image data and corresponding text information into the multimodal model for forward propagation, obtain the output of the multimodal model, extract attention weights in the attention layer of the multimodal model, and map the attention weights to the pixel level of the image to generate the attention heatmap. And the module for performing the following steps: Based on the attention heatmap, abnormal regions in the image data are extracted to obtain preliminary pseudo-label masks for the abnormal regions. The initial pseudo-label mask is optimized to generate labeled data corresponding to the abnormal region; the labeled data is used to guide the multimodal model to perform anomaly recognition and detection. The attention heatmap optimization module optimizes the attention heatmap using at least one of the following methods: A disease structure information database is introduced, and abnormal regions are screened based on the disease structure features in the disease structure information database in order to optimize the attention regions corresponding to the abnormal regions in the attention heatmap. Alternatively, a feature optimization method can be adopted for the attention heatmap based on the scale of the disease. Alternatively, the attention heatmap can be refined layer by layer using a feature pyramid containing deep feature layers.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the anomaly detection method for traffic infrastructure as described in any one of claims 1-4.

7. An electronic device, characterized in that, The system includes a memory and a processor, wherein the memory stores a computer program, and the processor, when calling the computer program in the memory, implements the steps of the anomaly detection method for traffic infrastructure as described in any one of claims 1-4.

Images