Steel bridge disease detection and identification method based on large language model

Abstract

The invention relates to a steel bridge disease detection and identification method based on a large language model, and belongs to the technical field of artificial intelligence and civil engineering crossing. According to the method, a cross-modal feature alignment mechanism is constructed through a pre-trained multi-modal large language model by fusing a steel bridge image and a field customized text prompt, and a cascade detection process of'component identification-disease classification-region segmentation 'is realized. Comprising the following steps: designing a structured text prompt word bank to enhance semantic consistency, and dynamically fusing general knowledge and instance features in combination with a mixed prompt mechanism; a multi-level cross-modal alignment strategy is adopted to generate an anomaly graph, and a disease area is accurately positioned; a visual prompt enhancement module is introduced to improve the multi-scale feature discrimination ability, and the robustness in a complex environment is adjusted and optimized through data self-adaption. Under the condition of few samples or even zero samples, high-sensitivity detection and pixel-level segmentation of steel bridge cracks, corrosion and other diseases are achieved, and the problems that a traditional method is low in efficiency, poor in generalization, high in labor cost and the like are effectively solved.

Description

Technical Field

[0001] The present invention relates to the fact that, as infrastructure continues to age and traffic pressures increase, steel bridges, as key transportation hubs, will inevitably develop structural defects such as cracks, corrosion, and fatigue damage during their long-term service. These problems are directly related to the safety and durability of bridges. Traditional steel bridge defect detection relies primarily on manual inspections and rule-based image processing techniques. However, these methods suffer from low detection efficiency, strong subjectivity, and significant environmental influences. Furthermore, the high cost of constructing large-scale, well-annotated defect image datasets severely limits the widespread application of deep learning technology in this field.

[0002] In recent years, zero-shot and few-shot learning methods have become important approaches to addressing data scarcity. Zero-shot learning leverages pre-trained models to directly identify unseen categories, while few-shot learning aims to rapidly adapt to new tasks with only a minimal number of labeled examples. Multimodal large language models (MLLMs), such as CLIP, achieve deep cross-modal correlation learning by simultaneously encoding image and text information, enabling the model to identify disease without requiring extensive labeled data.

[0003] Specifically, this type of model, through pre-training on large-scale image-text data, can capture and understand the relationship between the visual features of steel bridge defects and textual descriptions. With the help of carefully designed structured prompt text (Prompt Engineering), the model can not only complete the identification and defect classification of various steel bridge components under zero-sample conditions, but also maintain a high detection accuracy when faced with new or unknown defects. At the same time, through strategies such as data enhancement, prompt word optimization, and dynamic threshold matching, the model's robustness and generalization ability in complex environments are further improved. In the technical field, specifically, a steel bridge defect detection and identification method based on a large language model. Background Art

[0004] With the continuous aging of infrastructure and increasing traffic pressure, steel bridges, as key transportation hubs, are inevitably prone to structural defects such as cracks, corrosion, and fatigue damage during their long-term service. These problems are directly related to the safety and durability of bridges. Traditional steel bridge defect detection relies primarily on manual inspections and rule-based image processing techniques. However, these methods suffer from low detection efficiency, strong subjectivity, and significant environmental influences. Furthermore, the high cost of constructing large-scale, well-annotated defect image datasets severely limits the widespread application of deep learning technology in this field.

[0005] In recent years, zero-shot and few-shot learning methods have become important approaches to addressing data scarcity. Zero-shot learning leverages pre-trained models to directly identify unseen categories, while few-shot learning aims to rapidly adapt to new tasks with only a minimal number of labeled examples. Multimodal large language models (MLLMs), such as CLIP, achieve deep cross-modal correlation learning by simultaneously encoding image and text information, enabling the model to identify disease without requiring extensive labeled data.

[0006] Specifically, this type of model, through pre-training on large-scale image-text data, is able to capture and understand the relationship between the visual features and textual descriptions of steel bridge defects. Leveraging carefully designed structured prompts (Prompt Engineering), the model not only recognizes and classifies defects in steel bridge components with zero-shot detection, but also maintains high detection accuracy even for new or unknown defects. Furthermore, strategies such as data augmentation, prompt word optimization, and dynamic threshold matching further enhance the model's robustness and generalization capabilities in complex environments. Summary of the Invention

[0007] In order to solve the problems of the prior art, the present invention provides a steel bridge disease detection and identification method based on a large language model.

[0008] In order to solve the above technical problems, the present invention is implemented through the following technical solutions: In the first aspect, a method for detecting and identifying steel bridge defects based on a large language model comprises the following steps:

[0009] Step 1: Generate multimodal features for steel bridge images using a pre-trained multimodal large language model. Input the defect image and domain-specific text prompts into the model to generate a unified embedding space feature representation that includes both image and text information.

[0010] Step 2: Using the structured prompt word optimization module, we designed specialized text prompt words based on the types, causes, and external manifestations of steel bridge defects. This improved the model's defect classification capabilities under low-sample conditions by applying semantic consistency constraints.

[0011] Step 3: Use the multi-view data enhancement module to generate enhanced prompt words containing multi-angle, multi-scale, and multi-directional visual features to capture the diverse morphological characteristics of steel bridge defects;

[0012] Step 4: Insert a hybrid prompt mechanism into the image encoder and text encoder of the pre-trained model to generate instance-specific prompts by splicing static prompts and dynamic prompts, and input them into the Transformer layer for feature fusion;

[0013] Step 5: Using the cross-modal alignment module, a linear projection layer is added after the image encoder to calculate the similarity between the image patch and the text description of the disease to generate an anomaly map. Multi-level feature maps are then aggregated to locate the diseased area.

[0014] Step 6: Introduce a visual cue enhancement module, construct positive and negative sample pairs through contrastive learning, add a trainable token before the visual Transformer input, and combine it with attention weighting to achieve multi-scale feature fusion;

[0015] Step 7: Based on the cascade detection process, steel bridge component identification, defect classification, and region segmentation are performed in sequence to output the defect type, location, and precise boundary information.

[0016] In this application, the intelligence level and practicality of steel bridge defect detection are significantly improved through multimodal feature fusion and structured prompt optimization. Its core role is to overcome the traditional method's dependence on massive labeled data, and use the cross-modal understanding ability of the pre-trained model to deeply integrate steel bridge images with field-customized text descriptions to achieve a cascade detection process of "component recognition-disease classification-region segmentation". Through the hybrid prompt mechanism and cross-modal alignment strategy, the model accurately captures defect features under few or even zero sample conditions, effectively solving the detection challenges brought by complex lighting, multi-angle shooting and unknown defect types; combined with multi-scale visual prompt enhancement and data adaptive adjustment, it significantly improves the recognition sensitivity and segmentation accuracy of defects such as tiny cracks and localized corrosion. This method not only greatly reduces the cost and false detection rate of manual inspections, but also adapts to different service environments and defect forms through high generalization, providing efficient and reliable technical support for steel bridge structure safety assessment and maintenance decision-making, and promoting bridge detection towards automation and intelligence.

[0017] In a specific implementation of the first aspect, in step 1, the text prompt generated by the multimodal feature includes:

[0018] Descriptive text for steel bridge components, including the names and functional characteristics of beams, welds, and bolted connections;

[0019] Classification text for damage types, covering the causes, morphology and severity classification of cracks, corrosion and fatigue damage;

[0020] A hybrid prompt template that combines engineering terms with natural language descriptions, such as "linear cracks in the weld of a steel bridge, with a width of 0.1-0.3mm."

[0021] In a specific implementation of the first aspect, the multi-view data enhancement module in step 3 includes:

[0022] Randomly rotate, scale, and mirror the diseased image to generate multi-scale and multi-directional features;

[0023] Environmental enhancement based on simulated lighting conditions adjusts image brightness, contrast, and noise levels to match the actual detection scene.

[0024] In a specific implementation of the first aspect, the hybrid prompt mechanism in step 4 is specifically:

[0025] The static prompt (Ps) contains general disease description text, and the dynamic prompt (P_D) extracts instance-related features from the input image through the self-attention mechanism. Ps and P_D are linearly concatenated and input into the Transformer layer.

[0026] In a specific implementation of the first aspect, in the cross-modal alignment module of step 5, the method for generating an anomaly graph includes:

[0027] Calculate the cosine similarity between the image block features and the normal / abnormal text embeddings;

[0028] The feature maps of different levels are unified to the original image size through bilinear interpolation, and the weighted summation is used to generate the final anomaly map M.

[0029] In a specific implementation of the first aspect, the visual cue enhancement module in step 6 further includes:

[0030] Add a trainable token to the front end of the visual transformer's input sequence, interacting with the image patch features through a self-attention mechanism;

[0031] Hierarchical attention weights are used to fuse shallow detail features with deep semantic features to enhance the detection ability of minor diseases.

[0032] In a specific implementation of the first aspect, the cascade detection process of step 7 is specifically as follows:

[0033] Use the component identification module to locate key structural areas of steel bridges and narrow the scope of disease detection;

[0034] A dynamic threshold matching strategy is used within the component area to screen suspected diseased areas;

[0035] Combined with the segmentation algorithm, the pixel-level boundary of the diseased area is extracted and the segmentation results that can be quantitatively evaluated are output.

[0036] In a specific implementation of the first aspect, the pre-trained model is a multimodal large language model based on the CLIP architecture, whose image encoder adopts Vision Transformer (ViT) and the text encoder adopts Transformer network.

[0037] In a second aspect, an electronic device is provided, characterized in that it includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the steps of a steel bridge defect detection and identification method based on a large language model as described in any one of claims 1 to 8.

[0038] In a third aspect, a computer-readable storage medium is provided, characterized in that a computer program is stored therein, and when the computer program is executed by a processor, the steps of the steel bridge defect detection and identification method based on a large language model as described in any one of claims 1 to 8 are implemented.

[0039] The beneficial effects of the present invention are:

[0040] 1. By integrating multimodal large language models with structured domain knowledge, an efficient, robust and low-dependency innovative solution is provided for steel bridge defect detection. Its core advantages are reflected in the following aspects: First, by utilizing cross-modal feature alignment and dynamic prompting mechanisms, the dependence on large-scale labeled data is significantly reduced, so that the model can accurately identify various types of defects in complex environments under zero or few sample conditions, effectively solving the problem of insufficient generalization ability of traditional methods due to data scarcity; secondly, through the cascaded "component positioning-disease classification-region segmentation" process, combined with multi-scale visual prompt enhancement and adaptive threshold matching strategy, the detection of small cracks and local defects is realized. The high-sensitivity detection and pixel-level boundary extraction of hidden defects such as local corrosion greatly improve the reliability and engineering practicality of the detection results; in addition, the design based on hybrid prompts and cross-modal interaction enables the model to dynamically adapt to different service environments, shooting angles and unknown defect types, and maintain stable output under interference such as lighting changes and component deformation, significantly enhancing the environmental robustness of the system; finally, this method greatly reduces the cost of manual inspections and the risk of subjective misjudgment through automated detection and intelligent analysis, providing real-time and scalable technical support for steel bridge structural health monitoring, which has important practical value for extending the service life of bridges and ensuring the safety of transportation infrastructure. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a schematic diagram of the overall framework structure of the present invention. DETAILED DESCRIPTION

[0042] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0043] like Figure 1A steel bridge defect detection and identification method based on a large language model is shown.

[0044] The present invention achieves steel bridge disease identification and detection through the following technical effects:

[0045] 1. Multimodal feature generation

[0046] This approach simultaneously feeds steel bridge defect images and domain-specific textual cues into a pre-trained model (e.g., CLIP), thereby obtaining multimodal feature representations in a unified embedding space. This approach can fully integrate image and textual information, improving the accuracy of defect detection even with limited sample size.

[0047] 2. Optimize related prompt words

[0048] This paper designs a set of specialized text prompts for steel bridge defects, covering various defect types, causes, and external manifestations. This strategy ensures high semantic consistency of the prompts in the embedding space, thereby enhancing the model's recognition capabilities in low-sample scenarios.

[0049] 3. Data adjustment for different perspectives

[0050] To address the diversity of steel bridge defect images, this paper proposes a data augmentation scheme to generate visual cues from multiple angles, scales, and directions. These augmentation methods help the model capture the characteristics of defects in various directions and sizes, improving its ability to adapt to complex scenarios.

[0051] The above are some optimizations related to component recognition.

[0052] 4. New prompt mechanism

[0053] A new prompt layer is inserted into the image encoder and text encoder of pre-trained models such as CLIP. By concatenating static prompts PS and dynamic prompts PD, a hybrid prompt P = PS + PD is generated and input into the Transformer layer. This design combines universality and instance specificity, enhancing the model's adaptability to different abnormal patterns.

[0054] 5. Add transmembrane alignment module

[0055] To align the embedding spaces of images and text, a linear projection layer is added after the image encoder.

[0056] The anomaly map is generated by calculating the similarity between the image block and the normal or abnormal text. The final anomaly map M is aggregated by multi-level feature maps.

[0057] The above is the optimization related to disease identification

[0058] 6. Enhanced visual cues

[0059] Currently, prompts are only described in text, and visual features are insufficient, so contrastive learning is performed to enhance the discriminative power of features by constructing positive and negative sample pairs.

[0060] A learnable visual prompt is proposed: a trainable token is added before the ViT input, and the block features at different levels are weighted by attention to achieve multi-scale feature fusion.

[0061] Example:

[0062] 1. Data preparation and input configuration

[0063] Image input: Collect RGB images of the steel bridge surface, adjust the resolution to 1024×768 pixels, and use JPG or PNG format.

[0064] Text prompt library: Build a text library containing 50 standard templates, such as:

[0065] Component template: "Steel bridge {beam body / weld / bolt}, located at {main beam midspan / connection}, function is {load-bearing / structural connection}."

[0066] Disease template: "Corrosion on the beam, manifested as reddish-brown patches, area ≥ 5cm 2}, and the severity is {moderate}."

[0067] 2. Multimodal model initialization

[0068] Pre-trained model: Use the CLIP model (ViT-B / 16 as the image encoder and a 12-layer Transformer as the text encoder) and load the official pre-trained weights.

[0069] Feature alignment parameters: the image block size is 16×16 pixels, the maximum length of text encoding is set to 77 tokens, and the embedding space dimension is 512.

[0070] 3. Data enhancement and training configuration

[0071] Image enhancement:

[0072] Rotation angle range: ±30°, zoom ratio: 0.8-1.2 times, flip probability: 50% each in horizontal / vertical directions.

[0073] Brightness adjustment range: ±20%, contrast adjustment coefficient: 0.9-1.1, Gaussian noise standard deviation σ=0.05.

[0074] Text enhancement: Generate text descriptions three times the amount of original data through synonym replacement (such as "crack" → "craze", "rust" → "corrosion").

[0075] 4. Implementation of hybrid prompt mechanism

[0076] Static prompts (Ps): 10 common descriptions are predefined, such as “typical defect areas of steel bridges”, which are encoded as fixed-length 512-dimensional vectors.

[0077] Dynamic Prompt (P_D): Extract 16×16 image patch features from the input image and generate dynamic embedding with a dimension of 512 through a self-attention layer.

[0078] Feature fusion: Ps and P_D are concatenated into a 1024-dimensional vector and input into the Transformer layer of the 4-head attention mechanism for interaction.

[0079] 5. Cross-modal alignment and anomaly map generation

[0080] Linear projection layer: The fully connected layer parameters are 512×512, and the activation function is ReLU.

[0081] Similarity calculation: Calculate the cosine similarity between each image block and the “normal / abnormal” text, with the threshold set to 0.75.

[0082] Multi-level aggregation: Feature maps are extracted from the 4th, 8th, and 12th layers of ViT and bilinearly interpolated to the original image size with weighting coefficients of 0.2, 0.3, and 0.5 respectively.

[0083] 6. Visual Cue Enhancement Module

[0084] Trainable Token: Four trainable tokens (dimension 512) are inserted at the front end of the ViT input sequence and input into the 12-layer Transformer together with the image block.

[0085] Attention fusion: The attention weights of the shallow layer (3rd layer) and the deep layer (9th layer) are weighted in a ratio of 0.3:0.7 to output multi-scale features.

[0086] 7. Cascade detection process execution

[0087] Part recognition: Use the pre-trained model to locate the part area and output the valid area with a mask confidence ≥ 0.8.

[0088] Defect detection: Calculate the anomaly map within the component area, and use the dynamic threshold to take the top 10% of the response value as the candidate defect area.

[0089] Segmentation network: A lightweight U-Net (encoder 4 layers, decoder 4 layers, channel number [64, 128, 256, 512]) is used, the input size is 256×256, and the pixel-level segmentation result is output.

[0090] 8. Model training and deployment

[0091] Training parameters:

[0092] Optimizer: AdamW, initial learning rate 1e-4, weight decay 0.01, batch size = 32.

[0093] Learning rate scheduling: cosine annealing, period 50 epochs, minimum learning rate 1e-6.

[0094] Training strategy: Freeze the CLIP backbone and train only the hint layer, projection layer, and U-Net for 100 epochs.

[0095] Deployment configuration:

[0096] Model format: exported to ONNX, accelerated using TensorRT.

[0097] Hardware platform: NVIDIA Jetson AGX Xavier, 32GB memory, inference frame rate ≥ 10FPS (resolution 1024×768).

[0098] 9. Effect Verification

[0099] Test data: Contains 500 steel bridge images, covering six types of defects, including cracks, corrosion, and fatigue, and annotated in accordance with the "Highway Bridge Maintenance Code" (JTG / T J21-2023).

[0100] Performance indicators:

[0101] Zero-shot detection accuracy: 89.2% (Top-1), segmentation IoU: 76.5%.

[0102] Few-shot fine-tuning (20 labeled images): The accuracy is improved to 93.7% and the IoU reaches 81.3%.

[0103] Real-time performance: The edge device inference delay is ≤100ms / frame, meeting the real-time requirements of drone inspections.

[0104] In summary, the present invention has the following working principles:

[0105] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A steel bridge defect detection and identification method based on a large language model, characterized in that: The following steps are involved: Step 1: Generate multimodal features for steel bridge images using a pre-trained multimodal large language model. Input the defect image and domain-specific text prompts into the model to generate a unified embedding space feature representation that includes both image and text information. Step 2: Using the structured prompt word optimization module, we designed specialized text prompt words based on the types, causes, and external manifestations of steel bridge defects. This improved the model's defect classification capabilities under low-sample conditions by applying semantic consistency constraints. Step 3: Use the multi-view data enhancement module to generate enhanced prompt words containing multi-angle, multi-scale, and multi-directional visual features to capture the diverse morphological characteristics of steel bridge defects; Step 4: Insert a hybrid prompt mechanism into the image encoder and text encoder of the pre-trained model to generate instance-specific prompts by splicing static prompts and dynamic prompts, and input them into the Transformer layer for feature fusion; Step 5: Using the cross-modal alignment module, a linear projection layer is added after the image encoder to calculate the similarity between the image patch and the text description of the disease to generate an anomaly map. Multi-level feature maps are then aggregated to locate the diseased area. Step 6: Introduce a visual cue enhancement module, construct positive and negative sample pairs through contrastive learning, add a trainable token before the visual Transformer input, and combine it with attention weighting to achieve multi-scale feature fusion; Step 7: Based on the cascade detection process, steel bridge component identification, defect classification, and region segmentation are performed in sequence to output the defect type, location, and precise boundary information.

2. The method for detecting and identifying steel bridge defects based on a large language model according to claim 1, characterized in that: In step 1, the text prompt generated by the multimodal feature includes: Descriptive text for steel bridge components, including the names and functional characteristics of beams, welds, and bolted connections; Classification text for damage types, covering the causes, morphology and severity classification of cracks, corrosion and fatigue damage; A hybrid prompt template that combines engineering terms with natural language descriptions, such as "linear cracks in the weld of a steel bridge, with a width of 0.1-0.3mm." 3. The method for detecting and identifying steel bridge defects based on a large language model according to claim 1 is characterized by: The multi-view data enhancement module in step 3 includes: Randomly rotate, scale, and mirror the diseased image to generate multi-scale and multi-directional features; Environmental enhancement based on simulated lighting conditions adjusts image brightness, contrast, and noise levels to match the actual detection scene.

4. The method for detecting and identifying steel bridge defects based on a large language model according to claim 1 is characterized by: The hybrid prompt mechanism in step 4 is specifically as follows: The static prompt (Ps) contains general disease description text, and the dynamic prompt (P_D) extracts instance-related features from the input image through the self-attention mechanism. Ps and P_D are linearly concatenated and input into the Transformer layer.

5. The method for detecting and identifying steel bridge defects based on a large language model according to claim 1 is characterized by: In the cross-modal alignment module of step 5, the method for generating anomaly graphs includes: Calculate the cosine similarity between the image block features and the normal / abnormal text embeddings; The feature maps of different levels are unified to the original image size through bilinear interpolation, and the weighted summation is used to generate the final anomaly map M.

6. The method for detecting and identifying steel bridge defects based on a large language model according to claim 1 is characterized by: The visual cue enhancement module of step 6 further includes: Add a trainable token to the front end of the visual transformer's input sequence, interacting with the image patch features through a self-attention mechanism; Hierarchical attention weights are used to fuse shallow detail features with deep semantic features to enhance the detection ability of minor diseases.

7. The method for detecting and identifying steel bridge defects based on a large language model according to claim 1 is characterized by: The cascade detection process of step 7 is specifically as follows: Use the component identification module to locate key structural areas of steel bridges and narrow the scope of disease detection; A dynamic threshold matching strategy is used within the component area to screen suspected diseased areas; Combined with the segmentation algorithm, the pixel-level boundary of the diseased area is extracted and the segmentation results that can be quantitatively evaluated are output.

8. The method for detecting and identifying steel bridge defects based on a large language model according to claim 1 is characterized by: The pre-trained model is a multimodal large language model based on the CLIP architecture, whose image encoder adopts VisionTransformer (ViT) and text encoder adopts Transformer network.

9. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method realizes the steps of a steel bridge defect detection and identification method based on a large language model as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that A computer program is stored, and when the computer program is executed by a processor, the steps of the steel bridge defect detection and identification method based on a large language model according to any one of claims 1 to 8 are implemented.

Images