A bridge damage assessment method and system based on a visual segmentation model and a medium

Abstract

This invention relates to the field of bridge structural safety inspection technology, and in particular to a bridge damage assessment method, system, and medium based on a visual segmentation model. The method includes: acquiring prior structural information and post-damage images of the bridge to be inspected; matching the image to be inspected generated based on the post-damage images with the prior structural information and a preset damage semantic knowledge base to generate multimodal prompts; inputting the fused multimodal prompts and the image to be inspected into a visual segmentation model with input parameters frozen, and adaptively iterating to obtain pixel-level segmentation masks for each damage candidate region; superimposing each pixel-level segmentation mask onto the image to be inspected, and inputting it along with the damage analysis prompt template into a visual language model to generate a structured damage assessment report. This invention utilizes the general segmentation capability of the visual segmentation model to achieve rapid and accurate pixel-level segmentation and recognition of sudden bridge damage, and automatically generates a structured damage assessment report.

Description

Technical Field

[0001] This invention relates to the field of bridge structural safety inspection technology, and in particular to a bridge damage assessment method, system and medium based on a visual segmentation model. Background Technology

[0002] Bridges are a key component of transportation infrastructure. Under sudden natural disasters such as earthquakes and floods, as well as other unnatural disasters, bridge structures may suffer sudden damage such as impacts and collapses, requiring rapid and accurate damage assessment to support emergency decision-making and repair scheduling.

[0003] Currently, various technical approaches have been developed in the field of bridge damage detection, mainly consisting of three categories and their following shortcomings: First, there are methods based on traditional deep learning, such as YOLO and Faster R-CNN models, for example, a bridge crack detection method, system, and storage medium based on the YOLO algorithm (publication number CN119107317B), which uses the YOLO algorithm for bridge crack detection; and a bridge damage identification method based on deep learning (publication number CN115984641A), which uses transfer learning and the AlexNet model to identify cracks, defects, and corrosion. The core limitations of these approaches are: heavy reliance on large amounts of labeled data; the ability to identify only damage categories seen during training; insufficient accuracy of the output rectangular bounding box; and a lack of engineering semantic understanding capabilities. First, there are several challenges in bridge damage detection. First, vibration-based damage identification requires dense sensor deployment, leading to high costs and the inability to acquire visual damage information. Second, existing solutions apply basic visual large-scale models like SAM to bridge inspection, such as the bridge surface crack extraction method based on visual large-scale model distillation (publication number CN118365582B) and the bridge surface crack pixel-level identification method based on visual large-scale model SAM (publication number CN118379237B). However, these methods downgrade the visual large-scale model to a traditional small model through distillation or fine-tuning, losing zero-sample generalization ability and semantic understanding. Furthermore, in the field of bridge damage detection, sudden bridge damage data is naturally scarce. Existing supervised methods cannot handle unknown damage morphologies, resulting in low identification accuracy and a disconnect between visual perception and semantic cognition, failing to automatically generate structured assessment reports to support emergency decision-making.

[0004] Therefore, this invention provides a new bridge damage assessment method that can accurately identify sudden bridge damage and generate a structured damage assessment report without retraining a large visual model. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a bridge damage assessment method, system, and medium based on a visual segmentation model. It can utilize the universal segmentation capabilities of the visual segmentation model to achieve rapid and accurate pixel-level segmentation and identification of sudden bridge damage, and automatically generate a structured damage assessment report.

[0006] According to a first aspect of the present invention, a bridge damage assessment method based on a visual segmentation model is provided, comprising the following steps: S100: Obtain the prior structural information and damaged image of the bridge to be detected, and perform preprocessing operations on the damaged image to generate the image to be detected; the prior structural information includes bridge type, component information, material information and component spatial topology.

[0007] S200, the image to be detected is matched with the prior structural information and the preset damage semantic knowledge base to generate multimodal prompts; the multimodal prompts include text prompts based on bridge type, component information, material information and damage semantic descriptions matched from the damage semantic knowledge base, as well as visual prompts and geometric prompts based on the spatial topology of the components.

[0008] S300 takes the fused multimodal cue and the preset visual segmentation model with the input parameters of the image to be detected frozen, outputs each damage candidate region and its corresponding initial segmentation mask, and performs adaptive iterative segmentation on the damage candidate regions with segmentation confidence less than a preset threshold to obtain the pixel-level segmentation mask of each damage candidate region.

[0009] S400: After superimposing the pixel-level segmentation mask of each damage candidate region onto the image to be detected, it is input into the preset visual language model along with the damage analysis prompt template to generate a structured damage assessment report that conforms to the damage analysis prompt template.

[0010] According to a second aspect of the present invention, a bridge damage assessment system based on a visual language model is provided, the system comprising: The data acquisition and preprocessing module is used to acquire the structural prior information and damaged images of the bridge to be inspected, and to perform preprocessing operations on the damaged images to generate the images to be inspected; the structural prior information includes bridge type, component information, material information and component spatial topology.

[0011] The multimodal prompt generation module is used to match the image to be detected with prior structural information and a preset damage semantic knowledge base to generate multimodal prompts. The multimodal prompts include text prompts based on bridge type, component information, material information and damage semantic descriptions matched from the damage semantic knowledge base, as well as visual and geometric prompts based on the spatial topology of the components.

[0012] The adaptive iterative segmentation module is used to freeze the multimodal cue and the input parameters of the image to be detected into a preset visual segmentation model, output each damage candidate region and its corresponding initial segmentation mask, and perform adaptive iterative segmentation on the damage candidate regions with segmentation confidence less than a preset threshold to obtain the pixel-level segmentation mask of each damage candidate region.

[0013] The assessment report generation module is used to overlay the pixel-level segmentation mask of each damage candidate region onto the image to be detected, and then input it together with the damage analysis prompt template into a preset visual language model to generate a structured damage assessment report that conforms to the damage analysis prompt template.

[0014] According to a third aspect of the present invention, a non-transitory computer-readable storage medium is provided, wherein at least one instruction or at least one program is stored therein, the at least one instruction or the at least one program being loaded and executed by a processor to implement the above-described bridge damage assessment method based on a visual segmentation model.

[0015] The present invention has at least the following beneficial effects: This invention provides a bridge damage assessment method based on a visual segmentation model. First, it acquires prior structural information and damaged images of the bridge to be detected. The damaged images are then preprocessed to generate images for detection, effectively solving the image quality degradation problem caused by harsh environmental conditions after damage, thus maintaining good detection performance even under adverse conditions. Next, the images for detection are matched with prior structural information and a pre-defined damage semantic knowledge base to generate multimodal cues including visual, textual, and geometric cues. The fused multimodal cues and the input parameters of the images for detection are then frozen into a visual segmentation model. The multimodal cues guide the segmentation behavior of the visual segmentation model, allowing the incorporation of bridge structural cognition and damage classification knowledge without requiring retraining or fine-tuning of the visual segmentation model. This method achieves generalized recognition of unknown damage types in zero-shot scenarios, solving the reliance of traditional supervised methods on large amounts of labeled data. During the segmentation process, adaptive iterative segmentation is performed on damage candidate regions with low segmentation confidence to obtain pixel-level segmentation masks for each damage candidate region. The segmentation quality is continuously improved through segmentation confidence evaluation and iterative fine segmentation mechanisms. While maintaining the general segmentation capability of the large model, pixel-level accurate extraction of bridge damage regions is achieved, thereby improving the accuracy of subsequent damage region information assessment. Finally, the pixel-level segmentation masks are superimposed on the image to be detected and input into the visual language model along with the damage analysis prompt template to generate a structured damage assessment report. This achieves a leap from pure visual perception to damage semantic understanding, which is conducive to timely adjustment of emergency decisions and repair scheduling. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart of a bridge damage assessment method based on a visual segmentation model provided in Embodiment 1 of the present invention; Figure 2 A flowchart of the visual cues generation steps provided in Embodiment 1 of the present invention; Figure 3 This is a flowchart of the geometric hint generation steps provided in Embodiment 1 of the present invention; Figure 4 This is a flowchart of the adaptive iterative segmentation steps provided in Embodiment 1 of the present invention; Figure 5 This is a schematic diagram of the structure of the bridge damage assessment system based on a visual language model provided in Embodiment 2 of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Example 1 Embodiment 1 of the present invention provides a bridge damage assessment method based on a visual segmentation model, such as... Figure 1 As shown, the method includes the following steps: S100: Acquire the prior structural information and damaged images of the bridge to be inspected, and perform preprocessing operations on the damaged images to generate the image to be inspected; the prior structural information includes bridge type, component information, material information, and component spatial topology. The component spatial topology refers to the spatial geometric contours, position coordinates, and topological connections of bridge components obtained from digital information carriers; digital information carriers include, but are not limited to, BIM models and design drawings.

[0020] For example, the bridge type is a beam bridge, arch bridge, or cable-stayed bridge; the component information includes the type and location of components such as main beams, piers, bridge decks, and bearings; and the material information includes concrete, steel structure, etc.

[0021] Specifically, the sources of the images obtained after the damage include, but are not limited to, drone aerial photography, fixed cameras, handheld devices, and satellite remote sensing images.

[0022] Specifically, the steps for preprocessing the damaged image are as follows: S101, several damaged images of the bridge to be detected are subjected to image dehazing and image denoising processing to obtain a preliminary restored image corresponding to each damaged image; the several damaged images include image sequences acquired from different perspectives. In specific implementation, existing image dehazing algorithms, such as the AOD-Net dehazing algorithm, are used for image dehazing processing, and existing image denoising algorithms, such as the DnCNN denoising algorithm based on convolutional neural networks, are used for image denoising processing to suppress atmospheric scattering effects caused by suspended particles such as dust and smoke, as well as noise interference introduced by the sensor; S102, each initially restored image undergoes contrast stretching and histogram equalization processing to obtain an enhanced image corresponding to each damaged image. Contrast stretching and histogram equalization are used to improve image visibility under low-light conditions and enhance the grayscale difference between the damaged area and the background. Those skilled in the art are familiar with the specific processing steps of contrast stretching and histogram equalization, and will not elaborate further here.

[0023] S103: Extract feature points from each enhanced image, register feature points between different enhanced images, and fuse and stitch the registered enhanced images according to overlapping areas to generate a panoramic image covering the bridge to be detected. Feature points are a standard term in the field of image registration and stitching, including but not limited to Speeded-Up Robust Features (SURF) and Oriented FAST and Rotated BRIEF (ORB).

[0024] S104, the panoramic image is uniformly scaled to a preset standard resolution size, completing the preprocessing operation. Those skilled in the art can set the standard resolution size according to actual needs.

[0025] The above-mentioned method effectively solves the problem of image quality degradation caused by harsh environments such as dust, smoke, and low light after damage by acquiring the prior structural information of the bridge and preprocessing the damaged images. This allows the system to maintain good detection performance even under harsh environmental conditions. At the same time, the introduction of multi-source heterogeneous bridge prior knowledge into the system provides a structured semantic foundation for subsequent multimodal prompt generation, significantly improving robustness in complex environments.

[0026] S200, the image to be detected is matched with the prior structural information and the preset damage semantic knowledge base to generate multimodal prompts; the multimodal prompts include text prompts based on bridge type, component information, material information and damage semantic descriptions matched from the damage semantic knowledge base, as well as visual prompts and geometric prompts based on the spatial topology of the components.

[0027] Specifically, the damage semantic knowledge base includes, but is not limited to, descriptions of damage types, bridge component terminology, and descriptions of damage severity. For example, damage type descriptions include concrete cracking, exposed rebar, component fracture, structural deformation, surface spalling, and component loss / collapse. Bridge component terminology includes T-beam web, box girder top slab, pier cap beam, abutment, and expansion joint. Damage severity descriptions include minor surface cracking, moderate localized damage, severe component fracture, and complete collapse. Based on the bridge type, component information, and material information, targeted descriptions are selected, such as concrete cracking and exposed rebar for concrete bridges, and buckling, component fracture, and weld cracking for steel bridges.

[0028] In other words, the logic for generating text prompts is as follows: based on the bridge type, component information, and material information, appropriate bridge component terminology is selected; based on the damage type, the corresponding high-probability damage type description and damage severity description are prioritized, and combined with the bridge component terminology, a natural language prompt paragraph is generated, which can simultaneously include positive descriptions and negative constraints. For example, a negative constraint is to ignore non-structural damage such as surface stains and water stains. For example, the generated text prompts are: This image is a photograph of a concrete beam bridge after earthquake damage. Please identify the damage in the following component areas: main beam, pier, bridge deck, bearing, cap beam; please focus on the following damage types: component fracture, concrete fragmentation, exposed rebar, bending deformation, shear deformation, component tilting and displacement; please classify the damage according to the following severity levels: minor surface cracks and slight local spalling are considered minor; obvious cracks, broken and detached protective layer, and exposed rebar are considered moderate; component fracture, severe deformation, and partial collapse are considered severe; complete component loss and overall collapse are considered severe; please accurately segment the pixel-level boundaries of each damaged area; ignore non-structural damage such as surface stains, water stains, moss, etc.

[0029] Furthermore, a standardized damage semantic knowledge base can be constructed using a multi-level classification system, including, for example, primary classifications, secondary classifications, and corresponding textual semantic descriptions. Primary classifications include, but are not limited to, fracture, fragmentation, deformation, missing, surface, and penetration categories; secondary classifications under the fracture category, such as component fracture and rebar fracture, correspond to damage semantic descriptions such as "the component shows obvious fracture, with visible concrete fragmentation and exposed rebar at the fracture surface" and "the stressed rebar is fractured, with a shear-like fracture surface"; secondary classifications under the deformation category, such as bending deformation and shear deformation, correspond to damage semantic descriptions such as "the component shows obvious bending deformation, deviating from the original axis" and "the component shows shear deformation, exhibiting oblique displacement."

[0030] In addition, when new damage patterns emerge, the system can adapt to new damage patterns simply by updating the damage type description in the damage semantic knowledge base or by adding a small number of reference images to the visual cue library. This eliminates the need to recollect data and retrain models, significantly reducing the system's maintenance and update costs.

[0031] As described above, by matching the image to be detected with prior structural information and a damage semantic knowledge base to generate multimodal prompts, the segmentation behavior of the visual segmentation model is guided by the multimodal prompts. The structural cognition and damage classification knowledge of the bridge can be injected without retraining or fine-tuning the visual segmentation model, realizing the generalized recognition of unknown damage types in zero-shot scenarios and solving the dependence of traditional supervised methods on a large amount of labeled data.

[0032] Specifically, such as Figure 2 As shown, the visual cue is generated through the following steps: S201, when the source of the component spatial topology is a BIM model or design drawings, extract the standard appearance features of each component and build a component feature reference library; this can be understood as: after rendering the multi-angle standard views of each component from the BIM model, use a pre-trained visual encoder, such as the CLIP visual encoder, to extract the component feature vector of each component's standard view and build a component feature reference library.

[0033] S203, extract the appearance features of the image to be detected and perform feature matching with the component feature reference library to locate the region of each component in the image to be detected, and use the located component regions as visual cues. In specific implementation, the image to be detected is segmented into spatial regions corresponding to the standard views of each component, and appearance feature vectors are extracted for each. Then, the cosine similarity between each spatial region and the feature vectors of each component in the component feature reference library is calculated to determine the approximate location region of each component in the image to be detected.

[0034] S205, when the source of the spatial topology of the components is an image taken before the damage, features of each component region are extracted from the image taken before the damage as a visual matching template. For example, collect 3 to 10 reference images of the bridge in its intact state before the damage, and interactively annotate or automatically extract each component region from the reference images.

[0035] S207, perform feature matching between the appearance features of the image to be detected and the visual matching template, locate the region of each component in the image to be detected, and use the located component regions as visual cues.

[0036] In another scenario, if there is no BIM model or design drawings, and no images of the state before damage are available, a pre-trained general object detection model, such as the Grounding DINO model, can be used. Using bridge component terminology as the text query, preliminary component localization can be performed in the post-damage images. The preliminary localization results can then be used as a downgraded alternative to visual cues.

[0037] The above content discloses different visual cue generation strategies, which realize adaptive component localization capabilities under different conditions. When rich prior information is available, high-precision region guidance can be obtained. When prior information is lacking, the system can still maintain basic operation through a degradation scheme, which significantly improves the compatibility and robustness of various application scenarios. Moreover, this strategy does not require model retraining for specific bridges. The general segmentation capability of the model can be adapted to specific bridge structures simply through cue engineering, which greatly reduces the deployment threshold and data dependence.

[0038] Furthermore, such as Figure 3 As shown, the geometric hints are generated through the following steps: S202, Based on the component spatial topology, construct a structural topology graph with each component as a node and the spatial positional relationships between components as edges; the spatial positional relationships include at least up / down, left / right, front / back, and connection type. For example, the connection type is a rigid connection, or a support and supported relationship.

[0039] Specifically, each node includes a component type label, expected spatial location, and expected size range.

[0040] S204, based on the structural topology diagram, generate the expected position mask of each component in the original image coordinate system of the image to be detected, and calculate the feature deviation between the expected position mask of each component and the corresponding region in the image to be detected; this can be understood as: extracting the actual feature distribution of the image to be detected within the area covered by the expected position mask, calculating the deviation between the actual feature distribution and the expected position mask, and quantizing the deviation using Mahalanobis distance or Euclidean distance.

[0041] S206, regions with feature deviations exceeding a preset deviation threshold are marked as candidate anomalous regions, and these candidate anomalous regions are encoded as geometric cues in the form of spatial attention masks. For example, candidate anomalous regions are component regions where the component position is offset, missing, or misaligned.

[0042] Specifically, the spatial attention mask is a binary matrix or a weight matrix, where candidate abnormal regions correspond to high attention values, and other regions correspond to low attention values. Those skilled in the art can set preset deviation thresholds, high attention values, and low attention values ​​according to actual needs, which will not be elaborated upon here.

[0043] As described above, geometric cues are generated based on the deviation between the actual features in the image to be detected and the standard expected position mask, which realizes the automated detection of component position anomalies. Furthermore, the spatial attention mask guides the preset visual segmentation model to focus on candidate anomaly regions in the subsequent process, effectively compensating for the impact of visual feature degradation in the image to be detected on damage recognition and significantly improving the detection accuracy of the system in complex post-damage environments.

[0044] S300 takes the fused multimodal cues and the preset visual segmentation model with frozen input parameters of the image to be detected, outputs each damage candidate region and its corresponding initial segmentation mask, and performs adaptive iterative segmentation on damage candidate regions with segmentation confidence scores below a preset threshold to obtain pixel-level segmentation masks for each damage candidate region. During the fusion process, visual cues in feature map form, text cues in text embedding vector form, and geometric cues in spatial mask form are dynamically fused using a Cross-Attention mechanism. During fusion, attention weights are assigned based on the relevance of each cue to the image to be detected, with higher-relevance cues receiving greater weights. The fused cue vector is used as the guiding condition for the preset visual segmentation model. The Cross-Attention mechanism can also be replaced with other learnable or non-learnable feature fusion methods.

[0045] By using a combination of visual, textual, and geometric cues to guide the system, prior knowledge of bridge structures and engineering domain knowledge, such as damage types and terminology, are encoded into the reasoning process of a pre-defined visual segmentation model. This enables the system to adapt to different bridge types, materials, and environmental conditions, significantly improving its cross-domain generalization ability.

[0046] Specifically, the preset visual segmentation model uses the SAM 2 model or a combination of Grounding DINO and SAM, or any other pre-trained visual model with zero-shot segmentation capability, and the model parameters remain frozen during inference. The Grounding DINO model is used for object detection, generating bounding boxes of objects based on text prompts, while the SAM model uses these bounding boxes to generate corresponding segmentation masks. The combination of the two enables a complete process from text description to pixel-level segmentation.

[0047] The above describes a method that uses a pre-defined visual segmentation model with frozen parameters and no retraining, combined with multimodal cues for adaptive iterative segmentation. This method maintains the general segmentation capabilities of the large model while achieving pixel-level precision extraction of bridge damage areas. Furthermore, it continuously improves segmentation quality through segmentation confidence assessment and iterative fine segmentation mechanisms. Compared to traditional rectangular bounding box methods, this method can accurately describe the geometric boundaries of irregular damage morphologies, thereby improving the accuracy of subsequent damage area assessment.

[0048] Specifically, the segmentation confidence score corresponding to the initial segmentation mask is calculated using the following formula: C = W1×P + W2×G + W3×L, where C is the segmentation confidence corresponding to the initial segmentation mask, P is the pixel feature consistency within the initial segmentation mask, G is the boundary clarity of the initial segmentation mask, L is the position matching degree between the candidate damage region and the component spatial topology corresponding to the initial segmentation mask, and W1, W2 and W3 are all preset weights.

[0049] Specifically, the pixel feature consistency is the mean similarity calculated based on the similarity between each pair of pixel feature vectors within the initial segmentation mask; where the similarity can be obtained by calculating the cosine similarity.

[0050] The boundary sharpness is the normalized mean of the gradient magnitudes of pixels at the boundaries of the initial segmentation mask. The Sobel operator is typically used to calculate the gradient magnitude of each boundary pixel. The original gradient magnitude range depends on the image grayscale level, such as 0-255, and is normalized to the 0-1 range.

[0051] The position matching degree is the intersection-union ratio or the ratio of overlapping pixels between the candidate region of damage corresponding to the initial segmentation mask and the expected region corresponding to the candidate region of damage in the component space topology.

[0052] The three dimensions mentioned above reflect the consistency of features within the segmented candidate damage region, the sharpness of gradient changes at the boundary, and whether the segmentation result falls within the expected component region. If an anomaly occurs, it indicates that it may be a real damage. By integrating the weighted evaluation of these three dimensions, a quantitative judgment on the reliability of the segmentation result is achieved, providing an accurate convergence judgment basis for adaptive iterative segmentation.

[0053] In a specific embodiment, the adaptive iterative segmentation of candidate damage regions with segmentation confidence levels less than a preset threshold is performed to obtain pixel-level segmentation masks for each candidate damage region, such as... Figure 4 As shown, it includes the following steps: S301, mark the damage candidate regions with segmentation confidence scores less than a preset threshold as low-confidence regions, and generate local secondary prompts for the low-confidence regions; the local secondary prompts include local visual context prompts, local text prompts, and local geometric context prompts corresponding to the low-confidence regions.

[0054] Specifically as follows: Local visual context cues: Extract image region features corresponding to the obtained non-low-confidence initial segmentation mask within a preset pixel range centered on the low-confidence region, such as a neighborhood of 50 pixels.

[0055] Local text hints: Based on the structural prior information of low-confidence regions, match the most similar damage semantic description from the damage semantic knowledge base.

[0056] Local geometric cues: Generate spatial attention masks that only cover low-confidence regions and their extended neighborhoods to narrow the attention scope of the pre-defined visual segmentation model.

[0057] S302, the preset visual segmentation model, which freezes the image to be detected and the local secondary cue input parameters, further segments the local area corresponding to the low-confidence region, outputs new candidate damage regions and corresponding new segmentation masks within this local area, and updates the segmentation confidence of each new segmentation mask. The calculation method for the segmentation confidence of the new segmentation mask is the same as that for the initial segmentation mask, and will not be elaborated further here.

[0058] S303, Repeat steps S301 to S302 until the convergence condition is met, and record the new segmentation mask generated in each iteration and the segmentation confidence of each new segmentation mask.

[0059] Specifically, the convergence condition is that the segmentation confidence of all new segmentation masks is not less than a preset threshold, or the maximum number of iterations is reached.

[0060] S304: For the new segmentation masks of the same new damage candidate region in different iteration rounds, weighted fusion is performed with the corresponding segmentation confidence as the weight, and the fused mask is post-processed to finally output the pixel-level segmentation mask of each damage candidate region.

[0061] Specifically, the same new damage candidate region refers to two new damage candidate regions whose spatial crossover ratio is greater than the preset crossover ratio threshold, where the default value of the preset crossover ratio threshold is 0.5.

[0062] Furthermore, post-processing of the fused mask includes removing fragments with an area smaller than a preset area threshold, filling voids, and smoothing boundaries. Those skilled in the art can set the preset area threshold according to actual needs.

[0063] As described above, through an adaptive iterative mechanism, local re-segmentation is performed only on regions with insufficient segmentation quality, avoiding computational redundancy caused by repeated reasoning on the entire image. This improves segmentation accuracy while maintaining high computational efficiency. Furthermore, it introduces triple cues of local visual context, local text description, and local geometric constraints, providing rich local prior information for the preset visual segmentation model. This effectively solves the problems of blurred segmentation boundaries and loss of local details under complex damage patterns. Moreover, by fusing multiple rounds of masks, it can output high-precision and high-reliability pixel-level damage segmentation results.

[0064] Furthermore, the S300 procedure also includes a cross-scale damage treatment step, including the following: S310, construct a multi-scale pyramid corresponding to the image to be detected; the multi-scale pyramid includes at least the original resolution scale, the first downsampling scale, and the second downsampling scale, with the scale decreasing sequentially. For example, the first downsampling scale and the second downsampling scale are 0.5 times the downsampling scale and 0.25 times the downsampling scale, respectively, that is, dividing it into three scales: large, medium, and small.

[0065] S320, input the images at each scale into the preset visual segmentation model with frozen parameters, and combine the multimodal cues to generate the damage candidate region set and the corresponding intermediate segmentation mask set at each scale.

[0066] S330 maps the intermediate segmentation mask sets at each scale to the original image coordinate system of the image to be detected, fuses the intermediate segmentation masks that partially overlap or contain each other at different scales, and outputs the fused multi-scale damage segmentation mask set.

[0067] The specific fusion rules are as follows: if a large-scale mask completely contains one or more medium-scale or small-scale masks, the large-scale mask is retained as the overall damage representation, and the medium-scale or small-scale mask is retained as its sub-region details; if masks of different scales partially overlap but do not constitute an inclusion relationship, the mask with higher segmentation confidence is retained; for masks that only appear at a single scale, they are directly retained.

[0068] Specifically, the multi-scale damage segmentation mask set contains damage information at each scale of the multi-scale pyramid. Specifically, the damage information at the original resolution scale is component-level damage information, such as identifying the fracture of an entire beam, the collapse of an entire slab, or the tilting of a bridge pier; the damage information at the first downsampling scale is region-level damage information, such as identifying localized concrete cracking, exposed rebar, or peeling of the protective layer; and the damage information at the second downsampling scale is pixel-level damage information, such as identifying micro-cracks or surface cracks.

[0069] The above-mentioned method overcomes the defects of missed damage detection or inaccurate localization at a single scale by constructing a multi-scale image pyramid and performing damage segmentation separately. By uniformly mapping the segmentation masks at each scale to the original image coordinate system and fusing them, complementary enhancement of multi-scale damage information is achieved. This not only preserves the overall structural integrity of large-scale damage but also takes into account the detail accuracy of small-scale damage, significantly improving the comprehensive recognition capability for multi-type and multi-scale composite damage scenes.

[0070] S400: After overlaying the pixel-level segmentation masks of each damage candidate region onto the image to be detected, this mask, along with the damage analysis prompt template, is input into a preset visual-language model to generate a structured damage assessment report that conforms to the damage analysis prompt template. The preset visual-language model (VLM) can be an LLaVA model or a Qwen2.5-VL model, or any pre-trained multimodal model with image-text reasoning capabilities.

[0071] Specifically, superimposing the pixel-level segmentation mask of each damage candidate region onto the image to be detected means superimposing the pixel-level segmentation mask output by the preset visual segmentation model onto the original image to be detected with different colors to generate a mask annotation image; different damage types are identified by different colors, and the mask number is also marked.

[0072] Specifically, the damage analysis prompt template includes output constraints on the preset visual language model in the following dimensions: damage type determination, location of the affected component and location, damage severity classification, damage area or length quantification, structural safety assessment, and generation of disposal suggestions. Damage type determination refers to identifying the damage type of the candidate area, such as cracks, fractures, fragmentation, deformation, collapse, and spalling; location of the affected component and location refers to determining the location of the damage on which component, such as the upper west side of pier No. 3; damage severity classification refers to assessing the severity of the damage according to preset levels such as minor, moderate, severe, and complete destruction; damage area or length quantification refers to estimating the physical area or length of the damage based on the number of pixels in the corresponding mask and the image scale; structural safety assessment refers to preliminarily assessing the impact on the overall structural safety of the bridge by combining the damage type, location, and severity; and disposal suggestion generation refers to generating disposal suggestions based on the assessment results, such as recommending immediate traffic closure, restricting heavy vehicle traffic, or arranging emergency repairs.

[0073] Preferably, the preset visual language model outputs the structured damage assessment report in a preset JSON format; the JSON format includes bridge identification, detection time, analysis list of each damaged area, overall safety level, and priority action recommendations.

[0074] Specifically as follows: { "damage_list": [ { "mask_id": "mask number (corresponding to the color in the image)", "damage_type": "Damage type (fracture / fragmentation / deformation / missing / peeling / cracks / water seepage / corrosion)", "component": "Name of the component", "location": "Specific location description on the component", "severity": "severity level (minor / moderate / severe / destructive)", "severity_reason": "Judgment basis", "estimated_area_ratio": "Estimated percentage of damaged area to component area (%)", "safety_impact": "Analysis of the impact on structural safety", "recommendation": "Suggested course of action" } ], "overall_safety_level": "Overall safety level (Safe / Needs Attention / Hazardous / Extremely Hazardous)", "overall_assessment": "Overall assessment description", "priority_actions": ["Action suggestions sorted by priority"] }

[0075] Furthermore, the structured damage assessment report in JSON format can be visualized and rendered to generate a comprehensive assessment report that includes image annotations, data statistics, and safety levels, and then pushed to the command and decision-making system, maintenance management system, or digital twin platform.

[0076] As described above, by overlaying pixel-level segmentation masks onto an image and then inputting them into a visual language model to generate a structured assessment report, a leap from pure visual perception to semantic understanding of damage has been achieved. This enables the system to automatically output structured information containing damage type, location, severity, safety impact, and handling recommendations, directly serving emergency decision-making and repair scheduling.

[0077] To better understand the damage assessment process of this invention, the following example is provided: Example: Rapid assessment of bridge damage after an earthquake Scenario Description: A 6.5-magnitude earthquake has occurred in a certain area, requiring a rapid damage assessment of a prestressed concrete continuous beam bridge in the region to determine whether emergency traffic can be opened.

[0078] Input data: 12 aerial images of the damaged bridge taken by drone, covering the entire bridge; the bridge's BIM model in IFC format; basic bridge information: a 5-span prestressed concrete continuous box girder bridge with a total length of 320 meters.

[0079] Damage assessment process: Image preprocessing: Due to the high concentration of dust in the air after the earthquake, the 12 UAV images were dehazed and stitched together to form a full-bridge orthophoto.

[0080] Visual cues generation: Render standard views of components such as main beams, piers, bridge decks, and supports from the BIM model, extract CLIP feature vectors to construct a component feature reference library; perform feature matching with the preprocessed images to determine the approximate location of each component in the image.

[0081] Text prompt generation: Select damage type descriptions related to earthquake damage from the damage semantic knowledge base, such as component fracture, concrete fragmentation, shear cracks, and bearing displacement, and combine them with bridge component terms such as prestressed box girder web, pier cap beam, and pot bearing, as well as the damage severity description, to generate text prompts in natural language form.

[0082] Geometric hint generation: Extract the structural topology of each component from the BIM model and generate the expected position mask of each component in the image coordinate system; detection found that the position of the third span panel deviated significantly from the expected position and was marked as a candidate abnormal area.

[0083] Adaptive iterative segmentation: Round 1: Using three types of fusion prompts for coarse segmentation, six suspected damaged areas were identified as candidate damaged areas.

[0084] Segmentation confidence assessment: If the segmentation confidence of the four candidate damaged regions is greater than 0.9, they are identified as damaged regions; if the segmentation confidence of the two candidate damaged regions is less than 0.7, they are identified as regions with ambiguous boundaries.

[0085] Round 2: Generate local secondary cueing for the two low-confidence regions and perform fine segmentation.

[0086] Segmentation confidence reassessment: The segmentation confidence of the two low-confidence regions is greater than 0.88, which meets the convergence condition.

[0087] VLM model semantic reasoning: The segmentation mask is superimposed on the preprocessed image and input into the VLM model along with the bridge information to generate a structured damage assessment report that conforms to the damage analysis prompt template. The structured damage assessment report is processed to output a comprehensive assessment report: Damage 1: Diagonal crack in the web of the main beam in the 3rd span - serious, affecting the bearing capacity of this span.

[0088] Damage 2: Partial cracking and collapse of the third span deck – destroyed, with an area of ​​approximately 2.8 square meters.

[0089] Damage 3: The upper part of the No. 4 pier has suffered concrete spalling and exposed reinforcing steel bars - moderate damage.

[0090] Damage 4: The displacement of support No. 3 exceeds the limit - serious.

[0091] Overall assessment: Dangerous. It is recommended to close the third crossroads and arrange emergency testing and repairs.

[0092] Results: The entire assessment process was completed within 2 hours of the damage, which significantly improved timeliness compared to the traditional manual inspection that takes 1-2 days.

[0093] Example 2 Embodiment 2 of the present invention provides a bridge damage assessment system based on a visual language model, such as... Figure 5 As shown, the system includes: The data acquisition and preprocessing module 100 is used to acquire the structural prior information and damaged images of the bridge to be detected, and to perform preprocessing operations on the damaged images to generate the images to be detected; the structural prior information includes bridge type, component information, material information and component spatial topology.

[0094] The multimodal prompt generation module 200 is used to match the image to be detected with prior structural information and a preset damage semantic knowledge base to generate multimodal prompts. The multimodal prompts include text prompts based on bridge type, component information, material information and damage semantic descriptions matched from the damage semantic knowledge base, as well as visual and geometric prompts based on the spatial topology of the components.

[0095] The adaptive iterative segmentation module 300 is used to freeze the multimodal cue and the input parameters of the image to be detected into a preset visual segmentation model, output each damage candidate region and its corresponding initial segmentation mask, and perform adaptive iterative segmentation on the damage candidate regions with segmentation confidence less than a preset threshold to obtain the pixel-level segmentation mask of each damage candidate region.

[0096] The assessment report generation module 400 is used to overlay the pixel-level segmentation mask of each damage candidate region onto the image to be detected, and input it together with the damage analysis prompt template into a preset visual language model to generate a structured damage assessment report that conforms to the damage analysis prompt template.

[0097] It should be noted that the information interaction and execution process between the above modules are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0098] Embodiments of the present invention also provide a non-transitory computer-readable storage medium that can be disposed in an electronic device to store at least one instruction or at least one program related to implementing a method in the method embodiments, wherein the at least one instruction or the at least one program is loaded and executed by the processor to implement the method provided in the above embodiments.

[0099] While specific embodiments of the invention have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of the invention. It should also be understood that various modifications can be made to the embodiments without departing from the scope and spirit of the invention. The scope of the invention is defined by the appended claims.

Claims

1. A bridge damage assessment method based on a visual segmentation model, characterized in that, The method includes the following steps: S100: Obtain the prior structural information and damaged image of the bridge to be detected, and perform preprocessing operations on the damaged image to generate the image to be detected; the prior structural information includes bridge type, component information, material information and component spatial topology; S200, the image to be detected is matched with the prior structural information and the preset damage semantic knowledge base respectively to generate multimodal prompts; The multimodal prompts include text prompts based on bridge type, component information, material information and damage semantic descriptions matched from a damage semantic knowledge base, as well as visual and geometric prompts based on component spatial topology; S300 takes the multimodal cue after fusion processing and the preset visual segmentation model with the input parameters of the image to be detected frozen, outputs each damage candidate region and its corresponding initial segmentation mask, and performs adaptive iterative segmentation on the damage candidate regions with segmentation confidence less than a preset threshold to obtain the pixel-level segmentation mask of each damage candidate region. S400: After superimposing the pixel-level segmentation mask of each damage candidate region onto the image to be detected, it is input into the preset visual language model along with the damage analysis prompt template to generate a structured damage assessment report that conforms to the damage analysis prompt template.

2. The bridge damage assessment method based on a visual segmentation model according to claim 1, characterized in that, The steps for preprocessing the damaged image are as follows: S101, perform image dehazing and image denoising processing on several damaged images of the bridge to be detected to obtain a preliminary restored image corresponding to each damaged image; the several damaged images include image sequences acquired from different perspectives; S102, perform contrast stretching and histogram equalization on each preliminary restored image to obtain the enhanced image corresponding to each damaged image; S103, extract feature points from each enhanced image, register feature points between different enhanced images, and fuse and stitch the registered enhanced images according to the overlapping areas to generate a panoramic image covering the bridge to be detected. S104, the panoramic image is uniformly scaled to a preset standard resolution size to complete the preprocessing operation.

3. The bridge damage assessment method based on a visual segmentation model according to claim 1, characterized in that, The visual cue is generated through the following steps: S201, when the source of the component spatial topology is a BIM model or design drawings, extract the standard appearance features of each component and build a component feature reference library. S203, extract the appearance features of the image to be detected and perform feature matching with the component feature reference library to locate the region of each component in the image to be detected, and use the located component regions as visual cues. S205, when the source of the component spatial topology is an image taken before the damage, the features of each component region are extracted from the image taken before the damage as a visual matching template. S207, perform feature matching between the appearance features of the image to be detected and the visual matching template, locate the region of each component in the image to be detected, and use the located component regions as visual cues.

4. The bridge damage assessment method based on a visual segmentation model according to claim 1, characterized in that, The geometric hints are generated using the following steps: S202, Based on the component spatial topology, construct a structural topology graph with each component as a node and the spatial positional relationship between components as an edge; the spatial positional relationship includes at least up / down, left / right, front / back and connection type. S204, Based on the structural topology diagram, generate the expected position mask of each component in the original image coordinate system of the image to be detected, and calculate the feature deviation between the expected position mask of each component and the corresponding region in the image to be detected. S206, regions with feature deviations exceeding a preset deviation threshold are marked as candidate abnormal regions, and the candidate abnormal regions are encoded in the form of spatial attention masks as geometric cues.

5. The bridge damage assessment method based on a visual segmentation model according to claim 1, characterized in that, The step of adaptively iteratively segmenting candidate damage regions with segmentation confidence scores less than a preset threshold to obtain pixel-level segmentation masks for each candidate damage region includes the following steps: S301, mark the damage candidate regions with segmentation confidence scores less than a preset threshold as low-confidence regions, and generate local secondary prompts for the low-confidence regions; the local secondary prompts include local visual context prompts, local text prompts, and local geometric context prompts corresponding to the low-confidence regions; S302, freeze the preset visual segmentation model of the image to be detected and the local secondary prompt input parameters, further segment the local range corresponding to the low confidence region, output the new damage candidate region and the corresponding new segmentation mask in the local range, and update the segmentation confidence corresponding to each new segmentation mask; S303, Repeat steps S301 to S302 until the convergence condition is met, and record the new segmentation mask generated in each iteration and the segmentation confidence of each new segmentation mask; S304: For the new segmentation masks of the same new damage candidate region in different iteration rounds, weighted fusion is performed with the corresponding segmentation confidence as the weight, and the fused mask is post-processed to finally output the pixel-level segmentation mask of each damage candidate region.

6. The bridge damage assessment method based on a visual segmentation model according to claim 1, characterized in that, The segmentation confidence score corresponding to the initial segmentation mask is calculated using the following formula: C = W1 × P + W2 × G + W3 × L, where C is the segmentation confidence corresponding to the initial segmentation mask, P is the pixel feature consistency within the initial segmentation mask, G is the boundary sharpness of the initial segmentation mask, L is the positional matching degree between the candidate damage region and the component spatial topology corresponding to the initial segmentation mask, and W1, W2, and W3 are all preset weights; wherein, the pixel feature consistency is the average similarity calculated based on the similarity between each pair of pixel feature vectors within the initial segmentation mask; the boundary sharpness is the normalized average gradient magnitude of the pixels at the boundary of the initial segmentation mask.

7. The bridge damage assessment method based on a visual segmentation model according to claim 1, characterized in that, The S300 procedure also includes cross-scale damage treatment steps, including the following: S310, Construct a multi-scale pyramid corresponding to the image to be detected; the multi-scale pyramid includes at least the original resolution scale, the first downsampling scale, and the second downsampling scale, and the scales decrease sequentially. S320, Input the images at each scale into the preset visual segmentation model with frozen parameters, and combine the multimodal prompts to generate the damage candidate region set and the corresponding intermediate segmentation mask set at each scale respectively; S330, the intermediate segmentation mask sets at each scale are uniformly mapped to the original image coordinate system of the image to be detected, and the intermediate segmentation masks that partially overlap or contain each other at different scales are fused to output the fused multi-scale damage segmentation mask set; the multi-scale damage segmentation mask set contains damage information at each scale in the multi-scale pyramid.

8. The bridge damage assessment method based on a visual segmentation model according to claim 1, characterized in that, The damage analysis prompt template includes output constraints on the preset visual language model in the following dimensions: damage type determination, location of the component and position, damage severity classification, damage area or length quantification, structural safety assessment and treatment suggestions generation; The preset visual language model outputs the structured damage assessment report in a preset JSON format; the JSON format includes bridge identification, detection time, analysis list of each damaged area, overall safety level, and priority action recommendations.

9. A bridge damage assessment system based on a visual language model, characterized in that, The system includes: The data acquisition and preprocessing module is used to acquire the structural prior information and damaged images of the bridge to be inspected, and to perform preprocessing operations on the damaged images to generate the images to be inspected; the structural prior information includes bridge type, component information, material information and component spatial topology; The multimodal prompt generation module is used to match the image to be detected with the prior structural information and the preset damage semantic knowledge base to generate multimodal prompts. The multimodal prompts include text prompts based on bridge type, component information, material information and damage semantic descriptions matched from the damage semantic knowledge base, as well as visual prompts and geometric prompts based on the spatial topology of the components. The adaptive iterative segmentation module is used to freeze the multimodal cue and the input parameters of the image to be detected into a preset visual segmentation model, output each damage candidate region and its corresponding initial segmentation mask, and perform adaptive iterative segmentation on the damage candidate regions with segmentation confidence less than a preset threshold to obtain the pixel-level segmentation mask of each damage candidate region. The assessment report generation module is used to overlay the pixel-level segmentation mask of each damage candidate region onto the image to be detected, and then input it together with the damage analysis prompt template into a preset visual language model to generate a structured damage assessment report that conforms to the damage analysis prompt template.

10. A non-transitory computer-readable storage medium storing at least one instruction or at least one program segment, characterized in that, The at least one instruction or the at least one program segment is loaded and executed by the processor to implement the bridge damage assessment method based on the visual segmentation model as described in any one of claims 1-8.

Images