Bridge diagnostic method and system fusing multi-modal perception and mechanics-enhanced atlas

Abstract

This invention discloses a bridge diagnosis method and system that integrates multimodal perception and mechanical enhancement graphs. The method includes: Step S1, acquiring bridge domain documents and defect inspection images to construct a domain multimodal fine-tuning dataset; Step S2, fine-tuning a pre-trained multimodal large model to obtain a defect perception model, and generating structured semantic query instructions from bridge defect images and user instructions; Step S3, constructing a knowledge graph with a three-layer structure including physical, specification, and instance layers, and annotating the mechanical transmission paths between components; Step S4, performing retrieval in the hierarchical knowledge graph using a mechanical topology weighting algorithm, integrating semantic scores and topological incremental scores diffused along the force transmission path to construct a retrieval context; Step S5, generating preliminary diagnostic opinions and comparing them with the specification layer clauses for consistency, executing self-correction logic, and outputting diagnostic results, thus completing the bridge diagnosis method and system that integrates multimodal perception and mechanical enhancement graphs.

Description

Technical Field

[0001] This invention relates to the field of bridge engineering technology, specifically to a bridge diagnosis method and system that integrates multimodal perception and mechanical enhancement mapping, belonging to the field of bridge structural health monitoring, disease detection and intelligent operation and maintenance technology. Background Technology

[0002] With the continuous increase in the number of highway, municipal, and railway bridges in my country, and the increasing service life of existing bridges, bridge structures are prone to various defects such as cracks, concrete spalling, steel corrosion, and component deformation under long-term traffic loads, environmental erosion, and material aging. If these defects are not detected and accurately assessed in a timely manner, they may lead to a decrease in the structural load-bearing capacity and even cause bridge safety accidents.

[0003] Currently, bridge defect detection and assessment mainly rely on a combination of manual inspection and structural health monitoring systems. Manual inspection typically involves professional inspectors observing, photographing, and recording the apparent defects of the bridge on-site, and making qualitative judgments based on experience. Structural health monitoring systems, on the other hand, continuously collect data on the bridge structure's response over a long period by deploying sensors for acceleration, strain, displacement, and temperature, and provide early warnings of its condition through threshold judgments or statistical analysis.

[0004] However, the aforementioned existing technologies still have many shortcomings in practical applications. First, due to the massive scale of multi-source heterogeneous data acquired by the monitoring system, existing methods mostly analyze single types of data, making it difficult to effectively integrate inspection images and text reports, thus easily creating information silos. Second, the occurrence and development of bridge defects are usually closely related to the mechanical transmission relationship between components, but existing diagnostic methods generally lack systematic modeling of the overall stress path of the bridge, making it difficult to provide an in-depth explanation of the causes of defects from the perspective of structural mechanics mechanisms. Third, although some intelligent methods introduce artificial intelligence models for image recognition or text generation, their generated results often lack solid engineering physical basis, and because they are not subject to mandatory logical verification with current national or industry technical specifications, the reliability of diagnostic recommendations is limited, making it difficult to directly use them in rigorous engineering decision-making processes. Summary of the Invention

[0005] To address the above technical problems, this invention provides a bridge diagnosis method and system that integrates multimodal sensing and mechanical enhancement mapping, the method comprising:

[0006] Step S1: Obtain bridge domain files and defect inspection images, and construct a domain multimodal fine-tuning dataset that includes image descriptions, visual question answering, and multiple-choice questions;

[0007] Step S2: Use the domain multimodal fine-tuning dataset to fine-tune the pre-trained multimodal large model to obtain a disease perception model. Combine the user instructions with the disease perception model to extract disease feature information from bridge disease images, including disease component type, disease morphology, and disease severity, and combine them into structured semantic query instructions. Step S3: Extract entities and relationships based on unstructured domain documents to construct a hierarchical knowledge graph containing a physical layer, a specification layer, and an instance layer, where the physical layer contains the mechanical transmission paths between components.

[0008] Step S4: Based on the structured semantic query instruction, a mechanical topology weighted algorithm is used to search in the hierarchical knowledge graph. The relevance of the search results of the target node and its neighboring nodes is enhanced by utilizing the force transmission relationship between components. The final source tracing score of the candidate entity is calculated, and a diagnostic search context is constructed.

[0009] Step S5: Generate a preliminary diagnostic opinion based on the diagnostic retrieval context, compare the preliminary diagnostic opinion with the digital text in the knowledge graph specification layer for consistency and execute self-correction logic, and output the final diagnostic report.

[0010] Preferably, in step S2, the instruction fine-tuning adopts a low-rank adaptive fine-tuning architecture, which injects a low-rank matrix adapter into the attention layer of the pre-trained multimodal large model to achieve semantic understanding of the visual features of bridge component defects while freezing the pre-trained weights.

[0011] Preferably, in step S3, the construction of the hierarchical knowledge graph includes an entity extraction unit and a relation extraction unit.

[0012] The entity extraction unit is used to identify entities such as physical components, disease mechanisms, materials, technical indicators, and maintenance methods from text, and add hierarchical tags to the entity descriptions;

[0013] The relationship extraction unit is used to identify the logical associations between entities. If energy transfer or structural support is involved, it is marked as a mechanical transfer relationship in the relationship keywords.

[0014] Preferably, in step S4, the mechanical topology weighting algorithm includes semantic initial screening, topology diffusion, and score fusion;

[0015] The initial semantic screening is used to calculate the vector similarity between the query command and the graph entity to obtain an initial semantic score;

[0016] The topology diffusion is used to identify upstream and downstream nodes associated with the seed node in the physical layer through mechanical transmission, and to calculate the topology increment score along the force transmission path; wherein, the topology increment score is calculated using an energy diffusion model, and its diffusion weight decreases as the physical distance or logical hop count between the search node and the seed node on the physical layer topology path increases.

[0017] The score fusion is used to linearly or nonlinearly fuse the initial semantic score and the topological incremental score according to a preset weight to obtain the final source tracing score, which is used to characterize the logical correlation of the diseased entity on the mechanical force transmission path.

[0018] Preferably, in step S5, the self-correction logic includes conflict detection and instruction rewriting;

[0019] The conflict detection is used to compare the measured values ​​in the preliminary diagnostic opinion with the mandatory thresholds of the normative clauses to determine whether there is a logical violation.

[0020] The instruction rewriting is used to extract the original text of the specification clause as a constraint when a logical violation is determined, and force the model to regenerate a diagnostic response that meets the specification requirements.

[0021] This invention also provides a bridge diagnostic system that integrates multimodal sensing and mechanical enhancement mapping to implement the above method. The system includes:

[0022] The multi-task dataset construction module is used to acquire bridge defect inspection data and perform multi-task feature annotation to build a domain-specific multimodal fine-tuning dataset.

[0023] The disease perception fine-tuning module is used to fine-tune the pre-trained multimodal large model with instructions to obtain the disease perception model and generate structured semantic query instructions.

[0024] The hierarchical knowledge graph module is used to construct a hierarchical knowledge graph containing a physical layer, a specification layer, and an instance layer, and to annotate the mechanical transmission path;

[0025] The mechanical topology retrieval module is used to perform retrieval based on the mechanical topology weighted algorithm, calculate the topology incremental score, calculate the final source tracing score of candidate entities, and construct a diagnostic retrieval context.

[0026] The standard anchoring and correction module is used to compare the preliminary diagnostic opinions with the standard layer provisions and execute self-correction logic to output the final diagnostic report.

[0027] Preferably, the topology incremental score is calculated using an energy diffusion model, and its diffusion weight decreases as the physical distance or logical hop count between the retrieval node and the seed node on the physical layer topology path increases.

[0028] Preferably, in the specification anchoring correction module, the digital provisions stored in the specification layer include explicit provisions in industry standards regarding crack limits, strength assessment, and reinforcement requirements.

[0029] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0030] To address the shortcomings of current bridge defect diagnosis methods, such as insufficient accuracy in complex mechanical environments, lack of physical logic tracing, and difficulty in aligning with industry standards, this invention provides a bridge diagnosis method and system that integrates multimodal perception and enhanced mechanical graphs. Through multi-task instruction fine-tuning, the model's ability to perceive sub-millimeter-level defect features is improved, and a hierarchical knowledge graph is used to construct deep connections between physics, knowledge, and data. The mechanical topology weighted algorithm introduced in this invention enables the system to understand the force transmission logic between components, achieving a leap from single-phenomenon recognition to deep mechanism deduction, significantly enhancing the scientific rigor of the diagnostic results.

[0031] Specifically, this invention designs a bridge diagnosis method and system that integrates multimodal perception and mechanical enhancement graph. The system extracts structured semantic instructions using a defect perception model, and then performs energy diffusion calculations along the force transmission path in a hierarchical knowledge graph using a mechanical topology algorithm. This enhances the retrieval weights of mechanically related nodes and generates a retrieval context with physical logic. The preliminary diagnostic opinions are then compared in real-time with digital provisions in the knowledge graph's specification layer. Once the measured indicators exceed a threshold, a conflict closure loop is triggered, using the specification provisions as exclusive constraint instructions to drive the model's self-correction and rewriting, ensuring the diagnostic conclusions are rigorous and compliant. Attached Figure Description

[0032] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is a schematic diagram of a bridge diagnosis method and system that integrates multimodal sensing and mechanical enhancement mapping according to an embodiment of the present invention;

[0034] Figure 2 This is a schematic diagram of the multi-task dataset construction module according to an embodiment of the present invention;

[0035] Figure 3 This is a flowchart illustrating the construction of a hierarchical knowledge graph according to an embodiment of the present invention;

[0036] Figure 4 This is a schematic diagram of the hierarchical knowledge graph module in an embodiment of the present invention;

[0037] Figure 5 This is a schematic diagram of the mechanical topology retrieval module according to an embodiment of the present invention;

[0038] Figure 6 This is a schematic diagram of the standard anchoring correction module according to an embodiment of the present invention. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0040] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are used only to distinguish one concept from another. Depending on the context, the words “if,” “when,” or “in response to determination” as used herein may be interpreted as “when…”, “when…”, or “in response to determination.”

[0041] As used herein, the terms "at least one," "multiple," "each," "any," etc., include "at least one," "two," or more, and "multiple" includes "two" or more. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0042] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.

[0043] Multimodal Large Language Model (MLLM) refers to a deep learning model that can process and understand information in multiple modalities (such as images and text). In this application, it mainly refers to a disease perception model that has been fine-tuned.

[0044] A knowledge graph is a semantic network that reveals the relationships between entities. In this application, it specifically refers to a bridge domain graph that includes a physical layer, a specification layer, and an instance layer.

[0045] As bridge infrastructure ages, traditional inspection methods face challenges such as insufficient perception accuracy and a lack of standardized guidelines for diagnostic opinions. Existing AI-based recognition solutions often focus on single image segmentation, and large models frequently exhibit engineering illusions when generating diagnostic reports, meaning their recommendations do not conform to stress logic or industry standards. To address these issues, this paper proposes a bridge diagnostic method and system that integrates multimodal perception and enhanced mechanical mapping.

[0046] Example 1

[0047] Reference Figure 1 This application provides an intelligent diagnosis method for bridge defects that integrates multimodal perception and mechanical enhancement mapping. The method includes:

[0048] Step S1: Obtain bridge domain files and defect inspection images, and construct a domain multimodal fine-tuning dataset that includes image descriptions, visual question answering, and multiple-choice questions.

[0049] Reference Figure 2 In the multi-task dataset construction module, the original inspection reports and design documents are first screened to extract typical defect images. For each defect image, manual annotations are performed based on the defect descriptions in the report, including descriptions of component type, defect type, and defect severity. The annotations cover no fewer than 20 typical components of the bridge's superstructure, substructure, and deck system. Subsequently, the manually annotated data is converted into a multi-task training format using a preset template to obtain the domain-specific multimodal fine-tuning dataset.

[0050] In this embodiment, the multi-task includes a description task (Caption), a visual question answering task (VQA), and a selection task (MCQ). The description task uses a disease semantic description template to combine disease attributes, component locations, and geometric dimensions into a structured technical commentary to cultivate the model's language generation ability. The visual question answering task generates heuristic questions based on labeled values ​​to improve the model's ability to locate and judge key features. The selection task designs multiple-choice questions with distractors to force the model to make subtle distinctions between similar disease features (such as shrinkage cracks and stress cracks).

[0051] In this embodiment, the task distribution ratio is set to Caption:VQA:MCQ = 4:4:2, and the average length of the text instructions is approximately 150 characters. This step ensures the accuracy of the underlying data through manual logic, establishing a visual-semantic alignment benchmark for the model.

[0052] Step S2: Use the domain multimodal fine-tuning dataset to fine-tune the pre-trained multimodal large model to obtain the disease perception model. Combine the user instructions with the disease perception model to extract disease feature information from bridge disease images, including disease component type, disease morphology and disease degree, and combine them into structured semantic query instructions.

[0053] The disease perception fine-tuning module employs LoRA technology for efficient parameter fine-tuning of the Qwen2.5-VL-7B-Instruct pre-trained multimodal large model. Without changing the original pre-training weights, low-rank matrix operators are inserted into the model's language projection layer and multi-head attention layer. Key parameter settings include: rank... The scaling factor is The target layer involves q_proj, v_proj, k_proj, etc. In the training process, the learning rate is set to 1e-4 with cosine decay, the batch size is 16, the training epochs are 8, the optimizer is AdamW, and QLoRA with quantization level 4 is enabled. After training, testing is performed using an expert annotation set not involved in the training, and quantitative evaluation is performed by calculating ROUGE-L and BLUE-4 scores (metrics used to evaluate text generation quality).

[0054] Step S3: Extract entities and relationships from unstructured domain documents and construct a hierarchical knowledge graph containing a physical layer, a specification layer, and an instance layer, where the physical layer contains the mechanical transmission paths between components.

[0055] Reference Figure 3 The hierarchical knowledge graph is constructed based on preset system prompts, driving a multimodal large model to automatically parse unstructured domain documents. In this embodiment, the domain documents cover unstructured data throughout the entire lifecycle of a bridge, specifically including bridge inspection reports, design drawings, maintenance technical specifications, reinforcement design documents, and related administrative regulations.

[0056] The hierarchical knowledge graph construction process executes an entity extraction unit. In this embodiment, the output format of the multimodal large model is forcibly constrained by the Prompt instruction set, requiring it to strictly identify and output entity types (entity_type) with predefined ontological structures. The entity types include: Components representing physical components of the physical layer (such as main beams, supports, piers, and other actual structural units); Materials representing material properties of the physical layer (such as C50 concrete, prestressed steel strands, etc.); Defects representing the defect mechanisms of the specification layer or instance layer (such as stress cracks, alkali-aggregate reaction, steel corrosion, etc.); Parameters representing technical indicators and measured values ​​of the instance layer (such as measured joint width of 0.25mm, span length, etc.); and Activities representing maintenance activities and standard clauses of the specification layer (such as cantilever casting, periodic inspection) and Standard_Org (such as JTG specification clauses, maintenance center). In this embodiment, in order to ensure the systematic nature of the knowledge system, the description fields of the entities are all assigned specific prefix hierarchical labels, which are selected from "physical layer", "specification layer" or "instance layer".

[0057] The hierarchical knowledge graph construction process executes a relationship extraction unit. In this embodiment, the system utilizes the `relationship_keywords` field in the prompt words to perform multi-dimensional logical classification of the extracted entity relationships. For physical layer associations involving energy transfer, support, physical constraints, or force causality, the keyword sequence must include "mechanical relationship," "mechanical transfer," or "structural support" to describe the physical chain of load transfer from the superstructure to the substructure via supports. For normative layer associations involving administrative constraints, implementation targets, industry standard regulations, or reinforcement and treatment schemes, the keyword sequence includes "management relationship," "treatment scheme," or "standard constraint" to clarify the normative basis for each disease treatment behavior. For instance layer associations involving on-site disease characterization, component feature description, or measurement value attribution, the keywords are defined as "semantic relationship," "feature description," or "data association," thereby anchoring specific detection data to specific physical components.

[0058] Reference Figure 4The hierarchical knowledge graph achieves structured integration of bridge defect diagnosis knowledge through multi-dimensional decoupling and cross-layer association between the physical layer, specification layer, and instance layer. The physical layer (structural force path) describes the physical topological connections and load transfer mechanisms between bridge components. For example, it abstracts the bridge structure as a mechanical transmission chain composed of "bridge deck system - main beam - support - pier." By defining semantic relationships such as "mechanical transmission" or "structural support," it maps energy flow in physical space, providing a structural topological foundation for mechanical attribution of defects. The specification layer (standards and rules) stores digitized bridge industry technical standards and judgment logic. Through associative edges, it maps physical layer entities (such as "main beam" and "support") to corresponding technical indicators (such as "crack width limit" and "displacement limit"). This layer transforms discrete text specifications into hard constraint nodes in the graph, supporting the system to perform automated and compliant defect level determination based on measured values. The instance layer (historical engineering cases) stores multi-source maintenance data and expert treatment experience, including historical disease characteristics, measured parameters and treatment conclusions, and performs cross-layer anchoring with the physical layer and the specification layer.

[0059] Step S4: Based on the structured semantic query instruction, a mechanical topology weighted algorithm is used to search in the hierarchical knowledge graph. The relevance of the search results of the target node and its neighboring nodes is enhanced by utilizing the force transmission relationship between components. The final source tracing score of the candidate entity is calculated, and a diagnostic search context is constructed.

[0060] Reference Figure 5 The aforementioned mechanical topology weighted algorithm performs semantic initial screening. In this embodiment, the semantic initial screening utilizes a preset vector encoder to convert the structured semantic query command output by the disease perception fine-tuning model into a high-dimensional semantic feature vector. It then calculates the similarity between the query vector and the entity vectors in the hierarchical knowledge graph in the vector database, obtains the Top-N seed entities, and calculates and records the initial semantic score of each seed entity. The preferred value for N is between 5 and 10.

[0061] The aforementioned mechanical topology weighted algorithm further performs neighbor traversal and association detection based on the seed entity. In this embodiment, the system calls the graph storage engine to query the attributes of the first-order associated neighbor nodes and corresponding associated edges of the seed entity, and parses the attribute dictionary of the associated edges to determine whether the associated edges contain mechanical attribute identifiers. The mechanical attribute identifiers are used to characterize the physical force transmission relationship between bridge components, specifically including physical logic such as load transmission, structural support, or energy dissipation. For example, when the seed node is "pier crack," the system automatically identifies the "support stress anomaly" node associated with it above through the mechanical transmission path, thereby logically associating the apparent crack with the deep stress structure anomaly in the diagnostic report.

[0062] For paths identified as containing the mechanical property identifier, the algorithm performs topology diffusion. In this embodiment, neighboring nodes receive incremental topology scores from the seed entity. The calculation method is shown in the following formula:

[0063]

[0064] Among them, the This represents the topological diffusion attenuation factor, used to quantify the logical decay of mechanical correlation as the topological path distance increases. In this embodiment, the... The preferred value range is 0.4 to 0.7, with a typical value of 0.5. For paths determined not to contain the mechanical attribute identifier, the algorithm executes the logic of preserving the original semantic score, that is, the neighboring nodes do not generate topological incremental scores, but only retain their original semantic relevance scores. If a node is associated with a seed node through multiple mechanical paths, its topological incremental score is the weighted sum of the scores of each path.

[0065] The described mechanical topology weighting algorithm performs weighted calculation of multidimensional scores through score fusion. In this embodiment, the final source tracing score is calculated for all activated candidate entities in the graph. The calculation method is shown in the following formula:

[0066]

[0067] Among them, the Represents semantic weight coefficients, the Let represent the topological weight coefficients, and the sum of their coefficients satisfies ... In this embodiment, the... The preferred value is 0.6, the The preferred value is 0.4. For neighboring nodes that are associated through a mechanical path but were not originally in the seed entity list, their initial semantic score is... The default value is 0.

[0068] The mechanical topology weighted algorithm is based on the final source tracing score. A global descending order is performed on all candidate entities involved, and the top K high-scoring entities and their associated path relationships are selected. In this embodiment, the preferred value of K is 5. The system extracts the Top-K entities and their corresponding physical layer mechanical properties, specification layer decision limits, and instance layer historical maintenance records, encapsulates them into structured text, and outputs the knowledge tracing and retrieval context for large model inference.

[0069] Step S5: Generate a preliminary diagnostic opinion based on the diagnostic retrieval context, compare the preliminary diagnostic opinion with the digital text in the knowledge graph specification layer for consistency and execute self-correction logic, and output the final diagnostic report.

[0070] Reference Figure 6 The self-correction logic uses a closed-loop mechanism of conflict detection and instruction rewriting to correct the initial conclusions generated by the large model for compliance. The execution process first obtains a preliminary diagnostic opinion generated based on the diagnostic retrieval context, then proceeds to semantic parsing and comparison, verifying the logical consistency between the text content and the digitized entries in the knowledge graph specification layer. If the determination result is no conflict, the final diagnostic report is directly output; if a logical violation is found, the system immediately suspends the current output stream and triggers the self-correction logic to enter the instruction rewriting stage. The rewritten response needs to return to the conflict determination node for a second check, forming a loop verification until the verification is passed or the preset retry limit is reached. If multiple rewrites still result in a violation, manual intervention logic is triggered.

[0071] The conflict detection is used to compare the measured values ​​in the preliminary diagnostic opinion with the mandatory thresholds of the specification layer to determine whether there are any logical violations. The system reverse-parses the entity parameters and their corresponding quantified values ​​from the preliminary diagnostic opinion text and performs consistency verification with the corresponding mandatory limit values ​​in the specification layer of the hierarchical knowledge graph constructed in step S3. The comparison logic covers the logical rationality verification of numerical limit judgment and qualitative level judgment. If the comparison result shows that the measured value violates the deterministic requirements of the specification clause, the system determines that an engineering illusion has occurred and forcibly triggers the subsequent rewriting process.

[0072] The instruction rewriting process is used to extract the original text of the specification clause as an exclusive constraint and inject it into the correction instruction when a logical violation is detected, forcing the model to regenerate a compliant response. This rewriting process requires the large model to execute in a restricted context and lower the temperature coefficient to enhance the determinism of the response. The large model must use the specification entries in the correction instruction as a priori logical benchmark to revise the disease classification, causal analysis, and maintenance recommendations in the preliminary diagnosis, ensuring that the diagnostic results are strictly consistent with industry standards in terms of technical indicators. If the rewritten response passes the second check, a final report is output. If multiple attempts fail to eliminate the conflict, the system will perform mandatory intervention and prompt manual review.

[0073] Example 2

[0074] A bridge diagnostic system integrating multimodal sensing and mechanical enhancement mapping, the system comprising:

[0075] The multi-task dataset construction module is used to acquire bridge defect inspection data and perform multi-task feature annotation to build a domain-specific multimodal fine-tuning dataset.

[0076] Reference Figure 2 In the multi-task dataset construction module, the original inspection reports and design documents are first screened to extract typical defect images. For each defect image, manual annotations are performed based on the defect descriptions in the report, including descriptions of component type, defect type, and defect severity. The annotations cover no fewer than 20 typical components of the bridge's superstructure, substructure, and deck system. Subsequently, the manually annotated data is converted into a multi-task training format using a preset template to obtain the domain-specific multimodal fine-tuning dataset.

[0077] In this embodiment, the multi-task includes a description task (Caption), a visual question answering task (VQA), and a selection task (MCQ). The description task uses a disease semantic description template to combine disease attributes, component locations, and geometric dimensions into a structured technical commentary to cultivate the model's language generation ability. The visual question answering task generates heuristic questions based on labeled values ​​to improve the model's ability to locate and judge key features. The selection task designs multiple-choice questions with distractors to force the model to make subtle distinctions between similar disease features (such as shrinkage cracks and stress cracks).

[0078] In this embodiment, the task distribution ratio is set to Caption:VQA:MCQ = 4:4:2, and the average length of the text instructions is approximately 150 characters. This step ensures the accuracy of the underlying data through manual logic, establishing a visual-semantic alignment benchmark for the model.

[0079] The disease perception fine-tuning module is used to fine-tune the pre-trained multimodal large model to obtain the disease perception model and generate structured semantic query instructions.

[0080] The disease perception fine-tuning module employs LoRA technology for efficient parameter fine-tuning of the Qwen2.5-VL-7B-Instruct pre-trained multimodal large model. Without altering the original pre-training weights, low-rank matrix operators are inserted into the model's language projection layer and multi-head attention layer. Key parameter settings include: rank of 8, scaling factor of 16, and target layers involving q_proj, v_proj, and k_proj. During training, the learning rate is set to 1e-4 with cosine decay, the batch size is 16, the training epochs are 8, the optimizer is AdamW, and QLoRA with a quantization level of 4 is enabled. After training, testing is performed using an expert annotation set not involved in training, and quantitative evaluation is performed by calculating ROUGE-L and BLUE-4 scores (indicators for evaluating text generation quality).

[0081] The hierarchical knowledge graph module is used to construct a hierarchical knowledge graph containing a physical layer, a specification layer, and an instance layer, and to annotate the mechanical transmission path.

[0082] Reference Figure 3 The hierarchical knowledge graph is constructed based on preset system prompts, driving a multimodal large model to automatically parse unstructured domain documents. In this embodiment, the domain documents cover unstructured data throughout the entire lifecycle of a bridge, specifically including bridge inspection reports, design drawings, maintenance technical specifications, reinforcement design documents, and related administrative regulations.

[0083] The hierarchical knowledge graph construction process executes an entity extraction unit. In this embodiment, the output format of the multimodal large model is forcibly constrained by the Prompt instruction set, requiring it to strictly identify and output entity types (entity_type) with predefined ontological structures. The entity types include: Components representing physical components of the physical layer (such as main beams, supports, piers, and other actual structural units); Materials representing material properties of the physical layer (such as C50 concrete, prestressed steel strands, etc.); Defects representing the defect mechanisms of the specification layer or instance layer (such as stress cracks, alkali-aggregate reaction, steel corrosion, etc.); Parameters representing technical indicators and measured values ​​of the instance layer (such as measured joint width of 0.25mm, span length, etc.); and Activities representing maintenance activities and standard clauses of the specification layer (such as cantilever casting, periodic inspection) and Standard_Org (such as JTG specification clauses, maintenance center). In this embodiment, in order to ensure the systematic nature of the knowledge system, the description fields of the entities are all assigned specific prefix hierarchical labels, which are selected from "physical layer", "specification layer" or "instance layer".

[0084] Reference Figure 4The hierarchical knowledge graph achieves structured integration of bridge defect diagnosis knowledge through multi-dimensional decoupling and cross-layer association between the physical layer, specification layer, and instance layer. The physical layer (structural force path) describes the physical topological connections and load transfer mechanisms between bridge components. For example, it abstracts the bridge structure as a mechanical transmission chain composed of "bridge deck system - main beam - support - pier." By defining semantic relationships such as "mechanical transmission" or "structural support," it maps energy flow in physical space, providing a structural topological foundation for mechanical attribution of defects. The specification layer (standards and rules) stores digitized bridge industry technical standards and judgment logic. Through associative edges, it maps physical layer entities (such as "main beam" and "support") to corresponding technical indicators (such as "crack width limit" and "displacement limit"). This layer transforms discrete text specifications into hard constraint nodes in the graph, supporting the system to perform automated and compliant defect level determination based on measured values. The instance layer (historical engineering cases) stores multi-source maintenance data and expert treatment experience, including historical disease characteristics, measured parameters and treatment conclusions, and performs cross-layer anchoring with the physical layer and the specification layer.

[0085] The mechanical topology retrieval module is used to perform retrieval based on the mechanical topology weighted algorithm, calculate the topology incremental score, calculate the final source tracing score of candidate entities, and construct a diagnostic retrieval context.

[0086] Reference Figure 5 The aforementioned mechanical topology weighted algorithm performs semantic initial screening. In this embodiment, the semantic initial screening utilizes a preset vector encoder to convert the structured semantic query command output by the disease perception fine-tuning model into a high-dimensional semantic feature vector. It then calculates the similarity between the query vector and the entity vectors in the hierarchical knowledge graph in the vector database, obtains the Top-N seed entities, and calculates and records the initial semantic score of each seed entity. The preferred value for N is between 5 and 10.

[0087] The aforementioned mechanical topology weighted algorithm further performs neighbor traversal and association detection based on the seed entity. In this embodiment, the system calls the graph storage engine to query the attributes of the first-order associated neighbor nodes and corresponding associated edges of the seed entity, and parses the attribute dictionary of the associated edges to determine whether the associated edges contain mechanical attribute identifiers. The mechanical attribute identifiers are used to characterize the physical force transmission relationship between bridge components, specifically including physical logic such as load transmission, structural support, or energy dissipation. For example, when the seed node is "pier crack," the system automatically identifies the "support stress anomaly" node associated with it above through the mechanical transmission path, thereby logically associating the apparent crack with the deep stress structure anomaly in the diagnostic report.

[0088] For paths identified as containing the mechanical property identifier, the algorithm performs topology diffusion. In this embodiment, neighboring nodes receive incremental topology scores from the seed entity. The calculation method is shown in the following formula:

[0089]

[0090] Among them, the This represents the topological diffusion attenuation factor, used to quantify the logical decay of mechanical correlation as the topological path distance increases. In this embodiment, the... The preferred value range is 0.4 to 0.7, with a typical value of 0.5. For paths determined not to contain the mechanical attribute identifier, the algorithm executes the logic of preserving the original semantic score, that is, the neighboring nodes do not generate topological incremental scores, but only retain their original semantic relevance scores. If a node is associated with a seed node through multiple mechanical paths, its topological incremental score is the weighted sum of the scores of each path.

[0091] The described mechanical topology weighting algorithm performs weighted calculation of multidimensional scores through score fusion. In this embodiment, the final source tracing score is calculated for all activated candidate entities in the graph. The calculation method is shown in the following formula:

[0092]

[0093] Among them, the Represents semantic weight coefficients, the Let represent the topological weight coefficients, and the sum of their coefficients satisfies ... In this embodiment, the... The preferred value is 0.6, the The preferred value is 0.4. For neighboring nodes that are associated through a mechanical path but were not originally in the seed entity list, their initial semantic score is... The default value is 0.

[0094] The mechanical topology weighted algorithm is based on the final source tracing score. A global descending order is performed on all candidate entities involved, and the top K high-scoring entities and their associated path relationships are selected. In this embodiment, the preferred value of K is 5. The system extracts the Top-K entities and their corresponding physical layer mechanical properties, specification layer decision limits, and instance layer historical maintenance records, encapsulates them into structured text, and outputs the knowledge tracing and retrieval context for large model inference.

[0095] The standard anchoring and correction module is used to compare the preliminary diagnostic opinions with the standard layer provisions and execute self-correction logic to output the final diagnostic report.

[0096] Reference Figure 6 The self-correction logic uses a closed-loop mechanism of conflict detection and instruction rewriting to correct the initial conclusions generated by the large model for compliance. The execution process first obtains a preliminary diagnostic opinion generated based on the diagnostic retrieval context, then proceeds to semantic parsing and comparison, verifying the logical consistency between the text content and the digitized entries in the knowledge graph specification layer. If the determination result is no conflict, the final diagnostic report is directly output; if a logical violation is found, the system immediately suspends the current output stream and triggers the self-correction logic to enter the instruction rewriting stage. The rewritten response needs to return to the conflict determination node for a second check, forming a loop verification until the verification is passed or the preset retry limit is reached. If multiple rewrites still result in a violation, manual intervention logic is triggered.

[0097] The conflict detection is used to compare the measured values ​​in the preliminary diagnostic opinion with the mandatory thresholds of the specification layer to determine whether there are any logical violations. The system reverse-parses the entity parameters and their corresponding quantified values ​​from the preliminary diagnostic opinion text and performs consistency verification with the corresponding mandatory limit values ​​in the specification layer of the hierarchical knowledge graph constructed in step S3. The comparison logic covers the logical rationality verification of numerical limit judgment and qualitative level judgment. If the comparison result shows that the measured value violates the deterministic requirements of the specification clause, the system determines that an engineering illusion has occurred and forcibly triggers the subsequent rewriting process.

[0098] The instruction rewriting process is used to extract the original text of the specification clause as an exclusive constraint and inject it into the correction instruction when a logical violation is detected, forcing the model to regenerate a compliant response. This rewriting process requires the large model to execute in a restricted context and lower the temperature coefficient to enhance the determinism of the response. The large model must use the specification entries in the correction instruction as a priori logical benchmark to revise the disease classification, causal analysis, and maintenance recommendations in the preliminary diagnosis, ensuring that the diagnostic results are strictly consistent with industry standards in terms of technical indicators. If the rewritten response passes the second check, a final report is output. If multiple attempts fail to eliminate the conflict, the system will perform mandatory intervention and prompt manual review.

[0099] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0100] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A bridge diagnostic method integrating multimodal sensing and mechanical enhancement mapping, characterized in that, The method includes: Step S1: Obtain bridge domain files and defect inspection images, and construct a domain multimodal fine-tuning dataset that includes image descriptions, visual question answering, and multiple-choice questions; Step S2: Use the domain multimodal fine-tuning dataset to fine-tune the pre-trained multimodal large model to obtain the disease perception model. Combine the user instructions with the disease perception model to extract disease feature information from bridge disease images, including disease component type, disease morphology and disease degree, and combine them into structured semantic query instructions. Step S3: Extract entities and relationships from unstructured domain documents and construct a hierarchical knowledge graph containing a physical layer, a specification layer, and an instance layer, where the physical layer contains the mechanical transmission paths between components; Step S4: Based on the structured semantic query instruction, a mechanical topology weighted algorithm is used to search in the hierarchical knowledge graph. The relevance of the search results of the target node and its neighboring nodes is enhanced by utilizing the force transmission relationship between components. The final source tracing score of the candidate entity is calculated, and a diagnostic search context is constructed. Step S5: Generate a preliminary diagnostic opinion based on the diagnostic retrieval context, compare the preliminary diagnostic opinion with the digital text in the knowledge graph specification layer for consistency and execute self-correction logic, and output the final diagnostic report.

2. The bridge diagnosis method integrating multimodal sensing and mechanical enhancement mapping according to claim 1, characterized in that, In step S2, the instruction fine-tuning adopts a low-rank adaptive fine-tuning architecture. By injecting a low-rank matrix adapter into the attention layer of the pre-trained multimodal large model, semantic understanding of the visual features of bridge component defects is achieved while freezing the pre-trained weights.

3. The bridge diagnosis method integrating multimodal sensing and mechanical enhancement mapping according to claim 1, characterized in that, In step S3, the construction process of the hierarchical knowledge graph includes an entity extraction unit and a relation extraction unit. The entity extraction unit is used to identify entities such as physical components, disease mechanisms, materials, technical indicators, and maintenance methods from text, and add hierarchical tags to the entity descriptions; The relationship extraction unit is used to identify the logical associations between entities. If energy transfer or structural support is involved, it is marked as a mechanical transfer relationship in the relationship keywords.

4. The bridge diagnosis method integrating multimodal sensing and mechanical enhancement mapping according to claim 1, characterized in that, In step S4, the mechanical topology weighting algorithm includes semantic screening, topology diffusion, and score fusion: The initial semantic screening is used to calculate the vector similarity between the query command and the graph entity to obtain an initial semantic score; The topology diffusion is used to identify upstream and downstream nodes associated with the seed node in the physical layer through mechanical transmission, and to calculate the topology increment score along the force transmission path; wherein, the topology increment score is calculated using an energy diffusion model, and its diffusion weight decreases as the physical distance or logical hop count between the search node and the seed node on the physical layer topology path increases. The score fusion is used to linearly or nonlinearly fuse the initial semantic score and the topological incremental score according to a preset weight to obtain the final source tracing score, which is used to characterize the logical correlation of the diseased entity on the mechanical force transmission path.

5. The bridge diagnosis method integrating multimodal sensing and mechanical enhancement mapping according to claim 1, characterized in that, In step S5, the self-correction logic includes conflict detection and instruction rewriting: The conflict detection is used to compare the measured values ​​in the preliminary diagnostic opinion with the mandatory thresholds of the normative clauses to determine whether there is a logical violation. The instruction rewriting is used to extract the original text of the specification clause as a constraint when a logical violation is determined, and force the model to regenerate a diagnostic response that meets the specification requirements.

6. A bridge diagnostic system integrating multimodal sensing and mechanical enhancement mapping, said system being used to implement the method described in any one of claims 1-5, characterized in that, The system includes: The multi-task dataset construction module is used to acquire bridge defect inspection data and perform multi-task feature annotation to build a domain-specific multimodal fine-tuning dataset. The disease perception fine-tuning module is used to fine-tune the pre-trained multimodal large model with instructions to obtain the disease perception model and generate structured semantic query instructions. The hierarchical knowledge graph module is used to construct a hierarchical knowledge graph containing a physical layer, a specification layer, and an instance layer, and to annotate the mechanical transmission path; The mechanical topology retrieval module is used to perform retrieval based on the mechanical topology weighted algorithm, calculate the topology incremental score, calculate the final source tracing score of candidate entities, and construct a diagnostic retrieval context. The standard anchoring and correction module is used to compare the preliminary diagnostic opinions with the standard layer provisions and execute self-correction logic to output the final diagnostic report.

7. The bridge diagnostic system integrating multimodal sensing and mechanical enhancement mapping according to claim 6, characterized in that, The topology incremental score is calculated using an energy diffusion model, and its diffusion weight decreases as the physical distance or logical hop count between the retrieval node and the seed node on the physical layer topology path increases.

8. The bridge diagnostic system integrating multimodal sensing and mechanical enhancement mapping according to claim 6, characterized in that, In the aforementioned standard anchoring correction module, the digital provisions stored in the standard layer include explicit provisions in industry standards regarding crack limits, strength assessment, and reinforcement requirements.

Images