Bridge structure health diagnosis agent construction method based on thinking chain and domain tool set

Abstract

The invention provides a bridge structure health diagnosis agent construction method based on a thinking chain and a domain tool set. According to the method, a language model with a causal reasoning capability is introduced, and a complete task chain process is constructed by combining target detection, segmentation and geometric analysis models for image processing, so that automation and intelligence from image perception to structure diagnosis are realized. The method supports multi-image input and multi-task cooperative processing, has good generalization ability and interpretability, and can effectively improve the efficiency and accuracy of structural health assessment.

Description

Technical Field

[0001] This invention belongs to the field of bridge health monitoring technology, specifically to a method for constructing an intelligent agent for bridge structure health diagnosis based on thought chains and domain toolsets. This method can be directly applied to technologies such as smart infrastructure, intelligent bridges, intelligent inspection of bridge defects, and intelligent bridge health diagnosis. Background Art

[0002] With the continuous development of my country's infrastructure, numerous bridge structures have undergone decades of service and are now subject to multiple influences, including environmental corrosion, material degradation, natural disasters, and human factors. These factors have resulted in various types of structural damage, including concrete cracks, surface spalling, exposed rebar, anchor cable corrosion, and fatigue cracks in steel structures. These damages not only affect the durability and service life of the structures but also pose a potential threat to operational safety. Therefore, conducting health status monitoring and diagnosis of in-service structures and formulating targeted maintenance and reinforcement strategies are crucial for improving structural operational safety and reducing maintenance costs.

[0003] Traditional bridge structure inspections rely primarily on manual labor, requiring the involvement of professionals for everything from on-site photography and damage identification to documenting and comprehensive assessment. This results in low efficiency, high costs, and highly subjective inspection results. With the continuous advancement of artificial intelligence and image processing technologies, computer vision-based structural damage identification methods have gained widespread application. By analyzing structural images, these technologies enable non-contact, high-precision identification of surface defects, significantly improving monitoring efficiency and safety.

[0004] While existing methods demonstrate strong capabilities for image-level damage detection, they still lack systematic diagnostic capabilities at the structural level, making it difficult to meet the practical needs of multi-source information fusion and global judgment in bridge safety assessment. Furthermore, the reliance of currently widely used supervised learning methods on large numbers of high-quality labeled samples limits their widespread application in civil engineering, where data acquisition and annotation costs are relatively high. In recent years, unsupervised and weakly supervised structural damage identification methods have gradually emerged, gradually alleviating the reliance on labeled data. However, existing methods still struggle to infer the overall health of a structure from multiple images and components.

[0005] With the development of multimodal models, intelligent systems that combine image and text information have become a research hotspot. These systems, with their cross-modal understanding and question-answering capabilities, are expected to play a supporting role in structural health diagnosis, improving assessment efficiency and accuracy. However, due to the scarcity of multimodal data resources in civil engineering, the application of multimodal models in this field is still in its early stages of exploration. Summary of the Invention

[0006] The purpose of the present invention is to solve the problems in the prior art and propose a method for constructing a bridge structure health diagnosis intelligent agent based on thinking chain and domain tool set.

[0007] The present invention is achieved through the following technical solutions. The present invention proposes a method for constructing a bridge structure health diagnosis intelligent agent based on a thought chain and a domain tool set. The method includes the following steps: Step 1: Build a knowledge cache of bridge structure damage attributes; Construct a reusable attribute knowledge cache for the three core visual concepts of bridges: components, materials, and damage. This cache achieves semantic alignment between images and text, supports visual knowledge transfer across multiple scenes and components, and provides structured prior semantic support for subsequent multimodal reasoning tasks. Step 2: Design a bridge attribute concept embedding module and visual semantic mapping mechanism; A learnable bridge attribute encoding library is designed, which represents structural visual concepts as keys and their corresponding prompt vectors as values, forming a set of key-value pairs. The parameters of the bridge attribute encoding library are optimized through three loss mechanisms to achieve image-driven adaptive selection and optimization of concept prompts. Step 3: Establish a multimodal bridge health diagnosis model embedded with bridge structural attribute knowledge; Each time, a batch of multimodal bridge structural health diagnosis data is input into the model to obtain the model output. The network loss value is calculated using the bridge attribute concept comprehensive loss function and the language model loss function. The gradient of the network loss with respect to the model parameters is obtained using the backpropagation algorithm, and the network parameters are updated using the gradient descent algorithm. This cycle is repeated, and the model parameters are updated until convergence, resulting in a multimodal bridge health diagnosis model embedded with bridge structural attribute knowledge. Step 4: Build a multimodal bridge health diagnosis agent system based on thought chain reasoning and domain-specific tool sets; Build a bridge health diagnosis intelligent agent system that integrates a large language model, a multimodal model, and a dedicated tool set; using bridge images and text task descriptions as input, the thinking chain reasoning mechanism driven by the large model schedules visual analysis tasks in stages, automatically selects and calls bridge-specific tool models, and gradually completes component identification, damage detection, and health assessment tasks.

[0008] Furthermore, the step 1 specifically includes the following steps: Step 1: Collect images of bridge structures and damage, label them, and label each image with the structure type, damage, and material. Build a visual attribute model of bridge structures based on "component-material-damage," and develop a series of visual concepts for these three visual attributes. Step 1 and 2: Construct a concept cache of bridge structure damage attributes; Use the visual language multimodal model CLIP to construct a bridge attribute concept cache to efficiently store and retrieve the core visual concepts of bridge structures; the visual concept library can be regarded as an attribute vocabulary Dictionary; According to the zero-shot CLIP zero-shot reasoning setting, in each Pre-join tips = "The photo is ..." constitutes text input, using the text encoder Encode predefined concept text input to obtain text concept features ,in , is the number of visual concepts, is the serial number index; Step 13: Record the bridge attribute visual concept cache as , among which key It is the visual feature corresponding to the text concept word; through the image encoder of CLIP Images in the database Encode and obtain visual features ,in , N The number of images cached for building the bridge property concept; Step 14: Features of each text concept and All visual concepts are expressed in the formula Perform matching, measure the matching degree between image features and text, and select the top- K 1 as a candidate; to ensure the accuracy of concept matching, a manual review process is introduced after the matching is completed; the high-similarity images selected by CLIP are screened and adjusted so that each visual concept can accurately correspond to the appropriate image features; for special concepts that are difficult for CLIP to accurately identify, typical images are manually selected and their visual features are directly extracted to ensure the reliability of the visual features of this category; the image features of each visual concept are averaged to obtain the visual representation of the concept, which is stored as a key in the concept cache and associated with the corresponding text concept. These key-value pairs are stored in the bridge attribute concept cache.

[0009] Furthermore, the step 2 is specifically as follows: Step 21: Design and construct a learnable bridge attribute encoding library; The structural visual concept is represented as a key and the corresponding prompt vector is represented as a value; each visual concept corresponds to one or more learnable prompt vectors, forming a key-value pair set , where each Represents visual features, Represents M learnable vectors; the goal of the bridge attribute encoding library is to allow each image to select a matching hint vector based on its own visual features, thereby enhancing the transfer ability of visual concepts; Step 22: Compute visual concept cues for the input image; For each input image , through the image encoder , extract its features , and then calculate the image features and the set of learnable visual concepts The cosine similarity of each visual concept in ; Select the Top- K 3 visual concepts, corresponding prompt vectors of these concepts Stitch them together to get the image Visual concept cues; Step 2 and 3: Calculate the feature cosine similarity to ensure the consistency of the prompt words obtained from the concept cache and the prompt words selected from the encoding library in the feature space; In order to ensure the stability of the learnable bridge attribute encoding library during the learning process, a consistency constraint is introduced; for each image , extract its features through the image encoder ; Then, the top-most similar bridge features to the image features are retrieved from the manually constructed bridge attribute concept cache based on cosine similarity. K 2 concepts; after obtaining the text features of these concepts, they are input into the large language model to generate prompt words, and the consistency is compared with the learnable prompt words; through the Euclidean distance constraint , ensuring the consistency of the cue words obtained from the concept cache and the cue words selected from the encoding library in the feature space; Step 24: Based on the aforementioned Euclidean distance constraint, a comprehensive loss function is designed to optimize the learnable bridge attribute encoding library, including matching loss and orthogonal loss; Step 25: Design a lightweight cross-modal mapping computational method for visual language; In order to achieve seamless integration of bridge image information into the language model, a lightweight cross-modal mapping method is designed. Based on the linear mapping structure and combined with the visual feature splicing operation, the high-dimensional image features from the visual encoder and structural health diagnosis module are compressed and mapped to the semantic space of the large language model.

[0010] Furthermore, the comprehensive loss function designed in step 24 specifically includes: Matching loss, that is, when matching visual concepts, matching loss is introduced to ensure that the image features match the selected visual concepts more closely. The calculation method is:

[0011] Orthogonal loss: In order to enhance the semantic diversity of prompts and reduce redundancy, an additional orthogonal loss is introduced to make the embeddings of different prompts orthogonal to each other. The orthogonal loss is calculated as follows:

[0012] Combining all loss terms, the optimized objective function is expressed as:

[0013] Where, Optimize the matching of visual concepts, Optimize prompt word selection and consistency.

[0014] Furthermore, the visual language lightweight cross-modal mapping calculation method of step 25 includes the following steps: Step 251: Image features output by the visual encoder Reconstruction is performed by splicing adjacent n tokens, and get compressed features ; Step 252: Compress the features Transform to the semantic space of the large language model through linear mapping to obtain cross-modal representation ;in, is the projection matrix is the bias term , is the hidden layer dimension of the language model.

[0015] Furthermore, the step three is specifically as follows: Step 31: Collect images related to bridge structural health diagnosis, classify and organize them, manually annotate them, and generate text descriptions to construct a multimodal dataset of image and text pairings. This dataset can be divided into three categories: one is a panoramic image showing the overall shape of the bridge, another is a local image focusing on various bridge components, and the third is a detailed image taken at close range. Step 32: Based on the construction of image-text paired data, further introduce structured bridge metadata to supplement the multi-dimensional attribute information of the components corresponding to the image; Step 3: Based on the semantic requirements of the bridge health diagnosis task, a comprehensive prompt template system is designed. Few-sample examples are introduced to enhance the model's understanding capabilities. Subsequently, the large language model is used to automatically generate "image-command-answer" ternary question and answer samples through prompt learning. This stage uses a combination of manual review and automatic screening to control the quality of the output results, ensuring that the question and answer content is logically rigorous and linguistically accurate, providing high-quality training data for the multimodal understanding task. Steps 3 and 4: Based on the constructed image-text pairing dataset, a multi-task learning strategy is used to train the model. The model consists of an image encoder, a text encoder, a structural attribute embedding module, a visual feature projection module, and a large language model. It fuses multimodal information through cross-modal projection and is guided by structural attribute knowledge. An end-to-end training strategy is used to jointly optimize the model parameters. Step 35: The objective function of network training is the autoregressive language model loss, as shown in the following formula:

[0016] Each round of training inputs a batch of multimodal bridge health diagnosis data, calculates the multi-task loss between the model output and the true label, and iteratively updates the parameters using backpropagation and gradient descent algorithms.

[0017] Furthermore, the step 4 is specifically as follows: Step 41: Integrate specialized model tools commonly used in bridge inspections for object detection, semantic segmentation, and geometric parameter calculation to form a callable model component library. Each tool model can accurately identify and quantitatively analyze key parts in the image, serving as the perception basis for the intelligent agent's reasoning process. Step 42: Introduce a chain-of-thought reasoning mechanism and establish a chain-of-thought reasoning method for the bridge structural health diagnosis agent system. This method breaks down the complex task into a multi-stage process: "identify components and damage - invoke tools - analyze results - generate reports." By designing prompt templates, the agent is guided to gradually invoke object detection, semantic segmentation, and damage analysis tools, achieving human-like structural health diagnosis reasoning. Step 43: Based on the requirements of the bridge structure health diagnosis task, a standardized task process template is designed to clarify the task objectives, reasoning structure, and tool calling sequence, supporting the intelligent agent to efficiently complete component identification, damage analysis, and structural assessment operations in the multi-stage task. Step 44: Based on the reasoning template and tool set, build an interactive intelligent agent operation system; the system can receive user instructions, automatically reason about task steps, call appropriate tools to complete image analysis and diagnosis, and generate a structured health assessment report, realizing a closed-loop processing flow from image input to diagnosis output.

[0018] Furthermore, the specific steps of establishing the bridge structure health diagnosis intelligent agent system thinking chain reasoning method in step 42 include: Step 421: By constructing a customized thought chain reasoning template, the task input format, reasoning process, tool calling method, and output result structure are clarified, guiding the intelligent agent to perform the diagnostic task according to the "step-by-step reasoning-tool calling-result summary" method. The template includes four parts: task goal description, task execution step guidance, interactive reasoning format design, and output structure specification. The task goal description is used to guide the intelligent agent to complete the bridge component and damage diagnosis task. The task execution steps are refined into component and damage identification, damage area extraction, geometric analysis, and structural impact assessment. Step 422: The interactive reasoning format adopts the "Thought-Action-Action Input-Observation" structure to achieve continuous reasoning of the thought chain. The output structure specification is used to unify the organization of the health assessment report and improve the readability and logic of the results. Step 4, 2, 3: Construct an evaluation mechanism for the bridge structure health diagnosis agent system. By setting up multiple rounds of test tasks and comparing the standard answers with the task outputs, the agent's multimodal task execution capabilities are quantitatively measured. The specific evaluation process includes: Constructing an evaluation task set: Setting up 50 typical task examples including component identification, damage detection, parameter extraction, image diagnosis, and result summary. Each task includes a standard description and expected output to ensure clear task objectives and logical processes; Recording execution logs and intermediate states: During the agent's execution, each step of its reasoning chain is recorded, including the intent understanding text, tool call sequence, intermediate observation results, and final output, forming a complete reasoning execution trajectory; Step 424: To uniformly evaluate the system's overall task execution capability, define the comprehensive execution efficiency index as the average of three indicators:

[0019] Where, Indicates the accuracy of intention recognition, Indicates the success rate of decision execution, Indicates the completeness of the reasoning chain; Step 425: Conduct multiple rounds of experiments for each task example, repeatedly record the agent's execution results, and reduce the interference caused by the randomness of the content generated by the large language model on the experiment; finally, by calculating the indicator values ​​of each round and displaying the fluctuations through box plots, verify the stability and effectiveness of the agent system in task understanding, tool calling, and reasoning organization from multiple dimensions.

[0020] The present invention also proposes an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the steps of the method for constructing a bridge structure health diagnosis intelligent agent based on a thinking chain and a domain tool set.

[0021] The present invention also proposes a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of the method for constructing a bridge structure health diagnosis intelligent agent based on a thinking chain and a domain tool set.

[0022] Beneficial effects of the present invention: 1. This invention supports the integration of "component-material-damage" attribute knowledge cache, improves the visual language model's ability to understand the structural semantics in bridge images, and achieves improved pre-perception accuracy for component identification and damage location.

[0023] 2. This invention supports the construction of a multimodal embedding mapping mechanism between images and language, realizes the deep integration of cross-modal semantic alignment and structural health information, and enhances the generalization ability and robustness of diagnostic reasoning.

[0024] 3. The present invention supports the dispatching of multiple types of specialized model tools (such as target detection, semantic segmentation, crack analysis, etc.) to realize the automatic identification of bridge component status and damage feature extraction, thereby improving the accuracy and efficiency of the diagnostic process.

[0025] 4. The present invention supports a multi-step semantic reasoning mechanism based on thought chains, simulates the expert-level causal analysis process, realizes damage level assessment and disease cause tracing, and enhances the logic and explainability of the diagnosis process.

[0026] 5. The present invention supports batch processing of multiple images and unified result generation, outputs structured diagnostic reports based on bridge operation and maintenance requirements, and realizes the informatization and intelligence of bridge health management. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0028] Figure 1 This is an overall flow chart of the bridge structure health diagnosis intelligent agent construction method based on the thinking chain and domain tool set.

[0029] Figure 2 This is a schematic diagram of the bridge attribute concept cache construction process.

[0030] Figure 3 This is a schematic diagram of the learning process of the bridge attribute concept encoding library.

[0031] Figure 4This is a schematic diagram of the construction process of a multimodal large model for bridge health diagnosis that embeds structural attribute knowledge.

[0032] Figure 5 This is a schematic diagram of the architecture of a multimodal bridge health diagnosis agent system based on thought chain reasoning and a dedicated tool set.

[0033] Figure 6 It is a schematic diagram of the reasoning process of the thinking chain of the bridge structure health diagnosis intelligent agent system.

[0034] Figure 7 This is a schematic diagram of the concrete crack detection results of the bridge health diagnosis intelligent agent.

[0035] Figure 8 This is a schematic diagram of the steel structure corrosion detection results of the bridge health diagnosis intelligent agent. DETAILED DESCRIPTION

[0036] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0037] This paper proposes a method for constructing an intelligent agent for bridge structural health diagnosis based on thought chaining and a domain toolkit. This method incorporates a language model capable of causal reasoning, combined with object detection, segmentation, and geometric analysis models for image processing, to construct a complete task chain process, achieving automation and intelligentization from image perception to structural diagnosis. This method supports multi-image input and multi-task collaborative processing, exhibits good generalization and interpretability, and can effectively improve the efficiency and accuracy of structural health assessments.

[0038] Specifically, combined Figures 1-8 The present invention proposes a method for constructing a bridge structure health diagnosis intelligent agent based on a thought chain and a domain tool set, and the method comprises the following steps: Step 1: Build a knowledge cache of bridge structure damage attributes; Construct a reusable attribute knowledge cache for the three core visual concepts of bridges: components, materials, and damage. This cache achieves semantic alignment between images and text, supports visual knowledge transfer across multiple scenes and components, and provides structured prior semantic support for subsequent multimodal reasoning tasks. The step 1 specifically includes the following steps: Step 1: Collect images of bridge structures and damage, annotate them, and label each image with the structural type, damage, and material. Construct visual attributes of bridge structures based on the "component-material-damage" model, and compile a series of visual concepts for these three visual attributes. The structural component concept library includes main beams, piers, abutments, guardrails, pavement, steel trusses, anchors, main cables, suspenders, suspender anchor heads, steel box girder U-ribs, supports, etc. The material concept library includes concrete, steel bars, steel plates, and polyethylene (PE). The damage concept library includes cracks, spalling, efflorescence, water seepage, oil seepage, exposed steel bars, rust, and deformation. Step 1 and 2: Construct a concept cache of bridge structure damage attributes; Use the visual language multimodal model CLIP to construct a bridge attribute concept cache to efficiently store and retrieve the core visual concepts of bridge structures; the visual concept library can be regarded as an attribute vocabulary Dictionary; According to the zero-shot CLIP zero-shot reasoning setting, in each Pre-join tips = "The photo is ..." constitutes text input, using the text encoder Encode predefined concept text input to obtain text concept features ,in , is the number of visual concepts, is the serial number index; Step 13: Record the bridge attribute visual concept cache as , among which key It is the visual feature corresponding to the text concept word; through the image encoder of CLIP Images in the database Encode and obtain visual features ,in , N The number of images cached for building the bridge property concept; Step 14: Features of each text concept and All visual concepts are expressed in the formula Perform matching, measure the matching degree between image features and text, and select the top- K1 as a candidate; to ensure the accuracy of concept matching, a manual review process is introduced after the matching is completed; the high-similarity images selected by CLIP are screened and adjusted so that each visual concept can accurately correspond to the appropriate image features; for special concepts that are difficult for CLIP to accurately identify (such as alkali), typical images are manually selected and their visual features are directly extracted to ensure the reliability of the visual features of this category; the image features of each visual concept are averaged to obtain the visual representation of the concept, which is stored as a key in the concept cache and associated with the corresponding text concept. These key-value pairs are stored in the bridge attribute concept cache.

[0039] Step 2: Design a bridge attribute concept embedding module and visual semantic mapping mechanism; A learnable bridge attribute encoding library is designed, which represents structural visual concepts as keys and their corresponding prompt vectors as values, forming a set of key-value pairs. The parameters of the bridge attribute encoding library are optimized through three loss mechanisms to achieve image-driven adaptive selection and optimization of concept prompts. The step 2 is specifically as follows: Step 21: Design and construct a learnable bridge attribute encoding library; The structural visual concept is represented as a key and the corresponding prompt vector is represented as a value; each visual concept corresponds to one or more learnable prompt vectors, forming a key-value pair set , where each Represents visual features, Represents M learnable vectors; the goal of the bridge attribute encoding library is to allow each image to select a matching hint vector based on its own visual features, thereby enhancing the transfer ability of visual concepts; Step 22: Compute visual concept cues for the input image; For each input image , through the image encoder , extract its features , and then calculate the image features and the set of learnable visual concepts The cosine similarity of each visual concept in ; Select the Top- K 3 visual concepts, corresponding prompt vectors of these concepts Stitch them together to get the image Visual concept cues; Step 2 and 3: Calculate the feature cosine similarity to ensure the consistency of the prompt words obtained from the concept cache and the prompt words selected from the encoding library in the feature space; In order to ensure the stability of the learnable bridge attribute encoding library during the learning process, a consistency constraint is introduced; for each image , extract its features through the image encoder ; Then, the top-most similar bridge features to the image features are retrieved from the manually constructed bridge attribute concept cache based on cosine similarity. K 2 concepts; after obtaining the text features of these concepts, they are input into a large language model (such as Qwen, the Qianwen model) to generate prompt words, and the consistency is compared with the learnable prompt words; through the Euclidean distance constraint , ensuring the consistency of the cue words obtained from the concept cache and the cue words selected from the encoding library in the feature space; Step 24: Based on the aforementioned Euclidean distance constraint, a comprehensive loss function is designed to optimize the learnable bridge attribute encoding library, including matching loss and orthogonal loss; The comprehensive loss function designed in step 24 specifically includes: Matching loss, that is, when matching visual concepts, matching loss is introduced to ensure that the image features match the selected visual concepts more closely. The calculation method is:

[0040] Orthogonal loss: In order to enhance the semantic diversity of prompts and reduce redundancy, an additional orthogonal loss is introduced to make the embeddings of different prompts orthogonal to each other. The orthogonal loss is calculated as follows:

[0041] Combining all loss terms, the optimized objective function is expressed as:

[0042] Where, Optimize the matching of visual concepts, Optimize prompt word selection and consistency.

[0043] Step 25: Design a lightweight cross-modal mapping computational method for visual language; To achieve seamless integration of bridge image information into language models, a lightweight cross-modal mapping method was designed. Based on a linear mapping structure and combined with visual feature concatenation, the high-dimensional image features from the visual encoder and structural health diagnosis module were compressed and mapped into the semantic space of the large language model. While preserving high-resolution details, this method effectively reduces feature sequence length and cross-modal computational overhead, thereby improving overall system training efficiency and inference performance.

[0044] The lightweight cross-modal mapping calculation method for visual language in step 25 is applicable to the structural health diagnosis task of high-resolution bridge images. It reduces the number of visual tokens while maintaining local structural information, effectively improving the training and inference efficiency of multimodal models. It specifically includes the following steps: Step 251: Image features output by the visual encoder Reconstruction is performed by splicing adjacent n token (preferably n = 4), and obtain the compression feature ; Step 252: Compress the features Transform to the semantic space of the large language model through linear mapping to obtain cross-modal representation ;in, is the projection matrix is the bias term , is the hidden layer dimension of the language model.

[0045] Step 3: Establish a multimodal bridge health diagnosis model embedded with bridge structural attribute knowledge; Each time, a batch of multimodal bridge structural health diagnosis data is input into the model to obtain the model output. The network loss value is calculated using the bridge attribute concept comprehensive loss function and the language model loss function. The gradient of the network loss with respect to the model parameters is obtained using the backpropagation algorithm, and the network parameters are updated using the gradient descent algorithm. This cycle is repeated, and the model parameters are updated until convergence, resulting in a multimodal bridge health diagnosis model embedded with bridge structural attribute knowledge. The step three is specifically as follows: Step 31: Collect images related to bridge structural health diagnosis, classify and organize them, manually annotate them, and generate text descriptions to construct a multimodal dataset of image and text pairings. This dataset can be divided into three categories: one is a panoramic image showing the overall shape of the bridge, another is a local image focusing on various bridge components, and the third is a detailed image taken at close range. Step 32: Based on the image-text pairing data, we further introduce structured bridge metadata to supplement the multi-dimensional attribute information of the components corresponding to the images, including but not limited to component name, material type, spatial location, bounding box coordinates, damage category and severity level. This structured labeling system not only provides a clear knowledge foundation for subsequent tasks, but also facilitates the detailed analysis of image semantic information, significantly enhancing the model's ability to perceive civil structural details. Step 3: Based on the semantic requirements of the bridge health diagnosis task, a comprehensive prompt template system is designed, covering various forms such as closed-ended questions and answers, multi-round dialogues, and visual command positioning. Few-sample examples are introduced to improve the model's understanding ability. Subsequently, with the help of large language models (such as Qwen and Qianwen large models), "image-command-answer" ternary question and answer samples are automatically generated through prompt learning. This stage uses a mechanism that combines manual review and automatic screening to control the quality of the output results, ensuring that the question and answer content is logically rigorous and the language is accurate, providing high-quality training data for multimodal understanding tasks. Steps 3 and 4: Based on the constructed image-text pairing dataset, a multi-task learning strategy is used to train the model, covering tasks such as visual question answering, visual localization, and object detection. The model consists of an image encoder, a text encoder, a structural attribute embedding module, a visual feature projection module, and a large language model. It fuses multimodal information through cross-modal projection and is guided by structural attribute knowledge. An end-to-end training strategy is used to jointly optimize the model parameters. Step 35: The objective function of network training is the autoregressive language model loss, as shown in the following formula:

[0046] Each round of training inputs a batch of multimodal bridge health diagnosis data, calculates the multi-task loss between the model output and the true label, and iteratively updates the parameters using backpropagation and gradient descent algorithms.

[0047] Step 4: Build a multimodal bridge health diagnosis agent system based on thought chain reasoning and domain-specific tool sets; Build a bridge health diagnosis intelligent agent system that integrates a large language model, a multimodal model, and a dedicated tool set; using bridge images and text task descriptions as input, the thinking chain reasoning mechanism driven by the large model schedules visual analysis tasks in stages, automatically selects and calls bridge-specific tool models, and gradually completes component identification, damage detection, and health assessment tasks.

[0048] The step 4 is specifically as follows: Step 41: Integrate specialized model tools commonly used in bridge inspections for object detection, semantic segmentation, and geometric parameter calculation to form a callable model component library. Each tool model can accurately identify and quantitatively analyze key parts in the image, serving as the perception basis for the intelligent agent's reasoning process. Step 42: To enhance the system's reasoning and decision-making capabilities, a chain-of-thought reasoning mechanism was introduced to establish a chain-of-thought reasoning method for the bridge structural health diagnosis agent system. This method decomposes the complex task into a multi-stage process: "identify components and damage - invoke tools - analyze results - generate reports." A prompt template was designed to guide the agent to gradually invoke object detection, semantic segmentation, and damage analysis tools, achieving human-like structural health diagnosis reasoning. The specific steps of establishing the thinking chain reasoning method of the bridge structure health diagnosis intelligent agent system in step 4.2 include: Step 421: By constructing a customized thought chain reasoning template, the task input format, reasoning process, tool calling method, and output result structure are clarified, guiding the intelligent agent to perform the diagnostic task according to the "step-by-step reasoning-tool calling-result summary" method. The template includes four parts: task goal description, task execution step guidance, interactive reasoning format design, and output structure specification. The task goal description is used to guide the intelligent agent to complete the bridge component and damage diagnosis task. The task execution steps are refined into component and damage identification, damage area extraction, geometric analysis, and structural impact assessment. Step 422: The interactive reasoning format adopts the "Thought-Action-Action Input-Observation" structure to achieve continuous reasoning of the thought chain. The output structure specification is used to unify the organization of the health assessment report and improve the readability and logic of the results. Through the above template mechanism, the reasoning consistency of the intelligent agent and the tool collaboration ability can be improved, thereby enhancing the task execution efficiency and versatility of the system.

[0049] Step 4, 2, 3: Construct an evaluation mechanism for the bridge structure health diagnosis agent system. By setting up multiple rounds of test tasks and comparing the standard answers with the task outputs, the agent's multimodal task execution capabilities are quantitatively measured. The specific evaluation process includes: Constructing an evaluation task set: Setting up 50 typical task examples including component identification, damage detection, parameter extraction, image diagnosis, and result summary. Each task includes a standard description and expected output to ensure clear task objectives and logical processes; Recording execution logs and intermediate states: During the agent's execution, each step of its reasoning chain is recorded, including the intent understanding text, tool call sequence, intermediate observation results, and final output, forming a complete reasoning execution trajectory; Step 424: To uniformly evaluate the system's overall task execution capability, define the comprehensive execution efficiency index as the average of three indicators:

[0050] Where, Indicates the accuracy of intention recognition, Indicates the success rate of decision execution, Indicates the completeness of the reasoning chain; Step 425: Conduct multiple rounds of experiments for each task example, repeatedly record the agent's execution results, and reduce the interference caused by the randomness of the content generated by the large language model on the experiment; finally, by calculating the indicator values ​​of each round and displaying the fluctuations through box plots, verify the stability and effectiveness of the agent system in task understanding, tool calling, and reasoning organization from multiple dimensions.

[0051] Step 43: Based on the requirements of the bridge structure health diagnosis task, a standardized task process template is designed to clarify the task objectives, reasoning structure, and tool calling sequence, supporting the intelligent agent to efficiently complete component identification, damage analysis, and structural assessment operations in the multi-stage task. Step 44: Based on the reasoning template and tool set, build an interactive intelligent agent operation system; the system can receive user instructions, automatically reason about task steps, call appropriate tools to complete image analysis and diagnosis, and generate a structured health assessment report, realizing a closed-loop processing flow from image input to diagnosis output.

[0052] The bridge structure health diagnosis intelligent agent system based on thought chain and special model tool proposed in this invention is applied to the identification of structural damage such as steel structure surface corrosion and concrete cracks and the generation of detection reports. The results of the intelligent agent operation are as follows: Figure 7 and Figure 8 shown.

[0053] The present invention also proposes an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the steps of the method for constructing a bridge structure health diagnosis intelligent agent based on a thinking chain and a domain tool set.

[0054] The present invention also proposes a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of the method for constructing a bridge structure health diagnosis intelligent agent based on a thinking chain and a domain tool set.

[0055] The memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus RAM (DR RAM). It should be noted that the memory used in the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0056] In the above embodiments, all or part of the embodiments can be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a high-density digital video disc (DVD)), or a semiconductor medium (eg, a solid state disc (SSD)).

[0057] During implementation, each step of the above method can be completed by an integrated logic circuit of the hardware in the processor or by instructions in the form of software. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it will not be described in detail here.

[0058] It should be noted that the processor in the embodiments of the present application can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above-described method embodiments can be completed by hardware integrated logic circuits in the processor or by software instructions. The above-described processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware components. The methods, steps, and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software modules can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above-described method.

[0059] The above is a detailed introduction to the bridge structure health diagnosis intelligent body construction method based on thinking chain and domain tool set proposed in the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A bridge structure health diagnosis agent construction method based on thought chain and domain tool set, characterized by: The method comprises the following steps: Step 1: Build a knowledge cache of bridge structure damage attributes; Construct a reusable attribute knowledge cache for the three core visual concepts of bridges: components, materials, and damage. This cache achieves semantic alignment between images and text, supports visual knowledge transfer across multiple scenes and components, and provides structured prior semantic support for subsequent multimodal reasoning tasks. Step 2: Design a bridge attribute concept embedding module and visual semantic mapping mechanism; A learnable bridge attribute encoding library is designed, which represents structural visual concepts as keys and their corresponding prompt vectors as values, forming a set of key-value pairs. The parameters of the bridge attribute encoding library are optimized through three loss mechanisms to achieve image-driven adaptive selection and optimization of concept prompts. Step 3: Establish a multimodal bridge health diagnosis model embedded with bridge structural attribute knowledge; Each time, a batch of multimodal bridge structural health diagnosis data is input into the model to obtain the model output. The network loss value is calculated using the bridge attribute concept comprehensive loss function and the language model loss function. The gradient of the network loss with respect to the model parameters is obtained using the backpropagation algorithm, and the network parameters are updated using the gradient descent algorithm. This cycle is repeated, and the model parameters are updated until convergence, resulting in a multimodal bridge health diagnosis model embedded with bridge structural attribute knowledge. Step 4: Build a multimodal bridge health diagnosis agent system based on thought chain reasoning and domain-specific tool sets; Build a bridge health diagnosis intelligent agent system that integrates a large language model, a multimodal model, and a dedicated tool set; using bridge images and text task descriptions as input, the thinking chain reasoning mechanism driven by the large model schedules visual analysis tasks in stages, automatically selects and calls bridge-specific tool models, and gradually completes component identification, damage detection, and health assessment tasks.

2. The method according to claim 1, characterized in that The step 1 specifically includes the following steps: Step 1: Collect images of bridge structures and damage, label them, and label each image with the structure type, damage, and material. Build a visual attribute model for bridge structures based on "component-material-damage," and develop a series of visual concepts for these three visual attributes. Step 1 and 2: Construct a concept cache of bridge structure damage attributes; Use the visual language multimodal model CLIP to construct a bridge attribute concept cache to efficiently store and retrieve the core visual concepts of bridge structures; the visual concept library can be regarded as an attribute vocabulary Dictionary; According to the zero-shot CLIP zero-shot reasoning setting, in each Pre-join tips = "Thephoto is ..." constitutes text input, using the text encoder Encode predefined concept text input to obtain text concept features ,in , is the number of visual concepts, is the serial number index; Step 13: Record the bridge attribute visual concept cache as , among which key It is the visual feature corresponding to the text concept word; through the image encoder of CLIP Images in the database Encode and obtain visual features ,in , N The number of images cached for building the bridge property concept; Step 14: Features of each text concept and All visual concepts are expressed in the formula Perform matching, measure the matching degree between image features and text, and select the top- K 1 as a candidate; to ensure the accuracy of concept matching, a manual review process is introduced after the matching is completed; the high-similarity images selected by CLIP are screened and adjusted so that each visual concept can accurately correspond to the appropriate image features; for special concepts that are difficult for CLIP to accurately identify, typical images are manually selected and their visual features are directly extracted to ensure the reliability of the visual features of this category; the image features of each visual concept are averaged to obtain the visual representation of the concept, which is stored as a key in the concept cache and associated with the corresponding text concept. These key-value pairs are stored in the bridge attribute concept cache.

3. The method according to claim 1, characterized in that The step 2 is specifically as follows: Step 21: Design and construct a learnable bridge attribute encoding library; The structural visual concept is represented as a key and the corresponding prompt vector is represented as a value; each visual concept corresponds to one or more learnable prompt vectors, forming a key-value pair set , where each Represents visual features, Represents M learnable vectors; the goal of the bridge attribute encoding library is to allow each image to select a matching hint vector based on its own visual features, thereby enhancing the transfer ability of visual concepts; Step 22: Compute visual concept cues for the input image; For each input image , through the image encoder , extract its features< img src='' class="img-anchor" img-id="QLYQS_22" / >, then calculate the image features and the set of learnable visual concepts The cosine similarity of each visual concept in ; Select the Top- K 3 visual concepts, corresponding prompt vectors of these concepts Stitch them together to get the image Visual concept cues; Step 2 and 3: Calculate the feature cosine similarity to ensure the consistency of the prompt words obtained from the concept cache and the prompt words selected from the encoding library in the feature space; In order to ensure the stability of the learnable bridge attribute encoding library during the learning process, a consistency constraint is introduced; for each image , extract its features through the image encoder ; Then, the top-most similar bridge features to the image features are retrieved from the manually constructed bridge attribute concept cache based on cosine similarity. K 2 concepts; after obtaining the text features of these concepts, they are input into the large language model to generate prompt words, and the consistency is compared with the learnable prompt words; through the Euclidean distance constraint , ensuring the consistency of the cue words obtained from the concept cache and the cue words selected from the encoding library in the feature space; Step 24: Based on the aforementioned Euclidean distance constraint, a comprehensive loss function is designed to optimize the learnable bridge attribute encoding library, including matching loss and orthogonal loss; Step 25: Design a lightweight cross-modal mapping computational method for visual language; To achieve seamless integration of bridge image information into language models, a lightweight cross-modal mapping method is designed. Based on the linear mapping structure and combined with the visual feature splicing operation, the high-dimensional image features from the visual encoder and structural health diagnosis module are compressed and mapped to the semantic space of the large language model.

4. The method according to claim 3, characterized in that The comprehensive loss function designed in step 24 specifically includes: Matching loss, that is, when matching visual concepts, matching loss is introduced to ensure that the image features match the selected visual concepts more closely. The calculation method is: Orthogonal loss: In order to enhance the semantic diversity of prompts and reduce redundancy, an additional orthogonal loss is introduced to make the embeddings of different prompts orthogonal to each other. The orthogonal loss is calculated as follows: Combining all loss terms, the optimized objective function is expressed as: Where, Optimize the matching of visual concepts, Optimize prompt word selection and consistency.

5. The method according to claim 3, characterized in that The visual language lightweight cross-modal mapping calculation method of step 25 includes the following steps: Step 251: Output the image features of the visual encoder Reconstruction is performed by splicing adjacent n tokens, and get compressed features ; Step 252: Compress the features Transform to the semantic space of the large language model through linear mapping to obtain cross-modal representation ;in, is the projection matrix is the bias term , is the hidden layer dimension of the language model.

6. The method according to claim 1, characterized in that The step three is specifically as follows: Step 31: Collect images related to bridge structural health diagnosis, classify and organize them, manually annotate them, and generate text descriptions to construct a multimodal dataset of image and text pairings. This dataset can be divided into three categories: one is a panoramic image showing the overall shape of the bridge, another is a local image focusing on various bridge components, and the third is a detailed image taken at close range. Step 32: Based on the image-text pairing data, structured bridge metadata is further introduced to supplement the multi-dimensional attribute information of the components corresponding to the images; Step 3: Based on the semantic requirements of the bridge health diagnosis task, a comprehensive prompt template system is designed. Few-sample examples are introduced to enhance the model's understanding capabilities. Subsequently, the large language model is used to automatically generate "image-command-answer" ternary question and answer samples through prompt learning. This stage uses a combination of manual review and automatic screening to control the quality of the output results, ensuring the logical rigor and linguistic accuracy of the question and answer content, providing high-quality training data for the multimodal understanding task. Steps 3 and 4: Based on the constructed image-text pairing dataset, a multi-task learning strategy is used to train the model. The model consists of an image encoder, a text encoder, a structural attribute embedding module, a visual feature projection module, and a large language model. It fuses multimodal information through cross-modal projection and is guided by structural attribute knowledge. An end-to-end training strategy is used to jointly optimize the model parameters. Step 35: The objective function of network training is the autoregressive language model loss, as shown in the following formula: Each round of training inputs a batch of multimodal bridge health diagnosis data, calculates the multi-task loss between the model output and the true label, and iteratively updates the parameters using backpropagation and gradient descent algorithms.

7. The method according to claim 1, characterized in that The step 4 is specifically as follows: Step 41: Integrate specialized model tools commonly used in bridge inspections for object detection, semantic segmentation, and geometric parameter calculation to form a callable model component library. Each tool model can accurately identify and quantitatively analyze key parts in the image, serving as the perception basis for the intelligent agent's reasoning process. Step 42: Introduce a chain-of-thought reasoning mechanism and establish a chain-of-thought reasoning method for the bridge structural health diagnosis agent system. This method breaks down the complex task into a multi-stage process: "identify components and damage - invoke tools - analyze results - generate reports." By designing prompt templates, the agent is guided to gradually invoke object detection, semantic segmentation, and damage analysis tools, achieving human-like structural health diagnosis reasoning. Step 43: Based on the requirements of the bridge structure health diagnosis task, a standardized task process template is designed to clarify the task objectives, reasoning structure, and tool calling sequence, supporting the intelligent agent to efficiently complete component identification, damage analysis, and structural assessment operations in the multi-stage task. Step 44: Based on the reasoning template and tool set, build an interactive intelligent agent operation system; the system can receive user instructions, automatically reason about task steps, call appropriate tools to complete image analysis and diagnosis, and generate a structured health assessment report, realizing a closed-loop processing flow from image input to diagnosis output.

8. The method according to claim 7, characterized in that The specific steps of establishing the thinking chain reasoning method of the bridge structure health diagnosis intelligent agent system in step 4.2 include: Step 421: By constructing a customized thought chain reasoning template, the task input format, reasoning process, tool invocation method, and output result structure are clarified, guiding the agent to perform the diagnostic task according to the "step-by-step reasoning-tool invocation-result summary" method. The template includes four parts: task goal description, task execution step guidance, interactive reasoning format design, and output structure specification. The task goal description is used to guide the agent to complete the bridge component and damage diagnosis task. The task execution steps are refined into component and damage identification, damage area extraction, geometric analysis, and structural impact assessment. Step 422: The interactive reasoning format adopts the "Thought-Action-Action Input-Observation" structure to achieve continuous reasoning of thought chains. The output structure specification is used to unify the organization of health assessment reports and improve the readability and logic of the results. Step 4, 2, 3: Construct an evaluation mechanism for the bridge structure health diagnosis agent system. By setting up multiple rounds of test tasks and comparing the standard answers with the task outputs, the agent's multimodal task execution capabilities are quantitatively measured. The specific evaluation process includes: Constructing an evaluation task set: Setting up 50 typical task examples including component identification, damage detection, parameter extraction, image diagnosis, and result summary. Each task includes a standard description and expected output to ensure clear task objectives and logical processes; Recording execution logs and intermediate states: During the agent's execution, each step of its reasoning chain is recorded, including the intent understanding text, tool call sequence, intermediate observation results, and final output, forming a complete reasoning execution trajectory; Step 424: To uniformly evaluate the system's overall task execution capability, define the comprehensive execution efficiency index as the average of three types of indicators: Where, Indicates the accuracy of intention recognition, Indicates the success rate of decision execution, Indicates the completeness of the reasoning chain; Step 425: Conduct multiple rounds of experiments for each task example, repeatedly record the agent's execution results, and reduce the interference caused by the randomness of the content generated by the large language model on the experiment; finally, by calculating the indicator values ​​of each round and displaying the fluctuations through box plots, verify the stability and effectiveness of the agent system in task understanding, tool calling, and reasoning organization from multiple dimensions.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium for storing computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images