Bridge structure health diagnosis agent construction method based on thought chain and domain tool set

Abstract

The application proposes a bridge structure health diagnosis intelligent agent construction method based on thought chain and field tool set. The method introduces a language model with causal reasoning ability, combines target detection, segmentation and geometric analysis models for image processing, constructs a complete task chain process, and realizes automation and intelligentization from image perception to structure diagnosis. The method supports multi-image input, multi-task collaborative processing, has good generalization ability and interpretability, and can effectively improve the efficiency and accuracy of structure health evaluation.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of bridge health monitoring, in particular relates to a bridge structure health diagnosis intelligent agent construction method based on thought chain and domain tool set. The technical field to which the method can be directly applied includes smart infrastructure, intelligent bridge, intelligent bridge disease patrol, intelligent bridge health diagnosis, etc. BACKGROUND

[0002] With the continuous development of infrastructure in China, a large number of bridge structures have experienced decades of service, and are facing multiple influences such as environmental corrosion, material degradation, natural disasters and human factors, resulting in various types of structural damage including concrete cracks, surface spalling, exposed reinforcement, anchor corrosion and steel structure fatigue cracks. These damages not only affect the durability and service life of the structure, but also pose a potential threat to the operation safety. Therefore, it is of great significance to carry out health monitoring and diagnosis of in-service structures and develop targeted maintenance and reinforcement strategies to improve the operation safety of the structure and reduce maintenance costs.

[0003] Traditional bridge structure patrol work mainly relies on manual means. Professional personnel are required to participate in the entire process from field shooting, damage identification to text recording and comprehensive evaluation, which is low in efficiency and high in cost, and the detection results have strong subjectivity. With the continuous progress of artificial intelligence and image processing technology, structure damage identification methods based on computer vision have been widely applied. This kind of technology realizes non-contact and high-precision identification of structure surface defects through image analysis, greatly improving the efficiency and safety of monitoring.

[0004] Although existing methods have strong ability in image-level damage detection, they still lack systematic diagnosis capability at the structure level, making it difficult to meet the actual needs of multi-source information fusion and global judgment in bridge safety evaluation. At the same time, the dependence of the widely used supervised learning method on a large number of high-quality labeled samples limits its promotion and application in the field of civil engineering, because the data acquisition and labeling cost in this field is relatively high. In recent years, unsupervised and weakly supervised structure damage identification methods have gradually emerged, gradually alleviating the dependence on labeled data. However, existing methods still cannot infer the overall health status of the structure from multiple image and component information.

[0005] With the development of multi-modal models, intelligent systems combining image and text information have gradually become a research hotspot. Such systems have cross-modal understanding and question-answering capabilities, which are expected to play an auxiliary role in structure health diagnosis tasks and improve evaluation efficiency and accuracy. However, due to the scarcity of multi-modal data resources in the field of civil engineering, the application of multi-modal models in this field is still in the preliminary exploration stage. SUMMARY

[0006] The purpose of this invention is to solve the problems in the prior art and propose a method for constructing an intelligent agent for bridge structure health diagnosis based on thought chain and domain toolset.

[0007] This invention is achieved through the following technical solution: This invention proposes a method for constructing an intelligent agent for bridge structure health diagnosis based on thought chains and domain toolsets. The method includes the following steps:

[0008] Step 1: Construct a knowledge cache of bridge structural damage attributes;

[0009] A reusable attribute knowledge cache is constructed for the three core visual concepts of bridges: "components-materials-damage". This cache realizes semantic alignment between images and text, supports visual knowledge transfer between multiple scenarios and components, and provides structured prior semantic support for subsequent multimodal reasoning tasks.

[0010] Step 2: Design the bridge attribute concept embedding module and visual semantic mapping mechanism;

[0011] Design a learnable bridge attribute coding library, representing structural visual concepts as keys and their corresponding cue vectors as values, forming a set of key-value pairs; optimize the parameters of the bridge attribute coding library through three loss mechanisms to achieve image-driven adaptive selection and optimization of concept cues;

[0012] Step 3: Establish a multimodal model for bridge health diagnosis that embeds knowledge of bridge structural attributes;

[0013] Each time, a batch of multimodal bridge structural health diagnosis data is input into the model, the model output is obtained, and the network loss value is calculated using the bridge attribute concept comprehensive loss function and the language model loss function. The gradient of the network loss with respect to the model parameters is obtained using the backpropagation algorithm, and the network parameters are updated using the gradient descent algorithm. This process is repeated until the model parameters converge, resulting in a multimodal bridge health diagnosis model with bridge structural attribute knowledge embedded.

[0014] Step 4: Construct a multimodal bridge health diagnosis intelligent agent system based on thought chain reasoning and domain-specific toolsets;

[0015] A bridge health diagnosis intelligent agent system is constructed that integrates a large language model, a multimodal model, and a dedicated toolset. Taking bridge images and text task descriptions as input, the system uses a thought chain reasoning mechanism driven by the large model to schedule visual analysis tasks in stages, automatically select and call bridge-specific tool models, and gradually complete the tasks of component identification, damage detection, and health assessment.

[0016] Furthermore, step one specifically includes the following steps:

[0017] Step 11: Collect images of the bridge structure and its damage, label them, and name the structure type, damage and material of each image; construct the visual attributes of the bridge structure based on "component-material-damage", and write a series of visual concepts for these three types of visual attributes;

[0018] Steps 1 and 2: Constructing a bridge structural damage attribute concept cache; utilizing the Visual Language Multimodal Model (CLIP) to construct a bridge attribute concept cache for efficient storage and retrieval of core visual concepts of the bridge structure; the visual concept library can be viewed as an attribute vocabulary. The dictionary; according to the CLIP zero-shot inference setting, in each Add prompt The text input is "The photo is …", which is then encoded using a text encoder. Encode predefined conceptual text input to obtain textual conceptual features. ,in , For the number of visual concepts, Indexed by serial number;

[0019] Step 13: Cache and record the visual concept of bridge attributes as... , among them key These are visual features corresponding to textual conceptual words; obtained through CLIP's image encoder. Images from the database Encode to obtain visual features ,in , N The number of images cached for constructing the bridge attribute concept;

[0020] Step 14: Features of Each Text Concept and All visual concepts within are expressed using formulas The matching process measures the degree of match between image features and text, selecting the top-ranked text with the highest similarity scores. K Visual features of 1 are used as candidates; to ensure the accuracy of concept matching, a manual review process is introduced after matching is completed; the high similarity images selected by CLIP are screened and adjusted so that each visual concept can accurately correspond to the appropriate image features; for special concepts that CLIP cannot accurately identify, typical images are manually selected and their visual features are directly extracted to ensure the reliability of the visual features of this category; the mean of the image features of each visual concept is calculated to obtain the visual representation of the concept and stored as a key in the concept cache, and associated with the corresponding text concept. These key-value pairs are stored in the bridge attribute concept cache.

[0021] Furthermore, step two specifically involves:

[0022] Step 21: Design and build a learnable bridge attribute coding library;

[0023] Structural visual concepts are represented as keys, and their corresponding cue vectors are represented as values; each visual concept corresponds to one or more learnable cue vectors, forming a set of key-value pairs. Each of them Indicates visual characteristics, This represents M learnable vectors; the goal of this bridge attribute encoding library is to enable each image to select a matching cue vector based on its own visual features, thereby enhancing the transferability of visual concepts.

[0024] Step 22: Calculate the visual conceptual cues of the input image;

[0025] For each input image Through the image encoder Extract its features Then, the image features are calculated and compared with the set of learnable visual concepts. Cosine similarity of various visual concepts Select the Top-Matching feature that best matches the image characteristics. K Three visual concepts, and corresponding cue vectors for these concepts. By piecing them together, we obtain the image representation. Visual conceptual cues;

[0026] Steps 2 and 3: Calculate the feature cosine similarity to ensure the consistency between the prompt words obtained from the concept cache and the prompt words selected from the encoding library in the feature space;

[0027] To ensure the stability of the learnable bridge attribute encoding library during the learning process, a consistency constraint is introduced; for each image... Features are extracted through an image encoder. Then, based on cosine similarity, the top-most similar images to the bridge attribute concept cache are retrieved from the manually constructed cache. K Two concepts are identified; after obtaining the textual features of these concepts, they are input into a large language model to generate prompt words, and their consistency is compared with learnable prompt words; Euclidean distance constraints are then applied. This ensures the consistency of the prompt words obtained from the concept cache with the prompt words selected from the encoding library in the feature space;

[0028] Step 24: Based on the aforementioned Euclidean distance constraint, design a comprehensive loss function to optimize the learnable bridge attribute encoding library, including matching loss and orthogonal loss;

[0029] Step 25: Design a lightweight cross-modal mapping computation method for visual language;

[0030] To achieve seamless integration of bridge image information into the language model, a lightweight cross-modal mapping method is designed. Based on a linear mapping structure and combined with visual feature stitching operations, high-dimensional image features from the visual encoder and structural health diagnosis module are compressed and mapped to the semantic space of the large language model.

[0031] Furthermore, the comprehensive loss function designed in step two or four specifically includes:

[0032] Matching loss, in the context of matching visual concepts, ensures a closer match between image features and the selected visual concept by introducing a matching loss. The calculation method is as follows:

[0033]

[0034] Orthogonality Loss: To enhance the semantic diversity of cues and reduce redundancy, an additional orthogonality loss is introduced, making the embeddings of different cues mutually orthogonal. The orthogonality loss is calculated as follows:

[0035]

[0036] Combining all loss terms, the optimized objective function is expressed as:

[0037]

[0038] In the formula, Optimize the matching of visual concepts. Optimize the selection and consistency of prompt words.

[0039] Furthermore, the lightweight cross-modal mapping calculation method for visual language in step two-five includes the following steps:

[0040] Step 251: Extract image features from the visual encoder. Reconstruct by splicing adjacent elements. n Each token yields a compressed feature. ;

[0041] Step 252: Compress the aforementioned features By transforming the representation to the semantic space of a large language model through linear mapping, a cross-modal representation is obtained. ;in, Projection matrix For bias terms , This represents the hidden layer dimension of the language model.

[0042] Furthermore, step three specifically includes:

[0043] Step 31: Collect images related to bridge structural health diagnosis, classify and organize them, manually annotate them, and generate text descriptions to construct a multimodal dataset with paired images and text. This dataset can be divided into three categories: one is a panoramic view showing the overall shape of the bridge, another is a local view focusing on various components of the bridge, and the third is a detailed image taken at close range.

[0044] Step 32: Based on the construction of image-text matching data, further introduce structured bridge metadata to supplement the multi-dimensional attribute information of the components corresponding to the images;

[0045] Step 33: Combining the semantic requirements of the bridge health diagnosis task, a comprehensive prompt template system is designed, and few sample examples are introduced to improve the model's understanding ability. Subsequently, with the help of a large language model, three-dimensional question-and-answer samples of "image-instruction-answer" are automatically generated through prompt learning. This stage adopts a mechanism that combines manual review and automatic screening to control the quality of the output results, ensuring that the question-and-answer content is logically rigorous and linguistically accurate, and providing high-quality training data for multimodal understanding tasks.

[0046] Steps 3 and 4: Based on the constructed image-text matching dataset, a multi-task learning strategy is used to train the model. The model consists of an image encoder, a text encoder, a structural attribute embedding module, a visual feature projection module, and a large language model. Multimodal information is fused through cross-modal projection and guided by structural attribute knowledge. An end-to-end training strategy is used to jointly optimize the model parameters.

[0047] Step 35: The objective function for network training is the autoregressive language model loss, as shown in the following formula:

[0048]

[0049] Each training round inputs a batch of multimodal bridge health diagnostic data, calculates the multi-task loss between the model output and the true label, and iteratively updates the parameters using backpropagation and gradient descent algorithms.

[0050] Furthermore, step four specifically involves:

[0051] Step 41: Integrate commonly used specialized model tools for bridge inspection, including object detection, semantic segmentation, and geometric parameter calculation, to form a callable model component library; each tool model can accurately identify and quantify key parts in the image, serving as the perceptual basis for the agent's reasoning process;

[0052] Step 42: Introduce a thought chain reasoning mechanism to establish a thought chain reasoning method for the intelligent agent system for bridge structural health diagnosis. Decompose complex tasks into a multi-stage process of "identifying components and damage - calling tools - analyzing results - generating reports". By designing prompt templates, guide the intelligent agent to gradually call target detection, semantic segmentation and damage analysis tools to achieve human-like structural health diagnosis reasoning.

[0053] Step 43: Based on the requirements of bridge structural health diagnosis tasks, design a standardized task flow template, clarify the task objectives, reasoning structure and tool call sequence, and support the agent to efficiently complete component identification, damage analysis and structural assessment operations in multi-stage tasks.

[0054] Step 44: Based on the reasoning template and toolset, construct an interactive intelligent agent operation system; the system can receive user instructions, automatically reason about task steps, call appropriate tools to complete image analysis and diagnosis, and generate a structured health assessment report, realizing a closed-loop processing flow from image input to diagnostic output.

[0055] Furthermore, the specific steps of establishing the bridge structure health diagnosis intelligent agent system's thinking chain reasoning method in step four-two include:

[0056] Step 421: By constructing a custom thought chain reasoning template, the input format, reasoning process, tool invocation method, and output result structure of the task are clarified, guiding the agent to execute the diagnostic task in the manner of "step-by-step reasoning - tool invocation - result summary". The template includes four parts: task objective description, task execution step guidance, interactive reasoning format design, and output structure specification. The task objective description is used to guide the agent to complete the diagnostic task of bridge components and damage. The task execution steps are detailed into component and damage identification, damage area extraction, geometric analysis, and structural impact assessment.

[0057] Step 422: The interactive reasoning format adopts the "Thought-Action-Action Input-Observation" structure to realize continuous reasoning similar to a thought chain. The output structure specification is used to unify the organization of health assessment reports and improve the readability and logic of the results.

[0058] Step 423: Construct an evaluation mechanism for the intelligent agent system for bridge structural health diagnosis. This involves setting up multiple rounds of test tasks and comparing the results with standard answers to quantify the agent's multimodal task execution capabilities. The specific evaluation process includes: constructing an evaluation task set: setting 50 typical task examples including component identification, damage detection, parameter extraction, image diagnosis, and result summarization. Each task includes a standard description and expected output to ensure clear task objectives and logical flow; recording execution logs and intermediate states: recording each step of the agent's reasoning chain during execution, including intent understanding text, tool call sequences, intermediate observation results, and final output, forming a complete reasoning execution trajectory.

[0059] Step 424: To uniformly evaluate the overall task execution capability of the system, the comprehensive execution efficiency index is defined as the average of three types of indicators:

[0060]

[0061] In the formula, Indicates the accuracy of intent recognition. Indicates the success rate of decision execution. Indicates the completeness of the reasoning chain;

[0062] Step 425: Conduct multiple rounds of experiments for each task example, repeatedly record the execution results of the agent, and reduce the interference caused by the randomness of the content generated by the large language model on the experiment; finally, calculate the index value of each round and show the fluctuation by box plot, and verify the stability and effectiveness of the agent system in terms of task understanding, tool calling and reasoning organization from multiple dimensions.

[0063] The present invention also proposes an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the bridge structure health diagnosis intelligent agent construction method based on thought chain and domain toolset.

[0064] The present invention also proposes a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of the bridge structure health diagnosis intelligent agent construction method based on thought chain and domain toolset.

[0065] The beneficial effects of this invention are:

[0066] 1. This invention supports the fusion of "component-material-damage" attribute knowledge caching, which enhances the visual language model's ability to understand the structural semantics in bridge images and improves the accuracy of pre-sensing for component identification and damage localization.

[0067] 2. This invention supports the construction of a multimodal embedding mapping mechanism between images and language, realizing cross-modal semantic alignment and deep integration of structural health information, and enhancing the generalization ability and robustness of diagnostic reasoning.

[0068] 3. This invention supports scheduling multiple specialized model tools (such as target detection, semantic segmentation, crack analysis, etc.) to achieve automated identification of bridge component status and extraction of damage features, thereby improving the accuracy and efficiency of the diagnostic process.

[0069] 4. This invention supports a multi-step semantic reasoning mechanism based on thought chain, simulates an expert-level causal analysis process, realizes damage level assessment and disease cause tracing, and enhances the logic and interpretability of the diagnostic process.

[0070] 5. This invention supports batch processing of multiple images and generation of unified results, and outputs structured diagnostic reports in conjunction with bridge operation and maintenance needs, thereby realizing the informatization and intelligentization of bridge health management. Attached Figure Description

[0071] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0072] Figure 1 This is an overall flowchart of the construction method for a bridge structure health diagnosis intelligent agent based on thought chain and domain toolset.

[0073] Figure 2 This is a schematic diagram illustrating the bridge attribute concept cache construction process.

[0074] Figure 3 This is a schematic diagram illustrating the learning process of the bridge attribute concept encoding library.

[0075] Figure 4 This is a schematic diagram illustrating the construction process of a multimodal bridge health diagnosis multimodal large model with embedded structural attribute knowledge.

[0076] Figure 5 This is a schematic diagram of a multimodal bridge health diagnosis intelligent agent system architecture based on thought chain reasoning and a dedicated toolset.

[0077] Figure 6 This is a schematic diagram of the reasoning process of the intelligent system for diagnosing the health of bridge structures.

[0078] Figure 7 This is a schematic diagram of the concrete crack detection results from the bridge health diagnostic intelligent system.

[0079] Figure 8 This is a schematic diagram of the steel structure corrosion detection results from the intelligent bridge health diagnosis system. Detailed Implementation

[0080] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0081] This invention proposes a method for constructing an intelligent agent for bridge structural health diagnosis based on thought chains and domain toolsets. This method introduces a language model with causal reasoning capabilities, combined with target detection, segmentation, and geometric analysis models for image processing, to construct a complete task chain process, achieving automation and intelligence from image perception to structural diagnosis. This method supports multiple image inputs and multi-task collaborative processing, possesses good generalization ability and interpretability, and can effectively improve the efficiency and accuracy of structural health assessment.

[0082] Specifically, in combination Figures 1-8 This invention proposes a method for constructing an intelligent agent for bridge structural health diagnosis based on thought chains and domain toolsets. The method includes the following steps:

[0083] Step 1: Construct a knowledge cache of bridge structural damage attributes;

[0084] A reusable attribute knowledge cache is constructed for the three core visual concepts of bridges: "components-materials-damage". This cache realizes semantic alignment between images and text, supports visual knowledge transfer between multiple scenarios and components, and provides structured prior semantic support for subsequent multimodal reasoning tasks.

[0085] Step one specifically includes the following steps:

[0086] Step 11: Collect images of bridge structures and their damage, and label them, specifying the structural type, damage, and material of each image; construct visual attributes for bridge structures based on "component-material-damage," and develop a series of visual concepts for these three categories; the structural component concept library includes main beams, piers, abutments, guardrails, pavement, steel trusses, anchorages, main cables, suspenders, suspender anchor heads, steel box girder U-ribs, supports, etc.; the material concept library includes concrete, reinforcing steel, steel plates, and PE; the damage concept library includes cracks, spalling, efflorescence, water seepage, oil seepage, exposed reinforcing steel, corrosion, and deformation.

[0087] Steps 1 and 2: Constructing a bridge structural damage attribute concept cache; utilizing the Visual Language Multimodal Model (CLIP) to construct a bridge attribute concept cache for efficient storage and retrieval of core visual concepts of the bridge structure; the visual concept library can be viewed as an attribute vocabulary. The dictionary; according to the CLIP zero-shot inference setting, in each Add prompt The text input is "The photo is …", which is then encoded using a text encoder. Encode predefined conceptual text input to obtain textual conceptual features. ,in , For the number of visual concepts, Indexed by serial number;

[0088] Step 13: Cache and record the visual concept of bridge attributes as... , among them key These are visual features corresponding to textual conceptual words; obtained through CLIP's image encoder. Images from the database Encode to obtain visual features ,in , N The number of images cached for constructing the bridge attribute concept;

[0089] Step 14: Features of Each Text Concept and All visual concepts within are expressed using formulas The matching process measures the degree of match between image features and text, selecting the top-ranked text with the highest similarity scores. K Visual features of 1 are used as candidates; to ensure the accuracy of concept matching, a manual review process is introduced after matching is completed; the high similarity images selected by CLIP are screened and adjusted so that each visual concept can accurately correspond to the appropriate image features; for special concepts that CLIP cannot accurately identify (such as alkali), typical images are manually selected and their visual features are directly extracted to ensure the reliability of the visual features of this category; the mean of the image features of each visual concept is calculated to obtain the visual representation of the concept and stored as a key in the concept cache, and associated with the corresponding text concept. These key-value pairs are stored in the bridge attribute concept cache.

[0090] Step 2: Design the bridge attribute concept embedding module and visual semantic mapping mechanism;

[0091] Design a learnable bridge attribute coding library, representing structural visual concepts as keys and their corresponding cue vectors as values, forming a set of key-value pairs; optimize the parameters of the bridge attribute coding library through three loss mechanisms to achieve image-driven adaptive selection and optimization of concept cues;

[0092] Step two specifically involves:

[0093] Step 21: Design and build a learnable bridge attribute coding library;

[0094] Structural visual concepts are represented as keys, and their corresponding cue vectors are represented as values; each visual concept corresponds to one or more learnable cue vectors, forming a set of key-value pairs. Each of them Indicates visual characteristics, This represents M learnable vectors; the goal of this bridge attribute encoding library is to enable each image to select a matching cue vector based on its own visual features, thereby enhancing the transferability of visual concepts.

[0095] Step 22: Calculate the visual conceptual cues of the input image;

[0096] For each input image Through the image encoder Extract its features Then, the image features are calculated and compared with the set of learnable visual concepts. Cosine similarity of various visual concepts Select the Top-Matching feature that best matches the image characteristics. K Three visual concepts, and corresponding cue vectors for these concepts. By piecing them together, we obtain the image representation. Visual conceptual cues;

[0097] Steps 2 and 3: Calculate the feature cosine similarity to ensure the consistency between the prompt words obtained from the concept cache and the prompt words selected from the encoding library in the feature space;

[0098] To ensure the stability of the learnable bridge attribute encoding library during the learning process, a consistency constraint is introduced; for each image... Features are extracted through an image encoder. Then, based on cosine similarity, the top-most similar images to the bridge attribute concept cache are retrieved from the manually constructed cache. K Two concepts are identified; after acquiring the textual features of these concepts, they are input into a large language model (such as Qwen, the Thousand Questions Large Model) to generate prompt words, and their consistency is compared with learnable prompt words; Euclidean distance constraints are then applied. This ensures the consistency of the prompt words obtained from the concept cache with the prompt words selected from the encoding library in the feature space;

[0099] Step 24: Based on the aforementioned Euclidean distance constraint, design a comprehensive loss function to optimize the learnable bridge attribute encoding library, including matching loss and orthogonal loss;

[0100] The comprehensive loss function designed in step two of the above steps specifically includes:

[0101] Matching loss, in the context of matching visual concepts, ensures a closer match between image features and the selected visual concept by introducing a matching loss. The calculation method is as follows:

[0102]

[0103] Orthogonality Loss: To enhance the semantic diversity of cues and reduce redundancy, an additional orthogonality loss is introduced, making the embeddings of different cues mutually orthogonal. The orthogonality loss is calculated as follows:

[0104]

[0105] Combining all loss terms, the optimized objective function is expressed as:

[0106]

[0107] In the formula, Optimize the matching of visual concepts. Optimize the selection and consistency of prompt words.

[0108] Step 25: Design a lightweight cross-modal mapping computation method for visual language;

[0109] To achieve seamless integration of bridge image information into a language model, a lightweight cross-modal mapping method is designed. Based on a linear mapping structure and combined with visual feature stitching operations, high-dimensional image features from the visual encoder and structural health diagnosis module are compressed and mapped to the semantic space of a large language model. While ensuring high-resolution detail preservation, this method effectively reduces the feature sequence length and cross-modal computational overhead, thereby improving the overall system's training efficiency and inference performance.

[0110] The lightweight cross-modal mapping calculation method for visual language in step two five is suitable for structural health diagnosis tasks of high-resolution bridge images. It reduces the number of visual tokens while maintaining local structural information, effectively improving the training and inference efficiency of multimodal models. Specifically, it includes the following steps:

[0111] Step 251: Extract image features from the visual encoder. Reconstruct by splicing adjacent elements. nOne token (which can be taken in the example) n = 4), to obtain compression features ;

[0112] Step 252: Compress the aforementioned features By transforming the representation to the semantic space of a large language model through linear mapping, a cross-modal representation is obtained. ;in, Projection matrix For bias terms , This represents the hidden layer dimension of the language model.

[0113] Step 3: Establish a multimodal model for bridge health diagnosis that embeds knowledge of bridge structural attributes;

[0114] Each time, a batch of multimodal bridge structural health diagnosis data is input into the model, the model output is obtained, and the network loss value is calculated using the bridge attribute concept comprehensive loss function and the language model loss function. The gradient of the network loss with respect to the model parameters is obtained using the backpropagation algorithm, and the network parameters are updated using the gradient descent algorithm. This process is repeated until the model parameters converge, resulting in a multimodal bridge health diagnosis model with bridge structural attribute knowledge embedded.

[0115] Step three specifically involves:

[0116] Step 31: Collect images related to bridge structural health diagnosis, classify and organize them, manually annotate them, and generate text descriptions to construct a multimodal dataset with paired images and text. This dataset can be divided into three categories: one is a panoramic view showing the overall shape of the bridge, another is a local view focusing on various components of the bridge, and the third is a detailed image taken at close range.

[0117] Step 32: Building upon the constructed image-text pairing data, structured bridge metadata is further introduced to supplement the multi-dimensional attribute information of the components corresponding to the images, including but not limited to component name, material type, spatial location, bounding box coordinates, damage category, and severity level. This structured labeling system not only provides a clear knowledge foundation for subsequent tasks but also helps to achieve fine-grained analysis of image semantic information, significantly enhancing the model's ability to perceive details of civil structures.

[0118] Step 33: Combining the semantic requirements of the bridge health diagnosis task, a comprehensive prompt template system is designed, covering various forms such as closed-ended question-and-answer, multi-turn dialogue, and visual command localization. Few-sample examples are introduced to enhance the model's understanding ability. Subsequently, using a large language model (such as Qwen, the Thousand Questions Big Model), three-dimensional question-and-answer samples of "image-command-answer" are automatically generated through prompt learning. This stage employs a mechanism combining manual review and automatic screening to control the quality of the output results, ensuring that the question-and-answer content is logically rigorous and linguistically accurate, providing high-quality training data for the multimodal understanding task.

[0119] Steps 3 and 4: Based on the constructed image-text matching dataset, a multi-task learning strategy is used to train the model, covering tasks such as visual question answering, visual localization, and object detection. The model consists of an image encoder, a text encoder, a structural attribute embedding module, a visual feature projection module, and a large language model. It fuses multimodal information through cross-modal projection and is guided by structural attribute knowledge; an end-to-end training strategy is used to jointly optimize the model parameters.

[0120] Step 35: The objective function for network training is the autoregressive language model loss, as shown in the following formula:

[0121]

[0122] Each training round inputs a batch of multimodal bridge health diagnostic data, calculates the multi-task loss between the model output and the true label, and iteratively updates the parameters using backpropagation and gradient descent algorithms.

[0123] Step 4: Construct a multimodal bridge health diagnosis intelligent agent system based on thought chain reasoning and domain-specific toolsets;

[0124] A bridge health diagnosis intelligent agent system is constructed that integrates a large language model, a multimodal model, and a dedicated toolset. Taking bridge images and text task descriptions as input, the system uses a thought chain reasoning mechanism driven by the large model to schedule visual analysis tasks in stages, automatically select and call bridge-specific tool models, and gradually complete the tasks of component identification, damage detection, and health assessment.

[0125] Step four specifically involves:

[0126] Step 41: Integrate commonly used specialized model tools for bridge inspection, including object detection, semantic segmentation, and geometric parameter calculation, to form a callable model component library; each tool model can accurately identify and quantify key parts in the image, serving as the perceptual basis for the agent's reasoning process;

[0127] Step 42: To enhance the system's reasoning and decision-making capabilities, a thought chain reasoning mechanism is introduced to establish a thought chain reasoning method for the intelligent agent system for bridge structural health diagnosis. Complex tasks are decomposed into a multi-stage process of "identifying components and damage - calling tools - analyzing results - generating reports". By designing prompt templates, the intelligent agent is guided to gradually call target detection, semantic segmentation, and damage analysis tools to achieve human-like structural health diagnosis reasoning.

[0128] Step 4.2, establishing the bridge structural health diagnosis intelligent agent system's thought chain reasoning method, includes the following specific steps:

[0129] Step 421: By constructing a custom thought chain reasoning template, the input format, reasoning process, tool invocation method, and output result structure of the task are clarified, guiding the agent to execute the diagnostic task in the manner of "step-by-step reasoning - tool invocation - result summary". The template includes four parts: task objective description, task execution step guidance, interactive reasoning format design, and output structure specification. The task objective description is used to guide the agent to complete the diagnostic task of bridge components and damage. The task execution steps are detailed into component and damage identification, damage area extraction, geometric analysis, and structural impact assessment.

[0130] Step 422: The interactive reasoning format adopts a "Thought-Action-Action Input-Observation" structure to achieve continuous reasoning similar to a thought chain. The output structure standard is used to unify the organization of health assessment reports, improving the readability and logic of the results. Through the above template mechanism, the coherence of agent reasoning and the collaborative ability of tools can be improved, thereby enhancing the system's task execution efficiency and versatility.

[0131] Step 423: Construct an evaluation mechanism for the intelligent agent system for bridge structural health diagnosis. This involves setting up multiple rounds of test tasks and comparing the results with standard answers to quantify the agent's multimodal task execution capabilities. The specific evaluation process includes: constructing an evaluation task set: setting 50 typical task examples including component identification, damage detection, parameter extraction, image diagnosis, and result summarization. Each task includes a standard description and expected output to ensure clear task objectives and logical flow; recording execution logs and intermediate states: recording each step of the agent's reasoning chain during execution, including intent understanding text, tool call sequences, intermediate observation results, and final output, forming a complete reasoning execution trajectory.

[0132] Step 424: To uniformly evaluate the overall task execution capability of the system, the comprehensive execution efficiency index is defined as the average of three types of indicators:

[0133]

[0134] In the formula, Indicates the accuracy of intent recognition. Indicates the success rate of decision execution. Indicates the completeness of the reasoning chain;

[0135] Step 425: Conduct multiple rounds of experiments for each task example, repeatedly record the execution results of the agent, and reduce the interference caused by the randomness of the content generated by the large language model on the experiment; finally, calculate the index value of each round and show the fluctuation by box plot, and verify the stability and effectiveness of the agent system in terms of task understanding, tool calling and reasoning organization from multiple dimensions.

[0136] Step 43: Based on the requirements of bridge structural health diagnosis tasks, design a standardized task flow template, clarify the task objectives, reasoning structure and tool call sequence, and support the agent to efficiently complete component identification, damage analysis and structural assessment operations in multi-stage tasks.

[0137] Step 44: Based on the reasoning template and toolset, construct an interactive intelligent agent operation system; the system can receive user instructions, automatically reason about task steps, call appropriate tools to complete image analysis and diagnosis, and generate a structured health assessment report, realizing a closed-loop processing flow from image input to diagnostic output.

[0138] The bridge structural health diagnosis intelligent agent system based on thought chain and dedicated model tools proposed in this invention was applied to the identification and detection report generation of structural damage such as steel structure surface corrosion and concrete cracks. The results of the intelligent agent's operation are as follows: Figure 7 and Figure 8 As shown.

[0139] The present invention also proposes an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the bridge structure health diagnosis intelligent agent construction method based on thought chain and domain toolset.

[0140] The present invention also proposes a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of the bridge structure health diagnosis intelligent agent construction method based on thought chain and domain toolset.

[0141] The memory in this application embodiment can be volatile memory or non-volatile memory, or it can include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory used in the methods described in this invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0142] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., high-density digital video discs (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).

[0143] In implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software. The steps of the method disclosed in the embodiments of this application can be directly implemented by a hardware processor, or by a combination of hardware and software modules in the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are omitted here.

[0144] It should be noted that the processor in the embodiments of this application can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method embodiments can be completed by the integrated logic circuitry in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied as execution by a hardware decoding processor, or as a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above methods.

[0145] The above provides a detailed description of the bridge structure health diagnosis intelligent agent construction method based on thought chain and domain toolset proposed in this invention. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A method for building a bridge structural health diagnosis agent based on thought chain and domain toolset, characterized in that, The method comprises the following steps: Step one: build bridge structure damage attribute knowledge cache; Build a reusable attribute knowledge cache for the "component-material-damage" three core visual concepts of the bridge. This cache realizes the semantic alignment of images and texts, supports visual knowledge transfer between multiple scenes and components, and provides structured prior semantic support for subsequent multi-modal reasoning tasks; Step two: design bridge attribute concept embedding module and visual semantic mapping mechanism; Design a learnable bridge attribute encoding library, which represents the structure visual concept as a key and the corresponding prompt vector as a value, forming a key-value pair set. Through three loss mechanisms, the parameters of the bridge attribute encoding library are optimized to realize the adaptive selection and optimization of concept prompts driven by images; Step three: establish a bridge health diagnosis multi-modal model embedded with bridge structure attribute knowledge; Each time a batch of multi-modal bridge structure health diagnosis data is input into the model, the model output is obtained, the network loss value is calculated using the bridge attribute concept comprehensive loss function and the language model loss function; the gradient of the network loss to the model parameters is obtained using the back propagation algorithm, and the network parameters are updated using the gradient descent algorithm; repeat the above steps until the model parameters converge, and obtain the bridge health diagnosis multi-modal model embedded with bridge structure attribute knowledge; Step four: build a multi-modal bridge health diagnosis agent system based on thought chain reasoning and domain-specific toolset; Build a bridge health diagnosis agent system that integrates large language models, multi-modal models, and specialized toolsets. With bridge images and text task descriptions as input, the thought chain reasoning mechanism driven by large models schedules visual analysis tasks in stages, automatically selects and calls bridge-specific tool models, and gradually completes component identification, damage detection, and health assessment tasks; The step two is specifically: Step two one: design and build a learnable bridge attribute encoding library; The structure visual concept is represented as a key, and the prompt vector corresponding to the structure visual concept is represented as a value; each visual concept corresponds to one or more learnable prompt vectors, forming a key-value pair set wherein each represents a visual feature, represents M learnable vectors; the goal of the bridge attribute encoding library is to enable each image to select a prompt vector matching the image based on its own visual features, thereby enhancing the migration ability of the visual concept; Step two two: calculate the visual concept prompt of the input image; For each input image Through the image encoder Extract its features Then, the image features are calculated and compared with the set of learnable visual concepts. Cosine similarity of various visual concepts Select the Top-Matching feature that best matches the image characteristics. K Three visual concepts, and corresponding cue vectors for these concepts. By piecing them together, we obtain the image representation. Visual conceptual cues; Step two three: calculate the feature cosine similarity to ensure consistency between the prompt words obtained from the concept cache and the prompt words selected from the encoding library in the feature space; To ensure the stability of the learnable bridge attribute encoding library during the learning process, a consistency constraint is introduced; for each image , its features are extracted by the image encoder ; then based on the cosine similarity, the top- K 2 concepts most similar to the image features are retrieved from the manually constructed bridge attribute concept cache; after obtaining the text features of these concepts, they are input into the large language model to generate prompt words, and a consistency comparison is made with the learnable prompt words; through the Euclidean distance constraint , the consistency of the prompt words obtained from the concept cache and the prompt words selected from the encoding library in the feature space is ensured; Step two four: based on the aforementioned Euclidean distance constraint, design a comprehensive loss function to optimize the learnable bridge attribute encoding library, including matching loss and orthogonal loss; Step two five: design a lightweight cross-modal mapping calculation method for visual language; To realize seamless connection of bridge image information to language models, a lightweight cross-modal mapping method is designed. Based on the linear mapping structure, combined with the visual feature splicing operation, the high-dimensional image features from the visual encoder and the structure health diagnosis module are compressed and mapped to the semantic space of the large language model.

2. The method of claim 1, wherein, The step one specifically comprises the following steps: Step one one: collect bridge structure images and their damage, label each image's structure type, damage, and material; build bridge structure visual attributes based on "component-material-damage", and write a series of visual concepts for the three types of visual attributes; Step one two: build bridge structure damage attribute concept cache; use visual language multi-modal model CLIP to build bridge attribute concept cache to efficiently store and retrieve the core visual concept of bridge structure; the visual concept library can be regarded as the dictionary of attribute vocabulary According to the inference setting of zero-shot CLIP, a prompt ="The photo is …” is added in front of each text input to form a text input, which is encoded by a text encoder to encode the pre-defined concept text input and obtain the text concept feature , where , is the number of visual concepts, and is the serial number index; Step one three: cache the bridge attribute visual concept as wherein key is the visual feature corresponding to the text concept word; the image encoder of CLIP encodes the images in the database to obtain the visual features wherein , N is the number of images used to construct the bridge attribute concept cache; Step one four: each text concept feature All visual concepts in The formula Match the image features with the text to measure the matching degree between the image features and the text, and select the top- K 1 visual features with the highest similarity score as candidates; To ensure the accuracy of concept matching, an artificial review link is introduced after matching is completed; For high similarity images selected by CLIP, filter and adjust each visual concept to accurately correspond to appropriate image features; For special concepts that CLIP cannot accurately identify, manually select typical images and directly extract their visual features to ensure the reliability of the visual features of this category; Calculate the mean value of the image features of each visual concept to obtain the visual representation of the concept, and store it as the key in the concept cache, and associate it with the corresponding text concept. These key-value pairs are stored in the bridge attribute concept cache.

3. The method of claim 1, wherein, The comprehensive loss function designed in step two four specifically includes: The matching loss, i.e., ensuring that the image features are more closely matched with the selected visual concept by introducing a matching loss when matching the visual concept, is calculated as follows: The orthogonal loss: In order to enhance the semantic diversity of the prompt and reduce redundancy, an additional orthogonal loss is introduced to make the embeddings of different prompts orthogonal to each other. The orthogonal loss is calculated as follows: The objective function of the optimization is represented as follows: In the formula, Optimizing the matching of visual concepts, Optimizing the selection and consistency of cue words.

4. The method of claim 1, wherein, The visual language lightweight cross-modal mapping calculation method of step two five includes the following steps: Step two five one: image features output by the visual encoder reconstruction is performed by concatenating adjacent n tokens to obtain compressed features ; Step two five two: compress the features Transform to large language model semantic space by linear mapping to get cross-modal representation ; wherein is a projection matrix is a bias term , is the language model hidden layer dimension.

5. The method of claim 1, wherein, The step three is specifically as follows: Step three one: Collect bridge structure health diagnosis related images, and after classification, arrangement, artificial labeling and text description generation, construct an image-text paired multi-modal data set; The data set can be divided into three categories: one is a panoramic view showing the overall shape of the bridge, the other is a local view focusing on various components of the bridge, and the third is a close-up view of the details; Step three two: On the basis of constructing the image-text paired data, structured bridge metadata is further introduced to supplement the multi-dimensional attribute information of the components corresponding to the images; Step three three: A wide-ranging prompt template system is designed in combination with the semantic requirements of the bridge health diagnosis task, and a few-shot example is introduced to improve the model's understanding ability. Then, by means of a large language model, a three-element question and answer sample of "image-instruction-answer" is automatically generated through prompt learning. In this stage, a combination of manual review and automatic screening is used to control the quality of the output results, ensuring that the question and answer content is logically rigorous and linguistically accurate, providing high-quality training corpus for multi-modal understanding tasks; Step three four: Based on the constructed image-text paired data set, a multi-task learning strategy is used to train the model. The model includes an image encoder, a text encoder, a structure attribute embedding module, a visual feature projection module and a large language model. Through cross-modal projection, multi-modal information is fused and guided by structure attribute knowledge. An end-to-end training strategy is used to jointly optimize the model parameters; The objective function of the network training is the loss of the autoregressive language model, as shown in the following formula: Each round of training inputs a batch of multi-modal bridge health diagnosis data, calculates the multi-task loss between the model output and the true label, and iteratively updates the parameters using backpropagation and gradient descent algorithm.

6. The method of claim 1, wherein, The step four is specifically as follows: Step four one: Integrate target detection, semantic segmentation and geometric parameter calculation bridge inspection commonly used special model tools to form a callable model component library; Each tool model can accurately identify and quantify the key parts in the image as the perception basis in the agent reasoning process; Step four two: Introduce a thinking chain reasoning mechanism to establish a bridge structure health diagnosis agent system thinking chain reasoning method, which divides the complex task into a multi-stage process of "identifying components and damage-calling tools-analyzing results-generating reports"; By designing prompt templates, the agent is guided to gradually call target detection, semantic segmentation and damage analysis tools to realize human-like structure health diagnosis reasoning. Step four three: in combination with the task requirements of bridge structure health diagnosis, a standardized task flow template is designed to clearly define the task objectives, reasoning structure and tool calling sequence, supporting the agent to efficiently complete the component identification, damage analysis and structure evaluation operations in multi-stage tasks; Step four four: based on the reasoning template and tool set, an interactive agent running system is constructed; the system can receive user instructions, automatically reason task steps, call appropriate tools to complete image analysis and diagnosis, and generate a structured health evaluation report, realizing a closed-loop processing flow from image input to diagnosis output.

7. The method of claim 6, wherein, The specific steps of the bridge structure health diagnosis agent system thought chain reasoning method in step four two include: Step four two one: by constructing a self-defined thought chain reasoning template, the input format, reasoning flow, tool calling mode and output result structure of the task are clearly defined, guiding the agent to execute the diagnosis task in the manner of "step-by-step reasoning-tool calling-result summarizing"; the template includes four parts: task objective description, task execution step guidance, interactive reasoning format design and output structure specification, among which the task objective description is used to guide the agent to complete the diagnosis task of bridge components and damage, and the task execution steps are refined into component and damage identification, damage area extraction, geometric analysis and structure influence evaluation; Step four two two: the interactive reasoning format adopts the structure form of "Thought-Action-Action Input-Observation", realizing continuous reasoning in the form of thought chain, and the output structure specification is used to unify the organization form of health evaluation report, improving the readability and logicality of the results; Step four two three: an evaluation mechanism for the bridge structure health diagnosis agent system is constructed, the multi-modal task execution ability of the agent is quantitatively measured based on the comparison of standard answers and task output, and the specific evaluation process includes: constructing evaluation task set: 50 typical task examples including component identification, damage detection, parameter extraction, image diagnosis and result summarizing are set, each task contains standard description and expected output, ensuring clear task objectives and logical flow; recording execution logs and intermediate states: during the execution of the agent, each step of the reasoning chain is recorded, including intention understanding text, tool calling sequence, intermediate observation results and final output, forming a complete reasoning execution trajectory; Step four two four: to unify the overall task execution ability of the evaluation system, the comprehensive execution efficiency index is defined as the average value of three types of indexes: In the formula, represents the intention recognition accuracy, represents the decision execution success rate, represents the reasoning chain completeness; Step four two five: for each task example, multiple rounds of experiments are carried out, and the execution results of the agent are repeatedly recorded to reduce the interference caused by the randomness of the generated content of the large language model; finally, the stability and effectiveness of the agent system in task understanding, tool calling and reasoning organization are verified from multiple dimensions by calculating the index value of each round and showing the fluctuation in the box plot. 8.An electronic device comprising a memory and a processor, the memory storing a computer program, wherein, The processor executes the computer program to realize the steps of the method of any one of claims 1-7.

9. A computer readable storage medium for storing computer instructions, characterized in that, The computer instructions are executed by the processor to realize the steps of the method of any one of claims 1-7.

Images