Bridge disease image recognition method and system based on prompt words and large model

Abstract

This invention relates to the field of bridge defect image recognition technology, and discloses a method and system for bridge defect image recognition based on prompt words and a large model. The method includes: constructing a structured domain knowledge base; generating recognition prompt words by calling a bridge component recognition prompt word template; inputting the image and recognition prompt words into a fine-tuned multimodal large model to identify components and output standard component codes; querying the knowledge base based on the codes to obtain component names, defect definition sets, visual feature description sets, and negative constraint text sets; filling in dynamic prompt words by calling a visual diagnostic prompt word template; inputting the image and dynamic prompt words into the model to concurrently perform defect identification and localization on each component; summarizing the results and inputting them into a text generation model to generate a defect identification report. This invention improves the reliability of bridge defect image recognition by combining a structured domain knowledge base with dynamic prompt words.

Description

Technical Field

[0001] This invention relates to the field of bridge defect image recognition technology, and more specifically, to a method and system for bridge defect image recognition based on prompt words and large models. Background Technology

[0002] With the continuous growth in the number of bridges and the increasing demand for digital inspection, the identification of bridge appearance defects is gradually developing from manual inspection to image-based and intelligent methods.

[0003] Current methods for detecting bridge surface defects typically include manual visual inspection and recognition methods based on image processing or deep learning models. Manual inspection relies heavily on human experience, and the results are easily influenced by subjective factors. While recognition methods based on image processing or deep learning models can improve automation, the complex component types and diverse manifestations of defects in bridge surface scenes, along with the similarities between non-defect features such as stains, shadows, and construction joints and defect features such as cracks and spalling, mean that existing technologies still suffer from insufficient reliability in complex scenarios. In recent years, multimodal large-scale models have demonstrated strong capabilities in visual understanding and reasoning; however, directly applying general-purpose models to bridge defect image recognition still often results in insufficient adaptability to specialized scenarios. Summary of the Invention

[0004] In view of this, the present invention proposes a method and system for bridge defect image recognition based on prompt words and large models, aiming to solve the problem of insufficient reliability of existing bridge defect image recognition technology in complex bridge appearance scenarios.

[0005] In one aspect, this invention proposes a method for bridge defect image recognition based on prompt words and a large model, comprising: A structured domain knowledge base is constructed, and a component table, a defect definition table, and a negative constraint table are established based on a relational database. The component table stores the unique code, standard terminology, and classification information of bridge components. The defect definition table is associated with the component code in the component table and stores the standard defect name and typical visual feature description of each bridge component. The negative constraint table is associated with the component code in the component table and stores the non-defect scene description of each bridge component. The bridge component identification prompt word template is called, and the bridge component identification prompt words are generated according to the standard terminology of bridge components in the component table. The bridge appearance defect image to be identified and the bridge component identification prompt words are input into the multimodal large model after the bridge appearance defect data is fine-tuned. The bridge components in the bridge appearance defect image are identified, and the bridge component identification results and the corresponding standard component codes are output. The structured domain knowledge base is queried according to the standard component code to obtain the component name, defect definition set, visual feature description set and negative constraint text set of the corresponding bridge component; Call the visual diagnostic prompt word template containing placeholders, fill the component name, defect definition set, visual feature description set and negative constraint text set into the corresponding placeholders according to preset fields, and generate dynamic prompt words corresponding to each bridge component; The bridge's external defects images and the dynamic prompts corresponding to each bridge component are input into the fine-tuned multimodal large model. Defect identification and defect localization are performed concurrently on each bridge component to obtain the defect identification results and defect localization results corresponding to each bridge component. Summarize the defect identification and location results for each bridge component, call the report generation prompt word template, input the summarized results into the text generation model, and generate a bridge defect identification report.

[0006] Furthermore, when constructing a structured domain knowledge base, the following are included: Component tables, disease definition tables, and negative constraint tables are established based on relational databases; The component table stores the unique codes, standard terms, and classification information of bridge components; The defect definition table stores the standard defect names and typical visual feature descriptions corresponding to each bridge component. The negative constraint table stores the descriptions of non-defect scenarios corresponding to each bridge component.

[0007] Furthermore, when the defect definition table is associated with the component codes in the component table, and the negative constraint table is associated with the component codes in the component table, it includes: Establish corresponding association records for each bridge component; enter the defect definition and visual feature description bound to the corresponding bridge component in the defect definition table; enter the negative constraint text content bound to the corresponding bridge component in the negative constraint table.

[0008] Further, when calling the bridge component identification prompt word template and generating bridge component identification prompt words based on the standard terminology of bridge components in the component table, and inputting the bridge appearance defect image to be identified and the bridge component identification prompt words into the multimodal large model after fine-tuning the bridge appearance defect data, the process includes: Deploy a fine-tuning platform, select an open-source multimodal large model as the base model on the fine-tuning platform, and upload the bridge appearance defect training dataset; Set the number of training rounds, output image size, and learning rate, and perform model training based on the set fine-tuning parameters to obtain a fine-tuned multimodal large model; Call the bridge component identification prompt word template, fill the standard terms of bridge components in the component table into the corresponding placeholders according to the preset fields, and generate bridge component identification prompt words; The bridge appearance defect image to be identified and the bridge component identification prompts are input into the fine-tuned multimodal large model to identify the bridge components in the bridge appearance defect image and output the bridge component identification results and the corresponding standard component codes.

[0009] Furthermore, when performing model training based on the set fine-tuning parameters, it also includes: After training is completed, the trained model is tested using a pre-prepared validation dataset to perform quantitative and qualitative performance evaluations. The quantitative assessment includes accuracy assessment, information coverage assessment, and logical structure assessment; the qualitative performance assessment includes reasoning ability assessment, output structure and standardization assessment, and business knowledge professionalism assessment. When the quantitative evaluation results meet the preset standards and the qualitative results meet the preset requirements, the fine-tuning is considered complete.

[0010] Furthermore, when querying the structured domain knowledge base based on the standard component code, the process includes: The database is queried in real time according to the standard component code to retrieve the disease definition, visual feature description and negative constraint text content bound to the standard component code, and the component name corresponding to the standard component code is extracted.

[0011] Furthermore, when invoking a visual diagnostic cue template containing placeholders, it includes: A preset visual diagnostic prompt template includes placeholders for component names, disease definitions, visual feature descriptions, and negative constraints. The retrieved component names, defect definition sets, visual feature description sets, and negative constraint text sets are filled into the corresponding placeholders according to the preset field correspondence, generating dynamic prompt words corresponding to the current bridge component.

[0012] Furthermore, when inputting the bridge's external defect images and the corresponding dynamic prompts for each bridge component into the fine-tuned multimodal large model, the process of identifying and locating defects in each bridge component includes: For each of the identified bridge components, the images of the bridge's external defects and the corresponding dynamic prompts are input into the fine-tuned multimodal large model, and defect identification and localization are performed. The defect identification results and defect localization results for each bridge component are then output.

[0013] Furthermore, the defect identification and location results for each bridge component are summarized, the report generation prompt template is called, and the summarized results are input into the text generation model to generate a bridge defect identification report, including: The defect identification and location results for each bridge component are compiled into an input list. The input list is input into the text generation model by calling the report generation prompt word template; The bridge defect identification report is output based on the text generation model.

[0014] Compared with existing technologies, the advantages of this invention are as follows: By constructing a structured domain knowledge base containing a component table, a defect definition table, and a negative constraint table, and by calling the bridge component identification prompt word template during the bridge component identification stage, bridge component identification prompt words are generated based on the standard terminology of bridge components in the component table. This enables the multimodal large model to output bridge component identification results and corresponding standard component codes within a preset range of bridge components. After identifying the bridge components, the corresponding component name, defect definition, visual feature description, and negative constraint text content are retrieved based on the standard component code. Then, combined with the visual diagnostic prompt word template, a matching result with each bridge component is generated. The system provides corresponding dynamic prompts, enabling the multimodal large model to combine domain knowledge content corresponding to the current bridge component when performing bridge defect identification and localization. Simultaneously, by performing defect identification and localization on each bridge component in the same bridge exterior defect image, and summarizing the defect identification and localization results for each bridge component, a bridge defect identification report is generated. This forms a complete processing flow from knowledge construction, component identification prompts, component identification, knowledge retrieval, dynamic prompt generation, defect identification and localization to report output, facilitating the systematic implementation of bridge defect image recognition tasks.

[0015] On the other hand, this application also provides a bridge defect image recognition system based on prompt words and a large model, used to implement the above-mentioned bridge defect image recognition method based on prompt words and a large model, including: The structured domain knowledge base management module is configured to build a structured domain knowledge base, establishing a component table, a defect definition table, and a negative constraint table based on a relational database. The component table stores the unique code, standard terminology, and classification information of bridge components. The defect definition table is associated with the component code in the component table and stores the standard defect name and typical visual feature description corresponding to each bridge component. The negative constraint table is associated with the component code in the component table and stores the non-defect scene description corresponding to each bridge component. The bridge component identification module is configured to call the bridge component identification prompt word template and generate bridge component identification prompt words according to the standard terminology of bridge components in the component table. The module inputs the bridge appearance defect image to be identified and the bridge component identification prompt words into the multimodal large model after fine-tuning the bridge appearance defect data, identifies the bridge components in the bridge appearance defect image, and outputs the bridge component identification result and the corresponding standard component code. The domain knowledge retrieval module is configured to query the structured domain knowledge base based on the standard component code to obtain the component name, defect definition set, visual feature description set, and negative constraint text set of the corresponding bridge component; The dynamic prompt word generation module is configured to call a visual diagnostic prompt word template containing placeholders, fill the component name, disease definition set, visual feature description set and negative constraint text set into the corresponding placeholders according to preset fields, and generate dynamic prompt words corresponding to each bridge component. The defect identification and localization module is configured to input the bridge appearance defect images and dynamic prompts corresponding to each bridge component into the fine-tuned multimodal large model, concurrently perform defect identification and defect localization on each bridge component, and obtain the defect identification results and defect localization results corresponding to each bridge component. The defect identification report generation module is configured to summarize the defect identification results and defect location results corresponding to each bridge component, call the report generation prompt word template, input the summarized results into the text generation model, and generate a bridge defect identification report.

[0016] It is understandable that the aforementioned bridge defect image recognition system and method based on prompt words and large models have the same beneficial effects, and will not be elaborated further here. Attached Figure Description

[0017] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 A flowchart illustrating a bridge defect image recognition method based on prompt words and a large model, provided in an embodiment of the present invention; Figure 2 This is a functional block diagram of a bridge defect image recognition system based on prompt words and a large model, provided as an embodiment of the present invention. Detailed Implementation

[0018] Exemplary embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the disclosure to those skilled in the art. It should be noted that, unless otherwise specified, embodiments and features in the embodiments of the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0019] See Figure 1 As shown, this application proposes a method for bridge defect image recognition based on prompt words and a large model, including: S1: Construct a structured domain knowledge base, and establish a component table, a defect definition table, and a negative constraint table based on a relational database; the component table stores the unique code, standard terminology, and classification information of bridge components; the defect definition table is associated with the component code in the component table and stores the standard defect name and typical visual feature description of each bridge component; the negative constraint table is associated with the component code in the component table and stores the non-defect scene description of each bridge component. S2: Call the bridge component identification prompt word template, and generate bridge component identification prompt words according to the standard terminology of bridge components in the component table. Input the bridge appearance defect image to be identified and the bridge component identification prompt words into the multimodal large model after fine-tuning the bridge appearance defect data, identify the bridge components in the bridge appearance defect image, and output the bridge component identification results and the corresponding standard component codes. S3: Query the structured domain knowledge base based on the standard component code to obtain the component name, defect definition set, visual feature description set, and negative constraint text set of the corresponding bridge component; S4: Call the visual diagnostic prompt template containing placeholders, fill the component name, defect definition set, visual feature description set and negative constraint text set into the corresponding placeholders according to the preset fields, and generate dynamic prompts corresponding to each bridge component; S5: Input the bridge appearance defect images and the dynamic prompts corresponding to each bridge component into the fine-tuned multimodal large model, and perform defect identification and defect localization concurrently on each bridge component to obtain the defect identification results and defect localization results corresponding to each bridge component. S6: Summarize the defect identification results and defect location results for each bridge component, call the report generation prompt word template, input the summarized results into the text generation model, and generate a bridge defect identification report.

[0020] Specifically, the structured domain knowledge base is a collection of professional knowledge data on bridge defects built on a relational database. The component table stores basic information about bridge components. The unique code is a field that uniquely identifies a bridge component and can be generated according to preset coding rules, such as a combination of component category code and sequential number. The standard terminology is the unified name corresponding to the component, and the classification information is a record of the category to which the component belongs. The defect definition table is associated with the component table through the component code or the primary key and foreign key corresponding to the component code. It is used to store the standard defect name and typical visual feature description corresponding to the component. The typical visual feature description can include the linear extension shape of cracks, the edge shape of peeling, the color characteristics and distribution of rust. The negative constraint table is also associated with the component table. It is used to store scene descriptions that are similar in appearance to defects but are not defects, such as construction joints, surface stains, water stains, shadows, texture changes and reflective areas, etc. The negative constraint text is the content that provides textual constraints on non-defect features. In the bridge component identification stage, the bridge component identification prompt word template is first invoked, and bridge component identification prompt words are generated based on the standard terminology of bridge components in the component table. The bridge component identification prompt word template is a pre-set component identification text framework, which includes at least a candidate component field. The candidate component field contains the set of standard terminology of bridge components stored in the component table, thereby limiting the identification range of bridge components in the current image to the preset candidate component set. After generating the bridge component identification prompt words, the image of the bridge appearance defects to be identified and the bridge component identification prompt words are input together into the multimodal large model after fine-tuning the bridge appearance defects data, so as to output the identification results of visible components in the image and the corresponding standard component codes.The multimodal large model fine-tuned using bridge appearance defect data refers to a model obtained by fine-tuning parameters based on a general visual language model using a bridge appearance defect training dataset. During fine-tuning, parameters such as the number of training epochs, output image size, and learning rate can be set in the fine-tuning platform. The number of training epochs can be set from 3 to 10 epochs depending on the sample size and model convergence. The output image size can be set to 448×448, 512×512, or 768×768 based on the basic model input specifications. The learning rate can be set from 1×10⁻⁵ to 5×10⁻⁵. Parameters can be jointly determined by combining quantitative and qualitative evaluation results from the validation dataset. Quantitative evaluation can employ accuracy, information coverage, and... For logical structure-related indicators, qualitative evaluation can be combined with reasoning ability, the degree of output structure and standardization, and professional business knowledge for manual review. When the verification results meet the preset requirements, fine-tuning is considered complete. After obtaining the standard component code, the system queries the knowledge base in real time based on the standard component code to extract the corresponding component name, disease definition set, visual feature description set, and negative constraint text set. The disease definition set and visual feature description set can be one or more records related to the current component. Real-time query means that after obtaining the component code, a database retrieval is initiated directly and the data content bound to the code is returned. The visual diagnostic prompt word template containing placeholders is a prompt word framework with predefined field positions. This includes placeholders for component names, disease definitions, visual feature descriptions, and negative constraints. Dynamic filling involves writing the retrieved content into the corresponding positions of the template according to a preset field order. The structured format can be text format arranged by field segments, key-value pairs, or lists, as long as the meaning of the fields remains fixed and the boundaries between different fields are clear. After filling, a dynamic prompt word corresponding to the current bridge component is obtained. Then, the original bridge appearance disease image and the dynamic prompt word are input together into the multimodal large model. Disease identification and disease localization are performed on each bridge component. The disease identification result can include disease name, disease category, or disease presence / absence judgment. The disease localization result... The results can include a location description of the affected area, bounding box coordinates, mask area or target area marking information; when the image contains multiple bridge components, dynamic prompts can be generated for each component and fed into the model for processing in parallel; finally, the defect identification results and defect location results corresponding to each component are summarized into an input list, which can include component name, component code, defect name, location result and supplementary description, and then the report generation prompt template is called to input the input list into the text generation model, and the text generation model outputs a bridge defect identification report, which can be a structured text report and includes component information, defect items, location information, defect cause analysis and treatment suggestions.

[0021] In some embodiments of this application, constructing a structured domain knowledge base includes: Component tables, disease definition tables, and negative constraint tables are established based on relational databases; The component table stores the unique codes, standard terms, and classification information of bridge components; The standard defect name and typical visual feature description corresponding to each bridge component are stored in the defect definition table; The negative constraint table stores the descriptions of non-defect scenarios corresponding to each bridge component.

[0022] Specifically, the structured domain knowledge base can be established and maintained by a server or database management unit, while the relational database can use MySQL, PostgreSQL, or other database systems that support inter-table joins. The component table archives basic information about bridge components. A unique code is an identifier field that distinguishes each bridge component and can be generated by preset coding rules, such as combining component category and sequential number. Standard terminology is a unified naming convention for the same component, and classification information records the category to which the component belongs. The defect definition table stores standard defect names and typical visual feature descriptions corresponding to different bridge components. Typical visual feature descriptions can include the extension shape of cracks, the edge state of spalling, the color characteristics and distribution of rust, etc. The negative constraint table stores scene descriptions that are similar in appearance to defects but are not considered defects, such as construction joints, stains, water stains, shadows, reflective areas, or changes in material texture, for retrieval during subsequent searches. Both the defect definition table and the negative constraint table are linked by component codes. The associated fields are linked to the component table. These associated fields can be set using primary keys and foreign keys, allowing each bridge component to retrieve its own specific defect definition information and non-defect scene information. During database construction, technicians can input, verify, and update defect names, visual feature descriptions, and non-defect scene descriptions for different bridge components based on bridge inspection specifications, existing defect samples, and historical inspection records. When a new component type or defect type is added, the database can be expanded by adding records without changing the existing table structure. The specific content of each field can be determined based on bridge defect image samples, manual annotation results, and engineering experience. If a component has multiple common defects, multiple corresponding records can be created for the same component code in the defect definition table. If a component has multiple easily confused non-defect scenes, multiple corresponding records can be created in the negative constraint table. This ensures that subsequent searches based on component codes can return the defect definition set, typical visual feature set, and non-defect scene description set corresponding to that component.

[0023] In some embodiments of this application, when the defect definition table is associated with the component code in the component table, and the negative constraint table is associated with the component code in the component table, the following is included: Establish corresponding association records for each bridge component; enter the defect definition and visual feature description bound to the corresponding bridge component in the defect definition table; enter the negative constraint text content bound to the corresponding bridge component in the negative constraint table.

[0024] Specifically, the association between the defect definition table and the component codes in the component table, as well as the association between the negative constraint table and the component codes in the component table, refers to using the component codes in the component table as a unified index identifier for bridge components. Corresponding association fields are set in the defect definition table and the negative constraint table, respectively, to allow unified retrieval of defect information and non-defect scenario information for the same bridge component. Establishing corresponding association records for each bridge component can be understood as pre-creating corresponding data records for each type of bridge component. For example, separate records are created for different components such as bridge decks, main beams, diaphragms, supports, and piers, and each record is bound to its corresponding component code. When entering the defect definition and visual feature description bound to the corresponding bridge component in the defect definition table, the defect definition can be the standard name of common defects of that component, and the visual features... The description can be a textual description of the appearance characteristics corresponding to the defect, such as the extension state of cracks, the edge shape of peeling, the color characteristics and distribution of rust, etc. If the same bridge component corresponds to multiple defects, multiple defect definition records can be created for the same component code. When entering the negative constraint text content bound to the corresponding bridge component in the negative constraint table, the negative constraint text content refers to the scene description that is similar to the appearance of the defect but does not belong to the defect, such as construction joints, stains, water stains, shadows, reflective areas, or changes in material texture, etc. If the same bridge component has multiple easily confused non-defect scenes, multiple negative constraint records can be created for the same component code. Through the above association method, after obtaining the bridge component code, all defect definitions, visual feature descriptions, and negative constraint text content bound to it can be directly queried based on the component code.

[0025] In some embodiments of this application, the process of calling a bridge component identification prompt template and generating bridge component identification prompts based on standard terminology for bridge components in the component table, and inputting the bridge appearance defect image to be identified and the bridge component identification prompts into a multimodal large model fine-tuned by the bridge appearance defect data, includes: Deploy the fine-tuning platform, select an open-source multimodal large model as the base model on the fine-tuning platform, and upload the bridge appearance defect training dataset; Set the number of training rounds, output image size, and learning rate, and perform model training based on the set fine-tuning parameters to obtain a fine-tuned multimodal large model; Call the bridge component identification prompt word template, fill the standard terms of bridge components in the component table into the corresponding placeholders according to the preset fields, and generate bridge component identification prompt words; Input the bridge appearance defect image to be identified and the bridge component identification prompts into the fine-tuned multimodal large model, identify the bridge components in the bridge appearance defect image, and output the bridge component identification results and the corresponding standard component codes.

[0026] Specifically, the multimodal large-scale model fine-tuned with bridge appearance defect data refers to a dedicated model for bridge defect recognition tasks, formed by using an open-source multimodal large-scale model as the base model and updating parameters on a fine-tuning platform in conjunction with the bridge appearance defect training dataset. The fine-tuning platform can be a training environment that supports visual language model training, parameter configuration, and result verification, such as LLaMA-Factory. The base model can be an open-source multimodal large-scale model with image-text joint understanding capabilities, such as Qwen3-VL-32B-Thinking. The training dataset consists of pre-prepared bridge appearance defect samples, whose original materials can come from long-term accumulated defect data from infrastructure inspection units. Data processing, sample selection, and annotation are completed before training to meet the needs of model fine-tuning. In the parameter setting phase, the number of training epochs, output image size, and learning rate are the main fine-tuning parameters. The number of training epochs is used to control the model's learning rate on the training data. The number of iterations for the training dataset can be determined comprehensively based on the training sample size, the convergence of the loss function, and the validation results. For example, when the sample size is moderate, 3 to 10 iterations can be used as the initial range, and then adjusted according to the performance of the trained model on the validation dataset. If the number of training iterations is too small and the model does not fully learn the bridge defect features, it can be increased appropriately. If the number of training iterations is too large and overfitting occurs, it can be decreased appropriately. The output image size is used to control the uniform resolution of the input image before it enters the base model. This parameter is usually matched with the visual input specifications supported by the base model, and can be set to, for example, 448×448, 512×512, or 768×768. The specific value can be determined according to the default input size of the base model, memory resources, and the requirements for preserving bridge defect details. The learning rate is used to control the magnitude of model parameter updates during training. It can be set in combination with the base model size, the number of training samples, and the convergence stability. For example, it can be initially set to 1×10. -5 Up to 5×10 -5The values ​​within the range are then adjusted based on the verification results. If the learning rate is too large, causing training oscillations, the learning rate is reduced; if the learning rate is too small, causing slow convergence, it is appropriately increased. After setting the above parameters, model training is performed based on the set fine-tuning parameters. After training, the model is tested using a pre-prepared verification dataset. The fine-tuning results are confirmed through quantitative evaluation and qualitative effect comparison analysis. Quantitative evaluation indicators include accuracy, information coverage, and logical structure, while qualitative effect indicators include reasoning ability, output structure and standardization, and professional business knowledge. When the quantitative evaluation reaches the preset standard and the qualitative effect meets the preset requirements, the fine-tuning is considered complete, and a multimodal large model for subsequent bridge component identification, defect identification, and defect localization is obtained. In the bridge component identification stage, the bridge component identification prompt word template is a pre-set component identification text framework, which includes at least a candidate component field. The candidate component field is used to write the standard terminology of bridge components stored in the component table. When generating bridge component identification prompts, the system reads standard terminology data from the component table and fills it into the corresponding positions in the bridge component identification prompt template according to the preset field correspondence, forming a candidate set of bridge components corresponding to the current image. The preset fields can be organized by item listing, list arrangement, or field-based organization, as long as they can clearly define the scope of identifiable bridge components. Subsequently, the image of the bridge appearance defects to be identified and the bridge component identification prompts are input together into the fine-tuned multimodal large model. The model identifies the bridge components in the image within the candidate set of bridge components and outputs the bridge component identification results. After obtaining the bridge component name, the system further extracts the standard component code corresponding to the bridge component name according to the pre-established correspondence in the component table, or the multimodal large model directly outputs the standard component code, thereby obtaining the bridge component identification results and the corresponding standard component codes for subsequent structured domain knowledge retrieval steps.

[0027] In some embodiments of this application, when performing model training based on the set fine-tuning parameters, the method further includes: After training is completed, the trained model is tested using a pre-prepared validation dataset to perform quantitative and qualitative performance evaluations. Quantitative assessment includes accuracy assessment, information coverage assessment, and logical structure assessment; qualitative performance assessment includes reasoning ability assessment, output structure and standardization assessment, and business knowledge professionalism assessment. When the quantitative evaluation results meet the preset standards and the qualitative results meet the preset requirements, the fine-tuning is considered complete.

[0028] Specifically, after setting fine-tuning parameters such as the number of training epochs, output image size, and learning rate, the fine-tuning platform performs iterative training on the basic multimodal large model based on the bridge appearance defect training dataset. After training, it calls a pre-divided validation dataset to test the trained model. The validation dataset is independent of the training dataset; its samples can come from images and corresponding annotation information from the same bridge defect data source that were not used in training, or from subsequently added data samples that have undergone manual verification, to ensure the comparability and stability of the validation results. During the testing process, the bridge appearance defect images from the validation dataset are first input into the trained model to obtain... The model output is then compared with the standard labeled results in the validation dataset to complete both quantitative and qualitative performance evaluations. Accuracy evaluation assesses the consistency between the model output and the standard labeled content; information coverage evaluation assesses whether the model output includes pre-defined key disease information items; logical structure evaluation assesses the organization, coherence, and completeness of the output; reasoning ability evaluation assesses whether the model can form an analysis process that meets the requirements of the bridge disease identification task based on the input image and prompts; and output structuring and standardization evaluation assesses whether the model output format conforms to... The evaluation assesses whether the pre-defined field organization and terminology are consistent. The professional knowledge assessment evaluates whether the output content conforms to professional expressions and common judgment habits in the field of bridge defect detection. Pre-defined standards and requirements can be set before model training begins, based on the size of the validation dataset, historical model test results, and actual business requirements. Specifically, this can be determined by first performing a baseline test on the same validation dataset using an un-fine-tuned base model or an existing defect identification model to obtain the corresponding accuracy, information coverage, and logical structure evaluation results. Then, based on these baseline results, the conditions for completing fine-tuning are set, such as requiring the fine-tuned model to meet certain criteria. The model must achieve the preset minimum score in each of the quantitative assessments, or improve by a predetermined margin relative to the baseline model. At the same time, it must pass the manual review in the three qualitative assessments of reasoning ability, output structure and standardization, and professional business knowledge. The manual review can be conducted by bridge defect detection technical personnel who score or judge the level item by item according to the preset evaluation form. When the quantitative assessment results meet the preset standards and the qualitative effects meet the preset requirements, the model obtained from the current training is determined to be a fine-tuned multimodal large model. If the preset standards or preset requirements are not met, the model returns to the fine-tuning parameter setting stage, and the number of training rounds, learning rate, or input image size are adjusted before retraining and verification.

[0029] In some embodiments of this application, querying a structured domain knowledge base based on standard component codes includes: The database is queried in real time based on the standard component code. The disease definition, visual feature description and negative constraint text content bound to the standard component code are retrieved, and the component name corresponding to the standard component code is extracted.

[0030] Specifically, the standard component code is standardized identification information output during the bridge component identification stage, serving as the sole index for subsequent knowledge retrieval. After obtaining the standard component code, the system initiates a real-time query request to the relational database. Real-time query means that immediately after component identification is completed and the component code is output for the current bridge appearance defect image, a database query statement is invoked based on that component code to retrieve data records associated with that component code from the component table, defect definition table, and negative constraint table. During the query process, the component record corresponding to the standard component code can be located in the component table first, and the component name in that record can be extracted. Then, based on the same component code, the defect definition and visual feature description bound to that bridge component can be retrieved in the defect definition table, and the negative constraint text content bound to that bridge component can be retrieved in the negative constraint table. The defect definition describes the possible defects that may occur on that bridge component. The standardized description of the disease type includes visual feature descriptions, which are textual descriptions of the disease's appearance in the image, such as the morphological features of cracks, the edge state of peeling, and the color changes and distribution characteristics of rust. The negative constraint text content describes scenes that are similar in appearance to the disease but are not considered diseases, such as construction joints, stains, water stains, shadows, reflective areas, or changes in material texture. Since the same bridge component may correspond to multiple disease types and multiple non-disease scenes, the database query results can be one or more records. After the retrieval is completed, the system can collect multiple disease definitions, multiple visual feature descriptions, and multiple negative constraint texts corresponding to the same component code to form disease definition sets, visual feature description sets, and negative constraint text sets, which, together with the extracted component name, serve as input content for the subsequent dynamic prompt word generation stage.

[0031] In some embodiments of this application, calling a visual diagnostic prompt template containing placeholders includes: A preset visual diagnostic prompt template includes placeholders for component names, disease definitions, visual feature descriptions, and negative constraints. The retrieved component names, defect definition sets, visual feature description sets, and negative constraint text sets are filled into the corresponding placeholders according to the preset field correspondence, generating dynamic prompt words corresponding to the current bridge component.

[0032] Specifically, the visual diagnostic prompt template is a pre-configured text template with reserved field positions corresponding to the bridge defect diagnosis task. These field positions are used to receive content such as component name, defect definition, visual feature description, and negative constraint text. The component name placeholder is used to write the name of the currently identified bridge component, the defect definition placeholder is used to write one or more standard defect names and their definitions corresponding to the bridge component, the visual feature description placeholder is used to write the appearance description information corresponding to the defect definition, and the negative constraint placeholder is used to write the non-defect scene description related to the bridge component. A pre-defined visual diagnostic prompt template containing the aforementioned placeholders can be pre-prepared by technical personnel during the system deployment phase. Its content is organized using a fixed field order and a fixed semantic framework to ensure clear field boundaries and stable semantic relationships when subsequent content is written. During dynamic prompt generation, the system receives the component names, disease definition sets, visual feature description sets, and negative constraint text sets returned from the domain knowledge retrieval phase. It then writes each item into the corresponding position in the template according to the pre-defined field correspondence. If the disease definition set or visual feature description set contains multiple records, they can be arranged sequentially according to a pre-defined sorting rule and then the corresponding words can be filled in. The sorting rules for each segment can be determined based on the prevalence of the defect in the bridge component, the preset input order, or the database return order, as long as they remain consistent during system operation. Similarly, if the negative constraint text set contains multiple records, they can be organized in the same way and then filled into negative constraint placeholders. The filling method can be either field-by-field replacement or template splicing. Specifically, the search results are written into the corresponding field areas according to the mapping relationship between field names and template positions, thus forming a complete text prompt that corresponds one-to-one with the current bridge component. The "dynamic" in dynamic prompt words refers to the fact that the content of the prompt words is not fixed. The dynamic prompts are not fixed, but change with the standard component codes identified and the knowledge content retrieved from those codes. That is, the component names, defect definitions, visual feature descriptions, and negative constraint text content corresponding to different bridge components are different, and the generated prompts are also different accordingly. In this embodiment, the generated dynamic prompts can be organized in the form of segmented text, list text, or field-based text, as long as the content of each field can be clearly distinguished and the logical order between fields is maintained. Then, the dynamic prompts and the original bridge appearance defect images are input into the multimodal large model for subsequent defect identification and defect localization.

[0033] In some embodiments of this application, when inputting images of bridge exterior defects and dynamic prompts corresponding to each bridge component into a finely tuned multimodal large model, and performing defect identification and localization on each bridge component, the process includes: For the identified bridge components, the bridge appearance defect images and corresponding dynamic prompts are input into the fine-tuned multimodal large model, and defect identification and defect localization are performed. The defect identification results and defect localization results for each bridge component are output.

[0034] Specifically, after completing bridge component identification and dynamic prompt word generation, the system establishes corresponding diagnostic tasks for multiple bridge components identified from the same bridge appearance defect image. Each diagnostic task consists of input data composed of the original bridge appearance defect image and the dynamic prompt word corresponding to the current bridge component. Inputting the same bridge appearance defect image is to preserve the complete contextual information of the area where the bridge component is located. Inputting the corresponding dynamic prompt word is to limit the bridge component category, defect definition range, visual feature description content, and non-defect scene constraints for the current task. During execution, the system sends the input data corresponding to each bridge component to a fine-tuned multimodal large model. The multimodal large model performs defect identification and defect localization on the current bridge component based on the image content and dynamic prompt word content. Defect identification refers to outputting whether the current bridge component has defects and what kind of defects exist. The disease identification results can include a disease presence / absence judgment, disease category, disease name, or multiple candidate disease items. Disease localization refers to outputting the location range of the disease in the image. The disease localization results can take the form of location description, target box coordinates, region contour, mask region, or other information forms that can characterize the disease location. When the bridge appearance disease image contains multiple bridge components, the system can call the corresponding dynamic prompt words one by one according to the component list output in the component identification stage and perform model inference separately. Alternatively, it can schedule and process the diagnosis tasks of multiple bridge components in parallel according to the system's computing power, as long as each bridge component corresponds one-to-one with its dynamic prompt words. After the model outputs, the system records the disease identification results and disease localization results corresponding to each bridge component, and organizes them according to component name, component code, or preset output order to form the structured input data required for the subsequent disease identification report generation stage.

[0035] In some embodiments of this application, when summarizing the defect identification results and defect location results corresponding to each bridge component, calling the report generation prompt word template, and inputting the summarized results into the text generation model to generate a bridge defect identification report, the process includes: The defect identification and location results for each bridge component are compiled into an input list. The report generation prompt template is invoked to generate a model from the input list of text input; Output a bridge defect identification report based on a text generation model.

[0036] Specifically, after completing the identification and localization of defects in each bridge component, the system first summarizes and organizes the output results corresponding to each bridge component to form an input list. This input list is a structured data set built for the report generation stage. It records the component name, component code, defect identification results, and defect localization results of each bridge component according to a preset field order. Defect identification results can include the presence or absence of defects, defect name, defect category, or defect description information. Defect localization results can include the location description of the defect area, target box coordinates, area outline, or mask area information. When multiple bridge components are included in the same bridge appearance defect image, the results can be arranged according to the component order formed during the component identification stage, the component code order, or the preset report output order to ensure consistent organization of subsequent report content. The report generation prompt template is a pre-defined report generation text framework used to specify the field structure, semantic order, and expression method used in the report output. It can include component information fields, defect information fields, localization information fields, and analysis description fields, thus enabling the text generation model to organize and output the input content in a unified format. When calling the report... When generating the prompt word template, the system writes each item in the aforementioned input list into the corresponding position in the template according to the field correspondence, forming a text input for the report generation task, and then sends this text input to the text generation model. The text generation model is the model that performs the organization and output of the report text. It can be a standalone text generation model or a text output unit after a multimodal large model switches to text generation processing mode, as long as it can generate a complete report based on the input list and the report generation prompt word template. During the generation process, the text generation model outputs the corresponding bridge defect identification report in sequence based on the identification and positioning results of each bridge component recorded in the input list. The bridge defect identification report can be a structured text report, the content of which can include the name of the bridge component, the identified defect item, the location of the defect, and the corresponding cause analysis and treatment suggestions. The cause analysis and treatment suggestions are generated based on the defect identification and positioning results already in the input list and the preset organization rules in the report generation prompt word template. When the defect information corresponding to multiple bridge components is written and the text generation is completed, a bridge defect identification report for the current bridge appearance defect image is obtained.

[0037] In another preferred embodiment based on the above embodiments, see [reference] Figure 2 As shown, this embodiment provides a bridge defect image recognition system based on prompt words and a large model, including: The structured domain knowledge base management module is configured to build a structured domain knowledge base, establishing a component table, a defect definition table, and a negative constraint table based on a relational database. The component table stores the unique code, standard terminology, and classification information of bridge components. The defect definition table is associated with the component code in the component table and stores the standard defect name and typical visual feature description corresponding to each bridge component. The negative constraint table is associated with the component code in the component table and stores the non-defect scene description corresponding to each bridge component. The bridge component recognition module is configured to call the bridge component recognition prompt word template and generate bridge component recognition prompt words based on the standard terminology of bridge components in the component table. The module inputs the bridge appearance defect image to be recognized and the bridge component recognition prompt words into the multimodal large model after fine-tuning the bridge appearance defect data, identifies the bridge components in the bridge appearance defect image, and outputs the bridge component recognition result and the corresponding standard component code. The domain knowledge retrieval module is configured to query the structured domain knowledge base based on the standard component code to obtain the component name, defect definition set, visual feature description set, and negative constraint text set of the corresponding bridge component; The dynamic prompt word generation module is configured to call a visual diagnostic prompt word template containing placeholders, fill the component name, defect definition set, visual feature description set and negative constraint text set into the corresponding placeholders according to preset fields, and generate dynamic prompt words corresponding to each bridge component. The defect identification and localization module is configured to input images of bridge exterior defects and dynamic prompts corresponding to each bridge component into a finely tuned multimodal large model, concurrently perform defect identification and localization on each bridge component, and obtain the defect identification results and defect localization results corresponding to each bridge component. The defect identification report generation module is configured to summarize the defect identification results and defect location results corresponding to each bridge component, call the report generation prompt word template, input the summarized results into the text generation model, and generate a bridge defect identification report.

[0038] It is understandable that, through the collaborative efforts of the structured domain knowledge base management module, bridge component identification module, domain knowledge retrieval module, dynamic prompt word generation module, defect identification and location module, and defect identification report generation module, this implementation method can integrate bridge component identification prompts, domain knowledge invocation, dynamic prompt word generation, defect identification and location, and report output into the same system framework. The system comprises several modules: a structured domain knowledge base management module for unified storage and maintenance of bridge component information, defect definition information, and non-defect scene information, facilitating subsequent retrieval based on component codes; a bridge component recognition module for generating identification prompts by calling bridge component recognition prompt templates and combining them with standard terminology from the component table, enabling the multimodal large model to output bridge component recognition results and corresponding standard component codes within a preset range of bridge components; a domain knowledge retrieval module for directly retrieving defect definitions, visual feature descriptions, and negative constraint text content corresponding to identified bridge components; a dynamic prompt generation module for further organizing the retrieval results into dynamic prompts corresponding to the current bridge component, ensuring that subsequent defect recognition and localization processes align with the professional knowledge content of the current component; a defect recognition and localization module for outputting defect recognition and localization results for multiple bridge components; and a defect recognition report generation module for summarizing and organizing the results for each component and outputting a bridge defect recognition report. Through this modular setup, the input-output relationships between each processing stage are clearly defined, enabling the systematic deployment of bridge defect image recognition tasks.

[0039] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the protection scope defined by the present invention.

Claims

1. A method for image recognition of bridge defects based on prompt words and large models, characterized in that, include: Construct a structured domain knowledge base, and establish component tables, disease definition tables, and negative constraint tables based on a relational database; The component table stores the unique codes, standard terms, and classification information of bridge components. The defect definition table is associated with the component codes in the component table and stores the standard defect names and typical visual feature descriptions corresponding to each bridge component. The negative constraint table is associated with the component codes in the component table and stores the non-defect scene descriptions corresponding to each bridge component. The bridge component identification prompt word template is called, and the bridge component identification prompt words are generated according to the standard terminology of bridge components in the component table. The bridge appearance defect image to be identified and the bridge component identification prompt words are input into the multimodal large model after the bridge appearance defect data is fine-tuned. The bridge components in the bridge appearance defect image are identified, and the bridge component identification results and the corresponding standard component codes are output. The structured domain knowledge base is queried according to the standard component code to obtain the component name, defect definition set, visual feature description set and negative constraint text set of the corresponding bridge component; Call the visual diagnostic prompt word template containing placeholders, fill the component name, defect definition set, visual feature description set and negative constraint text set into the corresponding placeholders according to preset fields, and generate dynamic prompt words corresponding to each bridge component; The bridge's external defects images and the dynamic prompts corresponding to each bridge component are input into the fine-tuned multimodal large model. Defect identification and defect localization are performed concurrently on each bridge component to obtain the defect identification results and defect localization results corresponding to each bridge component. Summarize the defect identification and location results for each bridge component, call the report generation prompt word template, input the summarized results into the text generation model, and generate a bridge defect identification report.

2. The bridge defect image recognition method based on prompt words and large models according to claim 1, characterized in that, When building a structured domain knowledge base, the following are included: Component tables, disease definition tables, and negative constraint tables are established based on relational databases; The component table stores the unique codes, standard terms, and classification information of bridge components; The defect definition table stores the standard defect names and typical visual feature descriptions corresponding to each bridge component. The negative constraint table stores the descriptions of non-defect scenarios corresponding to each bridge component.

3. The bridge defect image recognition method based on prompt words and large models according to claim 2, characterized in that, When the defect definition table is associated with the component codes in the component table, and the negative constraint table is associated with the component codes in the component table, the association includes: Establish corresponding association records for each bridge component; enter the defect definition and visual feature description bound to the corresponding bridge component in the defect definition table; enter the negative constraint text content bound to the corresponding bridge component in the negative constraint table.

4. The bridge defect image recognition method based on prompt words and large models according to claim 3, characterized in that, The process of calling the bridge component identification prompt template and generating bridge component identification prompts based on the standard terminology of bridge components in the component table, and inputting the bridge appearance defect image to be identified and the bridge component identification prompts into the multimodal large model after fine-tuning the bridge appearance defect data, includes: Deploy a fine-tuning platform, select an open-source multimodal large model as the base model on the fine-tuning platform, and upload the bridge appearance defect training dataset; Set the number of training rounds, output image size, and learning rate, and perform model training based on the set fine-tuning parameters to obtain a fine-tuned multimodal large model; Call the bridge component identification prompt word template, fill the standard terms of bridge components in the component table into the corresponding placeholders according to the preset fields, and generate bridge component identification prompt words; The bridge appearance defect image to be identified and the bridge component identification prompts are input into the fine-tuned multimodal large model to identify the bridge components in the bridge appearance defect image and output the bridge component identification results and the corresponding standard component codes.

5. The bridge defect image recognition method based on prompt words and large models according to claim 4, characterized in that, When performing model training based on the set fine-tuned parameters, it also includes: After training is completed, the trained model is tested using a pre-prepared validation dataset to perform quantitative and qualitative performance evaluations. The quantitative assessment includes accuracy assessment, information coverage assessment, and logical structure assessment; the qualitative performance assessment includes reasoning ability assessment, output structure and standardization assessment, and business knowledge professionalism assessment. When the quantitative evaluation results meet the preset standards and the qualitative results meet the preset requirements, the fine-tuning is considered complete.

6. The bridge defect image recognition method based on prompt words and large models according to claim 5, characterized in that, When querying the structured domain knowledge base based on the standard component code, the following is included: The database is queried in real time according to the standard component code to retrieve the disease definition, visual feature description and negative constraint text content bound to the standard component code, and the component name corresponding to the standard component code is extracted.

7. The bridge defect image recognition method based on prompt words and large models according to claim 6, characterized in that, When calling a visual diagnostic cue template that includes placeholders, the following is included: A preset visual diagnostic prompt template includes placeholders for component names, disease definitions, visual feature descriptions, and negative constraints. The retrieved component names, defect definition sets, visual feature description sets, and negative constraint text sets are filled into the corresponding placeholders according to the preset field correspondence, generating dynamic prompt words corresponding to the current bridge component.

8. The bridge defect image recognition method based on prompt words and large models according to claim 7, characterized in that, The bridge's external defects images and the corresponding dynamic prompts for each bridge component are input into the fine-tuned multimodal large model. When performing defect identification and localization on each bridge component, the process includes: For each of the identified bridge components, the images of the bridge's external defects and the corresponding dynamic prompts are input into the fine-tuned multimodal large model, and defect identification and localization are performed. The defect identification results and defect localization results for each bridge component are then output.

9. The method for bridge defect image recognition based on prompt words and large models according to claim 8, characterized in that, The report summarizes the defect identification and location results for each bridge component, calls the report generation prompt template, inputs the summarized results into the text generation model, and generates a bridge defect identification report, including: The defect identification and location results for each bridge component are compiled into an input list. The input list is input into the text generation model by calling the report generation prompt word template; The bridge defect identification report is output based on the text generation model.

10. A bridge defect image recognition system based on prompt words and a large model, used to implement the bridge defect image recognition method based on prompt words and a large model as described in any one of claims 1-9, characterized in that, include: The structured domain knowledge base management module is configured to build a structured domain knowledge base, establishing component tables, disease definition tables, and negative constraint tables based on a relational database; The component table stores the unique codes, standard terms, and classification information of bridge components. The defect definition table is associated with the component codes in the component table and stores the standard defect names and typical visual feature descriptions corresponding to each bridge component. The negative constraint table is associated with the component codes in the component table and stores the non-defect scene descriptions corresponding to each bridge component. The bridge component identification module is configured to call the bridge component identification prompt word template and generate bridge component identification prompt words according to the standard terminology of bridge components in the component table. The module inputs the bridge appearance defect image to be identified and the bridge component identification prompt words into the multimodal large model after fine-tuning the bridge appearance defect data, identifies the bridge components in the bridge appearance defect image, and outputs the bridge component identification result and the corresponding standard component code. The domain knowledge retrieval module is configured to query the structured domain knowledge base based on the standard component code to obtain the component name, defect definition set, visual feature description set, and negative constraint text set of the corresponding bridge component; The dynamic prompt word generation module is configured to call a visual diagnostic prompt word template containing placeholders, fill the component name, disease definition set, visual feature description set and negative constraint text set into the corresponding placeholders according to preset fields, and generate dynamic prompt words corresponding to each bridge component. The defect identification and localization module is configured to input the bridge appearance defect images and dynamic prompts corresponding to each bridge component into the fine-tuned multimodal large model, concurrently perform defect identification and defect localization on each bridge component, and obtain the defect identification results and defect localization results corresponding to each bridge component. The defect identification report generation module is configured to summarize the defect identification results and defect location results corresponding to each bridge component, call the report generation prompt word template, input the summarized results into the text generation model, and generate a bridge defect identification report.

Images