A full-automatic labeling method for infrastructure appearance disease image

Abstract

This invention relates to the field of intelligent image annotation technology, and discloses a fully automated annotation method for images of infrastructure appearance defects. The method includes: constructing a domain knowledge base and generating dynamic prompts; preprocessing the image; inputting the image and dynamic prompts into a multimodal large-scale model to obtain preliminary annotation results; constructing a defect knowledge graph; using a graph retrieval-enhanced intelligent agent in conjunction with the defect knowledge graph to verify and correct the causal analysis and treatment suggestions in the preliminary annotation results; sampling and reviewing the corrected results; and outputting structured annotation data after passing the review. This invention solves the problems of insufficient professionalism, standardization, and consistency of annotation data in existing technologies by constructing a domain knowledge base and a defect knowledge graph, combined with a collaborative process of dynamic prompt generation, multimodal large-scale model preliminary annotation, graph retrieval-enhanced intelligent agent correction, and random sampling review.

Description

Technical Field

[0001] This invention relates to the field of intelligent image annotation technology, and more specifically, to a fully automated annotation method for images of infrastructure appearance defects. Background Technology

[0002] With the development of digital and intelligent identification technologies for infrastructure inspection, images of external defects in infrastructure such as bridges, tunnels, and road surfaces have been widely used for training and optimizing defect identification models. Model training typically relies on a large amount of labeled data as a foundation. The annotation content of infrastructure defect images not only involves defect type identification but also includes professional information such as defect description, cause analysis, and treatment suggestions. Therefore, high requirements are placed on the professionalism, standardization, and consistency of the labeled data.

[0003] In existing technologies, image annotation methods for infrastructure defects mainly include manual annotation and automated annotation. Manual annotation typically relies on professionals to judge and label each image individually, which suffers from low processing efficiency, high labor costs, and difficulty in standardizing annotation criteria among different personnel. While automated annotation can improve processing speed, in the specialized scenario of infrastructure defects, it is still prone to inaccurate identification results, non-standard text descriptions, and inconsistencies in logic. Especially when directly using general multimodal models to output annotation results, content inconsistent with domain knowledge may appear, thus affecting the usability of the annotated data.

[0004] Meanwhile, as infrastructure defect identification models develop towards lower cost and greater autonomy, the quality requirements for professional training data are further increasing. If the labeled data itself lacks unified standards or cannot accurately reflect the correspondence between defect types and related text information, it will affect the subsequent model training effect and practical application value. Summary of the Invention

[0005] In view of this, the present invention proposes a fully automated annotation method for infrastructure appearance defect images, aiming to solve the problem that the existing technology lacks the professionalism, consistency and standardization of infrastructure appearance defect image annotation data, and it is difficult to reliably obtain annotation data that meets the requirements of professional standards.

[0006] In one aspect, this invention proposes a fully automated annotation method for images of infrastructure appearance defects, comprising: Obtain infrastructure maintenance specifications, standards, and disease manuals; extract component types, disease definitions, disease descriptions, causal information, and remedial measures; construct non-disease constraint information based on misidentification results from historical identification tasks; and construct a domain knowledge base based on the extracted component types, disease definitions, disease descriptions, causal information, remedial measures, and the aforementioned non-disease constraint information. Based on the annotation task corresponding to the image to be annotated, information is extracted from the domain knowledge base and filled into the preset prompt word template to generate dynamic prompt words; Acquire images to be labeled and perform preprocessing, including disease type identification, image classification, format unification, and task creation; The preprocessed image and the dynamic prompt words are input into the multimodal large model to obtain preliminary annotation results including disease type, disease description, cause analysis and treatment suggestions; A disease knowledge graph is constructed based on the aforementioned infrastructure maintenance specifications, standards, and disease manual. The preliminary annotation results are input into the graph retrieval enhancement generation agent. The graph retrieval enhancement generation agent and the disease knowledge graph verify and correct the cause analysis and treatment suggestions in the preliminary annotation results to obtain the corrected annotation results. The corrected annotation results are randomly sampled according to a preset sampling ratio and input into a general large language model for review. After the review reaches a preset review threshold, the annotation text corresponding to the corrected annotation results is paired with the original image, stored, and output in a structured manner to obtain structured annotation data.

[0007] Furthermore, the construction of the domain knowledge base includes: Establish a component table for storing component types and a disease definition table for storing disease definitions and descriptions, based on infrastructure maintenance specifications, maintenance standards and disease manuals. Based on the misidentification results in historical identification tasks, non-disease constraint information is summarized, and a negative constraint table is established to store non-disease constraint information. The domain knowledge base is constructed based on the component table, the disease definition table, and the negative constraint table.

[0008] Furthermore, generating the dynamic prompt words includes: The content to be identified is determined based on the annotation task corresponding to the image to be annotated; Extract information corresponding to the annotation task from the component table, the disease definition table, and the negative constraint table; The extracted information is filled into the component identification prompt word template, the disease identification prompt word template, and the report generation prompt word template in a preset order to obtain the dynamic prompt words.

[0009] Furthermore, the process of acquiring and preprocessing the image to be labeled includes: Obtain the image to be labeled and determine whether the image to be labeled carries a disease type label; If the image to be labeled carries a disease type label, then the image to be labeled is classified according to the disease type label; If the image to be labeled does not carry a disease type label, the multimodal large model is called to identify the disease type of the image to be labeled, and the identification result is written into the corresponding label and then classified. Standardize the format of the categorized images to be labeled; Annotation tasks are created based on the images to be annotated after the format has been standardized.

[0010] Furthermore, when the preprocessed image and the dynamic prompts are input into the multimodal large model to obtain preliminary annotation results containing disease type, disease description, causal analysis, and treatment suggestions, the following are included: Based on the sequence of component identification, disease identification, and report generation, a multimodal large model is invoked to analyze the preprocessed image and output structured preliminary annotation results containing disease type, disease description, cause analysis, and treatment suggestions.

[0011] Furthermore, when constructing a disease knowledge graph based on the aforementioned infrastructure maintenance specifications, standards, and disease manuals, the following is included: The rule and attribute information in the infrastructure maintenance specifications, standards and disease manuals are extracted, and the rule and attribute information is transformed into reasoning rules and attribute constraints using a knowledge graph as a carrier to obtain a disease knowledge graph.

[0012] Furthermore, when inputting the preliminary annotation results into the graph retrieval enhancement generation agent, the process includes: Using the disease knowledge graph as a knowledge source, and combining the understanding and generation capabilities of a large-scale language model, a graph retrieval-enhanced generative agent is constructed. The preliminary annotation results are input into the graph retrieval enhancement generation agent, so that the graph retrieval enhancement generation agent outputs the corresponding verification results or correction results.

[0013] Furthermore, when the graph retrieval enhancement-generated agent and the disease knowledge graph verify and correct the causal analysis and treatment suggestions in the preliminary annotation results, the verification and correction of the causal analysis includes: Using the disease type in the preliminary annotation results as the starting point, the corresponding standard causal chain is retrieved in the disease knowledge graph; The causal analysis is compared with the standard causal chain to determine whether there is a reasonable causal relationship between the causal analysis and the disease type. In cases of causal inversion, logical jumps, or inconsistencies with domain rules, a replacement causal analysis is generated and replaces the original causal analysis in the preliminary annotation results.

[0014] Furthermore, when the graph retrieval enhancement generating agent and the disease knowledge graph verify and correct the causal analysis and treatment suggestions in the preliminary annotation results, the verification and correction of the treatment suggestions includes: The treatment recommendations are matched with the standard treatment measures for the corresponding disease type in the disease knowledge graph to determine whether the treatment process, materials and methods in the treatment recommendations are consistent with the standard treatment measures. In case of inconsistency, generate alternative handling suggestions and replace the original handling suggestions in the preliminary annotation results.

[0015] Furthermore, when randomly sampling the corrected annotation results according to a preset sampling ratio and inputting them into a general large language model for review, the process includes: The text in the corrected annotation results is randomly sampled according to a preset sampling ratio to form a sampling dataset. The sampled dataset is input into an independent general-purpose large language model for review, and the review conclusion is output based on consistency, professionalism and logic. After the review reaches the preset review threshold, the annotation text corresponding to the corrected annotation result is paired with the original image, stored, and output in a structured manner to obtain structured annotation data.

[0016] Compared with existing technologies, the beneficial effects of this invention are as follows: By acquiring infrastructure maintenance specifications, standards, and disease manuals, extracting component types, disease definitions, disease descriptions, causal information, and treatment measures, and combining this with misidentification results from historical identification tasks to construct non-disease constraint information, a domain knowledge base for infrastructure appearance disease image annotation can be formed; by extracting information from the domain knowledge base and filling it into a preset prompt word template according to the annotation task corresponding to the image to be annotated, dynamic prompt words can be generated, improving the matching of prompt content with the current annotation task; and by performing disease type identification, image classification, and format standardization on the image to be annotated, the invention can achieve the following: The first step, task creation, can improve the standardization of the image processing process. By inputting preprocessed images and dynamic prompts into a multimodal large model, preliminary annotation results containing disease types, disease descriptions, causal analysis, and treatment suggestions can be obtained. By constructing a disease knowledge graph and combining it with graph retrieval-enhanced generative agents to verify and correct the causal analysis and treatment suggestions in the preliminary annotation results, the consistency between the annotation results and domain knowledge can be improved. By randomly sampling the corrected annotation results according to a preset sampling ratio and inputting them into a general large language model for review, the quality control of the output results can be performed, thereby obtaining structured annotation data. Attached Figure Description

[0017] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 A flowchart illustrating a fully automated annotation method for infrastructure appearance defects images provided in this embodiment of the invention. Detailed Implementation

[0018] Exemplary embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the disclosure to those skilled in the art. It should be noted that, unless otherwise specified, embodiments and features in the embodiments of the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0019] See Figure 1 As shown, this application proposes a fully automated annotation method for images of infrastructure appearance defects, including: S1: Obtain infrastructure maintenance specifications, standards, and disease manuals; extract component types, disease definitions, disease descriptions, causal information, and remedial measures; construct non-disease constraint information based on misidentification results from historical identification tasks; and construct a domain knowledge base based on the extracted component types, disease definitions, disease descriptions, causal information, remedial measures, and non-disease constraint information. S2: Based on the annotation task corresponding to the image to be annotated, extract information from the domain knowledge base and fill it into the preset prompt word template to generate dynamic prompt words; S3: Acquire the image to be labeled and perform preprocessing, including disease type identification, image classification, format unification and task creation; S4: Input the preprocessed image and dynamic prompts into the multimodal large model to obtain preliminary annotation results including disease type, disease description, cause analysis and treatment suggestions; S5: Construct a disease knowledge graph based on infrastructure maintenance specifications, standards, and disease manuals; S6: Input the preliminary annotation results into the graph retrieval enhancement generation agent. The graph retrieval enhancement generation agent and the disease knowledge graph verify and correct the cause analysis and treatment suggestions in the preliminary annotation results to obtain the corrected annotation results. S7: Randomly sample the corrected annotation results according to the preset sampling ratio, and input them into the general large language model for review. After the review reaches the preset review threshold, pair the annotation text corresponding to the corrected annotation results with the original image, store it, and output it in a structured manner to obtain structured annotation data.

[0020] Specifically, infrastructure maintenance specifications, standards, and defect manuals can be current specifications, maintenance operation standards, and defect assessment data for infrastructure such as bridges, tunnels, pavements, or slopes. Extracting component types, defect definitions, defect descriptions, causal information, and remedial measures refers to compiling professional knowledge from the aforementioned data that can be used for program calls. Component types characterize the infrastructure part to which the defect belongs, such as beams, piers, linings, pavement surfaces, or protective components; defect definitions define the meaning of the corresponding defect assessment; and defect descriptions define the appearance of the defect. Causal information is used to limit the causes or triggering conditions of disease formation, while treatment measures are used to limit the treatment plan corresponding to the disease type. Constructing non-disease constraint information based on misidentification results in historical identification tasks refers to summarizing and organizing the misjudgment situations that occurred in the historical identification process of the multimodal large model to form constraints to limit the output of misidentification. Non-disease constraint information is used to limit scene information that is easily misidentified as a disease but is actually not a disease, such as stains, water stains, shadows, construction marks, or visual features formed by changes in lighting. Based on the extracted component type, disease definition, The construction of a domain knowledge base, which includes disease descriptions, causal information, treatment measures, and non-disease constraint information, refers to organizing and storing this information according to a pre-defined data structure. The domain knowledge base can be constructed using relational databases, key-value databases, or other structured storage methods. It provides a unified source of professional knowledge for prompt word generation and result verification. A labeling task is a processing task established for a single image to be labeled or a collection of similar images. The labeling task at least determines the labeling fields to be output and the corresponding recognition objects. Based on the labeling task corresponding to the image to be labeled, information is extracted from the domain knowledge base and filled into a pre-defined prompt word template to generate dynamic prompt words. This involves pre-setting a prompt word template with a fixed instruction framework and, based on the component type, disease type, or scene category corresponding to the current labeling task, calling relevant knowledge content from the domain knowledge base and filling it into the corresponding position in the template to form text instructions adapted to the current image recognition requirements. Dynamic prompt words differ from fixed prompt words; the content of dynamic prompt words changes with the task to which the image to be labeled belongs, thus enabling the multimodal large model to obtain domain constraints and output requirements matching the current task when performing image understanding.Acquiring and preprocessing images to be labeled refers to receiving raw images from inspection terminals, image acquisition devices, or historical datasets, and performing preprocessing on these raw images to suit subsequent automatic labeling. Disease type identification can be understood as determining the category of a disease in an image. For images already labeled with disease types, classification can be completed directly based on the existing labels. For images without disease type labels, a multimodal large model can be used to first identify the corresponding disease type, and then the identification results are written into the corresponding labels before classification. Image classification refers to categorizing and organizing images according to disease type, component type, or task category. Format unification refers to unifying images from different sources and with different encoding methods. Alternatively, images with different file extensions can be converted to a unified file format and naming convention, such as JPEG or PNG, and the image identification field can be standardized for subsequent batch retrieval. Task creation refers to generating corresponding annotation task records based on the preprocessed images. Inputting the preprocessed images and dynamic prompts into a multimodal large model yields preliminary annotation results containing disease type, disease description, causal analysis, and treatment suggestions. This refers to the joint processing of image information and text instruction information by the multimodal large model, enabling the model to output structured text results based on image content. Disease type indicates the disease category corresponding to the image, and disease description reflects the visual representation of the disease in the image. The visual manifestation and causal analysis are used to explain the cause-and-effect relationships of diseases, while the treatment suggestions provide corresponding treatment content. Preliminary annotation results can be field-based text, table-structured results, or JSON format results. Constructing a disease knowledge graph based on infrastructure maintenance specifications, standards, and disease manuals refers to organizing disease types, causal information, treatment measures, and corresponding relationships into nodes and associations, thus forming a searchable and reasonable graph structure. Nodes in the disease knowledge graph can correspond to disease names, component names, causal factors, or treatment measures, while associations can correspond to causal relationships, applicability relationships, or constraint relationships. Graph retrieval enhanced generative agents refer to… The processing unit, which combines graph retrieval and language generation capabilities, can retrieve standard knowledge content related to the current disease type based on the disease knowledge graph after receiving the initial annotation results. It then compares the retrieval results with the causal analysis and treatment suggestions in the initial annotation results to complete the verification and correction. Verification and correction refer to identifying whether there are inconsistencies in causal logic, missing content, or inconsistencies with the standards in the causal analysis, and whether there are inconsistencies in the treatment suggestions with the disease type, incorrect treatment direction, or inconsistencies with standard measures. When the above situations occur, the corresponding content is replaced, supplemented, or adjusted to obtain the corrected annotation results.Randomly sampling the corrected annotation results according to a preset sampling ratio and inputting them into a general large language model for review means selecting a portion of the annotated text as a sample for inspection from all corrected annotation results according to a pre-set ratio. The preset sampling ratio can be set according to the data scale, review cost, and quality control requirements. For example, when the amount of annotation data in a single batch is large, it can be 5% to 10%; when the amount of annotation data in a single batch is small or the quality requirements for the results are high, it can be 10% to 20%. The specific value can be determined through historical batch review experience, trial operation statistics, or manual verification results. The preset review threshold is a threshold parameter used to determine whether the corrected annotation results of the current batch meet the output conditions. The preset review threshold can be adopted by using a combination of methods. Evaluation metrics include pass rate, field consistency rate, or logical correctness rate. For example, a pass rate of 90%, 95%, or other pre-set values ​​can be used as the passing condition. The preset review threshold can be determined based on the actual application scenario's requirements for annotation accuracy, through preliminary testing, experience setting, or multiple rounds of verification statistical results. When the review reaches the preset review threshold, it indicates that the current batch of corrected annotation results meets the predetermined quality requirements. At this point, the annotation text corresponding to the corrected annotation results is paired one-to-one with the original image, and stored and structured output according to a unified data format. The structured annotation data can be saved in the form of JSON files, database records, or tabular field data for use in disease identification model training, verification, or data management.

[0021] In some embodiments of this application, constructing a domain knowledge base includes: Establish a component table for storing component types and a disease definition table for storing disease definitions and descriptions, based on infrastructure maintenance specifications, maintenance standards and disease manuals. Based on the misidentification results in historical identification tasks, non-disease constraint information is summarized, and a negative constraint table is established to store non-disease constraint information. A domain knowledge base is constructed based on the component table, the disease definition table, and the negative constraint table.

[0022] Specifically, based on infrastructure maintenance specifications, standards, and disease manuals, a component table is established to store component types, and a disease definition table is established to store disease definitions and descriptions. This refers to using the professional content in infrastructure maintenance specifications, standards, and disease manuals as data sources, and establishing interconnected structured data tables in a relational database. The component table stores basic attribute information of infrastructure components. Unique codes are used to distinguish different components; these unique codes can be in the form of numerical codes, alphanumeric codes, or a combination of alphanumeric codes. Standard terminology refers to the unified name used for the corresponding component in the specification documents. Classification information refers to the category level to which the component belongs, such as bridge components, tunnel components, pavement components, or further subdivided into beams, piers, linings, surface layers, protective components, etc. The disease definition table stores disease knowledge content corresponding to the components and establishes a correspondence between the disease definition table and the component table through associated component identifiers, so that each disease record can point to its corresponding component. Standard disease names refer to the unified naming of diseases used in the specification documents or disease manuals. Typical visual features refer to the characteristics of the disease in the image. The observable manifestations include linear fracture textures corresponding to cracks, local defect boundaries corresponding to peeling, and strip-like or sheet-like watermarks corresponding to water seepage. Based on the misidentification results from historical recognition tasks, non-disease constraint information is summarized, and a negative constraint table is established to store this information. This involves summarizing and organizing the misjudgments that occurred during the historical recognition process of the multimodal large model, and writing the summarized easily confused non-disease scene information into the negative constraint table. The negative constraint table stores scene descriptions that should not be judged as defects. Non-disease scene descriptions can include stains, water stains, shadows, changes in lighting, construction marks, or surface attachments, etc., to constrain the model output. In the specific establishment process, a component list can be compiled according to the specification documents and written into the component table. Then, based on the defect definitions and visual manifestations corresponding to each component, it can be written into the defect definition table. Finally, non-disease scene information is compiled based on the misidentification results from historical recognition tasks and written into the negative constraint table. The associated component identifier can use the unique code in the component table as the association field, thereby enabling the defect definition information and non-disease constraint information under the same component to be associated and managed with the corresponding component.

[0023] In some embodiments of this application, generating dynamic prompt words includes: The content to be identified is determined based on the annotation task corresponding to the image to be annotated; Extract information corresponding to the annotation task from the component table, the defect definition table, and the negative constraint table; The extracted information is filled into the component identification prompt template, the disease identification prompt template, and the report generation prompt template in a preset order to obtain dynamic prompts.

[0024] Specifically, determining the content to be identified based on the annotation task corresponding to the image to be annotated means first reading the task information corresponding to the image to be annotated, and then clarifying the scope of objects to be identified and the output content of the current image. The annotation task may include the infrastructure type to which the image belongs, the component category, the scope of the disease category to be judged, and the text fields to be output. The content to be identified may be one or more of the following: component name, disease type, disease manifestation, cause information, and treatment measures. Extracting information corresponding to the annotation task from the component table, disease definition table, and negative constraint table means retrieving the standard terms and classification information in the component table, the standard disease name and typical visual features in the disease definition table, and the non-disease scene description in the negative constraint table based on the component category or disease category determined in the annotation task. The information corresponding to the annotation task refers to the information content that matches the component, suspected disease type, or scene characteristics of the current image. The preset order refers to the pre-set information organization order. The preset order can be arranged in the order of component information, disease definition information, and negative constraint information, or in the order of recognition target, judgment basis, and exclusion conditions. Arrangement; Component identification prompt word templates, disease identification prompt word templates, and report generation prompt word templates refer to pre-set text frames corresponding to different identification stages. The component identification prompt word template may include component name determination requirements and candidate component terms, the disease identification prompt word template may include disease name, visual features, and exclusion conditions, and the report generation prompt word template may include output field requirements for disease description, cause analysis, and treatment suggestions. Filling the extracted information into each prompt word template in a preset order means replacing or writing the retrieved field content into the corresponding placeholder positions in the template to form prompt word content corresponding to the current annotation task. For example, filling in the candidate component name in the component identification prompt word template, filling in the definition and visual features of the corresponding disease in the disease identification prompt word template, and filling in the disease description, cause, and treatment items to be output in the report generation prompt word template. Dynamic prompt words refer to prompt word content generated in real time according to the annotation task corresponding to different images to be annotated. Compared with fixed prompt words, dynamic prompt words have at least some field content that changes with the component to which the image belongs, the disease category, or the task requirements.

[0025] In some embodiments of this application, the process of acquiring and preprocessing the image to be labeled includes: Acquire the image to be labeled and determine whether the image to be labeled carries a disease type label; If the image to be labeled carries a disease type label, then the image to be labeled is classified according to the disease type label; If the image to be labeled does not carry a disease type label, the multimodal large model is called to identify the disease type of the image to be labeled, and the identification result is written into the corresponding label and then classified. Standardize the format of the categorized images to be labeled; Annotation tasks are created based on the images to be annotated after the format has been standardized.

[0026] Specifically, acquiring the image to be labeled refers to receiving infrastructure appearance images provided by inspection equipment, mobile terminals, drone acquisition equipment, or existing disease image datasets. The image to be labeled can be a single image or a collection of images imported in batches. Determining whether the image to be labeled carries a disease type label involves reading the image file's auxiliary fields, filename identifiers, database records, or external index information to determine if the image to be labeled has been pre-labeled with a disease category. A disease type label refers to marker information that characterizes the category to which a disease belongs in the image, such as category identifiers for cracks, spalling, seepage, and exposed rebar. When the image to be labeled carries a disease type label, classifying the image according to the disease type label means assigning the image to the corresponding image set or task category according to the disease category corresponding to the label. For example, images labeled "cracks" are assigned to the crack image set, and images labeled "spalling" are assigned to the spalling image set. When the image to be labeled does not carry a disease type label, a multimodal large model is used to identify the disease type of the image to be labeled, and the identification result is written into the corresponding label before classification. This means inputting the image to be labeled into an image set with image processing capabilities. The multimodal large model with high resolution outputs disease category results corresponding to the images. These disease category results are then written into the image's label field, index record, or file association information, so that the images to be labeled have disease type identifiers that can be retrieved and classified. The format of the categorized images to be labeled is standardized, which means normalizing images from different sources, with different file extensions, different encoding rules, or different resolution organization methods to meet unified data storage and processing requirements. Format standardization can include unified file format, unified naming rules, unified storage path, unified image size description method, or unified metadata field name. For example, images in different formats such as BMP and TIFF can be converted to JPEG or PNG formats and renamed according to preset naming rules. Based on the format-standardized images to be labeled, corresponding labeling tasks are created. This means generating corresponding task records for each image or each group of similar images. The labeling task can include information such as image identifier, disease category to which the image belongs, image storage location, type of field to be output, and task number, thus forming a one-to-one or one-to-one correspondence between the images to be labeled and a group of corresponding task items.

[0027] In some embodiments of this application, when inputting preprocessed images and dynamic prompts into a multimodal large model to obtain preliminary annotation results containing disease type, disease description, causal analysis, and treatment suggestions, the process includes: Based on the sequence of component identification, disease identification, and report generation, a multimodal large model is invoked to analyze the preprocessed image and output structured preliminary annotation results containing disease type, disease description, cause analysis, and treatment suggestions.

[0028] Specifically, based on the sequence of component identification, defect identification, and report generation, the multimodal large model is invoked to analyze the preprocessed image. This means that the preprocessed image and corresponding dynamic prompts are sequentially input into the multimodal large model according to a pre-set processing flow, allowing the multimodal large model to complete the understanding of the image content and text output in stages. Component identification refers to judging the infrastructure objects presented in the image to determine the component category corresponding to the defect, such as bridge beams, piers, tunnel linings, road surface layers, or slope protection components. Defect identification refers to judging the type of defect present in the image based on the determined component category, in order to clarify the category of the defect reflected in the image. Report generation refers to generating text content corresponding to the defect according to preset output requirements based on the component identification results and defect identification results. Sequential invocation means that the multimodal large model first outputs the component identification results, then outputs the defect identification results based on the component identification results, and finally outputs the defect description, cause analysis, and treatment suggestions based on the component identification results and defect identification results, so that the output content has a clear generation layer. The disease type is used to characterize the category of the disease in the image; the disease description is used to characterize the visible appearance of the disease in the image, such as morphological features, distribution status, positional relationship, or surface changes; the causal analysis is used to characterize the causal information related to the formation of the disease; and the treatment suggestion is used to characterize the treatment content corresponding to the disease category. The structured preliminary annotation result refers to the result data organized according to pre-defined fields. The structured preliminary annotation result includes at least the disease type field, disease description field, causal analysis field, and treatment suggestion field, and the content of each field is output according to a fixed format to facilitate recording, reading, and processing. The fixed format can be a key-value pair format, a table structure format, or a JSON format, for example, organized in the field order of disease type—disease description—causal analysis—treatment suggestion. In some embodiments, after receiving image information and dynamic prompts, the multimodal large model can first extract the visual features in the image, and then combine the component range, disease definition, typical visual features, and output requirements defined in the dynamic prompts to generate the corresponding field content, thereby forming the structured preliminary annotation result.

[0029] In some embodiments of this application, when constructing a disease knowledge graph based on infrastructure maintenance specifications, maintenance standards, and disease manuals, the following are included: Rule and attribute information is extracted from infrastructure maintenance specifications, standards, and disease manuals. Knowledge graphs are then used as a carrier to transform the rule and attribute information into reasoning rules and attribute constraints, resulting in a disease knowledge graph.

[0030] Specifically, extracting rule and attribute information from infrastructure maintenance specifications, standards, and disease manuals involves organizing and structurally expressing the disease judgment conditions, the correspondence between diseases and components, the association between diseases and causes, and the correspondence between diseases and treatment measures in relevant specification documents, standard documents, and disease data. Rule information refers to information that reflects the logic of disease judgment or relational logic, such as common diseases corresponding to a certain type of component, the causes usually associated with a certain type of disease, and the treatment measures applicable to a certain type of disease. Attribute information refers to information used to describe the characteristics of the diseased object itself, such as disease name, component name, visual appearance, cause category, treatment category, and constraints. Using a knowledge graph as a carrier to transform rule and attribute information into reasoning rules and attribute constraints means abstracting various objects in disease knowledge into nodes in the graph, and abstracting the correspondence, subordination, association, or restriction relationships between objects into edges between nodes, thereby forming a graph-structured knowledge organization form. Nodes can include component nodes, disease nodes, cause nodes, and treatment nodes. Edges can include relation types such as occurring at, corresponding to, causing, or applicable to. Inference rules refer to logical judgment rules pre-set according to the content of the specification, used to represent the logical associations that can be established between different nodes. For example, once the disease type is determined, the cause information and treatment measures corresponding to the disease type can be retrieved along the graph relationship. Attribute constraints refer to the limiting conditions attached to nodes or relationships, used to limit the scope of application, object range, or matching conditions of diseases, causes, or treatment measures. For example, limiting a certain type of disease to only specific components, or limiting a certain type of treatment measures to only correspond to a specific disease type. In the specific construction process, component names, disease names, cause descriptions, and treatment content can be extracted from infrastructure maintenance specifications, maintenance standards, and disease manuals. Then, the extraction results can be processed by terminology unification, relationship sorting, and field standardization. After that, they can be written into a graph database or other graph structure storage carrier according to the preset graph pattern to obtain a disease knowledge graph.

[0031] In some embodiments of this application, when inputting the preliminary annotation results into the graph retrieval enhancement generative agent, the following steps are included: Using disease knowledge graphs as knowledge sources, and combining the understanding and generation capabilities of large-scale language models, we construct graph retrieval-enhanced generative agents. The initial annotation results are input into the graph retrieval enhancement generative agent, so that the graph retrieval enhancement generative agent outputs the corresponding verification results or correction results.

[0032] Specifically, using a disease knowledge graph as a knowledge source and combining it with the understanding and generation capabilities of a large-scale language model to construct a graph retrieval-enhanced generative agent refers to connecting a pre-constructed disease knowledge graph to a large-scale language model with text understanding, logical analysis, and text generation capabilities to form a processing unit capable of performing graph retrieval, result comparison, and text correction. The knowledge source refers to the data source that provides the agent with professional knowledge. In this embodiment, the knowledge source is a disease knowledge graph, which stores disease types, component types, causal information, treatment measures, and the relationships between disease types, component types, causal information, and treatment measures. The understanding capability of the large-scale language model is... This refers to the ability to perform semantic parsing, field recognition, and logical judgment on input text content. Generation capability refers to the ability to output verification or corrected text based on retrieval results. Graph retrieval enhancement in the generative agent refers to, before or during the processing of preliminary annotation results by a large language model, retrieving node information, relationship information, and constraint information corresponding to the disease type, cause analysis, and treatment suggestions from the disease knowledge graph based on the preliminary annotation results. Then, the retrieved graph knowledge is provided to the large language model for comprehensive judgment, so that the output result is based not only on the language model's own parameters but also on the disease knowledge graph. Explicit knowledge; when constructing a graph retrieval-enhanced generative agent, a graph database interface, a knowledge retrieval module, and a large language model invocation module can be combined and deployed. The graph database interface is used to access the disease knowledge graph, the knowledge retrieval module is used to generate search conditions based on the input content and return matching results, and the large language model invocation module is used to receive preliminary annotation results and graph retrieval results and output the corresponding text; inputting the preliminary annotation results into the graph retrieval-enhanced generative agent means that the disease type, disease description, cause analysis, and treatment suggestions in the preliminary annotation results are passed to the agent as input content. The agent parses the fields that need to be verified and, based on the parsing results, retrieves the disease knowledge... The graph retrieves causal information and treatment measures corresponding to the current disease type. The verification result refers to the judgment result formed on whether the preliminary annotation result is consistent with the knowledge content in the disease knowledge graph. The verification result can include results such as consistent, inconsistent, partially consistent, logically conflicting, or missing items. The correction result refers to the result obtained by the graph retrieval enhancement generation agent regenerating or adjusting the corresponding fields based on the retrieved graph knowledge when it is determined that the preliminary annotation result has inconsistencies, logical errors, or missing content. The correction result can be the correction result of the causal analysis field, the correction result of the treatment suggestion field, or the result that contains correction content of multiple fields at the same time.

[0033] In some embodiments of this application, when the graph retrieval-enhanced intelligent agent and the disease knowledge graph verify and correct the causal analysis and treatment suggestions in the preliminary annotation results, the verification and correction of the causal analysis includes: Starting with the disease type in the preliminary annotation results, the corresponding standard causal chain is retrieved in the disease knowledge graph; The causal analysis is compared with the standard causal chain to determine whether there is a reasonable causal relationship between the causal analysis and the disease type. When there are causal inversions, logical jumps, or inconsistencies with domain rules, a replacement causal analysis is generated and replaces the original causal analysis in the initial annotation results.

[0034] Specifically, starting with the disease type in the preliminary annotation results, the search retrieves the corresponding standard causal chain in the disease knowledge graph. This means first reading the disease type field already determined in the preliminary annotation results and using the disease type as a search condition in the graph. The search then queries the disease knowledge graph for causal nodes and relationship paths associated with the disease type. A standard causal chain refers to a sequence of causal relationships pre-established in the disease knowledge graph for a specific disease type. The standard causal chain characterizes the standardized causal relationship between inducing factors and the resulting disease, such as sequential relationships like environmental effects, material degradation, abnormal structural stress, and surface damage evolution. Comparing the causal analysis with the standard causal chain involves semantically parsing the causal analysis text in the preliminary annotation results, extracting the causal elements, acting objects, and logical relationships involved, and then matching them with the node content, relationship direction, and constraints in the standard causal chain to determine if the causal analysis is consistent with the knowledge rules corresponding to the current disease type. Consistency means that the causes recorded in the causal analysis are consistent. The content, causal direction, and logical order match the standard causal chain and conform to the domain rules in the disease knowledge graph. In cases of consistency, the original causal analysis in the preliminary annotation results is retained unchanged, and the original causal analysis is written into the corrected annotation results as the verified causal analysis. Inconsistency refers to situations where there is causal reversal, logical jump, or inconsistency with domain rules. Causal reversal means that the disease result is mistakenly written as the cause, or the subsequent result is mistakenly taken as the preceding cause. Logical jump means that the causal analysis lacks necessary intermediate links, resulting in a lack of complete deductive relationship between the cause and the disease result. Inconsistency with domain rules means that the content of the causal analysis does not conform to the disease causal rules recorded in the infrastructure maintenance specifications, maintenance standards, and disease manuals. When inconsistency is identified, the graph retrieval enhanced generative agent regenerates and replaces the causal analysis based on the standard causal chain obtained from the retrieval, and uses the replaced causal analysis to replace the original causal analysis in the preliminary annotation results. The replaced causal analysis refers to the standardized text content generated based on the causal nodes and relationship paths in the disease knowledge graph that match the current disease type.

[0035] In some embodiments of this application, when the graph retrieval-enhanced generative agent and the disease knowledge graph verify and correct the causal analysis and treatment suggestions in the preliminary annotation results, the verification and correction of the treatment suggestions includes: The treatment recommendations are matched with the standard treatment measures for the corresponding disease type in the disease knowledge graph to determine whether the treatment process, materials and methods in the treatment recommendations are consistent with the standard treatment measures. In case of inconsistency, generate alternative treatment suggestions and replace the original treatment suggestions in the preliminary annotation results.

[0036] Specifically, matching treatment recommendations with standard treatment measures for the corresponding disease type in the disease knowledge graph involves first reading the disease type and treatment recommendation fields from the preliminary annotation results, and then retrieving the standard treatment measures associated with the disease type from the disease knowledge graph based on the disease type. Standard treatment measures refer to standardized treatment content pre-organized and written into the disease knowledge graph according to infrastructure maintenance specifications, maintenance standards, and disease manuals. Standard treatment measures include information such as treatment processes, treatment materials, construction methods, and applicable conditions. Determining whether the treatment processes, materials, and methods in the treatment recommendations are consistent with the standard treatment measures involves semantic parsing of the treatment recommendations in the preliminary annotation results, extracting the process content, material names, and treatment methods from the treatment recommendations, and matching them item by item with the corresponding elements in the standard treatment measures. Consistency means that the treatment processes, materials, and methods in the treatment recommendations match the standard treatment measures corresponding to the disease type, and are not... If the application scope exceeds the range recorded in the disease knowledge graph, and there is consistency, the original treatment suggestion in the preliminary annotation results will be retained unchanged, and the original treatment suggestion will be written into the revised annotation results as the verified treatment suggestion. Inconsistency refers to the treatment process, materials, or methods in the treatment suggestion not matching the standard treatment measures, or the existence of incorrect treatment objects, incompatible treatment methods, or unsuitable material selection. When inconsistency is identified, the graph retrieval enhanced generation agent generates a replacement treatment suggestion based on the retrieved standard treatment measures, and uses the replacement treatment suggestion to replace the original treatment suggestion in the preliminary annotation results. The replacement treatment suggestion refers to the standardized text content formed according to the standard treatment measures corresponding to the disease type. The replacement treatment suggestion can be the treatment text corresponding to the standard treatment measures directly, or it can be the treatment text formed by deleting and supplementing non-standard processes, materials, or methods while retaining the correct content in the original treatment suggestion.

[0037] In some embodiments of this application, when randomly sampling the corrected annotation results according to a preset sampling ratio and inputting them into a general large language model for review, the process includes: The text in the corrected annotation results is randomly sampled according to a preset sampling ratio to form a sampling dataset. The sampled dataset is input into an independent general-purpose large language model for review, and the review conclusions are output based on consistency, professionalism and logic. After the review reaches the preset review threshold, the annotation text corresponding to the corrected annotation result will be paired with the original image, stored and output in a structured manner to obtain structured annotation data.

[0038] Specifically, randomly sampling the text in the corrected annotation results according to a preset sampling ratio to form a sampling inspection dataset means randomly selecting a portion of the annotated text as a quality inspection sample from a batch of completed and corrected annotation results according to a pre-set ratio. The preset sampling ratio can be set according to the amount of data in a single batch, the distribution of defect types, the allocation of audit resources, and quality control requirements. For example, when the number of corrected annotation results in a single batch is large, the preset sampling ratio can be set to 5% to 10%; when the number of corrected annotation results in a single batch is small or the output quality requirements are high, the preset sampling ratio can be set to 10% to 20%. The sampling ratio can also be adjusted based on historical audit records. For example, when the pass rate of several consecutive batches is high, the preset sampling ratio can be appropriately reduced. When the error rate of a certain type of defect is high in historical audits, the sampling ratio for the corresponding defect type can be increased. Random sampling refers to selecting samples from the text set corresponding to the corrected annotation results using random number selection with equal probability or stratified random selection, in order to avoid the sampled samples being concentrated in a certain defect category, a certain component category, or a certain batch source. The sampling dataset refers to the data set composed of the annotated text obtained by sampling. The sampling dataset can retain the defect type, component category, and other data corresponding to the annotated text. The document type, task number, or image identification information is used to facilitate corresponding verification during the review process. Inputting the sampled dataset into an independent general-purpose large language model for review means inputting the labeled text from the sampled dataset into a general-purpose large language model different from the aforementioned multimodal large model and graph retrieval enhanced generative agent. The general-purpose large language model then judges the quality of the labeled text content. Independence means that the review model is separated from the aforementioned generation and correction stages in terms of model source, parameter configuration, or calling process, thus ensuring the independence of the review conclusions. Consistency means that the disease types, disease descriptions, causal analyses, and treatment suggestions in the labeled text correspond to each other in content, without... In cases of inconsistencies, contradictions, or mismatches between disease types and text content, professionalism refers to the use of terminology, disease descriptions, causal descriptions, and treatment content in the labeled text conforming to the professional expression requirements of infrastructure maintenance specifications, maintenance standards, and disease manuals. Logicality refers to the existence of a reasonable causal relationship and correspondence between the causal analysis and disease types, and between treatment recommendations and disease types, without logical breaks, causal confusion, or incorrect treatment directions. The review conclusion can include results such as pass, fail, field issues, logical problems, or need for correction, and the pass rate for the corresponding batch can be further calculated.The preset review threshold is a threshold parameter used to determine whether the current batch of corrected annotation results meets the output conditions. The preset review threshold can use the review pass rate as the criterion. It can be set based on previous trial operation results, manual review results, historical data statistics, or actual business quality requirements. For example, a review pass rate of 90%, 95%, or 98% can be used as the preset review threshold. When the review reaches the preset review threshold, it indicates that the current batch of corrected annotation results meets the predetermined quality requirements. At this point, the annotation text corresponding to the corrected annotation results is paired one-to-one with the original image according to image identifiers or task numbers, and stored and output in a structured manner according to a unified data format. The structured annotation data can be saved in the form of JSON files, database table records, or fielded text records.

[0039] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the protection scope of the above content of the present invention.

Claims

1. A fully automated annotation method for images of infrastructure appearance defects, characterized in that, include: Obtain infrastructure maintenance specifications, standards, and disease manuals, and extract component types, disease definitions, disease descriptions, causal information, and treatment measures; Non-disease constraint information is constructed based on the misidentification results in historical identification tasks; a domain knowledge base is constructed based on the extracted component types, disease definitions, disease descriptions, causal information, treatment measures, and the aforementioned non-disease constraint information. Based on the annotation task corresponding to the image to be annotated, information is extracted from the domain knowledge base and filled into the preset prompt word template to generate dynamic prompt words; Acquire images to be labeled and perform preprocessing, including disease type identification, image classification, format unification, and task creation; The preprocessed image and the dynamic prompt words are input into the multimodal large model to obtain preliminary annotation results containing disease type, disease description, cause analysis and treatment suggestions; A disease knowledge graph is constructed based on the aforementioned infrastructure maintenance specifications, standards, and disease manual. The preliminary annotation results are input into the graph retrieval enhancement generation agent. The graph retrieval enhancement generation agent and the disease knowledge graph verify and correct the cause analysis and treatment suggestions in the preliminary annotation results to obtain the corrected annotation results. The corrected annotation results are randomly sampled according to a preset sampling ratio and input into a general large language model for review. After the review reaches a preset review threshold, the annotation text corresponding to the corrected annotation results is paired with the original image, stored, and output in a structured manner to obtain structured annotation data.

2. The fully automated annotation method for infrastructure appearance defects images according to claim 1, characterized in that, When constructing the domain knowledge base, the following are included: Establish a component table for storing component types and a disease definition table for storing disease definitions and descriptions, based on infrastructure maintenance specifications, maintenance standards and disease manuals. Based on the misidentification results in historical identification tasks, non-disease constraint information is summarized, and a negative constraint table is established to store non-disease constraint information. The domain knowledge base is constructed based on the component table, the disease definition table, and the negative constraint table.

3. The fully automated annotation method for infrastructure appearance defects images according to claim 2, characterized in that, When generating the dynamic prompt words, the following steps are included: The content to be identified is determined based on the annotation task corresponding to the image to be annotated; Extract information corresponding to the annotation task from the component table, the disease definition table, and the negative constraint table; The extracted information is filled into the component identification prompt template, the disease identification prompt template, and the report generation prompt template in a preset order to obtain the dynamic prompts.

4. The fully automated annotation method for infrastructure appearance defects images according to claim 3, characterized in that, The process of acquiring and preprocessing the image to be labeled includes: Obtain the image to be labeled and determine whether the image to be labeled carries a disease type label; If the image to be labeled carries a disease type label, then the image to be labeled is classified according to the disease type label; If the image to be labeled does not carry a disease type label, the multimodal large model is called to identify the disease type of the image to be labeled, and the identification result is written into the corresponding label and then classified. Standardize the format of the categorized images to be labeled; Annotation tasks are created based on the images to be annotated after the format has been standardized.

5. The fully automated annotation method for infrastructure appearance defects images according to claim 4, characterized in that, When the preprocessed image and the dynamic prompts are input into a multimodal large model to obtain preliminary annotation results containing disease type, disease description, causal analysis, and treatment suggestions, the results include: Based on the sequence of component identification, disease identification, and report generation, a multimodal large model is invoked to analyze the preprocessed image and output structured preliminary annotation results containing disease type, disease description, cause analysis, and treatment suggestions.

6. The fully automated annotation method for infrastructure appearance defects images according to claim 5, characterized in that, When constructing a disease knowledge graph based on the aforementioned infrastructure maintenance specifications, standards, and disease manuals, the following is included: The rule and attribute information in the infrastructure maintenance specifications, standards and disease manuals are extracted, and the rule and attribute information is transformed into reasoning rules and attribute constraints using a knowledge graph as a carrier to obtain a disease knowledge graph.

7. The fully automated annotation method for infrastructure appearance defects images according to claim 6, characterized in that, When inputting the preliminary annotation results into the graph retrieval enhancement agent, the following steps are included: Using the disease knowledge graph as a knowledge source, and combining the understanding and generation capabilities of a large-scale language model, a graph retrieval-enhanced generative agent is constructed. The preliminary annotation results are input into the graph retrieval enhancement generation agent, so that the graph retrieval enhancement generation agent outputs the corresponding verification results or correction results.

8. The fully automated annotation method for infrastructure appearance defects images according to claim 7, characterized in that, When the graph retrieval-enhanced intelligent agent and the disease knowledge graph verify and correct the causal analysis and treatment suggestions in the preliminary annotation results, the verification and correction of the causal analysis includes: Using the disease type in the preliminary annotation results as the starting point, the corresponding standard causal chain is retrieved in the disease knowledge graph; The causal analysis is compared with the standard causal chain to determine whether there is a reasonable causal relationship between the causal analysis and the disease type. In cases of causal inversion, logical jumps, or inconsistencies with domain rules, a replacement causal analysis is generated and replaces the original causal analysis in the preliminary annotation results.

9. The fully automated annotation method for infrastructure appearance defects images according to claim 8, characterized in that, When the graph retrieval-enhanced intelligent agent and the disease knowledge graph verify and correct the causal analysis and treatment suggestions in the preliminary annotation results, the verification and correction of the treatment suggestions includes: The treatment recommendations are matched with the standard treatment measures for the corresponding disease type in the disease knowledge graph to determine whether the treatment process, materials and methods in the treatment recommendations are consistent with the standard treatment measures. In case of inconsistency, generate alternative handling suggestions and replace the original handling suggestions in the preliminary annotation results.

10. The fully automated annotation method for infrastructure appearance defects images according to claim 9, characterized in that, When randomly sampling the corrected annotation results according to a preset sampling ratio and inputting them into a general large language model for review, the process includes: The text in the corrected annotation results is randomly sampled according to a preset sampling ratio to form a sampling dataset. The sampled dataset is input into an independent general-purpose large language model for review, and the review conclusion is output based on consistency, professionalism and logic. After the review reaches the preset review threshold, the annotation text corresponding to the corrected annotation result is paired with the original image, stored, and output in a structured manner to obtain structured annotation data.

Images