Engineering defect image automatic labeling method and system

Abstract

The invention discloses an engineering defect image automatic labeling method and system based on a visual language large model, and the method comprises the steps: 1, collecting an original image data set of a target domain, carrying out the standardization processing, and converting the original image data into unified semantic representation; step 2, based on the original image data set and the semantic representation after standardization processing, generating candidate tags and corresponding text descriptions; and step 3, based on the generated text description, calculating the defect category to which the inference image belongs in combination with rule matching and semantic similarity. According to the method, high-precision engineering defect image labeling is realized under the condition of no or a small amount of labeling, the labor cost is reduced, a high-quality training data set is constructed, and the method is suitable for identification and labeling tasks of various defect types such as cracks, peeling, steel bar exposure, corrosion and the like.

Description

Technical Field

[0001] This invention relates to a method and system for automatically annotating engineering defect images by using a large-scale visual language model (LVLM) combined with pattern matching and self-correction mechanisms, belonging to the interdisciplinary field of computer vision and structural safety inspection. Background Technology

[0002] With the development of artificial intelligence and computer vision technologies, the detection of defects in engineering structures has gradually shifted from manual inspection to automated detection. Traditional methods mainly rely on deep learning models to train and identify defect images. However, deep learning models are highly dependent on large-scale, high-quality labeled datasets. In actual engineering scenarios, the acquisition of defect images presents the following problems: First, the number of defect images in different engineering scenarios is limited, especially in special or rare defect categories, where labeled samples are scarce, leading to insufficient model generalization ability; second, manual labeling is time-consuming and labor-intensive, and the labeling results are easily affected by the experience level of the labelers, resulting in subjective bias and insufficient consistency; third, most existing automatic labeling methods rely on a single classifier or rule system, making it difficult to simultaneously guarantee the accuracy and robustness of the labeling.

[0003] In recent years, the development of visual language models has provided a new technical path for image annotation. These models can process both image and text information simultaneously, possessing cross-modal understanding capabilities, and theoretically can directly achieve image description and label generation. However, in engineering scenarios with inherent limitations, existing attempts based on visual language models still have shortcomings: first, the lack of standardized input and processing procedures makes it difficult to guarantee the consistency and reproducibility of generated labels; second, the lack of verification and correction mechanisms for ambiguous or easily confused samples; and third, the lack of effective integration with engineering data management systems, making it difficult to directly support subsequent model training and applications.

[0004] Therefore, a new automatic annotation method and system is needed that can combine the cross-modal capabilities of visual language models and introduce standardized preprocessing, candidate label generation, semantic reasoning, consistency verification, multi-round correction, and result aggregation and integration to form a closed-loop automated process, so as to solve the shortcomings of existing technologies in terms of efficiency, accuracy and application integration. Summary of the Invention

[0005] The technical problem to be solved by this invention is: how to achieve automatic annotation of engineering defect images by utilizing the cross-modal understanding capability of the Visual-Language Large Model (LVLM) and combining it with automated reasoning and result verification mechanisms in the absence of large-scale manually annotated data, thereby reducing manual costs and improving the efficiency of dataset construction.

[0006] To address the aforementioned technical problems, this invention provides an automatic annotation method for engineering defect images based on a large visual language model, comprising:

[0007] Step 1: Collect the original image dataset of the target domain and perform standardization processing to transform the original image data into a unified semantic representation;

[0008] Step 2: Based on the standardized original image dataset and semantic representation, generate candidate labels and corresponding text descriptions;

[0009] Step 3: Based on the generated text description, calculate the defect category to which the inferred image belongs by combining rule matching and semantic similarity.

[0010] The aforementioned automatic annotation method for engineering defect images based on a large visual language model includes, in step 1:

[0011] 11) Obtain the set of engineering defect images in the target domain. ;

[0012] 12) Use the standard score Z-score to detect outliers in the engineering defect image dataset I, and remove outliers;

[0013] 13) Standardize the image data after cleaning in step 12) to a specific range;

[0014] 14) Convert the normalized image to binary, then perform Base64 encoding, and encapsulate it into a unified input JSON structure to complete the semantic representation;

[0015] 15) Pre-set list of engineering defect categories Define a unified set of output field names, including tags, categories, and descriptions.

[0016] The aforementioned automatic annotation method for engineering defect images based on a large visual language model includes, in step 12), the following:

[0017] 121) For a given set of initial engineering image data The pixel values ​​of all images are combined into a one-dimensional array. The mean of the image dataset is calculated according to formula (1). and standard deviation :

[0018] (1)

[0019] (2)

[0020] Where M is the total number of pixels in all images, It is the first The value of each pixel. It is the square of the difference between each pixel value and the mean;

[0021] 122) Calculate the Z-Score for each image observation in the engineering defect image dataset:

[0022] (3)

[0023] in, It is the first Z-Score value of each pixel, It is the average of all pixel values. It is the standard deviation of all pixel values;

[0024] 123) Set threshold The Z-Score value of each data point is compared with a threshold. If the absolute value of the Z-Score value is greater than the threshold, the corresponding observation is marked as an outlier and removed. After removing the outliers, the statistical properties of the data are re-evaluated.

[0025] The aforementioned automatic image annotation method for engineering defects based on a large visual language model includes, in step 2:

[0026] 21) Design task prompt word templates for the input visual language model LVLM;

[0027] 22) Input the JSON structure generated in step 1 along with the prompt words into the LVLM model. The LVLM model will return a natural language description. ;

[0028] 23) Based on natural language description Extract keywords and the keywords With category list Compare and contrast to identify keywords. To determine if any of the categories in the list exist, obtain a set of candidate tags. ;

[0029] The aforementioned automatic image annotation method for engineering defects based on a large visual language model includes, in step 3:

[0030] 31) Establish a list of categories Corresponding pattern dictionary Pattern dictionary Includes keywords, synonyms, and typical expressions for each category, providing natural language descriptions. With keywords Perform matching and scoring to obtain the rule matching score. ;

[0031] 32) Calculate the semantic similarity between the text description and the category label to form preliminary image-description-candidate label entries;

[0032] 33) The natural language description generated in step 22) above. The candidate label set L generated in step 23) and the prior knowledge generated in step 3. Input the visual language model LVLM for inference.

[0033] The aforementioned automatic image annotation method for engineering defects based on a large visual language model includes, in step 32):

[0034] 321) Calculate the semantic similarity between the text description and the category label, and combine the natural language description... Each category label name is embedded into a semantic vector, and cosine similarity is calculated. :

[0035] (5)

[0036] in, It is a semantic vector embedded in the text description. It is a semantic vector embedded with category labels. The double vertical lines represent the norm of the vector, which is the Euclidean norm.

[0037] 322) Select the first one with the highest similarity Each category is used as a candidate label set to form the initial annotation entries of image-description-candidate label;

[0038] 323) Based on temperature parameters Normalization yields semantic probabilities :

[0039] (6)

[0040] Indicates the first Under the input text, and the first The semantic similarity of the input text is calculated for the first input text. Category labels;

[0041] 324) The design prompts are typical characteristics and descriptions of common defects in the prior knowledge base of the engineering field;

[0042] The aforementioned automatic image annotation method for engineering defects based on a large visual language model includes, in step 33), the following:

[0043] 331) Chained reasoning based on the visual language model LVLM is adopted, with natural language description as input. Candidate tag set Prior knowledge The output is the initial label. ;

[0044] (7)

[0045] in, The matching score output by LVLM; A semantic representation vector representing the features of the input image. Indicates the first The semantic vector of each candidate category label. Indicates a prompt word, Represents the set of all candidate categories, i.e. , The total number of categories, It is a set symbol belonging to, : The final predicted category label.

[0046] The aforementioned automatic image annotation method for engineering defects based on a large visual language model further includes, in step 33), the following:

[0047] 332) The rule matching score obtained in step 31) The semantic probabilities obtained in step 323) Confidence levels are assigned using weights. Fusion rule matching With semantic probability Result in a comprehensive score :

[0048] (8)

[0049] Pick As a preliminary confidence level; representing the index or position of the current image in the category set. , =0,1,2..., iterate through all possible candidate category k values ​​in the category set to find the comprehensive score. The maximum value in, and take the maximum value as Preliminary confidence level.

[0050] The aforementioned automatic annotation method for engineering defect images based on a large visual language model also includes:

[0051] Step 4: Verify and correct the consistency of the annotation results obtained after calculation and reasoning in Step 3;

[0052] Step 5: Based on the comprehensive confidence score, determine the uncertainty of the labels. For uncertain samples, perform multiple rounds of automatic processing and converge to a usable state.

[0053] The aforementioned automatic image annotation method for engineering defects based on a large visual language model includes, in step 4:

[0054] 41) Regarding the preliminary labels obtained in step 3 Construct verification prompt words Based on the given image and description, determine whether the label is reasonable and explain the reasoning. (Original image data) Generate description text With initial label Simultaneously, the Visual Language Model (LVLM) is input. After receiving the input, the LVLM outputs a judgment conclusion field, verdict. The range of values ​​for the verdict field includes:

[0055] {Reasonable: The current label matches the image and description;}

[0056] Modification needed: The current tag is reasonable, but there are better alternatives.

[0057] Unreasonable: The current label does not match the image and description.

[0058] The visual language big data model outputs suggested labels and brief reasons; these preliminary labels... When verdict is unreasonable, LVLM re-infers the most likely correct label, and the brief reasoning is generated by the language generation part of LVLM, explaining the reasons for the suggested modification.

[0059] 42) Map the verdict field of the decision conclusion to a consistency score. :

[0060] (9)

[0061] Combining confidence level Generate a comprehensive confidence score:

[0062] (10)

[0063] , These are fusion weight one and fusion weight two, respectively, which control the relative impact of consistency and confidence.

[0064] 43) Based on the result of the verdict field, execute different label correction rules:

[0065] If verdict= needs to be modified and the model provides suggested labels If so, the initial labels will be replaced with suggested labels, and the overall confidence score will be recalculated. ;

[0066] If verdict= is unreasonable, then mark the sample as uncertain and proceed to step 5 for processing;

[0067] If verdict = reasonable, then retain the current label and the overall confidence score. value.

[0068] The aforementioned automatic image annotation method for engineering defects based on a large visual language model includes, in step 5:

[0069] 51) Determine the uncertainty of the label and set a threshold. If the overall confidence score If no consensus is reached, it is recorded as an uncertain sample;

[0070] 52) The multi-round automatic processing includes: for uncertain samples, repeated updates and iterations are performed according to steps 2-4, with adjustments made during the iterations. The adjustments include: supplementing more specific task prompts, expanding the category pattern dictionary, and identifying key points for distinguishing easily confused categories; and recalculating the comprehensive confidence score after each round of updates and iterations. ;

[0071] 53) Set the maximum number of iterations If in any round If the label is not found, the label is accepted as the final result; if the threshold is not reached after the maximum number of rounds, the current label and original image data of the corresponding sample are retained, but the status is marked as uncertain. The sample is then placed in the uncertain directory for subsequent manual processing or the LVLM automatic labeling process is re-executed for the uncertain sample.

[0072] The aforementioned automatic image annotation method for engineering defects based on a large visual language model includes, in step 52):

[0073] 521) In the next round of inference, new task prompt phrases are automatically generated based on the results and keywords of the previous round of LVLM output, which are used to guide LVLM to focus on discriminative features;

[0074] 522) If in the previous round of LVLM output, the same new word appears repeatedly on at least 3 uncertain samples, and the corresponding LVLM confidence average value is... If the value is greater than 0.7, then the new word will be added to the corresponding category dictionary Ωk;

[0075] The aforementioned automatic image annotation method for engineering defects based on a large visual language model includes, in step 53), the following:

[0076] 531) The convergence condition is satisfied if any of the following conditions are met:

[0077]

[0078] Two consecutive rounds and Preferred ;

[0079] If the maximum number of rounds is reached and Reaching a high threshold High threshold 0.80 is preferred.

[0080] 532) If convergence occurs, then the final label will be... Final confidence level Number of iterations Write the data to a structured record and remove the sample from the uncertainty pool. ;

[0081] If this round involves dictionary expansion or weight updates, the changes will be recorded as a temporary version. Only after N manually verified positive examples will the changes be merged into the main dictionary to form a new version. This will occur within the maximum round. If convergence is still not achieved, the sample will be marked as long-term uncertain and submitted for manual review.

[0082] 533) The trigger condition for manual review is: if the convergence fails or the model has significant divergence, the sample is packaged together with the following information as a review task, including the image, descriptions of each round of LVLM, candidate labels and confidence scores for each round, and the results and suggestions given by the model.

[0083] The aforementioned automatic annotation method for engineering defect images based on a large visual language model also includes:

[0084] Step 6: Structurally summarize the confirmed annotation results and construct a dataset that can be directly used for training. The annotation results include all accepted samples obtained by convergence in Step 4 or Step 5, or confirmed by manual review.

[0085] The aforementioned automatic image annotation method for engineering defects based on a large visual language model includes, in step 6:

[0086] 61) Automatically create a labeling directory, automatically generate a classification directory according to the category list, and archive the confirmed samples; create an uncertain directory to store unconverged samples;

[0087] 62) Generate structured annotation records for each sample, including generating JSON structure and database schema;

[0088] 63) Automatically generate a dataset list, outputting a training list and a validation list. The validation list includes file paths and label pairs, and only contains confirmed samples; it also outputs a list of uncertain samples.

[0089] The aforementioned automatic image annotation method for engineering defects based on a large visual language model includes, in step 62), the following:

[0090] 621) Generate a JSON record for the sample, including image identifier, text description, final category, confidence level, verification conclusion, and iteration round;

[0091] 622) Generate a sample database schema table with fields consistent with those in the JSON structure record, but change the storage format from file to database table structure, and create indexes for image naming, final labels, text descriptions, etc., to facilitate retrieval and stratified sampling.

[0092] The aforementioned automatic image annotation method for engineering defects based on a large visual language model, in step 63), the step of generating a dataset list includes:

[0093] 631) Only samples with a confidence score greater than 0.8 are included in the training list, while other low-confidence or uncertain samples are exported separately for manual annotation or subsequent processing;

[0094] 632) Stratified sampling is used to ensure that the training / validation ratio of each class is consistent. If the sample size of a certain class is insufficient, data augmentation or supplementation tasks are triggered.

[0095] 633) The training list format is set to CSV format, and the training set images correspond one-to-one with the training list.

[0096] The aforementioned automatic annotation method for engineering defect images based on a large visual language model also includes:

[0097] Step 7: Use the dataset and annotation records output in Step 6 for system integration, model adjustment, or data platform sharing.

[0098] The aforementioned automatic image annotation method for engineering defects based on a large visual language model includes, in step 7:

[0099] 71) Connect the dataset and labeled records output in step 6 to the data management platform, including:

[0100] 711) Configure the data management platform system to access the application programming interface (API):

[0101] The input includes a JSON structure encoded from the input image and returns a task ID;

[0102] Receive the task ID and return the labeled status and partial results;

[0103] The output returns a complete JSON annotated record;

[0104] The interface for submitting manual review results includes image naming, final label, reviewer ID, and text suggestions;

[0105] 712) Batch export interface: Export training list and uncertain sample list;

[0106] 72) Using the confirmed samples as training data, perform lightweight optimization of the LVLM task prompts based on the training data. On the basis of existing labeled data, make minor adjustments to the prompts to make the LVLM model more adaptable to specific engineering scenarios, including:

[0107] 721) Input the training list into the LVLM model for minor adjustments. The adjustments are only made to the prompt word "Prompt". During the adjustment process, the LVLM model generates labeled output based on the input and compares the results with the manual annotations of the corresponding images to calculate the image accuracy. This is used to evaluate the initial performance of the prompt word in specific engineering scenarios.

[0108] 722) Periodically evaluate the performance of prompt words using manual annotation results and model adjustment results. If the model performance deteriorates or the error rate of a certain type is greater than 20%, trigger the prompt word correction task and record the prompt word correction.

[0109] 73) Upload the adjusted annotation data and records to the data management platform. The data management platform updates the prompt template and pattern dictionary based on actual usage feedback. The feedback includes the results of manual review, statistics of model output errors, and problems found in engineering applications.

[0110] A computer system includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to perform the steps of the method described above.

[0111] The beneficial effects achieved by this invention are as follows: The method of this invention utilizes a large language model combined with image semantic representation to automatically annotate and combine engineering defect images. It adopts a technical path that combines image standardization processing, semantic representation generation and inference, and achieves efficient and automated annotation of engineering defect images. It solves the problems of traditional methods relying on large-scale manual annotation data, low efficiency and high cost, significantly improves the accuracy and robustness of annotation, reduces data construction costs, and can provide high-quality data support for subsequent model training and engineering applications, and has good applicability. Attached Figure Description

[0112] Figure 1 This is a flowchart of the automatic annotation method for engineering defect images based on a large visual language model in Embodiment 1 of the present invention;

[0113] Figure 2 This is the engineering defect dataset obtained after data cleaning in Embodiment 1 of the present invention;

[0114] Figure 3 This is a schematic diagram of the automatic annotation method in Embodiment 1 of the present invention;

[0115] Figure 4 This is a schematic diagram of the LVLM generation description and reasoning process in Embodiment 1 of the present invention;

[0116] Figure 5 This is a schematic diagram illustrating the automatic creation of labeled datasets and lists in Embodiment 1 of the present invention;

[0117] Figure 6 This is a schematic diagram of the final annotation accuracy in Embodiment 1 of the present invention;

[0118] Figure 7 This is a schematic diagram of the dataset in Embodiment 1 of the present invention. Detailed Implementation

[0119] The method of the present invention will be described in detail below with reference to the accompanying drawings.

[0120] Example 1

[0121] Reference Figure 1 This embodiment provides an automatic annotation method for engineering defect images based on a large visual language model, including the following steps:

[0122] Step 1: Collect the original image dataset of the target domain and perform standardization processing to transform the original image data into a unified semantic representation, including:

[0123] 11) Reference Figure 2 By using methods such as on-site photography, drone inspections, and existing image databases, a set of images of engineering defects in the target domain is obtained. Image formats include JPG, JPEG, and PNG, with a resolution of at least 1024×768;

[0124] 12) Use the standard score Z-score to detect outliers in the engineering defect image dataset I, and remove outliers;

[0125] 121) For a given set of initial engineering image data The pixel values ​​of all images are combined into a one-dimensional array. The mean of the image dataset is calculated according to formula (1). and standard deviation :

[0126] (1)

[0127] (2)

[0128] Where M is the total number of pixels in all images, It is the first The value of each pixel. It is the square of the difference between each pixel value and the mean;

[0129] 122) Calculate the Z-Score for each image observation in the engineering defect image dataset:

[0130] (3)

[0131] in, It is the first Z-Score value of each pixel, It is the average of all pixel values. It is the standard deviation of all pixel values;

[0132] 123) Set threshold Option 3 can be used. The Z-Score value of each data point is compared with the threshold. If the absolute value of the Z-Score value is greater than the threshold, the corresponding observation is marked as an outlier and removed. After removing the outliers, the statistical characteristics of the data are re-evaluated.

[0133] 13) Standardize the image data after cleaning in step 12) to a specific range, which in this embodiment is 0 to 1:

[0134] For a given non-abnormal image after processing, size unification and pixel normalization are performed to eliminate resolution and brightness differences. The image data is normalized to the range [0,1] using formula (4):

[0135] (4)

[0136] in It is normalized image data. It is the smallest pixel value in the original image data. It is the maximum pixel value in the original image data;

[0137] 14) Convert the normalized image to binary, then perform Base64 encoding, and encapsulate it into a unified input JSON structure to complete the semantic representation, such as:

[0138] {"image_id": "xxx"; "image_base64": "xxxxxx"; "metadata": {...}}

[0139] The JSON structure is used as the input format for the subsequent Visual Language Model (LVLM), as shown in the attached figure. Figure 2 As shown.

[0140] 15) Reference Figure 3 Preset list of engineering defect categories {Cracks, peeling, weathering, corrosion, leakage, no defects}, and define uniform output field names, including labels, categories, and descriptions, to ensure data consistency across stages.

[0141] Step 2, based on the standardized original image dataset and semantic representation, generates candidate labels and corresponding text descriptions, including the following steps:

[0142] 21) Reference Figure 4 Design task prompt word templates for the input visual language model LVLM, including:

[0143] {Role restriction: such as "You are an engineering testing expert";}

[0144] Task description: such as "Please analyze the input image and describe possible defects";

[0145] Category list: Cracks, peeling, weathering, rust, leakage, no defects;

[0146] Output format constraints: JSON format, fields include tag, category, and description;

[0147] 22) Input the JSON structure generated in step 1 along with the prompt words into the LVLM model. The LVLM model will return a natural language description. Natural Language Description For example, "obvious concrete cracks are visible in the image."

[0148] 23) Based on natural language description Extract keywords and the keywords With category list Compare and contrast to identify keywords. To determine if any of the categories in the list exist, obtain a set of candidate tags. {Cracks, peeling, weathering, corrosion, leakage, no defects, indeterminate};

[0149] Step 3, based on the generated text description, calculates the defect category of the inferred image by combining rule matching and semantic similarity, including the following steps:

[0150] 31) Establish a list of categories Corresponding pattern dictionary Pattern dictionary Includes keywords, synonyms, and typical expressions for each category, providing natural language descriptions. With keyword set Perform matching and scoring to obtain the rule matching score. ;

[0151] 32) Calculate semantic similarity between text descriptions and category labels to form preliminary image-description-candidate label entries, including:

[0152] 321) Calculate the semantic similarity between the text description and the category label, and combine the natural language description... Each category label name is embedded into a semantic vector, and cosine similarity is calculated. :

[0153] (5)

[0154] in It is a semantic vector embedded in the text description. It is a semantic vector embedded with category labels. The double vertical lines represent the norm of the vector, which is the Euclidean norm.

[0155] 322) Select the first one with the highest similarity Each category is used as a candidate label set to form the initial annotation entries for image-description-candidate labels. In this embodiment, [the following is taken]. =3;

[0156] 323) Based on temperature parameters Normalization yields semantic probabilities :

[0157] (6)

[0158] Indicates the first The candidate category or sample, that is, the first candidate category or sample. Under the input text, and the first The semantic similarity of the input text is calculated for the first input text. Category labels;

[0159] 324) The design prompts are typical characteristics and descriptions of common defects in the prior knowledge base of the engineering field;

[0160] Prior knowledge include:

[0161] Cracks: Linear texture, discontinuous grayscale, width < 2mm;

[0162] Peeling: Localized material loss, irregular edges;

[0163] Rust: Metal oxidation, resulting in a yellowish-brown color.

[0164] 33) The natural language description generated in step 22) above. The candidate label set L generated in step 23) and the prior knowledge generated in step 3. Input the visual language model LVLM for inference.

[0165] 331) Chained reasoning based on the visual language model LVLM is adopted, with natural language description as input. Candidate tag set Prior knowledge The output is the initial label. ;

[0166] (7)

[0167] in, The matching score output by LVLM; A semantic representation vector representing the features of the input image. Indicates the first The semantic vector of each candidate category label. Indicates a prompt word, Represents the set of all candidate categories, i.e. , The total number of categories, It is a set symbol belonging to, : The final predicted category label, i.e., the category with the highest score.

[0168] 332) The rule matching score obtained in step 31) The semantic probabilities obtained in step 323) Confidence levels are assigned using weights. Fusion rule matching With semantic probability Result in a comprehensive score :

[0169] (8)

[0170] Pick As a preliminary confidence level; representing the index or position of the current image in the category set. , =0,1,2..., iterate through all possible candidate category k values ​​in the category set to find the comprehensive score. The maximum value in, and take the maximum value as Preliminary confidence level.

[0171] Example 2

[0172] Based on the technical solution of Embodiment 1, it includes:

[0173] Step 4 involves verifying the consistency and self-correcting of the annotation results obtained after the calculation and reasoning in Step 3, including:

[0174] 41) Reference Figure 3 Regarding the preliminary labels obtained in step 3 Construct verification prompt words Based on the given image and description, determine whether the label is reasonable and explain the reasoning. (Original image data) Generate description text With initial label Simultaneously, the Visual Language Model (LVLM) is input. After receiving the input content, the LVLM outputs a judgment conclusion field, verdict. The value range of the judgment conclusion field verdict includes:

[0175] {Reasonable: The current label matches the image and description;}

[0176] Modification needed: The current tags are generally reasonable, but there are better alternatives.

[0177] Unreasonable: The current label does not match the image and description.

[0178] Simultaneously, the visual language model outputs suggested labels and brief reasons, the preliminary labels. When verdict is unreasonable, LVLM re-infers the most likely correct label. The brief reason is generated by the language generation part of LVLM, explaining the reason for the suggested modification, such as: "No white salt frost is seen in the image, the label 'Efflorescence' is unreasonable and should be 'Crack'."

[0179] 42) Map the verdict field to a consistency score. :

[0180] (9)

[0181] Combined with the confidence level in step 3 Composite confidence score:

[0182] (10)

[0183] , These are fusion weight one and fusion weight two, respectively, which control the relative impact of consistency and confidence.

[0184] 43) Based on the result of the verdict field, execute different label correction rules:

[0185] If verdict= needs to be modified and the model provides suggested labels If so, the initial labels will be replaced with suggested labels, and the overall confidence score will be recalculated. ;

[0186] If verdict= is unreasonable, then mark the sample as uncertain and proceed to step 5 for processing;

[0187] If verdict = reasonable, then retain the current label and the overall confidence score. value;

[0188] Step 5: Based on the comprehensive confidence score, determine the uncertainty of the labels. For uncertain samples, use a multi-round automated processing procedure to ultimately improve the quality of sample labeling and converge to a usable state, including:

[0189] 51) Determine the uncertainty of the label and set a threshold. =0.6, if the overall confidence score If no consensus is reached, it is recorded as an uncertain sample. A sample refers to an image being processed, and a label is a marker assigned to the image.

[0190] 52) Multi-round automated processing: For uncertain samples, repeat the update iterations according to steps 2-4, making only minor adjustments during the iterations. The adjustments include: adding more specific task prompts, slightly expanding the category pattern dictionary, and identifying key points for distinguishing easily confused categories; after each round of update iterations, recalculate the overall confidence score. ;

[0191] 521) In the next round of inference, new task prompt phrases are automatically generated based on the results and keywords of the previous round of LVLM to guide LVLM to focus on discriminative features. The format is as follows:

[0192] {Note: The model mentioned shadows in previous results; please focus on determining whether these are features of genuine material fracture, rather than pseudo-defects caused by lighting or stains. If possible, please provide specific visual evidence to support or refute these claims.}

[0193] 522) If in the previous round of LVLM output, the same new word appears repeatedly on at least 3 uncertain samples, and the corresponding LVLM confidence average value is... If the value is greater than 0.7, then the new word will be added to the corresponding category dictionary Ωk;

[0194] 53) Set the maximum number of iterations =10, if in any round If the label is accepted as the final result, then the current label and original image data of the corresponding sample are retained even after the maximum number of rounds. However, the status is marked as uncertain and the sample is placed in the uncertain directory for subsequent manual processing or to re-execute the LVLM automatic labeling process for uncertain samples. This process covers the entire process from description generation and preliminary classification to verification and calibration.

[0195] 531) The convergence condition is satisfied if any of the following conditions are met:

[0196]

[0197] Two consecutive rounds and Preferred ;

[0198] If the maximum number of rounds is reached and Reaching a high threshold High threshold 0.80 is preferred.

[0199] 532) If convergence occurs, then the final label will be... Final confidence level Number of iterations Write the data to a structured record and remove the sample from the uncertainty pool. ;

[0200] If this round involves dictionary expansion or weight updates, the changes will be recorded as a temporary version. Only after N=10 manually verified positive examples will the changes be merged into the main dictionary to form a new version. If this occurs in the maximum round... If convergence is still not achieved, the sample will be marked as long-term uncertain and submitted for manual review.

[0201] 533) Manual review triggering conditions: If the convergence fails or the model has significant divergence, the sample is packaged together with the following information as a review task, including the image, descriptions of each round of LVLM, candidate labels and confidence scores for each round, and the results and suggestions given by the model;

[0202] The manual review interface should include at least: the final label provided by the reviewer, the reason for the review, whether the newly added dictionary word was adopted, the reviewer's ID, and the review time. The manual confirmation result is simultaneously written back to the system and highlighted for subsequent model and dictionary updates. See step 6 for the fields in the manual review record.

[0203] Step 6: Structurally summarize the confirmed annotation results and construct a dataset that can be directly used for training. The annotation results include all accepted samples obtained by convergence in Step 4 or Step 5, or confirmed by manual review.

[0204] 61) Reference Figure 5 Automatically create a labeling directory, automatically generate a classification directory according to the category list, and archive the confirmed samples; create an uncertain directory to store unconverged samples;

[0205] 62) Generate structured labeled records for each sample, including generating JSON and database schema;

[0206] 621) The sample generates a JSON record containing image identifier, text description, final category, confidence level, verification conclusion, iteration round, etc.

[0207] 622) Generate a sample database schema table: The fields are consistent with those in the JSON records, but the storage format is changed from a file to a database table structure. Indexes are also created for image naming, final labels, and text descriptions to facilitate retrieval and stratified sampling.

[0208] 63) Automatically generate a dataset list, outputting a training list and a validation list. The validation list includes file paths and label pairs, and only contains confirmed samples. Also outputs a list of uncertain samples for subsequent processing.

[0209] The steps to generate a dataset list include:

[0210] 631) Reference Figure 6 Only samples with a confidence score greater than 0.8 are included in the training list, while other low-confidence or uncertain samples are exported separately for manual annotation or subsequent processing.

[0211] 632) Stratified sampling is used to ensure a consistent training / validation ratio for each class, such as a training:validation ratio of 4:1. If a class has insufficient samples, data augmentation or supplementation tasks are triggered.

[0212] 633) The training list format is set to CSV format, and the training set images correspond one-to-one with the training list;

[0213] Step 7: Use the dataset and annotation records output in Step 6 for system integration, model adjustment, or data platform sharing to form a complete closed loop, such as... Figure 7 As shown.

[0214] 71) Connect the dataset and annotation records output in step 6 to the data management platform to support automatic loading and task orchestration;

[0215] 711) Configure the data management platform system to access the Application Programming Interface (API):

[0216] The input includes a JSON structure encoded from the input image and returns a task ID;

[0217] Receive task ID and return status (not started, in progress, uncertain, completed) and partial results;

[0218] The output returns a complete JSON annotated record;

[0219] The interface for submitting manual review results includes image naming, final label, reviewer ID, and text suggestions;

[0220] 712) Batch export interface: Export training list and uncertain sample list;

[0221] 72) Reference Figure 7 Using confirmed samples as training data, the task prompts for LVLM are lightly optimized based on the training data. By making minor adjustments to the prompts on existing labeled data, the LVLM model is better adapted to specific engineering scenarios, including:

[0222] 721) Input the training list into LVLM for minor adjustments. The adjustments are only made for the prompt word Prompt. During the adjustment process, the model generates labeled output based on the input and compares the results with the manual annotations of the corresponding images to calculate the image accuracy. This is used to evaluate the initial performance of the prompt word in specific engineering scenarios.

[0223] 722) Periodically evaluate the performance of prompt words using manual annotation results and model adjustment results. If the model performance deteriorates or the error rate of a certain type is greater than 20%, trigger the prompt word correction task and record the prompt word correction.

[0224] 73) Upload the adjusted annotation data and records to the data management platform to achieve unified management and task scheduling. The data management platform updates the prompt template and pattern dictionary based on actual usage feedback to continuously improve the stability and consistency of automatic annotation. Feedback includes manual review results, model output error statistics, and problems found in engineering applications.

[0225] Example 3

[0226] A computer system includes a memory, a processor, and a computer program stored in the memory, characterized in that the processor executes the computer program to implement the steps of the method as described in claims 1-8.

[0227] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0228] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0229] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0230] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0231] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An automatic annotation method for engineering defect images based on a large visual language model, characterized in that, include: Step 1: Collect the original image dataset of the target domain and perform standardization processing to transform the original image data into a unified semantic representation; Step 2: Based on the standardized original image dataset and semantic representation, generate candidate labels and corresponding text descriptions; Step 3: Based on the generated text description, calculate the defect category to which the inferred image belongs by combining rule matching and semantic similarity.

2. The automatic annotation method for engineering defect images based on a large visual language model according to claim 1, characterized in that, Step 1 includes: 11) Obtain the set of engineering defect images in the target domain. ; 12) Use the standard score Z-score to detect outliers in the engineering defect image dataset I, and remove outliers; 13) Standardize the image data after cleaning in step 12) to a specific range; 14) Convert the normalized image to binary, then perform Base64 encoding, and encapsulate it into a unified input JSON structure to complete the semantic representation; 15) Pre-set list of engineering defect categories Define a unified set of output field names, including tags, categories, and descriptions.

3. The automatic annotation method for engineering defect images based on a large visual language model according to claim 2, characterized in that, Step 12) includes: 121) For a given set of initial engineering image data The pixel values ​​of all images are combined into a one-dimensional array. The mean of the image dataset is calculated according to formula (1). and standard deviation : （1）； （2）； Where M is the total number of pixels in all images. It is the first The value of each pixel. It is the square of the difference between each pixel value and the mean; 122) Calculate the Z-Score for each image observation in the engineering defect image dataset: （3）； in, It is the first Z-Score value of each pixel, It is the average of all pixel values. It is the standard deviation of all pixel values; 123) Set threshold The Z-Score value of each data point is compared with a threshold. If the absolute value of the Z-Score value is greater than the threshold, the corresponding observation is marked as an outlier and removed. After removing the outliers, the statistical properties of the data are re-evaluated.

4. The automatic annotation method for engineering defect images based on a large visual language model according to claim 1, characterized in that, Step 2 includes: 21) Design task prompt word templates for the input visual language model LVLM; 22) Input the JSON structure generated in step 1 along with the prompt words into the LVLM model. The LVLM model will return a natural language description. ; 23) Based on natural language description Extract keywords and the keywords With category list Compare and contrast to identify keywords. To determine if any of the categories in the list exist, obtain a set of candidate tags. .

5. The automatic annotation method for engineering defect images based on a large visual language model according to claim 4, characterized in that, Step 3 includes: 31) Establish a list of categories Corresponding pattern dictionary Pattern dictionary Includes keywords, synonyms, and typical expressions for each category, providing natural language descriptions. With keywords Perform matching and scoring to obtain the rule matching score. ; 32) Calculate the semantic similarity between the text description and the category label to form preliminary image-description-candidate label entries; 33) The natural language description generated in step 22) above. The candidate label set L generated in step 23) and the prior knowledge generated in step 3. Input the visual language model LVLM for inference.

6. The automatic annotation method for engineering defect images based on a large visual language model according to claim 5, characterized in that, Step 32) includes: 321) Calculate the semantic similarity between the text description and the category label, and combine the natural language description... Each category label name is embedded into a semantic vector, and cosine similarity is calculated. : （5）； in, It is a semantic vector embedded in the text description. It is a semantic vector embedded with category labels. The double vertical lines represent the norm of the vector, which is the Euclidean norm. 322) Select the first one with the highest similarity Each category is used as a candidate label set to form the initial annotation entries of image-description-candidate label; 323) Based on temperature parameters Normalization yields semantic probabilities : （6）； Indicates the first Under the input text, and the first The semantic similarity of the input text is calculated for the first input text. Category labels; 324) Design prompts are typical characteristics and descriptions of common defects in the prior knowledge base of the engineering field.

7. The automatic annotation method for engineering defect images based on a large visual language model according to claim 6, characterized in that, Step 33) includes: 331) Chained reasoning based on the visual language model LVLM is adopted, with natural language description as input. Candidate tag set Prior knowledge The output is the initial label. ; （7）； in, The matching score output by LVLM; A semantic representation vector representing the features of the input image. Indicates the first The semantic vector of each candidate category label. Indicates a prompt word, Represents the set of all candidate categories, i.e. , The total number of categories, It is a set symbol belonging to, : The final predicted category label.

8. The automatic annotation method for engineering defect images based on a large visual language model according to claim 5, characterized in that, Step 33) also includes: 332) The rule matching score obtained in step 31) The semantic probabilities obtained in step 323) Confidence levels are assigned using weights. Fusion rule matching With semantic probability Result in a comprehensive score : （8）； Pick As a preliminary confidence level; representing the index or position of the current image in the category set. , =0,1,2..., iterate through all possible candidate category k values ​​in the category set to find the comprehensive score. The maximum value in, and take the maximum value as Preliminary confidence level.

9. The automatic annotation method for engineering defect images based on a large visual language model according to claim 8, characterized in that, Also includes: Step 4: Verify and correct the consistency of the annotation results obtained after calculation and reasoning in Step 3.

10. A computer system comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method as described in claims 1-9.

Images