Abnormality detection sample labeling method and training method based on multi-modal large model

Abstract

The application provides an abnormality detection sample labeling method and a training method based on a multi-modal large model. The method provided by the application comprises the following steps: acquiring multiple types of normal images; generating a text description of a detection object in multiple normal images of each type; inputting the text description and the normal image into a multi-modal large model to generate a characteristic description set of the detection object in a normal state; classifying each subset in the characteristic description set according to a characteristic category, and collecting characteristic description subsets of the same characteristic category in multiple normal images of the same type to generate a detection standard set under each characteristic category; and generating an abnormality label corresponding to a to-be-labeled sample and an inference process text based on the detection standard set. The abnormality detection sample labeling method and the training method based on the multi-modal large model can automatically generate a sample abnormality label and an inference process.

Description

Technical Field

[0001] This application relates to the fields of image recognition and artificial intelligence technology, and in particular to an anomaly detection sample annotation and training method based on a multimodal large model. Background Technology

[0002] In the field of image anomaly detection, anomalies are clearly divided into two categories: structural anomalies and logical anomalies. Structural anomalies refer to physical defects in the object itself (such as scratches on industrial parts, morphological distortions of organs in medical images, and lesions on crop leaves), which usually have clear visual features. Logical anomalies, on the other hand, are defined as abnormal phenomena that violate logical constraints (such as incorrect number of parts in an industrial assembly line, inverted organ positions in medical images, and abnormal crop planting density in farmland). Their features are more subtle and often require the model to have reasoning capabilities to identify them.

[0003] With the increasing demand for automated detection across various fields, the requirements for the accuracy and interpretability of anomaly detection are becoming increasingly urgent, especially the need for a labeling and evaluation system that can simultaneously cover two types of anomalies. Existing technologies have significant limitations: first, manual labeling is inefficient and inconsistent, making it difficult to handle large-scale samples; second, single-modal automation methods (such as pure image recognition or pure text rule matching) cannot generate interpretable reasoning processes and lack adaptability to logical anomalies; third, existing evaluation systems focus primarily on the final result, lacking consideration of intermediate reasoning logic. Therefore, a technical solution that integrates multimodal information and supports the automatic generation of anomaly labels and reasoning processes is urgently needed. Summary of the Invention

[0004] In view of this, this application provides an anomaly detection sample labeling and training method based on a multimodal large model, which can realize the automatic generation of sample anomaly labels and inference processes.

[0005] Specifically, this application is implemented through the following technical solution:

[0006] The first aspect of this application provides a method for annotating anomaly detection samples based on a multimodal large model, the method comprising:

[0007] Acquire multiple types of normal images;

[0008] Generate text descriptions of detected objects in multiple normal images of various types;

[0009] The text description and normal image are input into a multimodal large model to generate a set of characteristic descriptions of the detected object in its normal state;

[0010] The subsets in the feature description set are classified according to the feature category. The feature description subsets of multiple normal images of the same type with the same feature category are summarized to generate the detection standard set under each feature category.

[0011] Based on the set of detection standards, anomaly labels and inference process text corresponding to the samples to be labeled are generated.

[0012] A second aspect of this application provides a method for training a large model based on multimodal anomaly-labeled samples, the training method comprising:

[0013] The labeled normal samples, structurally abnormal samples, and logically abnormal samples, along with their corresponding inference process text, are used to construct a multimodal fine-tuning dataset according to a preset format. The preset format includes an image input field, a text description field, a system prompt field, and an inference output target field.

[0014] Based on the multimodal fine-tuning dataset, supervised fine-tuning is performed on the pre-trained multimodal large model to learn the ability to generate anomaly recognition and reasoning processes under image-text association;

[0015] Based on supervised fine-tuning, a reinforcement learning dataset is constructed by selecting labeled samples that meet specified requirements, and the expected output containing only the answer is used as the reward target to perform reinforcement learning fine-tuning on the model.

[0016] During the training process, the original pre-trained image-text pairs of the multimodal large model are periodically introduced.

[0017] The anomaly detection sample annotation and training method based on a multimodal large model provided in this application can adaptively generate annotation standards for various types of images through a large model, ensuring that the annotation of images of the same type uses the same standard, avoiding the inconsistency of manual annotation, and also avoiding the uncontrollability of relying entirely on machine annotation. Specifically, by incorporating multiple types of normal images and extracting common features, the generated detection standards can cover the normal features of different categories of detection objects, avoiding the limitations of standards caused by single samples, and improving the adaptability of the method to multiple scenarios. By using a multimodal large model to complete the transformation from image to text and the summarization of common features, it replaces the manual summarization and standard setting of normal features in traditional manual annotation, reducing the error caused by human intervention, and realizing the automation of detection standard generation, significantly improving annotation efficiency. In addition, the generation of anomaly labels and inference process text based on structured detection standards provides a clear benchmark for anomaly judgment, ensuring the accuracy and consistency of labels, and the generation of inference process text makes the anomaly judgment logic traceable, solving the problem of lack of interpretability of results in traditional annotation. Attached Figure Description

[0018] Figure 1A flowchart of an embodiment of the anomaly detection sample annotation method based on a multimodal large model provided in this application;

[0019] Figure 2 The flowchart is for Embodiment 2 of the large model training method based on multimodal anomaly labeled samples provided in this application. Detailed Implementation

[0020] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0021] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used herein are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0022] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0023] The following specific embodiments are given to illustrate the technical solution of this application in detail.

[0024] Example 1

[0025] Figure 1 This is a flowchart of an embodiment of the anomaly detection sample annotation method based on a multimodal large model provided in this application. Please refer to... Figure 1 The method provided in this embodiment may include:

[0026] S101. Obtain multiple types of normal images.

[0027] It should be noted that a normal image refers to an image in which the object being detected is in a state without abnormalities, meaning that the structure, morphology, and spatial relationships of the object in the image fully comply with industry standards or design requirements (such as qualified parts in the industrial field, normal organ slices in the medical field, and healthy crops in the agricultural field). The multiple types are categorized according to the type of object being detected (such as electronic components, mechanical parts, human organs, crop varieties, etc.), the stage of development (such as the processing stage of parts, the growth cycle of crops, etc.), and morphological characteristics (such as different specifications of the same part, different physiological states of the same organ, different varieties of the same crop, etc.).

[0028] At the image acquisition level, this can be achieved in the following ways: First, real-world scene acquisition, such as in the quality inspection stage of a factory assembly line, where high-definition industrial cameras are used to capture images of products that have been judged to be qualified by humans or machines from multiple angles (such as front, side, and top views) to ensure that the complete features of the inspected object are captured. Second, screening of public datasets, where normal sample subsets without abnormal annotations are extracted from general or industry-specific datasets such as the MVTecAD industrial defect dataset and PascalVOC. This type of data is usually pre-cleaned and can be directly used as the base sample. Third, simulation generation, for scenarios where there are no actual acquisition conditions (such as the new product development stage), standard 3D renderings of the inspected object are generated using CAD software, or image editing tools are used to repair abnormal features (such as scratches and damage) in existing images to obtain normal images that meet the requirements.

[0029] To ensure the reliability of subsequent steps, each type of normal image must be acquired to meet specified requirements (e.g., at least 10 images), and the acquisition process must cover the typical state of the object being detected (e.g., its performance under different lighting and angles).

[0030] It's important to note that inputting a variety of normal images of different types and quantities can avoid errors caused by a single sample in the detection standard. For example, in electronic component inspection scenarios, if only normal images of chips are collected, the subsequent generated standard can only cover the anomaly types of chips; however, by supplementing with images of resistors and capacitors, the standard can simultaneously constrain the structural, parameter, and other characteristics of multiple types of components, significantly broadening the applicable scenarios for anomaly detection. Furthermore, a rich sample pool can provide sufficient data support for subsequent steps, thereby improving the accuracy and reliability of anomaly detection.

[0031] S102. Generate text descriptions of the detected objects in multiple normal images of various types.

[0032] It should be noted that text description refers to converting the visual features of detected objects in an image (such as shape, color, spatial relationships, texture, structural integrity, etc.) into natural language statements. The purpose of this step is to use a multimodal large model to convert the visual information of the image into machine-processable text information.

[0033] Specifically, the acquired normal images of various types are grouped by type (e.g., electronic components, mechanical parts, etc.), with each group containing multiple images. Then, each individual image within a group is processed. First, the image is input into the image encoder of a multimodal large-scale model, where the model automatically identifies the visual features of the detected object. Then, a text decoder maps these features into natural language descriptions. After generating text for a single image, the above process is repeated for multiple images of the same type, ultimately obtaining a set of text descriptions corresponding to each type. Specifically, when generating text descriptions for detected objects in multiple normal images of the same type, grouping by type is required. For the first image in a group, the multimodal large-scale model is input to generate a natural language description, and the feature categories (such as shape, color, quantity, etc.) contained in this description are extracted as a baseline. For the remaining images in the group, in addition to the input image itself, the feature categories of the first image are added as prompts, forcing the model to cover all prompted feature categories. During this process, the model can supplement new features, but must not omit the core suggested categories, ultimately ensuring that all text descriptions within the same type group maintain consistency in core feature categories, forming a standardized set of text descriptions. In this way, automated text generation is achieved through a multimodal large model, which significantly improves efficiency compared to manually generating text descriptions for images. At the same time, the model's accurate capture of visual features avoids the biases or omissions of manual descriptions. Furthermore, the unified natural language format provides a foundation for subsequent steps.

[0034] In addition, after generating textual descriptions of the detected objects in multiple normal images of various types, the process also includes:

[0035] (1) Perform semantic analysis and structural segmentation on the generated text description to obtain multiple text units.

[0036] It's important to note that a text unit refers to a linguistic unit segmented from a text description, independently carrying a single feature. These units can be words, phrases, or short sentences, each corresponding to a specific feature of the detected object (such as attributes, states, or environments). Specifically, Natural Language Processing (NLP) techniques (such as Transformer-based semantic parsing models) can be used to structure each text description, segmenting continuous natural language sentences into multiple independent text units (i.e., phrases or short sentences carrying a single feature). For example, the description of a rectangular silver chip with no scratches and good background lighting can be segmented into text units such as rectangle, silver, chip, no scratches, and good background lighting, each corresponding to a specific feature.

[0037] (2) Locate the text unit corresponding to the detection object in the text unit.

[0038] It should be noted that in this step, a domain-specific vocabulary library for the detected objects can be constructed, pre-collecting all possible names of detected objects and their common aliases in the target scene as a basic reference for recognition. Subsequently, the segmented text units are input into a domain-adapted Named Entity Recognition (NER) model (e.g., an industrial parts NER model fine-tuned based on BERT). This model automatically labels words belonging to the detected object ontology by learning the semantic features of the detected object terms. For example, see the example in step (1), where the chip is the detected object ontology. In addition, if there is ambiguity (such as multiple suspected object terms appearing in the same text), further verification is required by combining contextual semantic analysis (see the description of the relevant technology for details).

[0039] (3) Identify the relationship between other text units and the text units corresponding to the detection object.

[0040] Specifically, the essence of association is the descriptive and described relationship between text units. That is, through semantic association analysis (such as relationship recognition based on knowledge graphs), it is determined whether non-core text units use the text unit corresponding to the detected object as the descriptive subject. If the non-core text unit directly describes the inherent characteristics of the detected object, it is a valid association, including attribute relationships (such as silver describing the color of a chip) and state relationships (such as no scratches on the surface describing the structural integrity of a chip). If the non-core text unit describes the external environment of the detected object (such as good background lighting), it is an invalid association because it does not directly point to the characteristics of the detected object itself. Through this analysis, descriptive units centered on the detected object can be clearly identified, laying the foundation for subsequent removal of irrelevant information.

[0041] (4) Based on the association, filter out environmental feature descriptions that are not related to the detection object and redundant descriptions that do not appear repeatedly in multiple images.

[0042] It should be noted that this step involves classifying and counting associations to eliminate information that is meaningless to the generation of detection criteria. Specifically, for descriptions of environmental features unrelated to the detection object, descriptions with environmental relationships are identified from text units (i.e., features that only reflect the external environment of the detection object and are unrelated to its own quality, such as the description of good background lighting obtained in the previous step). Secondly, for redundant descriptions that do not appear repeatedly in multiple images, filtering is achieved through frequency statistics of text units in samples of the same type. Text units of multiple normal images of the same type are summarized, and the number of times and frequency of each text unit appear in samples of that type are counted. If the number and frequency are below a threshold, they are determined to be non-common features and belong to redundant descriptions.

[0043] It should also be noted that this step can be automated using relation classification models in natural language processing (such as BERT-based relation classifiers) and frequency statistics algorithms (such as word frequency statistics tools), which ensures that the screening criteria are consistent and the results are reproducible.

[0044] (5) Perform deletion processing on the environmental feature description and redundant description according to the preset rules.

[0045] It should be noted that the content of the preset rules is based on the core feature requirements of the detection object. In addition, when performing deletion, the descriptions to be processed selected in step (4) can be scanned in batches using natural language processing tools. For environmental feature descriptions, they can be directly marked and deleted by keyword matching; for redundant descriptions, units with insufficient frequency can be automatically deleted by comparing their frequency in similar samples with the preset threshold.

[0046] In this step, through automated semantic analysis and filtering, irrelevant information and non-common features are eliminated, ensuring that the feature description set generated subsequently only contains the stable and core features of the detected object, thus laying the foundation for building accurate detection standards.

[0047] S103. Input the text description and normal image into the multimodal large model to generate a set of characteristic descriptions of the detected object under normal conditions.

[0048] It should be noted that the feature description set is a set of common features that repeatedly appear in the normal state of the detected object, such as shape, structure, and quantity. In this step, the purified text after the deletion process in step S102 is used as input and is batch-input into the multimodal large model in the form of image-text pairs. Through the cross-modal understanding and inductive ability of the multimodal large model, the scattered image and text information is transformed into a structured set of common features (i.e., the feature description set).

[0049] Specifically, generating a feature description set of the detected object under normal conditions includes: generating text descriptions of the detected object for multiple normal images of the current type; inputting the multiple normal images and corresponding text descriptions into a multimodal large model to extract common feature terms of the detected object that appear repeatedly under normal conditions and have a frequency higher than a specified threshold; and summarizing the common feature terms of each normal image to generate the feature description set. In specific implementation, for multiple normal images of the current type, the cleaned text descriptions generated in step S102 are directly reused to avoid repeated generation and improve efficiency. After inputting these images and their corresponding cleaned texts into the multimodal large model in the form of image-text pairs, the model first performs feature parsing on each image-text pair. Visual features such as contours and colors can be extracted through an image encoder, and structured descriptions such as "shape: circle" and "quantity: 24" can be parsed through a text encoder. Both are uniformly mapped to feature category-feature value key-value pairs to ensure the consistency of cross-modal features. Subsequently, the model aggregates all image-text key-value pairs, counts the frequency of each key-value pair in samples of the same type, and identifies key-value pairs with frequencies above a specified threshold (e.g., 60%) as common features, while occasional features below the threshold are excluded. Finally, the extracted common features are aggregated according to attribute categories (e.g., shape, quantity, color, etc.) to form a structured set of feature descriptions.

[0050] Specifically, the process of extracting common features from a batch of normal images using a multimodal large model can be summarized in three steps: prompt-driven structured description generation, consistent output of multi-image descriptions, and extraction and aggregation of identical / similar descriptions. In practice, when multiple normal images and their corresponding cleaned text descriptions are input into the multimodal large model, a preset prompt constrains the model's output format, preventing description dimension chaos caused by the model's free generation. An example prompt is set as: "Based on the input image and text description, please output a description based solely on the core features of the detected object, using the following fixed structure: Shape: {}; Color: {}; Quantity: {}; Spatial Relationship: {}; Surface State: {}; If a feature has no clear information, fill in 'None'." This structured prompt forces the model to cover the same feature dimensions for all input images, ensuring that descriptions of different images have a comparable basis. Then, for each normal image of the current type, the multimodal large model outputs a structured text description centered on fixed fields based on the aforementioned prompts. For example, Image 1 → Shape: Circular; Color: Silver; Quantity: 24; Spatial Relationship: None; Surface Condition: Smooth. Image 2 → Shape: Perfect Circle; Color: Silver; Quantity: 24; Spatial Relationship: None; Surface Condition: No Scratches. Image 3 → Shape: Circular; Color: Silver-white; Quantity: 24; Spatial Relationship: None; Surface Condition: Complete. In this process, the feature descriptions of different images are mapped to the same semantic slots (such as "shape", "quantity", and "surface condition"), achieving strict alignment of description dimensions and laying the foundation for subsequent extraction of common features.

[0051] Furthermore, the structured descriptions of all images are summarized, statistically analyzed, and normalized according to field categories. For numerical or clearly defined single-value fields (such as "quantity"), the frequency of each value in all image descriptions is calculated. If the frequency of a certain value is greater than or equal to a preset threshold (such as 80%), it is directly identified as a common feature of that field. For example, if 85% of the descriptions for the "quantity" field are "24", then "quantity = 24" is determined to be a common feature. For fields with different expressions but similar semantics (such as "shape", "color", and "surface condition"), a semantic similarity calculation model (such as the BERT vector similarity model) is used to unify similar expressions into standard expressions. For example, in the "shape" field, the semantic similarity between "circle" and "perfect circle" is 92% (higher than the 85% similarity threshold), so it is unified as "circle"; in the "color" field, the semantic similarity between "silver" and "silver-white" is 88%, so it is unified as "silver". If some descriptions only appear in a very small number of images (such as "well-lit"), they are determined to be non-core features and are removed. After the above steps, the common features of all fields are aggregated to form a structured set of common features. Taking the above sample as an example, the final set of common features is: "Shape: circle; Color: silver; Quantity: 24; Spatial relationship: none; Surface condition: smooth, no scratches".

[0052] S104. Classify each subset in the feature description set according to the feature category, summarize the feature description subsets of multiple normal images of the same type with the same feature category, and generate a detection standard set under each feature category.

[0053] It should be noted that feature categories refer to the attribute classification of the detected object (i.e., describing the specific attribute dimensions of the detected object, such as shape, color, quantity, positional relationship, structural integrity, etc.), and the final classification is based on these attribute dimensions. The feature description set consists of multiple subsets, each subset being a feature category-feature value key-value pair. The subset itself is unrelated to the feature category and exists only as an independent element in the set.

[0054] During classification, the feature category of each key-value pair is determined directly based on the key value (i.e., feature category). For example, key-value pairs with "shape" as the key are classified into the "shape" category, and key-value pairs with "quantity" as the key are classified into the "quantity" category, without relying on a semantic classification model. Subsequently, all key-value pairs under the same feature category are aggregated, and semantic normalization is used to eliminate expression differences (e.g., unifying "rectangle" and "rectangle" as "rectangle") to form a standard expression for that category. Then, the standard expression is bound to the corresponding feature category to generate a detection standard for a single feature category. Finally, the detection standards for all feature categories are aggregated to form a complete set of detection standards. In this way, through key-value-based classification and semantic standardization, scattered feature descriptions are transformed into structured, executable detection rules, solving the problem of ambiguous detection benchmarks caused by inconsistent descriptions.

[0055] Specifically, the feature description set includes multiple subsets, each subset including descriptions of the detected object for the same common feature term in various normal images; generating the detection standard set under each feature category includes:

[0056] (1) Perform semantic normalization on the description content within each subset.

[0057] Specifically, semantic normalization addresses different feature descriptions (e.g., circle, perfect circle, undistorted circle) under the same feature category (i.e., the same key value, such as shape). This can be achieved through the following steps: First, a standard expression library is established based on a domain-specific dictionary. Second, models such as BERT are used to calculate the semantic similarity between different descriptions and the standard expression. Finally, descriptions with similarity exceeding a preset threshold are uniformly replaced with the standard expression, and necessary feature details are added (e.g., perfect circle and undistorted circle are unified as circle with no edge distortion, clearly defining the core feature of no distortion). In this step, the purpose of normalization is to eliminate expression differences and provide a unified foundation for subsequent classification and standard generation. For example, for the subset of shape features {circle, perfect circle, undistorted circle}, normalization unifies it to: circle with no edge distortion.

[0058] (2) Unify the dimensions of the normalized expressions under the same characteristic category.

[0059] Because normalized expressions under the same characteristic category (such as surface condition) may still have different perspectives (e.g., smooth surface without scratches, rust, or damage), they need to be further integrated into a standard expression with the same dimension. Specifically, core feature dimensions (such as surface integrity) are identified through semantic association analysis, and then all normalized expressions under this category are mapped to this dimension to form a comprehensive description covering each sub-feature (e.g., unifying smooth surface without scratches, rust, or damage into surface integrity: no scratches, no rust, no damage).

[0060] (3) Combine all expressions under each feature category with the corresponding common feature items to form a subset of the detection standard set.

[0061] It should be noted that this step transforms the categorized feature descriptions into standard units that can be directly used for detection through structured association. Specifically, during combination, the feature category is used as the core label, and all normalized expressions under that category are used as standard description content, and their original common feature terms are associated to form a structured subset of detection standards. For example, if the feature category is shape, the normalized expression under this category is: circle, no edge deformation, regular contour without indentation; the corresponding common feature term is: gear contour feature. The resulting subset after combination is: Shape Inspection Standard [Corresponding Common Feature: Gear Contour Feature]: Circular, no edge deformation; regular contour without depressions. For example, if the feature category is surface condition, and the normalized expression under this category is no depressions, no protrusions (derived from the normalization of {no depressions, smooth, no protrusions}), and the corresponding common feature is the surface integrity feature of the part, then the resulting subset is: Surface Condition Inspection Standard [Corresponding Common Feature: Surface Integrity Feature of Part]: No depressions, no protrusions, clearly indicating that the standard targets the core dimension of whether there are surface defects. Each subset contains three parts: the standard category name, the corresponding common feature, and a specific standard description. This clarifies which dimension of the inspection object the standard targets (e.g., shape, quantity), explains the source of the standard (based on which common feature), and provides a specific description of the normal state (normalized expression). This combination method makes each standard subset independent and clear.

[0062] (4) Summarize all subsets to obtain the set of detection criteria for each characteristic category of the object under normal conditions.

[0063] It should be noted that the summary here is not a simple splicing, but a structured integration according to the logical association of the feature categories, to ensure that the set of detection standards not only fully covers all core features, but is also clear and organized. Taking the gear as the detection object, after processing in steps (1)-(3) of S104, the detection standard subsets of each feature category have been obtained: shape category standard subset: round, complete edge; quantity category standard subset: number of teeth = 24; surface state category standard subset: no scratches, no defects. The structured integration of the above subsets is not a simple splicing, but a feature hierarchy based on the attribute logic of the detection object (from appearance to function, from whole to part), and then each standard subset is assigned to the corresponding level to form a hierarchical structure. The specific integration results are as follows: the first level of appearance features are assigned to the standard subsets that are directly related to the visual appearance of the detection object. After integration, it is expressed as: [Appearance features] shape = round, complete edge; this level focuses on the overall morphological features of the detection object, which is the most intuitive judgment dimension in anomaly detection (such as shape distortion and edge defects can be directly identified by visual observation). The second level of functional characteristics is categorized into a subset of standards directly related to the core functions of the object being tested, and is described as: [Functional Characteristic] Number of teeth = 24; this level focuses on the key parameters for the object to achieve its preset functions (such as the number of gear teeth directly affecting transmission accuracy), and is the core dimension for determining whether the object can function normally. The third level of surface integrity characteristics is categorized into a subset of standards related to the surface quality of the object being tested, and is described as: [Surface Integrity] No scratches, no defects; this level focuses on the microscopic state of the surface of the object being tested. Surface defects (such as scratches and defects) may affect the service life or safety of the object being tested, and are an important auxiliary judgment dimension.

[0064] Through the above hierarchical integration, the set of detection standards forms a logically progressive structure of appearance features, functional features, and surface integrity features. On the one hand, this facilitates the model to prioritize comparisons according to hierarchy when detecting anomalies (e.g., first determine whether the appearance and function are abnormal, and then determine the surface condition), thus improving detection efficiency. On the other hand, it facilitates the quick location of the dimension to which the anomaly belongs during manual review (e.g., determining that "number of teeth = 22" belongs to functional feature anomalies), reducing operational complexity. At the same time, it reserves hierarchical expansion space for subsequent addition of standard subsets (e.g., "material features" and "size features"), ensuring the scalability and practicality of the set of detection standards.

[0065] The following example further illustrates the process of forming a set of detection criteria after semantic normalization of the model's output text. Assuming a gear is the detection object, the multimodal large model outputs the following structured text descriptions for three normal gear images: Image 1: "Circular gear, 24 teeth, smooth surface without scratches"; Image 2: "Perfectly round gear, intact edges, 24 teeth"; Image 3: "Round gear, no surface defects, 24 teeth". First, semantic normalization is performed on the output text of these three images, extracting descriptions related to "shape"—"round," "perfectly round," "round," and "intact edges"—from the three images. Through semantic similarity calculation (e.g., BERT model calculates similarity ≥ 85%), "round" and "perfectly round" are determined to be similar expressions and unified as "round." Simultaneously, the feature "intact edges," related to shape integrity, is integrated, ultimately forming a normalized expression for the shape dimension: "round, intact edges." Secondly, descriptions related to "quantity" were extracted from the text of the three images—"24 teeth," "24 teeth," and "24 teeth." The core values ​​are consistent across all three, differing only in their wording ("24" versus "24"), and are standardized to "24 teeth." Further, descriptions related to "surface condition" were extracted from the text of the three images—"smooth surface without scratches" and "no defects." Through semantic association analysis, it was determined that "smooth" implicitly implies the basic attribute of "no defects." Both "no scratches" and "no defects" are core indicators of surface integrity, and after integration, a normalized expression for the surface condition dimension was formed: "no scratches, no defects."

[0066] Then, the normalized expressions are combined according to their characteristic categories and corresponding common feature terms to form a subset of detection standards for each characteristic category: Shape detection standard [corresponding common feature term: gear contour feature]: round, with complete edges; Quantity detection standard [corresponding common feature term: gear tooth count feature]: number of teeth = 24; Surface condition detection standard [corresponding common feature term: gear surface integrity feature]: no scratches, no defects. Finally, all the above subsets of detection standards are summarized to obtain a complete set of detection standards for the "gear" detection object. This set clarifies the judgment criteria for gears in the three core dimensions of shape, quantity, and surface condition under normal conditions. When performing anomaly detection on the annotated gear samples, this set will be used as a unified comparison basis to ensure the consistency and accuracy of anomaly judgment.

[0067] Furthermore, semantic normalization plays a crucial role in this process. On the one hand, normalization unifies the standard expressions for the same feature category, avoiding logical confusion caused by differences in expression during aggregation, and ensuring that different subsets can be directly related on the same dimension. On the other hand, normalization focuses on core features, making the aggregated standard set more concise and accurate, eliminating redundant expressions, and facilitating rapid comparison during subsequent anomaly detection. It should also be noted that the normalized standards have clear feature dimensions, and can be arranged in an orderly manner according to the attribute logic of the detected objects (such as individual features → group relationships) during aggregation, improving the systematicness and interpretability of the standard set, and providing a clear benchmark framework for manual review or model training.

[0068] S105. Generate anomaly labels and inference process text corresponding to the samples to be labeled based on the set of detection standards.

[0069] It's important to note that this step uses the detection standard set and the sample to be labeled as input. A multimodal large model transforms the visual features of the sample into textual descriptions, which are then compared item by item with the detection standard set to generate results in two steps. The generation of anomaly labels must be closely integrated with the detection standards. Based on the unmatched items obtained from the comparison, the label content is clearly defined by associating the feature categories in the standards. The generation of the inference text relies solely on the textualized visual features of the sample to be labeled and the factual descriptions of the unmatched items, without repeatedly referencing standard content. This approach ensures the accuracy of anomaly labels through the standards while keeping the inference process focused on the sample itself, avoiding information redundancy.

[0070] Specifically, generating anomaly labels based on the aforementioned set of detection standards includes:

[0071] (1) Input the image of the sample to be labeled into the multimodal large model to obtain the corresponding image attribute description information.

[0072] Specifically, as described above, the multimodal large model first extracts the visual features (such as contours, colors, textures, spatial relationships, etc.) of the samples to be labeled through an image encoder, and then transforms these visual features into structured natural language descriptions through a text decoder. The final output attribute information is presented in the form of key-value pairs of feature categories and feature values.

[0073] (2) Compare the image attribute description information with the detection standard set item by item to identify the unmatched standard items.

[0074] Specifically, the sample description is compared with the standard one by one according to the characteristic category (such as shape, quantity, surface condition). If a certain attribute in the sample description (such as ellipse) is inconsistent with the standard of the corresponding category (such as shape detection standard: circle), then the standard item is marked as unmatched.

[0075] It should be noted that the method for obtaining image attribute description information is described in the preceding section. After obtaining the image attribute description information, semantic similarity is calculated, and the attribute description of the sample is compared item by item with the corresponding feature category in the detection standard set, using the feature category as the unit (e.g., shape, quantity). If the semantic similarity between the sample description and the standard description of that category is ≥ a preset threshold (e.g., 80%), it is considered a match; otherwise, it is considered a non-match. For example, if the shape detection standard in the detection standard set is: shape: circle, no edge indentation, and the sample description is: shape: circle (edge ​​indentation), the semantic similarity with the standard is 60% (this is just an example, below the threshold), then it is considered a non-match.

[0076] (3) Generate corresponding exception tags based on the content of the unmatched standard items.

[0077] It should be noted that each unmatched standard item corresponds to an anomaly label focusing on its characteristic category. For example, unmatched items in the shape category correspond to a shape anomaly label, unmatched items in the quantity category correspond to a quantity anomaly label, and unmatched items in the surface state category correspond to a surface state anomaly label. If there are multiple unmatched items, a combined label is generated (e.g., shape anomaly + quantity anomaly + surface state anomaly). In addition, when a sample matches some standards and does not match others, the system outputs anomaly labels by category dimension. For example, if "shape" matches "quantity" but "surface state" does not match, the label is "surface state anomaly".

[0078] Specifically, the generation of the reasoning process text includes: if all standard items match, then extract the feature information in the image that is consistent with the detection standard in sequence to form feature description text, and add a first judgment statement at the end; if there are unmatched standard items, then extract the abnormal feature description corresponding to each unmatched item, identify the feature category to which it belongs, generate abnormal location information, and add a second judgment statement at the end; and concatenate the feature description information and the judgment statement to generate the final reasoning process text.

[0079] When the sample descriptions for all feature categories match the criteria, text generation follows the logic of feature restatement → judgment summary, outputting coherent sentences through templates or NLG models. Specifically, the normal features of the samples are described sequentially according to feature categories, and finally, a judgment statement is used to summarize that there are no anomalies. For example, "The sample to be labeled has a circular shape (no indentations at the edges), 24 teeth, and a smooth surface without scratches. All features meet the detection criteria, and it is judged to be without anomalies."

[0080] When there is only one unmatched standard item, text generation needs to clearly define the abnormal features → locate the abnormality in the appropriate category → and finally determine the result. This is achieved through abnormal information extraction → logical concatenation → natural language output. Specifically, it extracts the abnormal feature description (the specific features in the sample that do not conform to the standard) and abnormal location information (the feature category corresponding to the unmatched item) from the unmatched item. First, it describes the abnormal features of the sample, clarifies the abnormality location, and finally provides the determination statement. For example, "The shape of the sample to be labeled is circular, but there are indentations at the edges (abnormal feature). The abnormality is located in the shape category. Therefore, it is determined that the sample has a shape abnormality."

[0081] For different types of logical anomalies, corresponding structured reasoning process text templates are preset. The template design corresponds one-to-one with the feature categories (such as quantity, location, and spatial relationship) in the detection standard set, ensuring that the anomaly judgment logic is consistent with the standard dimensions. Specifically, in one possible implementation, the template format for the quantity anomaly template is: "The standard quantity is {X}, but only {Y} is detected in the image, which is judged as a quantity anomaly." Here, {X} is the normal quantity threshold under the corresponding category in the detection standard set (such as "2" or "24"), which is determined by the quantity class detection standard generated by S104; {Y} is the actual quantity value extracted from the attribute description of the sample image to be labeled (such as "1" or "22"), which is obtained through feature parsing of the sample image by a multimodal large model. The template format for the location anomaly template is: "The standard requires {A} to be located in {B}, but it is actually located in {C}, which is judged as a spatial location anomaly." Here, {A} is the category of the detected object (such as "circle" or "gear"); {B} is the normal location range specified in the detection standard set (such as "left (x1, y1) - (x2, y2) interval" or "unobstructed central area"); {C} is the actual location information of the detected object in the sample to be labeled (such as "right (x3, y3) - (x4, y4) interval" or "edge occlusion area"), which is determined by the coordinate analysis results of the sample image. The template format for the overlap anomaly template is: "The standard requires that objects do not overlap, but {object 1} and {object 2} are detected to have {overlapping area}, which is judged as an overlap anomaly." Here, {object 1} and {object 2} are the detected objects in the sample to be labeled that have spatial conflicts (such as "circle 1" and "rectangle 2"); {overlapping area} is the actual overlap ratio calculated by the image bounding box (such as "30%" or "more than 50%). This calculation result needs to be compared with the normal requirement of "no overlap" in the detection standard set to clarify the degree of anomaly.

[0082] It should be noted that if a multimodal large model is used to freely generate the inference process text, problems such as inconsistent expression dimensions (e.g., some texts mention different quantities, while others omit this core information) and inconsistent judgment logic (e.g., in the same anomaly scenario, some texts judge it as "insufficient quantity," while others judge it as "feature mismatch") are likely to occur. By using a pre-set structured template, the inference text is forced to cover the complete logical chain of standard requirements, actual features, and anomaly judgment. Furthermore, a unified expression framework is used for the same anomaly type, ensuring that the inference texts of different samples have a consistent logical structure and judgment criteria, avoiding ambiguity caused by free generation. In addition, templated output makes the semantic hierarchy of the inference text clear, allowing for quick location of key information such as "standard parameters," "actual deviations," and "anomaly categories" during manual review, reducing the cost of understanding. Simultaneously, variables such as {X} and {B} in the template are directly associated with the detection standard set, and variables such as {Y} and {C} correspond one-to-one with the attribute descriptions of the sample images. This allows for tracing back to the original standards and sample features through variables, ensuring the traceability of the inference process and meeting the requirement for transparency of judgment criteria in anomaly detection scenarios.

[0083] It should be noted that the template is only applied to the anomaly inference text generation stage (i.e., the step in S105 where the inference process is generated based on the detection standard set and sample features). In the previous stage of generating the detection standard set for ordinary images (S103-S104), no preset template is used. Instead, the multimodal large model's cross-modal inductive ability for a batch of normal images is relied upon to extract common features and form standards. This avoids the constraints of templates on standard generation, ensuring that the detection standard set can truly reflect the normal features of the detected objects in ordinary images, balancing the objectivity of the standards with the standardization of the inference text. The fluency and logic of the inference text rely on two technical paths. One is template filling, which involves using a preset fixed sentence template (e.g., standard requirement X, sample feature Y, does not meet the standard, is located in category Z, and is judged as an anomaly) for structured standards and features (e.g., feature category + standard description + sample features). The extracted information is then filled into the template to generate text, ensuring uniform formatting. Second, there is the NLG model generation. For complex scenarios (such as abnormal features that require more detailed description), the natural language generation model is called. The input is standard content + sample features + matching results, and the model automatically generates coherent and conversational reasoning text.

[0084] Furthermore, it should be noted that the samples to be labeled include simulated logically anomalous scene samples. Specifically, these simulated logically anomalous scene samples are generated by artificially constructing images whose geometric objects, such as their number and position, do not conform to normal logical rules. These images are used to train the model's ability to recognize non-visual defect anomalies. The generation of these logically anomalous scene samples includes:

[0085] (1) Determine the types of geometric objects to be simulated.

[0086] It should be noted that, based on the core logic of the detection scenario (such as spatial relationships between objects and the rationality of their quantity), basic geometric objects are selected as simulation objects to ensure that they can cover common logical anomaly dimensions (such as quantity anomalies, positional conflicts, and combination errors). These geometric objects, used to simulate detection objects in logical anomaly scenarios, are essentially abstracted standard shapes (such as circles, rectangles, triangles, and ellipses). These simulated objects can be analogous to parts, organs, crops, etc., in real-world scenarios. The selection of geometric objects is based on the logical constraints of the scenario. For example, a circle simulates a gear or screw hole; a rectangle simulates a chip or metal sheet; and a triangle simulates a support structure. The quantity, position, and relationships of these geometric objects simulate logical anomalies in real-world scenarios (such as missing, misaligned, and overlapping objects).

[0087] (2) Randomly set the quantity parameters of various geometric objects.

[0088] Quantity parameters are randomly generated by pre-setting normal and abnormal quantity ranges, where abnormal quantities are used to simulate logical anomalies (such as too many / too few quantities, or imbalanced proportions). Specifically, the normal quantity range is defined based on the reasonableness of the scenario, while the abnormal quantity range consists of values ​​that exceed the normal range.

[0089] (3) Generate a background image as the base image of the composite image, and determine the placement position of each type of geometric object based on preset rules.

[0090] It should be noted that a simple background image is generated as the basis for synthesis. The background image uses a solid color background image (such as white), a regular grid background image (for easy positioning by coordinates), or a simplified scene from a real production environment (such as a gray background of an assembly line) to avoid complex backgrounds interfering with the presentation of logical anomalies, thereby avoiding interference with the feature recognition of geometric objects. The preset rules specifically include the generation logic of normal and abnormal positions. The normal position is generated by establishing a two-dimensional coordinate system and assigning an independent coordinate interval to each type of object (such as defining an area by cell in a grid background image). It requires that the distance between intervals of different objects is not less than a specified multiple (such as 1.5 times) of the maximum side length of the object and that the center is aligned with the reference point to ensure that the position is regular and without conflict (objects do not overlap and the distance meets the preset threshold). The abnormal position is generated based on the normal position by parameter offset, such as moving the object coordinates into the interval of other objects (simulating overlap), exceeding the boundary of the background image (simulating boundary crossing), or exchanging preset intervals (simulating inversion), so as to distinguish between normal and abnormal spatial logic.

[0091] (4) Place each geometric object in the base map and add random textures to the image after placement to generate a composite image.

[0092] In specific implementation, each geometric object is assigned a unique position according to the quantity determined in step (2) (e.g., 3 circles, 2 rectangles) and the coordinate interval preset in step (3). Specifically, objects of the same type are sequentially assigned to the reference point in their respective intervals according to their serial numbers (e.g., the first circle corresponds to (x1, y1), the second circle corresponds to (x2, y2), ensuring that the distance between objects in the same interval meets the threshold requirement), and objects of different types fall into their respective independent intervals (e.g., circles in the left area, rectangles in the right area). After placement, a preset texture library is called based on the object type (e.g., circles are associated with gear tooth texture library, rectangles are associated with chip solder joint library), a texture style is randomly selected and scaled proportionally to the object surface (e.g., tooth texture covers 30% of the edge of the circle), and the texture transparency is adjusted (e.g., 50%-80%) to simulate the real lighting effect, and the core feature parameters of the object (e.g., shape, size, color value, coordinate positioning) are not changed throughout the process, ensuring that the anomalies of the synthesized image are only due to the quantity or position logic and are unrelated to the texture details.

[0093] (5) Generate a corresponding standard content description based on the synthesized image.

[0094] It should be noted that, based on the quantity, location, and other features of geometric objects in the image, a structured text description is generated, clearly labeling normal and abnormal features. Specifically, it is formatted as object type + quantity + location + texture (optional), highlighting logical anomalies (such as quantities not exceeding the normal range, overlapping locations, etc.).

[0095] It should be noted that the logical anomaly scene samples generated through the above steps focus on logical conflicts arising from non-visual defects (such as unreasonable quantities or irregular positions), and are highly similar to anomalies such as missing or misaligned parts in real-world scenarios. The generated synthetic images and corresponding descriptions can be used to train the model's sensitivity to violations of logical rules, supplementing the lack of logical anomaly data in real-world samples and improving the comprehensiveness of anomaly detection.

[0096] Specifically, generating corresponding standard content descriptions includes:

[0097] (i) Obtain the quantity, category, and positional relationship of various geometric objects in the synthesized image.

[0098] It should be noted that synthetic images can be processed using a lightweight object detection model. Since the geometric features of objects in synthetic images are simple (fixed shape and color, such as red circles and blue rectangles), the model can directly output object categories (e.g., red circles) using a pre-trained geometric shape + color classifier; the same number of categories can also be output. Furthermore, the bounding box coordinates of each object are obtained through an image coordinate system (e.g., a two-dimensional coordinate system established with the top-left corner of the image as the origin), and then the relationships are calculated using preset rules (the specifics are not elaborated here).

[0099] (ii) Fill the quantity, category and positional relationship information into the preset standard content description template supplement to generate a standard content description.

[0100] Specifically, a structured expression template can be designed, which includes fixed fields and placeholders. Then, the structured data output from step (i) is filled into the placeholders through string replacement, and a description is automatically generated.

[0101] It should be noted that for the standard content description generation of synthetic images (such as logically abnormal scene samples), two types of structured templates are preset to adapt to the description requirements of normal synthetic samples and abnormal synthetic samples, respectively. The template format of the standard content description template for normal synthetic samples is: "The image contains {quantity}{category}, the location is {coordinates / relationship}, and the surface {state}." Here, {quantity} is the actual number of geometric objects in the synthetic image obtained in step (i) (e.g., "2" or "3"); {category} is the type of geometric object (e.g., "circle" or "rectangle"); {coordinates / relationship} is the specific location information of the object in the base image (e.g., "(x1, y1) to (x2, y2) interval" or "no overlapping area on the left"); and {surface} is the surface texture state of the object (e.g., "no texture" or "mesh texture"). An example fill result is: "The image contains 2 circles, located in the interval (100, 100) to (200, 200), with no surface texture."

[0102] The template format for the standard content description template of the abnormal synthetic sample is: "The image should contain {standard quantity}{category}, but actually contains {abnormal quantity / state}, located as {abnormal type}." Here, {standard quantity} is the normal quantity threshold determined based on the aforementioned detection standard set (e.g., "2"); {category} is the geometric object type (e.g., "circle"); {abnormal quantity / state} is the actual abnormal feature of the object in the synthetic image (e.g., "3", "edge overlap"); and {abnormal type} is the abnormal category determined according to the detection standard set (e.g., "abnormal quantity", "abnormal positional overlap"). Example fill result: "The image should contain 2 circles, but actually contains 3, located as an abnormal quantity."

[0103] It should be noted that the detection standard set generated for ordinary normal images is obtained by extracting common features, semantically normalizing, and structurally summarizing the text descriptions of a batch of ordinary normal images through a multimodal large model (see S103-S104 for details). Its generation process relies on the cross-modal inductive ability of the large model and has no preset template constraints. Its main function is to establish the normal feature benchmark for ordinary image detection objects. The standard content description for synthetic images in this step is generated by directly filling in the known parameters (obtained in step (i)) such as the number, category, and position of objects in the synthetic image based on the preset template. Its generation process relies on the structural constraints of the template and its function is to quickly output the feature description and anomaly judgment information of synthetic images (especially logically abnormal synthetic samples), which is suitable for the efficiency requirements of batch generation of synthetic samples.

[0104] Based on the preceding description, when generating inference process text for samples to be labeled (including ordinary image samples and synthetic image samples) based on the detection standard set, a benchmark-output relationship is established between the two. The detection standard set serves as a comparison benchmark, used to compare each item with the image attribute description information of the sample to be labeled (e.g., determining whether "3 circles" in a synthetic image meets the normal threshold of "2 circles" in the standard set), locating unmatched anomalies. The standard content description template serves as the output tool. After the comparison is completed, the corresponding template is selected based on the matching result (all matches / there are unmatched items). The benchmark information of the detection standard set, the actual characteristics of the sample to be labeled, and the anomaly judgment result are filled into the template to generate structured inference process text (e.g., "The number of circles in the sample to be labeled is 3, which does not meet the requirement of '2 circles' in the detection standard set, and is judged as an anomaly in quantity"). In this way, both the efficiency of inference process text generation (relying on the template) and the accuracy of anomaly judgment (relying on the detection standard set) are ensured, achieving the unification of anomaly detection inference logic for ordinary images and synthetic images.

[0105] (iii) Call the preset reasoning process text template, and fill in the set abnormal feature content and corresponding judgment statement according to the known number, category and spatial position relationship information of geometric objects when the synthesized image is generated, to generate the reasoning process text that matches the image.

[0106] The templates include semantic judgment templates designed for different types of logical anomalies (such as quantity anomalies, overlap anomalies, spatial relationship errors, etc.). The system automatically matches the corresponding template based on abnormal keywords (such as "insufficient quantity" and "positional overlap") and fills the abnormal feature values ​​(such as overlap ratio and target quantity) into the template variables to generate coherent reasoning text. Furthermore, when generating logical anomaly sample image-text pairs, the annotation results are simultaneously set to error (i.e., the detection model cannot correctly identify them) to constitute real logical conflict samples. In addition, to enhance the logical anomaly detection capability, the system also includes a process of constructing negative samples, that is, generating descriptive text that is inconsistent with the image content by perturbing the numbers, categories, or spatial relationship words in the standard description, and forming negative sample image-text pairs with the image to form abnormal reasoning text and annotation labels.

[0107] It should also be noted that the method further includes a data augmentation process to enhance the ability to detect logical anomalies. The process includes: in the synthetic data of logical anomaly scenarios, constructing negative sample text that is inconsistent with the image content by perturbing and replacing numbers, shapes or spatial relation words in the standard content description; using the negative sample text and the synthetic image to form a new image-text sample pair, and generating corresponding anomaly labels and reasoning process text.

[0108] It should be noted that this process involves perturbing and replacing key information in the standard content description based on the synthetic data of logically abnormal scenarios, generating negative sample text that contradicts the image content, binding it with the synthetic image to form a new image-text sample pair, and generating corresponding abnormal labels and reasoning process text. Specifically, the perturbation objects are identified as core elements in the standard content description, such as numbers (e.g., quantity, size, coordinates), shapes (e.g., geometric types, structural features), and spatial relationships (e.g., position, relative relationships). Perturbation is performed based on rules indicating a clear contradiction between the replaced text and image content (e.g., if the image contains two circles, the text perturbs to three squares). Secondly, negative sample text is generated based on the original standard content description (with a 1:1 ratio of negative to positive samples) to ensure a clear conflict between the text description and the image content in terms of quantity, shape, or spatial relationship. Subsequently, the negative sample text is bound to the original synthetic image to form a new sample pair with inconsistent text and image. Finally, specific anomaly labels (e.g., quantity text does not match the image, shape text does not match the image) and reasoning process text are generated for this sample pair, clearly indicating the contradiction points (e.g., "the image contains two circles, the text describes three squares, both quantity and shape do not match").

[0109] In practice, based on the geometric object parameters (such as quantity, shape, coordinates, etc.) recorded during sample generation, the core elements to be perturbed in the standard content description are determined, including numerical categories (quantity 2, coordinate x=100), shape categories (circle, rectangle), and spatial relationship categories (left interval, no overlap). The perturbation rules are designed differently according to feature type. Among them, numerical categories can be adjusted by ±1 / ±2 numerical offset (e.g., quantity changed from 2 to 3) or ± percentage (e.g., coordinate changed from x=100 to x=110); shape categories use same-category substitution (e.g., circle changed to ellipse) or cross-category substitution (e.g., rectangle changed to triangle); spatial relationship categories use semantic inversion (e.g., left interval changed to right interval, no overlap changed to 30% overlap area) to ensure that the text and image content form a clear contradiction after perturbation (e.g., the image has 2 circles (left), and the text perturbation is 3 triangles (right)).

[0110] Secondly, negative sample text is generated at a 1:1 ratio of positive to negative samples. The text structure is ensured to be consistent through template construction (e.g., "The image contains {number of perturbations} {perturbation shapes} ({perturbation locations})"). The level of contradiction is categorized based on the number of perturbation dimensions (single-dimensional perturbation is considered a minor contradiction, while multi-dimensional perturbation is considered a severe contradiction). The negative sample text and the original synthesized image are bound together as new sample pairs using unique identifiers. Structured anomaly labels (e.g., quantity-shape-severe contradiction) and inference process text are generated simultaneously. The text template is: "Actual image features: {original parameters}; Text description features: {perturbation parameters}; The two have a logical conflict in the {contradiction dimension}, which is judged as an image-text inconsistency anomaly."

[0111] This process compensates for the model's insufficient ability to identify false text descriptions during conventional training by manually constructing negative feedback samples. Conventional training samples are mostly positive samples with consistent images and text, or samples with abnormal images but accurate text descriptions. Negative samples force the model to learn the two-way verification logic of image features and text descriptions, avoiding reliance on a single modality for judgment, thereby more accurately identifying logical anomalies.

[0112] It should be noted that this operation can enhance the model's sensitivity to subtle logical errors, improve its ability to capture inconsistencies in details by learning contradictory samples with small differences, improve the ability to judge cross-modal consistency between images and text, and encourage the model to establish a two-way verification mechanism between images and text. In addition, it can supplement the diversity and balance of training data, solve the problem of scarce and unevenly distributed inconsistent samples between images and text in real-world scenarios, and help enhance the robustness of the model.

[0113] In addition, it should be noted that the construction of negative samples follows the principle of realism, ensuring that the contradictions after perturbation conform to logical errors that may occur in real-world scenarios (such as labeling errors or parameter errors). Specifically, it covers the amplification of two types of samples: one is explicit contradiction samples (such as perturbing the number from 2 to 5, or changing the shape from a circle to a triangle), and the other is implicit contradiction samples (such as perturbing the number from 3 to 4 (a typo close to the threshold), or changing the spatial relationship from non-overlapping to edge tangency (a slight positional deviation)). The latter is specifically designed to simulate subtle logical errors that are easily overlooked in real-world scenarios.

[0114] The sample pairs generated by this process can be used in conjunction with other training data (such as positive samples and image anomaly samples) without adjusting the model input format. Through a unified <image-text-label> structure, negative samples can be directly integrated into the existing training process, complementing other samples. Simultaneously, when using template-based design for perturbation, additional subdivision rules are added for implicitly contradictory samples. For example, quantity perturbation is limited to ±1 (simulating counting error), and spatial relationship perturbation is controlled to have an overlap area ≤10% (simulating slight positional shifts). This ensures that perturbation focuses on the core logical dimension, avoiding the introduction of irrelevant features (such as random modification of color or texture). This guarantees the relevance of negative samples while comprehensively supplementing the diversity of logically anomalous samples in the training data by covering both explicit and implicit contradictory scenarios.

[0115] The method provided in this embodiment automates and increases the efficiency of anomaly labeling. It transforms image visual features into text descriptions using a multimodal large model and generates standardized detection standards based on normal samples. The entire process, from sample analysis to anomaly labeling and inference text generation, is completed automatically without manual intervention, significantly reducing the cost and error of manual labeling. This is particularly suitable for rapid sample processing in large-scale industrial inspection scenarios. Furthermore, it improves the accuracy and reliability of anomaly detection. By extracting common features from multiple types and quantities of normal images, it generates a set of detection standards covering multiple dimensions such as shape, quantity, and location, ensuring the comprehensiveness and stability of the standards. Simultaneously, through semantic normalization and cross-modal comparison techniques, it accurately identifies the differences between the sample to be labeled and the standards, making the generation of anomaly labels more targeted and avoiding misjudgments or omissions caused by ambiguous or singular standards. Furthermore, this method enhances the generalization ability and scene adaptability of anomaly detection. It not only supports the identification of structural anomalies (such as scratches and deformations), but also supplements the lack of logical anomaly samples in real data by simulating logical anomaly scene samples (such as quantity imbalance and position conflict). Combined with the data augmentation process, it constructs negative samples with inconsistent graphics and text, which further enhances the model's sensitivity to complex logical conflicts and enables it to adapt to diverse anomaly detection needs.

[0116] It should also be noted that the method provided in this application ensures the interpretability and traceability of the annotation results. The generated reasoning process text clearly records the complete logic of sample features—standard comparison—anomaly determination, which not only facilitates manual review of the rationality of the annotation results, but also provides a transparent decision basis for subsequent model training, solving the problem of difficult interpretation of results in traditional detection methods.

[0117] Example 2

[0118] Corresponding to the aforementioned embodiment of an anomaly detection sample annotation method based on a multimodal large model, this application also provides an embodiment of a large model training method based on multimodal anomaly annotation samples.

[0119] Figure 2 This is a flowchart of Embodiment 2 of the large model training method based on multimodal anomaly labeled samples provided in this application. Please refer to... Figure 2 The training method provided in this embodiment includes:

[0120] S201. The labeled normal samples, structurally abnormal samples, and logically abnormal samples, along with their corresponding inference process text, are used to construct a multimodal fine-tuning dataset according to a preset format. The preset format includes an image input field, a text description field, a system prompt field, and an inference output target field.

[0121] It should be noted that the three types of labeled samples (normal samples, structurally abnormal samples, and logically abnormal samples) and their corresponding inference process texts are integrated into a multimodal fine-tuning dataset according to a preset format. Normal samples are images of the detected object without anomalies and their corresponding inference texts without anomalies (e.g., all features meet the detection criteria, and are therefore considered normal); structurally abnormal samples are images with visual defects (e.g., scratches, deformations) and their corresponding inference texts (e.g., if the surface has scratches and does not meet the surface condition criteria, it is considered structurally abnormal); logically abnormal samples are images with logical conflicts in quantity, location, etc., and their corresponding inference texts (e.g., if the number is 3, exceeding the normal range by 1-2, it is considered logically abnormal). The preset format includes four core fields: an image input field (stores image paths or pixel data for the model to read visual information), a text description field (records the attribute description of the sample, such as the gear being elliptical and having 22 teeth), a system prompt field (fixed instructions, such as "Please determine whether it is abnormal based on the image and text description and generate an inference process," constraining the model's task direction), and an inference output target field (stores the expected output anomaly labels and inference text, such as "shape anomaly + quantity anomaly: the gear in the image is elliptical, which does not conform to the circular standard; the number of teeth is 22, which does not conform to the 24 standard, and is therefore judged as an anomaly"). This format unifies the sample input and output forms, providing structured data support for subsequent model fine-tuning.

[0122] S202. Based on the multimodal fine-tuning dataset, supervised fine-tuning is performed on the pre-trained multimodal large model to learn the ability to generate anomaly recognition and reasoning processes under image-text association.

[0123] Specifically, during training, the low-level parameters of the model's image encoder and text encoder are frozen (preserving general cross-modal understanding capabilities), and parameter optimization is performed only on the top output layer (the fully connected layer responsible for anomaly classification and text generation). During training, fine-tuning dataset samples are input in batches. The model generates inference output based on images, text descriptions, and system prompts. Then, using the target field of the inference output as a benchmark, the loss between the output and the target is calculated (cross-entropy loss is used for text generation, and multi-label classification loss is used for anomaly labels). The parameters are iteratively optimized through backpropagation. Through this process, the model gradually masters the matching logic between image features and text descriptions (such as recognizing the conflict between elliptical and circular standards) and the structured expression of the inference process (such as the sentence structure of standard → sample features → judgment result). Ultimately, it can automatically output anomaly labels and inference text that conform to the format for input image and text samples.

[0124] S203. Based on supervised fine-tuning, select labeled samples that meet the specified requirements to construct a reinforcement learning dataset, and use the expected output containing only the answer as the reward target to perform reinforcement learning fine-tuning on the model.

[0125] First, a reinforcement learning dataset is constructed by selecting high-difficulty samples (such as samples with subtle anomalies or multiple types of anomalies) from the supervised fine-tuning dataset. These samples comprise approximately 20% of the dataset to ensure targeted training. The reward objective of reinforcement learning focuses on the anomaly labels in the target field of the inference output (ignoring the details of the inference text): if the model's output anomaly label is completely consistent with the target, a positive reward (e.g., +1.0) is given; if it is partially consistent (e.g., missing an anomaly type), a partial reward (e.g., +0.5) is given; and if it is completely inconsistent, a negative reward (e.g., -1.0) is given. Using reinforcement learning algorithms such as PPO, the model adjusts its parameters based on the reward value after generating the inference output—for example, if the model misses a location anomaly label, the parameters are optimized through reward feedback to make it more likely to cover all anomaly types in subsequent outputs. This process complements the shortcomings of supervised learning in optimizing inference logic and improves the model's ability to identify complex anomaly scenarios.

[0126] S204. During the training process, periodically introduce image-text pairs from the original pre-training of the multimodal large model.

[0127] To prevent the model from degrading its general cross-modal understanding capabilities due to over-focusing on anomaly detection tasks, original pre-trained image-text pairs from the multimodal large model are periodically introduced during training. Specifically, after each specified round (e.g., 10 rounds) of training on the fine-tuning dataset, one round of training with original pre-trained samples is inserted. The proportion of original samples is controlled within a threshold (e.g., 10%-20%) to balance task-specific training with the preservation of general capabilities. This approach maintains the model's basic understanding of non-anomaly scenes (e.g., recognizing common objects and parsing general text), avoids a decline in generalization ability due to long-term training with anomaly samples (e.g., inability to understand simple shape descriptions), and reinforces the underlying logic of cross-modal associations, providing stable foundational capabilities for anomaly detection tasks.

[0128] The method provided in this embodiment, through a progressive strategy of comprehensive data support, specialized capability enhancement, precise optimization, and generalization assurance, enables multimodal large models to not only accurately identify various anomalies and generate interpretable reasoning processes, but also maintain good general cross-modal understanding capabilities, thus providing reliable model support for automated anomaly detection systems.

[0129] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for anomaly detection sample annotation based on a multimodal large model, characterized in that, The method includes: Acquire multiple types of normal images; Generate text descriptions of detected objects in multiple normal images of various types; The text description and normal image are input into a multimodal large model to generate a set of characteristic descriptions of the detected object in its normal state; The subsets in the feature description set are classified according to the feature category. The feature description subsets of multiple normal images of the same type with the same feature category are summarized to generate the detection standard set under each feature category. Based on the set of detection standards, anomaly labels and inference process text corresponding to the samples to be labeled are generated.

2. The method according to claim 1, characterized in that, The generation of the characteristic description set of the detected object under normal conditions includes: Generate text descriptions of the detected objects for each of the multiple normal images of the current type; The multiple normal images and corresponding text descriptions are input into a multimodal large model to extract common feature terms of the detected objects that recur in the normal state and have a frequency higher than a specified threshold. The common feature items of each normal image are summarized to generate the feature description set.

3. The method according to claim 1, characterized in that, The feature description set includes multiple subsets, each subset including descriptions of the detected object for the same common feature term in various normal images; generating the detection standard set under each feature category includes: Semantic normalization is performed on the descriptions within each subset; Dimensional unification is performed on the normalized expressions under the same characteristic category; Combine all expressions under each feature category with their corresponding common feature terms to form a subset of the detection standard set; By summing up all subsets, we obtain the set of detection criteria for each characteristic category of the object under normal conditions.

4. The method according to claim 1, characterized in that, The generation of anomaly labels based on the set of detection standards includes: The images of the samples to be labeled are input into the multimodal large model to obtain the corresponding image attribute description information; The image attribute description information is compared item by item with the detection standard set to identify unmatched standard items; Generate corresponding exception tags based on the content of the unmatched standard items.

5. The method according to claim 1, characterized in that, The generation of the reasoning process text includes: If all standard items match, then extract the feature information in the image that is consistent with the detection standard in turn to form feature description text, and add the first judgment statement at the end; If there are unmatched standard items, extract the abnormal feature description corresponding to each unmatched item, identify the feature category to which it belongs, generate abnormal location information, and add a second judgment statement at the end; The feature description information is concatenated with the decision statement to generate the final reasoning process text.

6. The method according to claim 1, characterized in that, After generating textual descriptions of the detected objects in multiple normal images of various types, the process also includes: Semantic analysis and structural segmentation are performed on the generated text description to obtain multiple text units; Locate the text unit corresponding to the detection object in the text unit; Identify the relationships between other text units and the text units corresponding to the detected object; Based on the aforementioned correlation, environmental feature descriptions that are irrelevant to the detection object and redundant descriptions that do not appear repeatedly in multiple images are filtered out. The environmental feature descriptions and redundant descriptions are deleted according to preset rules.

7. The method according to claim 1, characterized in that, The samples to be labeled include simulated logical anomaly scenario samples, the generation of which includes: Identify the types of geometric objects that need to be simulated; Randomly set the quantity parameters of various geometric objects; A background image is generated as the base image for the composite image, and the placement position of each type of geometric object is determined based on preset rules; Place the geometric objects in the base image, and add random textures to the image after placement to generate a composite image; Generate a corresponding standard content description based on the synthesized image.

8. The method according to claim 7, characterized in that, The generated corresponding standard content description includes: Obtain the quantity, category, and positional relationship of various geometric objects in the synthesized image; Fill the quantity, category, and positional relationship information into the preset standard content description template supplement to generate a standard content description; The preset reasoning process text template is invoked, and based on the known number, category, and spatial relationship information of geometric objects when the synthesized image is generated, the set abnormal feature content and corresponding judgment statements are filled in to generate the reasoning process text that matches the image. The templates include semantic judgment templates designed for different types of logical anomalies. The system automatically matches the corresponding template based on the anomaly keywords and fills the anomaly feature values ​​into the template variables to generate coherent reasoning text.

9. The method according to claim 1, characterized in that, The method further includes a data augmentation process for enhancing logical anomaly detection capabilities, the process comprising: In the synthetic data of logically abnormal scenarios, negative sample text that is inconsistent with the image content is constructed by perturbing and replacing numbers, shapes or spatial relation words in the standard content description. The negative sample text is used to construct a new image-text sample pair with the synthesized image, and corresponding anomaly labels and reasoning process text are generated.

10. A method for training large models based on multimodal anomaly-labeled samples, characterized in that, The method is applied to the anomaly detection sample annotation method as described in any one of claims 1 to 9, wherein the training method includes: The labeled normal samples, structurally abnormal samples, and logically abnormal samples, along with their corresponding inference process text, are used to construct a multimodal fine-tuning dataset according to a preset format. The preset format includes an image input field, a text description field, a system prompt field, and an inference output target field. Based on the multimodal fine-tuning dataset, supervised fine-tuning is performed on the pre-trained multimodal large model to learn the ability to generate anomaly recognition and reasoning processes under image-text association; Based on supervised fine-tuning, a reinforcement learning dataset is constructed by selecting labeled samples that meet specified requirements, and the expected output containing only the answer is used as the reward target to perform reinforcement learning fine-tuning on the model. During the training process, the original pre-trained image-text pairs of the multimodal large model are periodically introduced.

Images