An industrial anomaly classification method and system based on a visual expert and a multi-modal large language model

Abstract

The application discloses an industrial anomaly classification method and system based on a visual expert and a multi-modal large language model, and belongs to the field of industrial visual detection. The method adopts a two-stage cascade architecture: in the first stage, a visual expert module performs unsupervised anomaly detection on a to-be-detected sample image, and outputs an anomaly score map and an anomaly mask; an image-level anomaly score is calculated based on the anomaly score map, if the image-level anomaly score is lower than a preset detection threshold, a normal determination is directly outputted and the process is terminated; if the image-level anomaly score reaches or exceeds the threshold, the second stage is entered; in the second stage, a visual prompt map is generated based on the anomaly mask, and a normal reference image, the to-be-detected sample image, the visual prompt map and a structured text prompt are jointly inputted into a multi-modal large language model classifier, and an anomaly category, a severity level, an acceptability determination and a disposal suggestion are outputted. The application can reduce the calculation overhead caused by normal samples entering the large model, and realize fine classification and disposal decision assistance of industrial anomalies.

Description

Technical Field

[0001] This invention relates to the field of industrial visual inspection, and in particular to an industrial anomaly classification method and system based on visual experts and a multimodal large language model. Background Technology

[0002] Industrial anomaly detection typically involves two levels: image-level normality assessment and pixel-level anomaly localization. Existing visual anomaly detection methods can achieve high anomaly detection and localization performance on industrial vision benchmark datasets such as MVTec-AD and VisA, and can meet the speed requirements of online detection to a certain extent. Common methods include those based on pre-trained feature representations, those based on nearest neighbor matching from memory banks, those based on image reconstruction, and those based on diffusion model reconstruction errors.

[0003] However, in actual production lines, detecting an "abnormality" does not equate to making a quality decision. Different types of abnormalities typically correspond to different handling methods: for example, a bent wire may be resolved by trimming, a missing wire may require rework, a color change may originate from a design flaw or batch update, and a minor indentation may only require re-inspection or no intervention at all. Therefore, industrial inspection systems not only need to determine whether an abnormality exists, but also need to identify the type of abnormality, its severity, and whether intervention is necessary.

[0004] Simple visual anomaly detectors primarily rely on the visual distribution formed by normal samples in the training set for judgment, and their ability to utilize distribution shifts, normal process variations, and user-defined quality standards is limited. If the image to be tested is directly input into a multimodal large language model for anomaly judgment, although semantic knowledge can be utilized, it is easily affected by background regions, irrelevant textures, and normal samples. Furthermore, calling the large language model on a large number of normal samples increases inference costs and latency.

[0005] Multimodal large language models possess strong image understanding, natural language instruction following, and semantic reasoning capabilities, and can introduce user-defined categories, process rules, and quality standards through structured text prompts. However, multimodal large language models themselves may not have precise pixel-level anomaly localization capabilities; in industrial scenarios, anomaly regions are often small, low-contrast, or located in complex textured backgrounds, and the lack of clear spatial cues may affect classification stability.

[0006] Therefore, a technical solution is needed that combines the rapid detection and localization capabilities of visual experts with the semantic classification capabilities of multimodal large language models, enabling normal samples to pass quickly and allowing abnormal samples to complete interpretable anomaly classification and handling judgments under the constraints of spatial cues and structured language cues. Summary of the Invention

[0007] The purpose of this invention is to provide an industrial anomaly classification method and system based on visual experts and a multimodal large language model. This addresses the problem that existing industrial anomaly detection methods can only determine whether a product is abnormal, but cannot perform fine-grained classification of anomalies and output actionable handling suggestions. Directly calling a multimodal large language model to reason about all samples suffers from high computational overhead and a lack of precise spatial positioning guidance technology.

[0008] Specifically, the present invention provides an industrial anomaly classification method based on visual experts and a multimodal large language model, comprising the following steps: S1. Obtain the image of the sample to be tested. I q And the object category o, and select at least one normal reference image from the normal reference image library according to the object category o. I ref ; S2, the image of the sample to be tested I q The input is a visual expert module, which uses an unsupervised anomaly detection model to analyze the image of the test sample. I q Perform pixel-level anomaly detection and localization, and output an anomaly score map S and anomaly mask M; S3. Calculate the image-level anomaly score based on the anomaly score map S. s img The image-level anomaly score s img With preset detection threshold T detect Compare; if the image-level anomaly score s img Below the preset detection threshold T detect If the image-level anomaly score is not found, the normal judgment result will be output directly and the process will terminate; if the image-level anomaly score is not found, the normal judgment result will be output directly and the process will terminate. s img Reaching or exceeding the preset detection threshold T detect Then proceed to step S4; the anomaly mask M, after being generated, is also used for the image-level anomaly score. s img The calculation and visual cue diagram in step S4 I v The generation of; S4. Perform post-processing on the anomaly mask M and extract the anomaly region contours. Superimpose the extracted anomaly region contours onto the test sample image in the form of high-contrast contour lines. I q Above, generate visual cue images. I vThe color of the high-contrast contour line is similar to that of the image of the sample to be tested. I q The color difference of the background color meets the preset color difference threshold, and the outline only marks the outer boundary of the abnormal area; S5. Construct a structured text prompt based on the object category o. T structured The structured text prompt T structured It includes descriptions of normal objects, definitions of abnormal categories, classification strategy instructions, and output format constraints; S6. The normal reference image I ref The image of the sample to be tested I q The aforementioned visual cue diagram I v and the structured text prompts T structured The data is input to a multimodal large language model classifier, which outputs anomaly classification results, severity levels, acceptability judgments, and handling suggestions. The model parameters of the multimodal large language model classifier remain frozen during inference, eliminating the need for fine-tuning training for specific object categories or anomaly categories.

[0009] An industrial anomaly classification system based on visual experts and a multimodal large language model includes: The image acquisition module is configured to acquire images of the test samples. I q and object category information; The vision expert module is configured to analyze the image of the sample to be tested. I q Perform unsupervised anomaly detection and pixel-level anomaly localization, and output an anomaly score map S and anomaly mask M; The pre-filtering module is configured to calculate image-level anomaly scores based on the anomaly score map S. s img In the image-level anomaly score s img Below the preset detection threshold T detect If the error occurs, the normal judgment result is output directly and the process is terminated, only abnormal samples are passed to subsequent modules; The visual cue generation module is configured to perform post-processing on the anomaly mask M and extract the anomaly region contours, then overlay the extracted anomaly region contours onto the test sample image as high-contrast contour lines. I q Above, generate visual cue images. I v ; The text prompt building module is configured to construct a structured text prompt based on the object category o, which includes a description of normal objects, a definition of abnormal categories, classification strategy instructions, and output format constraints. T structured ; The multimodal large language model classification module is configured to receive normal reference images. I ref The image of the sample to be tested I q The aforementioned visual cue diagram I v and the structured text prompts T structured The system outputs anomaly classification results, severity level, acceptability judgment, and handling suggestions; the model parameters of the multimodal large language model classification module remain frozen during inference, eliminating the need for fine-tuning training for specific object categories or anomaly categories.

[0010] The beneficial effects provided by this invention are as follows: First, normal samples can be quickly output at the visual expert stage, reducing the number of calls to the multimodal large language model; second, abnormal samples have obtained pixel-level spatial cues before entering the multimodal large language model, which helps the model focus on abnormal areas; third, abnormal categories, severity thresholds, and acceptance criteria can be defined in natural language, making it easier to adapt to different production lines and products; fourth, the output results not only include abnormal categories, but also acceptability and handling suggestions, making it more suitable for industrial quality control processes. Attached Figure Description

[0011] Figure 1 This is a simplified flowchart of the method of the present invention; Figure 2 This is a schematic diagram of the system structure of the present invention; Figure 3 This diagram illustrates the process of generating a visual cue image, showing how a binary anomaly mask is obtained from the image under test and the anomaly score map, the contour set is extracted, and then superimposed onto the image under test to form a visual cue image.

[0012] Figure 4 This is a schematic diagram of the structure of structured text prompts, showing four components: normal object description, abnormal category definition, classification strategy instructions, and output constraints. Figure 5 To refine the data set construction and evaluation process, the diagram illustrates the process of forming an anomaly classification evaluation dataset from the original anomaly detection dataset after annotation correction, similar category merging, and removal of unsuitable evaluation categories, and then calculating the classification evaluation index based on this dataset. Figure 6 This is a flowchart example of an industrial anomaly classification method based on visual experts and a multimodal large language model. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0014] Before formally describing the present invention, a general description of the solution of the present invention will be given first to facilitate understanding.

[0015] Example 1 Please refer to Figure 1 This invention provides an industrial anomaly classification method based on visual experts and a multimodal large language model, comprising the following steps: S1. Obtain the image of the sample to be tested. I q And the object category o, and select at least one normal reference image from the normal reference image library according to the object category o. I ref ; As one embodiment, in this step, the image of the sample to be tested... I q Image acquisition devices such as industrial cameras, line scan cameras, area scan cameras, or microscopic imaging systems are used to acquire the images of the samples under test. I q It can be a grayscale image or a color image (such as an RGB three-channel image), and its resolution is determined by the actual production line inspection requirements. For example, it can be set to 1024×1024 pixels, 2048×2048 pixels, or other resolutions. The object category o is used to identify the type of the product to be tested (such as "bottle" for bottles, "cable" for cables, "pcb" for printed circuit boards, "metal_nut" for metal nuts, etc.), and can be automatically determined by production line station information, barcode information, host computer task information, or a vision classifier.

[0016] The normal reference image library R_o stores multiple defect-free normal sample images corresponding to object category o. At least one normal reference image is selected from the normal reference image library R_o based on object category o. I ref The selection method can be random selection, selection of the sample image to be tested, or selection of the sample image to be tested. I qThe image with the closest shooting conditions (e.g., same lighting, same angle) or a standard template image from a normal sample library can be selected. In one implementation, a normal reference image is selected by default; when there are significant process fluctuations in the appearance of normal samples (e.g., color differences between the same product in different batches), multiple (e.g., m images, m≥2) normal reference images can be selected to cover the normal range of variation. The number of reference images m selected can be set according to actual needs. The larger the value of m, the more fully the multimodal large language model understands the normal appearance baseline, but the inference cost also increases accordingly. Preferably, m=1 or m=3.

[0017] As a concrete example, suppose the current production line is inspecting bottle products, and the object category o="bottle". After the system obtains this category through the workstation information, it reads the normal image folder corresponding to the bottle category from the normal reference image library, and selects a standard bottle mouth image from it as the normal reference image. I ref The image shows the internal structure of a normal bottle neck, the complete shape of the bottle neck edge, and the normal color and texture.

[0018] Optionally, when the image of the sample to be tested is... I q Before inputting the image into the vision expert module, the system first inputs the image to be tested. I q Normal reference image I ref The anomaly score map S is uniformly scaled to n×n pixels, with n preferably being 448. Uniform scaling reduces classification bias caused by inconsistent input sizes across different models, ensuring that images received by the multimodal large language model have a consistent visual scale. The specific scaling formula is as follows:

[0019] Resize represents the scaling function.

[0020] S2, the image of the sample to be tested I q The input is a visual expert module, which uses an unsupervised anomaly detection model to analyze the image of the test sample. I q Perform pixel-level anomaly detection and localization, and output an anomaly score map S and anomaly mask M; As one embodiment, the visual expert module V_θ is the core component for anomaly detection and localization in the first stage of this invention. The visual expert module can be at least one of an unsupervised anomaly detector based on a diffusion model, an anomaly detector based on pre-trained feature representation, or an anomaly detector based on reconstruction error. Regardless of the specific implementation, the visual expert module must meet the following functional requirements: (1) It can receive the image of the sample to be tested. Iq As input; (2) It can output a pixel-level anomaly score map S (i.e. anomaly heatmap), where the value of each pixel represents the degree to which the location deviates from the normal distribution; (3) It can generate a binary anomaly mask M based on the anomaly score map S to mark the location of the anomaly area.

[0021] Specifically, the visual expert module calculates the anomaly score map S by estimating the deviation between the normal image distribution and the test sample image. Taking an anomaly detector based on reconstruction error as an example, the visual expert module first uses a reconstruction model (such as an autoencoder, U-Net, or diffusion model) trained only on normal samples to analyze the test sample image. I q Reconstruction is performed to obtain the reconstructed image I_rec; then calculation is performed. I q The pixel differences between I_rec and the target image (such as L2 distance, structural similarity difference, or feature difference) are used to obtain the anomaly score map S. Regions with larger values ​​in S indicate greater reconstruction error and are more likely to be anomalous regions. Taking an anomaly detector based on pre-trained feature representations as an example, the visual expert module uses a pre-trained deep neural network (such as ResNet, EfficientNet, or ViT) to extract local features from the image of the test sample. Then, each local feature is compared with its nearest neighbor features in the normal sample feature library, and the feature distance is calculated as the anomaly score, resulting in the anomaly score map S.

[0022] The numerical range of the anomaly score map S is typically normalized to the interval [0,1], where 0 represents perfect conformity to the normal distribution and 1 represents the maximum deviation from the normal distribution. The anomaly mask M generates a threshold by combining the anomaly score map S with a pixel-level anomaly mask. T mask The comparison yielded the following results:

[0023] Where (x,y) represents pixel coordinates. T mask A threshold is used to generate pixel-level anomaly masks, and its value is determined based on the output characteristics of the vision expert module, validation set calibration, or process tolerance. For example, when the anomaly score map S has been normalized to [0,1], T mask The possible value is 0.5; for anomalous heatmaps stored in 8-bit grayscale format (value range 0-255). T mask The possible value is 127. In actual production lines, T mask The threshold that optimizes the F1-score can be determined by searching on the validation set.

[0024] As a concrete example, suppose we employ an anomaly detector based on diffusion reconstruction error as the visual expert module, which is pre-trained on normal bottle samples. The image of the bottle opening to be tested... I q After inputting the image into the detector, the detector reconstructs the image through a reverse denoising process using a diffusion model, calculates the pixel differences between the original and reconstructed images, and generates an anomaly score map S (with the same resolution as the input image and values ​​normalized to [0,1]). If there is damage at the edge of the bottle opening, the pixels corresponding to the damaged area will show high values ​​in S (e.g., 0.85-0.95), while the normal area will show low values ​​(e.g., 0.05-0.15). Then... T mask Binarize S to 0.5 to obtain the anomaly mask M, where the pixel value of the damaged area is 1 and the pixel value of the rest of the area is 0.

[0025] S3. Calculate the image-level anomaly score based on the anomaly score map S. s img The image-level anomaly score s img With preset detection threshold T detect Compare; if the image-level anomaly score s img Below the preset detection threshold T detect If the image-level anomaly score is not found, the normal judgment result will be output directly and the process will terminate; if the image-level anomaly score is not found, the normal judgment result will be output directly and the process will terminate. s img Reaching or exceeding the preset detection threshold T detect Then proceed to step S4; the anomaly mask M, after being generated, is also used for the image-level anomaly score. s img The calculation and visual cue diagram in step S4 I v The generation of; As one embodiment, this step is a key pre-filtering step in the two-level cascaded architecture of the present invention. Image-level anomaly score s img It is a comprehensive measure of whether the entire image is abnormal, and its calculation method can be the maximum value, mean, quantile statistics, or a weighted fusion of multiple statistics from the anomaly score map S. In a preferred embodiment, the maximum value is used as the image-level anomaly score:

[0026] Here, H and W represent the height and width of the anomaly score map S, respectively (consistent with the input image size). Using the maximum value has the advantage of capturing the most significant anomalous signals in the image, avoiding the missed detection of local anomalies due to the mean being diluted by large areas of normal data.

[0027] In another implementation, the top k percentile (e.g., the top 10 percentile) can be used as the image-level anomaly score:

[0028] Where Q_{90}(S) represents the 90th percentile of the outlier score map S. In another implementation, a weighted fusion method can be used:

[0029] α, β, and γ are weighting coefficients that can be adjusted based on the performance of the validation set, for example, α=0.6, β=0.1, and γ=0.3.

[0030] Preset detection threshold T detect The threshold used to distinguish between normal and abnormal samples is determined by searching on the validation set for a threshold that balances the false positive and false negative rates. Specifically, this threshold can be set on a validation set containing known normal and abnormal samples, using different... T detect The detection accuracy, recall, and F1-score are calculated using these values, and the threshold with the optimal F1-score is selected as the threshold. T detect In one implementation, when s img When normalized to [0,1], T detect The typical value range is 0.3 to 0.7.

[0031] A key technical feature of this step is that the anomaly mask M, generated in step S2, is used for two purposes simultaneously: as input for calculating the image-level anomaly score (M can be used to calculate the area of ​​the anomaly region, the maximum anomaly response, etc.) and as input for generating the visual cue map in step S4. This "one-time calculation, two-time reuse" design avoids redundant calculations and improves overall detection efficiency.

[0032] like s img < T detect The system determines the current sample to be tested to be a normal sample, directly outputting a "normal" or "qualified" result, and recording the image number, object category, and detection score. This sample will not enter the subsequent multimodal large language model inference process, thus significantly reducing computational resource consumption and inference latency.s img ≥ T detect The system will pass the sample image to be tested, the anomaly score map, and the anomaly mask to subsequent modules for further anomaly classification.

[0033] As a specific example, using the bottle opening detection scenario described above, the maximum value of the anomaly score S is... s img =0.92. Preset detection threshold. T detect =0.5. Since 0.92 ≥ 0.5, the system determines that the bottle opening image is abnormal and proceeds to step S4 for further classification. At the same time, the abnormality mask M (marking the location of the damaged area) generated in step S2 is retained and passed to step S4 for the generation of the visual cue image.

[0034] S4. Perform post-processing on the anomaly mask M and extract the anomaly region contours. Superimpose the extracted anomaly region contours onto the test sample image in the form of high-contrast contour lines. I q Above, generate visual cue images. I v The color of the high-contrast contour line is similar to that of the image of the sample to be tested. I q The color difference of the background color meets the preset color difference threshold, and the outline only marks the outer boundary of the abnormal area; As one embodiment, this step generates a visual cue map from the anomaly mask M that can be stably understood by a multimodal large language model. I v The anomaly mask M is a binary image where pixels in the anomaly region have a value of 1, and pixels in the normal region have a value of 0. However, directly using the raw binary mask may contain noise, holes, or irregular boundaries, which is detrimental to the accuracy of subsequent contour extraction. Therefore, a series of post-processing operations are required on M.

[0035] Post-processing operations include: (1) Connected component analysis - identify all interconnected abnormal regions in M ​​and assign an independent label to each connected component; (2) Area filtering - calculate the area (i.e. the number of pixels) of each connected component, and filter out connected components with an area less than the preset threshold A_min. The value of A_min can be determined according to the image resolution and production line tolerance. For example, in a 1024×1024 image, A_min can be set to 100 pixels; (3) Hole filling - for normal holes inside the abnormal region (such as small unbroken connections in the middle of the damaged area), morphological closing operation or hole filling algorithm is used to fill them to make the abnormal region more complete; (4) Morphological smoothing - morphological opening operation or closing operation is used to smooth the boundary of the abnormal region and remove the jagged boundary.

[0036] After post-processing, the binary mask is used to extract the outer contour set B of the abnormal region using a boundary extraction algorithm (such as OpenCV's findContours function). In the case of multiple abnormal connected components, all contours can be retained, or, depending on actual needs, the contours with the largest area or those located in critical regions can be prioritized.

[0037] After contour extraction is completed, in the sample image to be tested I q The outline set B is drawn using preset colors, line widths, and transparency to form a visual cue image. I v The selection of the outline color should meet the requirement of high contrast with the image background: let the outline color be c_r=(R_r,G_r,B_r), and the image background color be c_b=(R_b,G_b,B_b), and the color difference ΔE between the two should satisfy ΔE≥40 (using the color difference calculation formula in the CIE L*a*b* color space). In a preferred embodiment, a red outline (RGB=(255,0,0)) is superimposed on the original image, with a line width w=2 pixels. For grayscale images, the single-channel image is first converted into a three-channel image to ensure that the visual cue image format is consistent, and then the outline is superimposed. When saving the visual cue image, the coordinate axes, borders, and extra white space are removed so that the image content received by the multimodal large language model only contains the test sample and its abnormal outline, reducing the interference of irrelevant visual elements on classification.

[0038] The outline only marks the outer boundary of the abnormal region without filling the inside of the abnormal region. This design can clearly indicate the location of the abnormality, while preserving the original image information (such as texture and color) inside the abnormal region, so that the multimodal large language model can perform more detailed semantic analysis.

[0039] As a specific example, the abnormal mask M passed in step S3 is post-processed as follows: First, connected component analysis is performed, revealing two connected components with areas of 1250 pixels and 35 pixels respectively; A_min is set to 100 pixels to filter out the small connected component of 35 pixels (determined as noise); hole filling and morphological closing operations (kernel size 3×3) are performed on the 1250-pixel connected component to make the boundary of the damaged area more complete and smooth; then, the outer contour is extracted using OpenCV's findContours function; finally, the contour is drawn on the original bottle neck image with a red outline (RGB=255,0,0) and a line width of 2 pixels to generate a visual cue image. I v The visual cue clearly indicates the location and extent of the damage to the bottle's rim.

[0040] S5. Construct a structured text prompt based on the object category o. T structured The structured text prompt T structured It includes descriptions of normal objects, definitions of abnormal categories, classification strategy instructions, and output format constraints; As one example, the structured text prompts constructed in this step T structured This is a semantic guidance framework for anomaly classification in a multimodal large language model. The prompts convey the following four key types of information to the model in natural language: (1) Normal object description: Define the normal appearance baseline of object category o, including the typical visual characteristics of the object (shape, color, texture), assembly relationship, shape of holes or edges, allowable range of process variations, etc. For example, for bottle objects, the normal object description is: "A normal bottle mouth includes the inner hole and the edge of the bottle mouth. The inner hole should be clean and free of foreign objects, and the edge of the bottle mouth should be flat and complete without cracks, debris or gaps." For cable objects, the normal object description is: "A normal cable includes an outer sheath, inner insulation layers of different colors and copper conductors. Each layer has a complete structure and the colors conform to the standard color matching scheme." (2) Definition of exception category: Give the set of candidate exception categories corresponding to object category o. C oEach category is defined by its name, location characteristics, morphological characteristics, severity threshold, and exclusion criteria. For example, the definition of anomaly categories for bottles may include: "Major damage – damage to the bottle opening edge covering 20% ​​or more of the circumference; Minor damage – damage or gaps to the bottle opening edge covering less than 20% of the circumference; Contamination – foreign objects, impurities, or contaminants are present in the inner hole." The definition of anomaly categories for cables may include: "Outer sheath cut – linear cut marks on the outer sheath surface; Outer sheath deformation – abnormal depressions or protrusions on the outer sheath surface; Missing conductor – insufficient number of internal copper conductors; Color mismatch – insulation color inconsistent with the standard color scheme." (3) Classification strategy instructions: These instructions direct the multimodal large language model to make judgments according to a specific decision-making order to reduce classification ambiguity. Classification strategy instructions typically employ hierarchical decision-making logic: the first level determines the location of the anomaly (whether it is located in a critical area); the second level determines the anomaly type based on morphological features; the third level determines the severity based on area, length, or process thresholds; and the fourth level determines acceptability based on user quality standards. For example, the classification strategy instruction for bottles is: "If the anomaly is located inside the inner hole or extends outward from the inner hole, it is determined to be contamination; if the anomaly is located at the edge of the bottle mouth and forms damage, it is determined to be a large or small damage based on the proportion of the circumference covered by the damage." (4) Output format constraints: Specify the output format of the model so that the subsequent program can parse it stably. The output format constraints can be simple constraints such as "only return the category name" or complex constraints such as "please output in the following JSON format: {'category':'broken', 'severity':'high','acceptability':'unacceptable', 'action':'rework'}. In one programmatic implementation, the above four parts are stored in the form of a template library. Each object category o corresponds to an independent template entry, containing a normal field (normal object description), a categories field (a list of candidate abnormal categories, each category containing a standard code, display name, and definition text), a strategy field (categorization strategy instructions), and an output field (output format constraints). After reading the above fields, the text prompt building module concatenates them into a complete structured text prompt in a preset order.

[0041] As a concrete example, for a bottle-type object (o="bottle"), the structured text hints are constructed. T structured as follows: "You are an industrial quality inspection expert. Please classify the following bottle opening images as anomalies."

[0042] Normal object description: A normal bottle mouth includes the inner hole and the edge of the bottle mouth. The inner hole should be clean and free of foreign objects, and the edge of the bottle mouth should be flat and intact without cracks, debris or gaps.

[0043] Anomaly category definition: Candidate anomaly categories are limited to the following three categories: (1) Major damage - damage to the edge of the bottle mouth covering 20% ​​or more of the perimeter; (2) Minor damage - damage to the edge of the bottle mouth or gap covering less than 20% of the perimeter; (3) Contamination - foreign objects, impurities or contaminants are present in the inner hole.

[0044] Classification strategy instructions: Please judge in the following order: First, judge the location of the abnormality - if the abnormality is located inside the inner hole or extends outward from the inner hole, it is judged as contamination; if the abnormality is located at the edge of the bottle mouth and forms a break, it is judged as a large break or a small break based on the proportion of the circumference covered by the break.

[0045] Output format constraints: Please output only the category name, i.e., 'major damage', 'minor damage', or 'contamination'. S6. The normal reference image I ref The image of the sample to be tested I q The aforementioned visual cue diagram I v and the structured text prompts T structured The data is input to a multimodal large language model classifier, which outputs anomaly classification results, severity levels, acceptability judgments, and handling suggestions. The model parameters of the multimodal large language model classifier remain frozen during inference, eliminating the need for fine-tuning training for specific object categories or anomaly categories.

[0046] As one example, this step fuses visual and textual information into a multimodal large language model, which then uses its powerful cross-modal understanding and reasoning capabilities to output the final anomaly classification result.

[0047] The order in which multimodal inputs are organized has a significant impact on the model's comprehension. In a preferred embodiment, the inputs are organized in the following fixed order: the first image is a normal reference image. I ref The second image is the sample image to be tested. I q The third image is a visual cue. I v Then add structured text prompts T structured This sequence allows the model to first establish a normal appearance baseline (through...). I ref Then observe the overall condition of the sample to be tested (through...).I q Then, use visual cues to locate abnormal areas (through...). I v Finally, semantic reasoning and classification decisions are made under the guidance of structured text prompts.

[0048] Multimodal large language model classifiers can be closed-source models (such as GPT-4o, GPT-4o-mini, Claude 3.5Sonnet, etc.) or open-source models (such as Qwen2-VL, LLaVA, InternVL, etc.), as long as they have the ability to accept multiple image inputs and follow natural language instructions. In the API interface of closed-source models, images are encoded as Base64 strings in PNG format and sent together with text prompts as image URL messages; in local open-source models, images are converted into visual tensors and then combined with text prompts by the model processor before being input into the model.

[0049] Model inference parameter settings have a significant impact on output stability. In a preferred embodiment, the generation temperature is set to 0 or close to 0 (e.g., 0.01) to reduce output fluctuations when inferring the same input multiple times; the maximum output length is limited to no more than 128 tokens to reduce irrelevant explanatory text and improve parsing efficiency.

[0050] After receiving input, the multimodal large language model classifier understands the semantic differences between normal and abnormal states based on object descriptions and anomaly category definitions in structured text prompts. It also combines spatial location information provided by visual cue maps and a normal appearance baseline provided by normal reference images, within a predefined set of anomaly categories. C o The category with the highest matching degree is selected as the classification result. The model output can be represented as:

[0051] Where P_φ represents the prediction probability (or equivalent matching score) of the multimodal large language model for the candidate anomaly category c given visual input and text prompts. C o This represents the set of candidate exception categories corresponding to object category o.

[0052] The model output includes anomaly classification results such as anomaly category (e.g., "major damage", "minor damage", or "contamination"), severity level (e.g., "high", "medium", "low"), acceptability judgment ("acceptable" or "unacceptable"), and handling recommendations (e.g., "no intervention required", "re-inspection", "repair", "rework", "scrap"). After parsing the model output, the system maps the natural language category names to standard category codes through the category name mapping table Γ_o, facilitating integration with MES (Manufacturing Execution System) or quality management system.

[0053] It is particularly important to emphasize that the multimodal large language model classifier of this invention keeps the model parameters frozen during inference, eliminating the need for fine-tuning training for specific object categories or anomaly categories (i.e., "zero-training" or "zero-sample" deployment). This characteristic fundamentally distinguishes it from methods that require fine-tuning or auxiliary training on downstream datasets, enabling this invention to be quickly deployed to new production lines or new product categories without collecting large amounts of labeled samples and retraining the model.

[0054] As a concrete example, a normal reference image of the bottle opening is used. I ref Image of the sample to be tested I q Red outline visual cue I v and the aforementioned structured text prompts T structured Input the GPT-4o model sequentially (temperature set to 0). The model outputs the following after inference: Category: Major damage; Severity: High; Acceptability: Unacceptable; Recommendation: Rework. The system parses the output, maps "major damage" to the standard code "BROKEN_LARGE" through the bottle category name mapping table Γ_bottle, and records the severity, acceptability, and handling recommendations in the detection log.

[0055] Example 2 It should be noted that in step S2, the visual expert module employs at least one of the following: an unsupervised anomaly detector based on a diffusion model, an anomaly detector based on pre-trained feature representation, or an anomaly detector based on reconstruction error. The visual expert module calculates an anomaly score map S by estimating the deviation between the normal image distribution and the image to be tested. When the maximum value, mean, quantile statistic, or weighted fusion image-level anomaly score of the anomaly score map S reaches or exceeds a preset detection threshold, the image to be tested is... I q It is determined to be abnormal, and a pixel-level abnormality mask M is generated based on the abnormality score map S.

[0056] As one embodiment, the specific implementation type of the visual expert module and the calculation method of the image-level anomaly score are as follows: (a) Specific implementation types of the visual expert module The vision expert module can be of at least one of the following three types: Type 1: Unsupervised anomaly detector based on diffusion model. The diffusion model models the data distribution of normal images by progressively adding noise to the image and then learning an inverse denoising process. During anomaly detection, the sample image to be tested... I q The input is a diffusion model, which performs an inverse denoising and reconstruction process to obtain the reconstructed image I_rec. Since the diffusion model is only trained on normal samples, its reconstruction of normal regions is relatively accurate, while its reconstruction error for abnormal regions is larger. The anomaly score map S can be calculated... I q The pixel difference between I_rec and I_rec is obtained:

[0057] Here, ||·|2² represents the square of the L2 distance. In practical implementations, the Structural Similarity Index (SSIM) or perceptual loss can also be used as a measure of difference. The diffusion model-based approach is sensitive to subtle anomalies and is suitable for high-precision detection scenarios.

[0058] Type 2: Anomaly detectors based on pre-trained feature representations. This type of method utilizes deep neural networks (such as ResNet, EfficientNet, ViT, etc.) pre-trained on large-scale datasets (such as ImageNet) as feature extractors. For the test sample image... I q The method extracts multi-scale local features from normal samples. Simultaneously, a normal feature library is extracted and stored from normal samples during the training phase. During anomaly detection, each local feature of the image under test is compared with its nearest neighbor feature in the normal feature library, and the feature distance is calculated as the anomaly score at that location, thus obtaining the anomaly score map S. Commonly used distance metrics include Euclidean distance and cosine distance. Methods based on pre-trained feature representations require no training and can directly use pre-trained models, making deployment convenient.

[0059] Type 3: Anomaly Detectors Based on Reconstruction Errors. This type of method employs reconstruction models such as autoencoders, variational autoencoders (VAEs), or U-Net trained on normal samples. The image of the sample to be tested... I qThe reconstruction model is input to obtain the reconstructed image I_rec, and the pixel difference between the two is calculated as the anomaly score map S. Compared with the diffusion model, the reconstruction capability of the traditional autoencoder is relatively limited, but its computational efficiency is higher, making it suitable for production lines with high detection speed requirements.

[0060] (II) Calculation method of image-level anomaly score Image-level anomaly score s img The calculation can be performed using any one or a combination of the following methods: (1) Maximum value: This method can capture the most significant anomalous signals in an image and is sensitive to strong local anomalies.

[0061] (2) Mean: This method reflects the overall degree of anomaly in an image, but it is not sensitive enough to localized, small-area anomalies.

[0062] (3) Quantile statistics: , where Q_p represents the p-th percentile (e.g., p=90, 95, or 99). This approach balances sensitivity to extreme outliers with robustness to noise.

[0063] (4) Weighted fusion: , where α+β+γ=1, and each weight coefficient can be adjusted according to the performance of the validation set.

[0064] when s img Reaching or exceeding the preset detection threshold T detect At that time, the sample image to be tested is judged as abnormal, and the abnormality score S is compared with the pixel-level threshold. T mask The comparison generates a binary anomaly mask M:

[0065] As a concrete example, an anomaly detector based on pre-trained feature representations (such as PatchCore) is used as the visual expert module. This detector uses a Wide ResNet-50 pre-trained on ImageNet as a feature extractor to extract and store a normal feature library from normal bottle samples. For the bottle opening image to be tested, its local features are extracted and then nearest neighbor matching is performed with the normal feature library to calculate the feature distance and obtain the anomaly score map S (with the same resolution as the input image and values ​​normalized to [0,1]). The image-level anomaly score is calculated using the maximum value method: s img =max(S)=0.87. Preset detection threshold. T detect=0.5, since 0.87 ≥ 0.5, the image is determined to be abnormal. Then... T mask Binarize S with a value of 0.5 to generate an anomaly mask M.

[0066] As a concrete example, an unsupervised anomaly detector based on a diffusion model (such as DiffusionAD) is used as the visual expert module. This detector is pre-trained on normal bottle samples. For the image of the bottle opening to be tested, the diffusion model performs an inverse denoising process to reconstruct the image, and the L2 distance between the original image and the reconstructed image is calculated to obtain the anomaly score map S. The image-level anomaly score is calculated using the 90th percentile. s img =Q_90(S)=0.78. Preset detection threshold. T detect =0.55, since 0.78 ≥ 0.55, the image is determined to be abnormal. T mask =0.5 generates an anomaly mask M.

[0067] like Figure 3 As shown, it should be noted that in step S4, the visual cue image... I v The generation method includes the following sub-steps: S41. Normalize and binarize the anomaly score map S or the anomaly mask M to obtain an initial binary mask M0. S42. Perform at least one of connected component analysis, area filtering, hole filling and morphological smoothing on the initial binary mask M0 to filter out noisy connected components with an area smaller than a preset threshold. S43. Extract the set of outer contours of the abnormal region B from the post-processed binary mask; S44, in the image of the sample to be tested I q The outer contour set B is drawn using preset colors, line widths, and transparency to form a visual cue image that marks abnormal areas. I v .

[0068] As one embodiment, visual cue diagram I v The specific steps for generating the sub-steps are as follows: S41: Normalization and Binarization. The anomaly score map S is usually a floating-point matrix, and the numerical range varies depending on the type of vision expert module—some are normalized to [0,1], while others are stored as grayscale values ​​of 0-255. Therefore, normalization is required first to map all values ​​to the [0,1] interval. If S has already been normalized to [0,1], it can be used directly; If S is stored as an 8-bit grayscale image (value range 0-255), then it is normalized by dividing by 255. After normalization, it is compared with a pixel-level threshold. T mask Binarization of the comparison:

[0069] in, This represents the normalized outlier score plot. T mask The value depends on the data source and the output characteristics of the vision expert module: for mask data that already represents the abnormal region with non-zero pixels (i.e., the vision expert module directly outputs the mask instead of the fractional image), a rule greater than 0 can be used. T mask =0); for floating-point fractional graphs normalized to [0,1], T mask Typical values ​​range from 0.3 to 0.7; for an 8-bit grayscale mask, a median grayscale threshold of 127 can be used, or it can be replaced with the value obtained from the validation set calibration. T mask .

[0070] S42: Post-processing. The initial binary mask M0 may contain noise, holes, or irregular boundaries, requiring post-processing to improve the accuracy of contour extraction. Post-processing operations include at least one of the following: Connectivity analysis: Use a connected component labeling algorithm (such as the Two-Pass algorithm or OpenCV's connectedComponents function) to identify all interconnected pixel regions in M0 and assign a unique label to each connected component.

[0071] Area filtering: Calculate the area (number of pixels) of each connected component, and filter out connected components with an area less than a preset threshold A_min (i.e., set the pixel value of that region to 0). The value of A_min is related to the image resolution. For example, for a 1024×1024 image, A_min can be set to 100 pixels; for a 448×448 image, A_min can be set to 20 pixels.

[0072] Hole filling: For holes inside abnormal regions (i.e., normal pixels surrounded by abnormal regions), morphological closing operations (dilation followed by erosion) or flooding filling algorithms are used to fill them, making the abnormal regions more complete.

[0073] Morphological smoothing: Morphological opening operations (erosion followed by dilation) or closing operations are used to smooth the boundaries of abnormal regions, removing jagged edges. The kernel size is typically set to 3×3 or 5×5.

[0074] S43: Extract the outer contour set B. Extract the outer contours of the anomalous regions from the post-processed binary mask. In one implementation, the OpenCV findContours function is used, with the contour retrieval mode set to RETR_EXTERNAL (extracting only the outermost contour) and the contour approximation method set to CHAIN_APPROX_SIMPLE (compressing horizontal, vertical, and diagonal segments, retaining only endpoints). The extracted outer contour set B = {b_1, b_2, ..., b_k}, where k is the number of anomalous connected components.

[0075] S44: Draw a contour to form a visual cue map. In the sample image to be tested... I q The outer contour set B is drawn using preset colors, line widths, and transparency. The contour colors should meet the requirement of high contrast with the image background. In a preferred embodiment, the contour is drawn using red (RGB=255,0,0), with a line width w=2 pixels and transparency α=1.0 (opaque). If the sample image to be tested is a grayscale image, it is first converted to a three-channel RGB image using OpenCV's cvtColor function before drawing the contour. After drawing, the coordinate axes, borders, and extra whitespace are removed before saving.

[0076] like Figure 4 As shown, it should be noted that in step S5, the structured text prompt... T structured The construction method includes the following sub-steps: S51. Based on the object category o, call the normal object description template to generate a normal object description field. The normal object description includes the typical visual features of the object category, assembly relationship, allowable appearance differences and normal process variation range. Specifically, the system reads the corresponding normal object description template from the prompt template library according to the object category o. The normal object description should include the following information: (1) typical visual characteristics of the object category - such as shape (circle, square, line, etc.), color (monochrome, multicolor, standard color scheme), texture (smooth, rough, textured, etc.); (2) assembly relationship - such as the connection method, relative position, hierarchical structure, etc. between the components; (3) allowable appearance differences - such as acceptable color deviation, dimensional tolerance, etc. between different batches of the same product; (4) normal process variation range - such as the normal appearance variation range caused by process fluctuations during production.

[0077] S52. Based on the object category o, call the anomaly category definition template to generate anomaly category definition fields. The anomaly category definition includes the name, location features, morphological features, area or length threshold, functional impact, and exclusivity discrimination boundary between different categories of the candidate anomaly category. Specifically, the system reads the corresponding anomaly category definition template from the prompt template library according to the object category o. Each candidate anomaly category must contain the following information: (1) Name - the standard name of the anomaly category (such as "damage", "scratches", "contamination", "deformation", "missing", etc.); (2) Location characteristics - the location where the anomaly usually occurs (such as "bottle mouth edge", "inside the inner hole", "outer sheath surface", etc.); (3) Morphological characteristics - the visual morphological description of the anomaly (such as "linear", "block", "spot", "gap", etc.); (4) Area or length threshold - a quantitative indicator used to distinguish the severity (such as "damage covering the perimeter reaches or exceeds 20%" is judged as severe); (5) Functional impact - the description of the impact of the anomaly on the product function (such as "affects sealing", "affects electrical connection", "affects only appearance", etc.); (6) Exclusivity discrimination boundary - the priority and exclusion rules when multiple anomaly categories may exist at the same time (such as "contamination category takes precedence over shape category" and "critical area missing category takes precedence over ordinary surface defect category").

[0078] S53. Based on the object category o, call the classification strategy instruction template to generate a classification strategy instruction field. The classification strategy instruction includes hierarchical decision rules that instruct the multimodal large language model to make judgments in sequence according to the abnormal location, abnormal form, severity threshold and user quality standard. Specifically, the system reads the corresponding classification strategy instruction template from the prompt template library according to the object category o. The classification strategy instruction adopts hierarchical decision logic, instructing the multimodal large language model to judge in the following order: (1) First level - judge whether the abnormal location belongs to the critical area (such as the edge of the bottle mouth, the inner hole, the cable conductor, etc.) based on the description of normal objects; (2) Second level - judge the abnormal type based on the location and morphological features in the definition of abnormal category; (3) Third level - judge the severity based on the area or length threshold in the definition of abnormal category; (4) Fourth level - judge the acceptability based on the user quality standard. The judgment result of the previous level serves as the context input for the judgment of the next level.

[0079] S54. The normal object description field, the abnormal category definition field, the classification strategy instruction field, and the output format constraint field are concatenated in a preset order to obtain the structured text prompt. T structured .

[0080] Specifically, the normal object description field, the exception category definition field, the classification strategy instruction field, and the output format constraint field are concatenated into a complete structured text prompt in a preset order. T structuredIn one implementation, the concatenation order is: normal object description → exception category definition → classification strategy instructions → output format constraints, with each part separated by a newline character or delimiter.

[0081] Structured text prompts T structured It is composed of five parts pieced together in sequence, as shown below:

[0082] Basic Description T base (o,g) (includes object category and visual cue pattern identifier), normal object description D normal (o) Candidate anomaly category definition set {D} cls (c)} Classification strategy instruction R strategy (o) and output format constraints R out Where g is the visual cue mode identifier, g=contour indicates that a visual cue image with outline is input, and g=none indicates that no visual cue image is input.

[0083] It should be noted that in step S6, after receiving the input, the multimodal large language model classifier, based on the structured text prompts... T structured Understanding the object descriptions and exception category definitions in the text helps to grasp the semantic differences between normal and abnormal states, in conjunction with the visual cue diagrams. I v The provided spatial positioning information and the normal reference image I ref The provided normal appearance baseline is based on a predefined set of anomaly categories. C o Select the category with the highest matching degree as the classification result; The severity level is based on the structured text prompt. T structured The severity criteria output is defined in [the standard], and the severity criteria include the percentage of abnormal area. percentage of abnormal lengths The abnormality covers at least one of the following: the proportion of critical areas covered by the abnormality; whether the abnormality affects structural integrity or electrical connectivity.

[0084] Wherein, H and W represent the sample image to be tested, respectively. I q Height and width, R key L(·) represents the critical structural region, and L(·) represents the contour length or the reference length defined by the process.

[0085] As one example, the classification decision mechanism and severity evaluation criteria of the multimodal large language model are as follows: (I) Classification Decision-Making Mechanism After receiving four types of input, the multimodal large language model classifier performs the following reasoning process: (1) through a normal reference image I ref Establish a visual baseline for normal appearance—the model understands “what the object should look like under normal circumstances”; (2) Through the test sample image I q Observe the overall visual appearance of the current sample; (3) Locate the spatial location of the abnormal area through the high-contrast outline in the visual cue image I_v - the red outline guides the model to focus attention on the abnormal area and reduce attention to the background and irrelevant areas; (4) Use structured text prompts T structured Acquire prior knowledge for classification—including the definition of normal states, the discrimination boundaries of candidate abnormal categories, hierarchical decision rules, and output format requirements.

[0086] After integrating the above four types of information, the model selects from a predefined set of anomaly categories. C o Select the category with the highest matching degree as the classification result:

[0087] in, This represents the predicted probability of a multimodal large language model for a candidate anomaly category c given the input conditions.

[0088] (ii) Severity assessment The severity level is quantitatively assessed based on the severity criteria defined in the structured text prompt. The severity criteria include at least one of the following: (1) Percentage of abnormal area The percentage of the area of ​​the abnormal region relative to the total area of ​​the image.

[0089]

[0090] Where ΣM(x,y) represents the total number of pixels with a value of 1 in the anomaly mask M (i.e., the area of ​​the anomaly region), and H and W represent the height and width of the image, respectively (in pixels). The value range is [0,1], with a larger value indicating a higher proportion of abnormal area. For example, when the abnormal area is 15,000 pixels and the total image area is 448 × 448 = 200,704 pixels, =15000 / 200704≈0.075 (approximately 7.5%).

[0091] (2) Percentage of abnormal lengths The proportion of the length of the abnormal region covering the critical structure to the total length of the critical structure.

[0092]

[0093] in, Indicates key structural areas (such as the edge of a bottle neck, the annular area of ​​a cable sheath, etc.), L(·) represents the contour length or the reference length defined by the process (unit: pixels or millimeters), B∩ This represents the intersection of anomaly contours and critical structural regions. The value range is [0,1], with a larger value indicating a higher coverage ratio of the anomaly on the critical structure. For example, for bottle opening damage detection, if the total perimeter of the bottle opening edge is 300 pixels and the arc length of the damage coverage is 75 pixels, then... =75 / 300=0.25 (25%).

[0094] (3) The proportion of abnormal coverage of critical areas: directly assess whether the abnormal area covers the critical functional areas of the product (such as sealing surfaces, electrical contact points, load-bearing structures, etc.) and the size of the coverage area.

[0095] (4) Whether the abnormality affects the structural integrity or electrical connectivity: a binary judgment indicator. If the abnormality damages the structural integrity of the product (such as crack penetration or gaps leading to insufficient strength) or electrical connectivity (such as conductor breakage or insulation layer damage leading to short circuit risk), the severity is directly judged as "high".

[0096] In one implementation, the severity levels are divided into three levels: low (minor anomaly, not affecting functionality), medium (anomaly affecting appearance or non-critical functions), and high (anomaly affecting structural integrity, electrical connectivity, or safety). Thresholds for each level are predefined in structured text prompts.

[0097] It should be noted that in step S6, the acceptability determination is based on the user's predefined acceptance criteria to determine whether the detected anomaly requires intervention. If the anomaly is a design update, a change in appearance within the allowable range of the process, or a minor anomaly that has no significant impact on the product's function, an acceptable result is output and a suggestion of no intervention or re-inspection is given. If the anomaly affects structural integrity, assembly integrity, electrical connectivity, or safety, an unacceptable result is output and an intervention suggestion is given. The handling suggestion includes at least one of the following: no intervention, re-inspection, repair, rework, or scrapping.

[0098] As one example, acceptability determination is a crucial step in making quality decisions regarding detected anomalies, based on user-defined acceptance criteria. These acceptance criteria are pre-set by production line quality engineers or process engineers according to product specifications, customer requirements, and industry standards, and are embedded in structured text prompts. Acceptance criteria typically include the following determination rules: Acceptable anomalies include design updates (such as intentional changes in appearance color during product iteration), appearance changes within the allowable range of the process (such as slight flow marks on the surface of injection molded parts, normal oxidation color difference on the surface of metal parts), or minor anomalies that have no significant impact on product function (such as minor scratches on non-sealed surfaces, minor indentations in non-critical areas). For acceptable anomalies, the system outputs an "Acceptable" result and provides handling suggestions of "No intervention required" or "Re-inspection" (such as recommending random sampling and re-inspection).

[0099] Unacceptable anomalies include those affecting structural integrity (e.g., through-cracks leading to insufficient strength, damage causing load-bearing structure failure), assembly integrity (e.g., dimensional deviations preventing assembly, missing positioning features), electrical connectivity (e.g., conductor breakage, insulation damage leading to short circuit risk), or safety (e.g., sharp edges, burrs potentially causing user injury). For unacceptable anomalies, the system outputs "Unacceptable" and provides specific intervention suggestions.

[0100] The suggested actions are executable instructions output by the system, and must include at least one of the following: No intervention required: If the anomaly is within the product's allowable tolerances and does not affect any functional or appearance requirements, it can be released directly.

[0101] Re-inspection: The abnormality is at a critical level. It is recommended to conduct a second confirmation by manual inspection or with higher precision testing equipment.

[0102] Repair: Abnormalities can be repaired through simple processes (such as removing burrs and cleaning surface contaminants), and the product can be used normally after repair.

[0103] Rework: Abnormalities require reprocessing or partial repair to meet the acceptance standards, and re-inspection is required after rework.

[0104] Scrap: If the abnormality is so severe that it cannot be repaired or the repair cost is too high, the product should be scrapped.

[0105] like Figure 5 As shown, it should be noted that the method also includes a refined dataset construction and evaluation step S7, as detailed below: S71. Perform category label correction and sample screening on the original anomaly detection dataset, including correcting incorrectly labeled samples in the original dataset, merging anomaly categories with highly similar visual features and unstable discrimination boundaries, removing anomaly categories with insufficient sample numbers to a preset threshold, and removing cross-category combined anomaly categories to generate a refined dataset suitable for anomaly classification tasks. Specifically, raw anomaly detection datasets (such as MVTec-AD, VisA, etc.) are typically designed for anomaly detection tasks, and their annotation methods may not be entirely suitable for evaluating anomaly classification tasks. Therefore, it is necessary to refine the raw datasets: (1) Correcting mislabeled samples: The images, masks, and category labels of each anomalous sample in the original dataset are manually reviewed. For samples that are clearly inconsistent with the category definition, their labels are corrected. For example, some "damaged" class samples in the MVTec-AD dataset are actually "contaminated" and should be corrected.

[0106] (2) Merging categories with highly similar visual features: For abnormal categories that are visually similar and difficult to form a stable exclusive discrimination boundary, they are merged into a unified category. For example, if “scratches” and “scratches” are visually difficult to distinguish and have similar process meanings, they can be merged into the category of “surface damage”.

[0107] (3) Remove categories with insufficient sample size: For abnormal categories with a sample size lower than the preset threshold (e.g., less than 5 images), they are removed from the evaluation or merged into the higher category because their statistical significance is insufficient.

[0108] (4) Remove cross-category combination anomalies: For samples containing multiple different types of anomalies (such as "damage" and "contamination" in the same image), if they cannot be clearly classified into a single category, they are removed from the classification assessment to avoid assessment ambiguity.

[0109] Taking the MVTec-AD dataset as an example, a refined dataset (which can be called MVTec-AC) suitable for anomaly classification tasks can be constructed. This process includes: correcting several mislabeled samples; merging anomaly categories with highly similar visual features (e.g., merging "bent" and "broken" into "deformation"); removing object categories containing only a single simple anomaly category; and removing combined anomaly categories that cannot provide specific severity information.

[0110] S72. Based on the refined dataset, calculate the anomaly classification accuracy Acc, macro-F1 score, Cohen's Kappa score κ, and confusion matrix. Calculate the evaluation index for each object category and take the average as the overall evaluation result.

[0111]

[0112]

[0113] in, This represents the number of test samples in object category o, where o represents the set of object categories. This represents the set of candidate exception categories under object category o. and These represent the true category and the standardized predicted category, respectively. Indicates the observational consistency rate. Represents the random consistency rate. [·] indicates an indicator function.

[0114] Based on the refined dataset, the following evaluation metrics are used to assess the performance of the anomaly classification method: (1) Accuracy The proportion of correctly classified samples in object category o out of the total number of samples.

[0115]

[0116] I[·] is an indicator function (value 1 when the condition is true, otherwise 0). The accuracy range is [0,1], and the higher the value, the more accurate the classification.

[0117] (2) Macro-F1: First, calculate the F1 score for each category, and then take the arithmetic mean of all categories. This method is more robust to class imbalance and will not mask the performance of small sample categories due to the advantage of large sample categories.

[0118]

[0119]

[0120]

[0121] in, = TP / (TP+FP) represents the precision of class c. = TP / (TP+FN) represents the recall rate of category c, where TP, FP, and FN represent the number of true positive, false positive, and false negative samples, respectively. C o This represents the set of candidate exception categories under object category o, where O represents the set of all object categories.

[0122] (3) Cohen's Kappa value κ: measures the degree of consistency between the classification result and the true label, taking into account the influence of random consistency, and the value range is [-1,1]. Usually, κ≥0.6 indicates good consistency.

[0123]

[0124] in, This indicates the observational consistency rate (i.e., accuracy rate). ), This represents the random consistency rate (i.e., the expected consistency rate when two evaluators classify randomly, calculated using the marginal probabilities of the confusion matrix). =1 indicates that they are completely identical. =0 indicates that consistency is equivalent to randomness. <0 indicates that consistency is lower than randomness.

[0125] (4) Confusion matrix: is a | |×| A confusion matrix is ​​a matrix where rows represent the true classes and columns represent the predicted classes. Each matrix element represents the number of true samples classified as a particular predicted class. A confusion matrix visually illustrates the classification confusion between different classes.

[0126] For multi-category anomaly classification tasks, calculations can be performed separately for each object category. Acc o , Macro-F1 o and κ o Then, the average value of all object categories is taken as the overall evaluation result to reduce the evaluation bias caused by differences in the number of categories and sample size.

[0127] As a concrete example, let's take the bottle class from the MVTec-AD dataset as an example: The set of candidate anomaly categories C for refined bottle types bottle = {'major damage', 'minor damage', 'contamination'}, where N is the total number of test samples. bottle =100.

[0128] After comparing the model's predictions with the true labels, a confusion matrix is ​​obtained, and Acc is calculated. bottle =0.88 (88%), Macro-F1 bottle =0.85, κ bottle =0.82.

[0129] The above indicators were calculated for all 15 object categories (O = 15), and the average was taken to obtain the overall evaluation results: Acc = 0.84, Macro-F1 = 0.80, κ = 0.80.

[0130] Final Reference Figure 6 , Figure 6 The present invention illustrates a specific process example of the method. First, the visual anomaly detector performs a normal / abnormal judgment and anomaly region location on the image to be tested. If the sample is normal, the normal result is directly output. If the sample is abnormal, a visual cue image with a red outline is generated and combined with the normal reference image, the image to be tested, and the text cue input into a multimodal large language model. Finally, the anomaly category and quality handling result are output.

[0131] Example 3 Please see Figure 2 , Figure 2 This is a schematic diagram of the system structure of the present invention.

[0132] An industrial anomaly classification system based on visual experts and a multimodal large language model includes: The image acquisition module is configured to acquire images of the test samples. I q and object category information; The vision expert module is configured to analyze the image of the sample to be tested. I q Perform unsupervised anomaly detection and pixel-level anomaly localization, and output an anomaly score map S and anomaly mask M; The pre-filtering module is configured to calculate image-level anomaly scores based on the anomaly score map S. s img In the image-level anomaly score s img Below the preset detection threshold T detect If the error occurs, the normal judgment result is output directly and the process is terminated, only abnormal samples are passed to subsequent modules; The visual cue generation module is configured to perform post-processing on the anomaly mask M and extract the anomaly region contours, then overlay the extracted anomaly region contours onto the test sample image as high-contrast contour lines. I q Above, generate visual cue image I_v; The text prompt building module is configured to construct a structured text prompt based on the object category o, which includes a description of normal objects, a definition of abnormal categories, classification strategy instructions, and output format constraints. T structured ; The multimodal large language model classification module is configured to receive normal reference images. I ref The image of the sample to be tested I q The aforementioned visual cue diagram Iv and the structured text prompts T structured The system outputs anomaly classification results, severity level, acceptability judgment, and handling suggestions; the model parameters of the multimodal large language model classification module remain frozen during inference, eliminating the need for fine-tuning training for specific object categories or anomaly categories.

[0133] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An industrial anomaly classification method based on visual experts and a multimodal large language model, characterized in that: Includes the following steps: S1. Obtain the image of the sample to be tested. I q And the object category o, and select at least one normal reference image from the normal reference image library according to the object category o. I ref ; S2, the image of the sample to be tested I q The input is a visual expert module, which uses an unsupervised anomaly detection model to analyze the image of the test sample. I q Perform pixel-level anomaly detection and localization, and output an anomaly score map S and anomaly mask M; S3. Calculate the image-level anomaly score based on the anomaly score map S. s img The image-level anomaly score s img With preset detection threshold T detect Compare; if the image-level anomaly score s img Below the preset detection threshold T detect If the image-level anomaly score is not found, the normal judgment result will be output directly and the process will terminate; if the image-level anomaly score is not found, the normal judgment result will be output directly and the process will terminate. s img Reaching or exceeding the preset detection threshold T detect Then proceed to step S4; the anomaly mask M, after being generated, is also used for the image-level anomaly score. s img The calculation and visual cue diagram in step S4 I v The generation of; S4. Perform post-processing on the anomaly mask M and extract the anomaly region contours. Superimpose the extracted anomaly region contours onto the test sample image in the form of high-contrast contour lines. I q Above, generate visual cue images. I v The color of the high-contrast contour line is similar to that of the sample image under test. I q The color difference of the background color meets the preset color difference threshold, and the outline only marks the outer boundary of the abnormal area; S5. Construct a structured text prompt based on the object category o. T structured The structured text prompt T structured It includes descriptions of normal objects, definitions of exception categories, classification strategy instructions, and output format constraints; S6. The normal reference image I ref The image of the sample to be tested I q The aforementioned visual cue diagram I v and the structured text prompts T structured The data is input to a multimodal large language model classifier, which outputs anomaly classification results, severity levels, acceptability judgments, and handling suggestions. The model parameters of the multimodal large language model classifier remain frozen during inference, eliminating the need for fine-tuning training for specific object categories or anomaly categories.

2. The industrial anomaly classification method based on visual experts and a multimodal large language model as described in claim 1, characterized in that, In step S2, the visual expert module employs at least one of an unsupervised anomaly detector based on a diffusion model, an anomaly detector based on pre-trained feature representation, or an anomaly detector based on reconstruction error; the visual expert module calculates the anomaly score map S by estimating the degree of deviation between the normal image distribution and the sample image to be tested; When the maximum value, mean, quantile statistical value, or weighted fusion image-level anomaly score of the anomaly score map S reaches or exceeds a preset detection threshold, the sample image to be tested will be... I q It is determined to be abnormal, and a pixel-level abnormality mask M is generated based on the abnormality score map S.

3. The industrial anomaly classification method based on visual experts and a multimodal large language model as described in claim 1, characterized in that: In step S4, the visual cue image I v The generation method includes the following sub-steps: S41. Normalize and binarize the anomaly score map S or the anomaly mask M to obtain an initial binary mask M0. S42. Perform at least one of connected component analysis, area filtering, hole filling and morphological smoothing on the initial binary mask M0 to filter out noisy connected components with an area smaller than a preset threshold. S43. Extract the set of outer contours of the abnormal region B from the post-processed binary mask; S44, in the image of the sample to be tested I q The outer contour set B is drawn using preset colors, line widths, and transparency to form a visual cue image that marks abnormal areas. I v .

4. The industrial anomaly classification method based on visual experts and a multimodal large language model as described in claim 1, characterized in that: In step S5, the structured text prompt T structured The construction method includes the following sub-steps: S51. Based on the object category o, call the normal object description template to generate a normal object description field. The normal object description includes the typical visual features of the object category, assembly relationship, allowable appearance differences and normal process variation range. S52. Based on the object category o, call the anomaly category definition template to generate anomaly category definition fields. The anomaly category definition includes the name, location features, morphological features, area or length threshold, functional impact, and exclusivity discrimination boundary between different categories of the candidate anomaly category. S53. Based on the object category o, call the classification strategy instruction template to generate a classification strategy instruction field. The classification strategy instruction includes hierarchical decision rules that instruct the multimodal large language model to make judgments in sequence according to the abnormal location, abnormal form, severity threshold and user quality standard. S54. The normal object description field, the abnormal category definition field, the classification strategy instruction field, and the output format constraint field are concatenated in a preset order to obtain the structured text prompt. T structured .

5. The industrial anomaly classification method based on visual experts and a multimodal large language model as described in claim 1, characterized in that, In step S6, after receiving the input, the multimodal large language model classifier, based on the structured text prompts... T structured Understanding the object descriptions and exception category definitions in the text helps to grasp the semantic differences between normal and abnormal states, in conjunction with the visual cue diagrams. I v The provided spatial positioning information and the normal reference image I ref The provided normal appearance baseline is based on a predefined set of anomaly categories. C o Select the category with the highest matching degree as the classification result; The severity level is based on the structured text prompt. T structured The severity criteria output is defined in [the standard], and the severity criteria include the percentage of abnormal area. ρ area percentage of abnormal lengths The abnormality covers at least one of the following: the proportion of critical areas covered by the abnormality; whether the abnormality affects structural integrity or electrical connectivity. Wherein, H and W represent the sample image to be tested, respectively. I q Height and width, R key L(·) represents the critical structural region, and L(·) represents the contour length or the reference length defined by the process.

6. The industrial anomaly classification method based on visual experts and a multimodal large language model as described in claim 1, characterized in that, In step S6, the acceptability determination is based on the user's predefined acceptance criteria to determine whether the detected anomaly requires intervention. If the anomaly is a design update, a change in appearance within the allowable range of the process, or a minor anomaly that has no significant impact on the product function, an acceptable result will be output and a suggestion of no intervention or re-inspection will be given. If the anomaly affects structural integrity, assembly integrity, electrical connectivity, or safety, an unacceptable result will be output and intervention recommendations will be given; the intervention recommendations include at least one of the following: no intervention, re-inspection, repair, rework, or scrapping.

7. The industrial anomaly classification method based on visual experts and a multimodal large language model as described in claim 1, characterized in that, The method also includes refining the dataset construction and evaluation step S7, as follows: S71. Perform category label correction and sample screening on the original anomaly detection dataset, including correcting incorrectly labeled samples in the original dataset, merging anomaly categories with highly similar visual features and unstable discrimination boundaries, removing anomaly categories with insufficient sample numbers to a preset threshold, and removing cross-category combined anomaly categories to generate a refined dataset suitable for anomaly classification tasks. S72. Based on the refined dataset, calculate the anomaly classification accuracy Acc, macro-F1 score, Cohen's Kappa score κ, and confusion matrix. Calculate the evaluation index for each object category and take the average as the overall evaluation result. in, This represents the number of test samples in object category o, where o represents the set of object categories. This represents the set of candidate exception categories under object category o. and These represent the true category and the standardized predicted category, respectively. Indicates the observational consistency rate. Represents the random consistency rate. [·] indicates an indicator function.

8. An industrial anomaly classification system based on visual experts and a multimodal large language model, characterized in that, include: The image acquisition module is configured to acquire images of the test samples. I q and object category information; The vision expert module is configured to analyze the image of the sample to be tested. I q Perform unsupervised anomaly detection and pixel-level anomaly localization, and output an anomaly score map S and anomaly mask M; The pre-filtering module is configured to calculate image-level anomaly scores based on the anomaly score map S. s img In the image-level anomaly score s img Below the preset detection threshold T detect If the error occurs, the normal judgment result is output directly and the process is terminated, only abnormal samples are passed to subsequent modules; The visual cue generation module is configured to perform post-processing on the anomaly mask M and extract the anomaly region contours, then overlay the extracted anomaly region contours onto the test sample image as high-contrast contour lines. I q Above, generate visual cue images. I v ; The text prompt building module is configured to construct a structured text prompt based on the object category o, which includes a description of normal objects, a definition of abnormal categories, classification strategy instructions, and output format constraints. T structured ; The multimodal large language model classification module is configured to receive normal reference images. I ref The image of the sample to be tested I q The aforementioned visual cue diagram I v and the structured text prompts T structured The system outputs anomaly classification results, severity level, acceptability judgment, and handling suggestions; the model parameters of the multimodal large language model classification module remain frozen during inference, eliminating the need for fine-tuning training for specific object categories or anomaly categories.

Images