Method for industrial defect diagnosis based on segmentation-aware vision-language model

Abstract

This invention, based on a segmentation-aware visual-language model, is an industrial defect diagnosis method belonging to the field of computer vision and image processing technology. First, it utilizes the CLIP-SAM cascaded cueing mechanism, leveraging the zero-shot generalization capability of the CLIP model for coarse-grained defect localization, and then guides the SAM model to generate pixel-level fine-grained segmentation masks. Second, it introduces a multi-scale cue learner and mask decoder, adaptively processing cross-scale defect features ranging from microscopic scratches to large-area stains through a multi-branch structure. Finally, it constructs a segmentation-aware semantic alignment (SASA) module, establishing a precise semantic association between the pixel-level segmentation mask and the text embedding through a bidirectional cross-attention mechanism, forcing the language model to focus on the true defect region when generating defect descriptions. This invention achieves superior defect detection accuracy and description consistency compared to existing methods on multiple industrial datasets, and is applicable to automated quality inspection in various intelligent manufacturing scenarios.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and image processing technology, and specifically relates to an industrial defect diagnosis method based on a segmentation-aware visual-language model. Background Technology

[0002] Industrial defect detection is a crucial step in ensuring product quality and production efficiency. Traditional methods rely on manual features, resulting in poor generalization ability. With the development of deep learning, methods based on convolutional neural networks or visual transformers, such as PatchCore and SPADE, have achieved good performance by building memory banks or feature pyramid matching. However, these methods typically follow a "one class, one model" approach, requiring large amounts of labeled data and struggling to adapt to new defect categories or product line changes, thus limiting their deployment in dynamic industrial environments.

[0003] In recent years, the emergence of Large Vision-Language Models (LVLMs) has brought new ideas to industrial diagnostics. By introducing language priors, such as AnomalyGPT and SPGDD-GPT, and leveraging CLIP's cross-modal alignment capabilities or learning through cueing, they have shown potential in zero-shot and few-shot defect detection. However, existing LVLM methods still face the following key challenges: First, the localization accuracy is insufficient. Existing visual encoders typically rely on fixed image patch-level similarity calculations, making it difficult to accurately locate tiny defects occupying only a few pixels. Second, descriptive illusions are severe. When inputting detection results into a large language model, existing methods often simply concatenate image features and text tokens, failing to establish deep semantic alignment. This results in descriptions that are inconsistent with the actual defect areas, and even fabricated "illusions." Third, scale adaptability is poor. Most frameworks treat defect localization and description as isolated stages, lacking robustness to handle cross-scale defects ranging from microscopic scratches to large-area stains.

[0004] Therefore, researching an automated industrial diagnostic method that can achieve precise positioning, consistent description, and adaptability to multi-scale defects is of great significance for improving the autonomy and reliability of intelligent manufacturing systems. Summary of the Invention

[0005] This invention proposes an industrial defect diagnosis method based on a segmentation-aware vision-language model (SAM-LLaVA), aiming to address the technical problems of insufficient localization accuracy, severe descriptive illusion, and poor scale adaptability of existing large vision-language models in industrial defect diagnosis. This invention achieves end-to-end optimization from coarse-grained localization to fine-grained segmentation and description generation by constructing a CLIP-SAM cascaded prompting mechanism, a multi-scale feature extraction architecture, and a segmentation-aware semantic alignment module.

[0006] An industrial defect diagnosis method based on a segmentation-perception vision-language model is characterized by the following steps: (1) Image acquisition and preprocessing: Acquire the RGB image of the industrial product to be diagnosed and perform normalization preprocessing. (2) Constructing a CLIP-SAM cascading hint mechanism: (2.1) Coarse-grained defect localization: Using the image encoder and text encoder of the pre-trained CLIP model, image features and predefined text features are generated. The text features include normal text features and abnormal text features. The contrast distance between the image features and the text features is calculated to generate a coarse-grained defect response heatmap. An initial coarse-grained mask is obtained by adaptive threshold binarization of the heatmap. (2.2) Fine-grained defect segmentation: The coarse-grained mask is used as a spatial cue and is input into the pre-trained SAM model along with the RGB image. The SAM's image encoder, cue encoder, and mask decoder generate multiple candidate masks, and the mask with the highest confidence is selected as the final fine-grained segmentation mask. (3) Multi-scale defect feature extraction: The coarse-grained mask is processed by a multi-scale cue learner to generate cue embeddings with three scales: global, regional, and detailed. The segmentation mask is predicted in parallel on three resolution branches (high, medium, and low) by a multi-scale mask decoder, and the output of the medium resolution branch is used as the final segmentation result. (4) Construct a segmentation-aware semantic alignment module: The fine segmentation mask is multiplied element-wise with the features extracted by the SAM image encoder to obtain defect-aware visual features. These visual features are then input together with the text features extracted by the CLIP model's text encoder into a bidirectional cross-attention module, where visual-to-text attention operations and text-to-visual attention operations are performed sequentially to generate semantically aligned visual and text features. (5) Defect description generation: The aligned visual and textual features are concatenated and input into a Large Language Model (LLM) to generate defect description text. (6) Model training: The segmentation-aware visual-language model is trained through steps (1) to (5), and the total loss function of the model is the weighted sum of segmentation loss and defect description text generation loss. (7) Model Deployment: After training is complete, in practical applications, input the image to be processed and execute the above steps (1)-(5).

[0007] The CLIP-SAM cascaded hint mechanism in step (2) involves using CLIP to generate a coarse segmentation mask through contrastive learning, and then using SAM to segment a more fine-grained defect segmentation mask map using the coarse segmentation mask of CLIP as a hint.

[0008] Step (2.1) further includes introducing a small number of normal image samples as a support set under a few-sample setting to construct a multi-scale normal feature memory bank; when step (2.1) is executed in step (7), the minimum cosine distance between the query image feature and the nearest feature in the memory bank is calculated to generate a context-aware anomaly score map, which is converted into modulation coefficients by the Sigmoid function to enhance or suppress the coarse-grained defect response heatmap.

[0009] The multi-scale cue learner in step (3) contains three parallel convolutional paths with different receptive fields, which are used to extract the global structure, regional context and local detail features of the coarse-grained mask, respectively, and the three features are concatenated along the channel dimension to form a multi-scale cue embedding.

[0010] In step (3), the multi-scale mask decoder supervises the outputs of the high, medium and low resolution branches simultaneously during the training phase. The segmentation loss is the weighted sum of the losses of each branch, with the medium resolution branch having the highest weight. In the inference phase of step (7), only the output of the medium resolution branch is used to improve efficiency.

[0011] The bidirectional cross-attention module in step (4) specifically refers to: (4.1) Using the defect-aware visual features as the query and the text features as the key and value, perform the first cross-attention operation to make the visual features focus on the most relevant semantic words, and obtain the enhanced visual features. (4.2) Using text features as queries and enhanced visual features as keys and values, perform a second cross-attention operation to focus text features on the most relevant visual regions, resulting in aligned text features.

[0012] The large language model in step (5) is Vicuna-7B-v1.5, and low-rank adaptation (LoRA) technology is used for efficient parameter fine-tuning, updating only the query and value projection matrices in the self-attention mechanism and the linear layers in the feedforward network.

[0013] In step (6), the segmentation loss includes a weighted combination of the Dice loss and the Focal loss of each branch of the multi-scale mask decoder; the defect description text generation loss is the autoregressive cross-entropy loss used to supervise the generation of defect description text.

[0014] The application of the method is characterized in that the method is applied to product quality inspection in industrial manufacturing, and the product quality inspection includes surface defect detection of electronic components, scratch detection of metal workpieces, defect identification of textile fabrics, and diagnosis of molding defects in injection molded parts.

[0015] Advantages of the invention This invention innovatively introduces a segmentation perception mechanism into a large vision-language model, systematically optimizing the industrial defect diagnosis process. First, this invention effectively solves the problem of inaccurate localization of minute defects in existing LVLM methods. By constructing a CLIP-SAM cascaded cue mechanism, it utilizes the zero-sample generalization capability of CLIP to provide coarse-grained spatial cues, which then guide SAM for pixel-level fine segmentation. This overcomes the limitation of CLIP's fixed image block granularity, achieving a leap in accuracy from "region-level localization" to "pixel-level segmentation."

[0016] Secondly, this invention significantly reduces the "illusion" phenomenon in descriptive text by constructing a segmentation-aware semantic alignment module. Unlike existing methods that simply stitch together image and text features, this invention establishes a precise correspondence between pixel-level segmentation masks and word-level semantics through bidirectional cross-attention, forcing the language model to focus on real defect areas when generating descriptions, thereby ensuring the authenticity and consistency of the descriptive content.

[0017] Third, this invention enhances robustness to cross-scale defects through a multi-scale cue learner and a mask decoder. By processing global, regional, and detailed features in parallel and employing multi-resolution branches for supervised learning, the model can simultaneously capture the fine structure of microscopic scratches and the overall morphology of large-area stains, improving its generalization ability in complex industrial scenarios.

[0018] Finally, this invention employs a parameter-efficient LoRA fine-tuning strategy, combined with end-to-end joint optimization, enabling the entire model to quickly adapt to new industrial scenarios with a small number of samples and to be easily deployed on edge devices with limited computing power, providing a practical and feasible technical path for automated and intelligent quality inspection in smart manufacturing. Attached Figure Description

[0019] Figure 1 This is the overall architecture and process framework of SAM-LLaVA of the present invention.

[0020] Figure 2 This is a schematic diagram of the multi-scale cue learner and mask decoder structure in this invention.

[0021] Figure 3 This is a schematic diagram of the segmentation-aware semantic alignment module structure in this invention.

[0022] Figure 4Image of a fabric with a defect featuring a black scratch.

[0023] Figure 5 For the present invention in Figure 4 The description results on the defective data are compared with the description results of other methods.

[0024] Figure 6 Image of a white pill that does not contain defects.

[0025] Figure 7 For the present invention in Figure 6 The description results on the defect-free data are compared with the description results of other methods. Detailed Implementation

[0026] Industrial defect diagnosis requires models to not only segment industrial defects but also generate corresponding textual descriptions. Addressing the shortcomings of existing Large Visual-Language Models (LVLMs) in industrial defect diagnosis, such as insufficient defect localization accuracy, inconsistencies between textual descriptions and visual evidence (i.e., illusions), and difficulty in adapting to multi-scale defects, this invention proposes a segmentation-aware visual-language framework—an industrial defect diagnosis method based on a segmentation-aware visual-language model (SAM-LLaVA). The complete workflow framework of this invention is as follows: Figure 1 As shown, it includes: Step 1. Image Acquisition and Preprocessing. RGB images of the product to be inspected are acquired using a camera at the industrial site, and then subjected to size normalization and numerical normalization.

[0027] Step 2. Construct a CLIP-SAM cascading hint mechanism to achieve defect localization from coarse to fine.

[0028] Step 3. Construct a multi-scale cue learner and a mask decoder to extract and fuse multi-scale features.

[0029] Step 4. Construct a segmentation-aware semantic alignment module to achieve precise alignment between pixel-level masks and text embeddings.

[0030] Step 5. Construct a large language model description generator to generate defect description text, and use LoRA technology for efficient parameter fine-tuning.

[0031] Step 6. Model training: The total loss function of the model is a weighted sum of the segmentation loss and the defect description text generation loss. Step 7. Model Deployment: In practical applications, input the image to be processed, and the model will automatically execute the above steps to achieve automated industrial defect diagnosis.

[0032] The following specific examples will further illustrate this point.

[0033] Example The core of this invention lies in enhancing the fine-grained localization capability and semantic consistency of the large vision-language model for industrial defects through a segmentation perception mechanism. The implementation of this invention will be described in detail below with reference to the accompanying drawings and specific algorithm steps.

[0034] 1. Industrial Image Acquisition and Preprocessing First, an RGB image of the product to be inspected is acquired using an industrial camera, denoted as . To ensure consistency of model input, for Size normalization and numerical normalization are performed to scale pixel values ​​to the [0,1] range.

[0035] 2. Construct a CLIP-SAM cascading suggestion mechanism The CLIP-SAM cascaded cueing mechanism of this invention aims to combine the zero-shot generalization capability of CLIP with the precise segmentation capability of SAM.

[0036] Coarse-grained localization: using a pre-trained CLIP image encoder and text encoder Predefine normal text descriptions (e.g., "a normal product") and abnormal text descriptions (e.g., "a defective product"). Calculate the contrast similarity between image features and text features to generate a coarse-grained defect heatmap. Through adaptive thresholding Binarization yields the initial coarse-grained mask. .

[0037] Fine-grained segmentation: As a spatial cue, with Input SAM together. SAM's image encoder Extracting image features The encoder is prompted. Encoding mask hint Mask decoder By fusing features and hints, multiple candidate masks and their IoU scores are generated, and the mask with the highest score is selected as the fine-grained segmentation mask. .

[0038] 3. Multi-scale cue learning and mask decoding To address cross-scale defects, this invention enhances the cue encoder and mask decoder of SAM in multiple scales, such as... Figure 2 As shown.

[0039] Multi-scale cue learner: This module receives It extracts features at different scales through three parallel convolutional paths: one uses a large convolutional kernel (such as 7×7) to extract the global structure. A method that uses a medium convolutional kernel (e.g., 3×3) to extract region context. One method uses small convolutional kernels (such as 1×1) to extract local details. The three features are spliced ​​along the channel to form a multi-scale cue embedding, which replaces the original SAM cue encoder output.

[0040] Multi-scale mask decoder: This decoder predicts masks in parallel across three resolution branches (high, medium, and low). During training, the output of each branch... , , Both are downsampled to the true mask at the corresponding resolution. Calculate the loss: , in, For branch weights, the medium-resolution branch has the highest weight (e.g., ...). To ensure the quality of the final output, only the output of the medium-resolution branch is used as the final segmentation result during inference.

[0041] 4. Segmentation-Aware Semantic Alignment Module To establish a precise relationship between the segmentation results and the descriptive text, this invention designs a SASA module, such as... Figure 3 As shown.

[0042] First, the fine segmentation mask SAM image features Element-wise multiplication yields visual features that focus only on the defective regions. Simultaneously, CLIP text features are extracted. Then, bidirectional cross-attention is performed: Visual → Text Alignment: For query (Q), For the key (K) and value (V), perform cross-attention to focus the visual features on the most relevant semantic words, resulting in... .

[0043] Text → Visual Alignment: with For query (Q), For the key (K) and value (V), perform cross-attention to focus text features on the most relevant visual regions, resulting in... .

[0044] Finally, the aligned and Concatenate the sequences as input to the large language model.

[0045] 5. Defect description generation and efficient parameter fine-tuning Alignment sequence output by the SASA module Input a large language model (Vicuna-7B-v1.5 in this example). The model generates defect descriptions word by word using an autoregressive approach. To ensure accuracy, cross-entropy loss is used for supervision. . To reduce training costs, LoRA (Local Alternative Relationship) is used to fine-tune the LLM. A low-rank adapter is injected into the LLM's Transformer layer, updating only the query (Q), value (V) projection matrices and the feedforward network layer; all other parameters are frozen. The model's total loss is a weighted sum of the segmentation loss and the generation loss. , in and Set all values ​​to 1.0.

[0046] 6. Experimental Verification and Effect Evaluation To verify the effectiveness of this invention, zero-shot defect detection experiments were conducted on two publicly available industrial defect detection datasets (MVTec-AD and VisA). This invention was compared with 11 existing methods, including traditional methods (such as PaDiM and PatchCore), LVLM-based methods (such as AnomalyGPT and Myriad), and CLIP-based methods (such as WinCLIP and AnomalyCLIP). Image-AUC and pixel-AUC were used as evaluation metrics in the experiments.

[0047] Experimental results show that the present invention achieves optimal performance across all metrics. In the MVTec-AD task, the image-AUC and pixel-AUC of the present invention reach 94.8% and 95.6%, respectively, representing improvements of 1.2% and 0.8% over the existing best methods. In the VisA task, the image-AUC and pixel-AUC of the present invention reach 86.9% and 95.7%, respectively, also outperforming all comparative methods. The experimental results are shown in Table 1. Table 1. Image-AUC and Pixel-AUC scores of the present invention and prior art. To further verify the advantages of this invention in defect description generation, the quality of text descriptions generated by this method was compared with that of four baseline methods (PandaGPT, MiniGPT-4, AnomalyGPT, and SPGDD-GPT), and GPT-4V was used to score the descriptions generated by each method.

[0048] Figure 4The image shown is of a fabric with a black scratch defect. Figure 5 The results of description generation for fabric scratch defect images are presented in comparison. PandaGPT generated a relatively long description, but only the first sentence, "anomaly exists," was valid; subsequent descriptions incorrectly assumed the fabric texture was "fingers," creating a severe hallucination, thus scoring only 20 points. MiniGPT-4 failed to detect any defects at all, producing a false negative result and scoring 0 points. AnomalyGPT correctly located the defect area on the bottom left and identified the object as fabric, but failed to provide details such as color, shape, or defect type, scoring 40 points. SPGDD-GPT showed significant improvement, correctly describing "gray woven fabric" and "black scratch," but lacked refined descriptions such as geometric morphology, scoring 80 points. In contrast, the method of this invention achieved the highest score of 95 points, accurately describing the scratch as located in the "central area" and having a "narrow and linear" shape, refining its "sharp edges" and "depth contrast" features, and clearly confirming that other areas were defect-free.

[0049] Figure 6 The image shown is of a white pill without any defects. Figure 7 The results of the description generation comparison for normal tablet images are presented. PandaGPT again produced severe hallucinations, incorrectly describing the tablet as "a cat sitting on a plate," scoring 30 points; MiniGPT-4, while judging no abnormalities, incorrectly described the tablet marking as the letter "f" on a red surface, scoring 40 points; AnomalyGPT gave a correct but extremely brief answer, simply describing it as "white tablet," scoring 60 points; SPGDD-GPT provided a more detailed description, including details such as a black background, centered tablet, "FF" mark, and red dots, scoring 85 points. The method of this invention achieved 96 points, further refining the tablet's "oval" shape, "smooth and uniform" surface texture, "serif" logo, and red dots as "normal manufacturing marks," and explicitly stating the absence of any cracks, discoloration, or deformation. The above qualitative comparison results fully verify the significant superiority of this invention in generating high-fidelity, hallucination-free, fine-grained defect descriptions.

Claims

1. An industrial defect diagnosis method based on a segmentation-perception vision-language model, characterized by: Includes the following steps: (1) Image acquisition and preprocessing: Acquire the RGB image of the industrial product to be diagnosed and perform normalization preprocessing. (2) Constructing a CLIP-SAM cascading hint mechanism: (2.1) Coarse-grained defect localization: Using the image encoder and text encoder of the pre-trained CLIP model, image features and predefined text features are generated. The text features include normal text features and abnormal text features. The contrast distance between the image features and the text features is calculated to generate a coarse-grained defect response heatmap. An initial coarse-grained mask is obtained by adaptive threshold binarization of the heatmap. (2.2) Fine-grained defect segmentation: The coarse-grained mask is used as a spatial cue and is input into the pre-trained SAM model along with the RGB image. The SAM's image encoder, cue encoder, and mask decoder generate multiple candidate masks, and the mask with the highest confidence is selected as the final fine-grained segmentation mask. (3) Multi-scale defect feature extraction: The coarse-grained mask is processed by a multi-scale cue learner to generate cue embeddings with three scales: global, regional, and detailed. The segmentation mask is predicted in parallel on three resolution branches (high, medium, and low) by a multi-scale mask decoder, and the output of the medium resolution branch is used as the final segmentation result. (4) Construct a segmentation-aware semantic alignment module: The fine segmentation mask is multiplied element-wise with the features extracted by the SAM image encoder to obtain defect-aware visual features. These visual features are then input together with the text features extracted by the CLIP model's text encoder into a bidirectional cross-attention module, where visual-to-text attention operations and text-to-visual attention operations are performed sequentially to generate semantically aligned visual and text features. (5) Defect description generation: The aligned visual and textual features are concatenated and input into a large language model (LLM) to generate defect description text. (6) Model training: The segmentation-aware visual-language model is trained through steps (1) to (5), and the total loss function of the model is the weighted sum of segmentation loss and defect description text generation loss. (7) Model Deployment: After training is complete, in practical applications, input the image to be processed and execute the above steps (1)-(5).

2. The method according to claim 1, characterized in that, The CLIP-SAM cascaded hint mechanism in step (2) involves using CLIP to generate a coarse segmentation mask through contrastive learning, and then using SAM to segment a more fine-grained defect segmentation mask map using the coarse segmentation mask of CLIP as a hint.

3. The method according to claim 1, characterized in that, Step (2.1) also includes introducing a small number of normal image samples as a support set under a few-sample setting to construct a multi-scale normal feature memory library; When step (2.1) is performed in step (7), the minimum cosine distance between the query image feature and the nearest feature in the memory is calculated to generate a context-aware anomaly score map. This score map is converted into modulation coefficients by the Sigmoid function to enhance or suppress the coarse-grained defect response heatmap.

4. The method according to claim 1, characterized in that, The multi-scale cue learner in step (3) contains three parallel convolutional paths with different receptive fields, which are used to extract the global structure, regional context and local detail features of the coarse-grained mask, respectively, and the three features are concatenated along the channel dimension to form a multi-scale cue embedding.

5. The method according to claim 1, characterized in that, In step (3), the multi-scale mask decoder supervises the outputs of the high, medium and low resolution branches simultaneously during the training phase. The segmentation loss is the weighted sum of the losses of each branch, with the medium resolution branch having the highest weight. In the inference phase of step (7), only the output of the medium resolution branch is used to improve efficiency.

6. The method according to claim 1, characterized in that, The bidirectional cross-attention module in step (4) specifically refers to: (4.1) Using the defect-aware visual features as the query and the text features as the key and value, perform the first cross-attention operation to make the visual features focus on the most relevant semantic words, and obtain the enhanced visual features. (4.2) Using text features as queries and enhanced visual features as keys and values, perform a second cross-attention operation to focus text features on the most relevant visual regions, resulting in aligned text features.

7. The method according to claim 1, characterized in that, The large language model in step (5) is Vicuna-7B-v1.5, and low-rank adaptive LoRA technology is used for efficient parameter fine-tuning, updating only the query and value projection matrices in the self-attention mechanism and the linear layers in the feedforward network.

8. The method according to claim 1, characterized in that, In step (6), the segmentation loss includes a weighted combination of the Dice loss and the Focal loss of each branch of the multi-scale mask decoder; the defect description text generation loss is the autoregressive cross-entropy loss used to supervise the generation of defect description text.

9. The application of the method according to claim 1, characterized in that, The method is applied to product quality inspection in industrial manufacturing, which includes surface defect detection of electronic components, scratch detection of metal workpieces, defect identification of textile fabrics, and diagnosis of molding defects in injection molded parts.

Images