A cross-model defect detection method

Abstract

The application discloses a cross-model defect detection method, which is suitable for automatic identification of surface defects in product update iteration in an industrial production line. The method uses the structural similarity of defect masks in the feature space of old-model products to construct prompt information, guiding the SAM model to complete efficient segmentation of the defect area in the new-model image. Then, the segmentation result is subjected to defect discrimination at the semantic level through a visual language model, realizing closed-loop judgment from structural perception to semantic cognition. The method effectively breaks through the dependence of traditional weak supervision methods on new-model labeling and prior knowledge, can fully reuse old-model data, avoids reacquisition, labeling and training of new-model images, and significantly reduces system deployment and update costs. Meanwhile, the application boundary of the prompt learning mechanism in the industrial vision scene is expanded, providing a solution with universality, high efficiency and interpretability for multi-model rapid adaptation defect detection.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and industrial intelligent inspection technology, and in particular to a cross-model product defect detection method and system based on the combination of image segmentation model and multimodal semantic understanding model, which is suitable for automatic identification of surface defects during product updates and iterations in industrial production lines. Background Technology

[0002] Surface defect detection is a crucial step in ensuring product quality during industrial production. With the development of intelligent manufacturing and Industry 4.0, machine vision technology is increasingly widely used in industrial quality inspection. Especially in industries such as electronics manufacturing, metal processing, and food packaging, the efficient and accurate identification and segmentation of surface defects has become a key technology for improving product quality and production efficiency.

[0003] Currently, mainstream visual defect detection methods are based on supervised training mechanisms using deep learning models, such as image segmentation models based on convolutional neural networks (CNNs) or fully convolutional networks (FCNs). These models typically require a large number of labeled images for training to achieve good segmentation results on target products; examples include Mask R-CNN and U-Net. Unsupervised models, on the other hand, detect anomalies by modeling the distribution of normal samples. These models usually employ a one-class-one-weight paradigm, such as CutPaste and EfficientAD, which have certain biases in adaptability and generalization to different products, making them unsuitable for detecting various models and iterative updates of products.

[0004] With the continuous advancement of industrialization, the product iteration cycle has shortened significantly, and its production efficiency also needs to be systematically improved under the drive of technological innovation. In actual industrial applications, the diversity of product models and frequent iterations pose two key challenges to the above methods: (1) Whenever the product model or process mold is changed, the system needs to re-collect images, label defects, and train the model. The process is cumbersome and time-consuming, making it difficult to adapt to the rapidly changing production rhythm; (2) New product models often lack sufficient defect samples, resulting in unstable model training effects and even the inability to learn effectively.

[0005] Therefore, there is an urgent need for an intelligent defect detection method that can be trained without new model samples, has cross-model adaptability, and can reuse old image information, in order to meet the core requirements of "rapid deployment and good generalization" in modern industrial quality inspection. Summary of the Invention

[0006] To address the above challenges, this invention proposes a cross-model defect detection model, SVL (Segmentation and Vision Language model), which combines a prompt-guided segmentation model with a large-scale visual language model. This method utilizes existing defect annotation information from older product models to construct prompts, which are then input into a pre-trained large-scale image segmentation model. This guides the model to segment defect regions in structurally similar but unannotated images of newer models. Subsequently, a large-scale visual language model performs a post-hoc evaluation of the segmentation results to improve the overall detection accuracy and robustness. This model can perform effective detection at the beginning of product manufacturing iterations by effectively reusing data from older product models. The specific steps are as follows:

[0007] Step 1: Visual image and dataset construction for different product models. A synthetic dataset is collected using industrial cameras (simulating multi-model defect scenarios); spatiotemporally aligned images of multiple product models are captured using industrial cameras on a real production line to construct a real dataset; old model samples (including positive and negative samples) and new model samples to be detected (including positive and negative samples) are also included. The negative samples of the old models are then labeled to facilitate subsequent suggestion construction. Both old and new model samples will be input into the SAM model.

[0008] Step 2: For the data input into the SAM model: negative samples of the old model will be used to construct prompts, guiding SAM to segment the new model samples. If similar defects exist, SAM will segment a mask image of the new model sample with the defects; otherwise, it will segment the entire object. The segmented mask image will be input into the Llama model for further processing.

[0009] Step 3: After the mask image containing only black and white pixels is input into the Llama model, the model focuses on the input mask image by combining the fixed language paradigm constraint model, and finally gives the fixed paradigm output to realize the binary discrimination of defects.

[0010] The specific details of each step are as follows:

[0011] In step 1, a Baslerac A2500-14gc industrial camera is used to acquire visual images of different product models. The acquired images are divided into two categories: old model samples (positive and negative samples) and new model samples (positive and negative samples). Simultaneously, during dataset creation, a small number of old model negative samples are selected as cue samples for the SVL model. The remaining old and new model samples will serve as a test set to verify the cross-model detection performance of the SVL model.

[0012] Step 1 primarily utilizes Labelme to process data from older models. The Labelme annotation tool is used to annotate the defective regions in the image with polygonal vertices, generating a JSON file containing contour coordinate information. A script then converts the JSON annotation data into a binary mask image of the same size as the original image, where pixel values ​​for defective regions are 1 and background regions are 0. Random rotation, mirroring, and brightness adjustment are applied to the original image and the corresponding mask image for data augmentation. Finally, a standardized dataset containing the original image file, the mask image file, and category labels is generated.

[0013] In step 2, to effectively utilize old model data as prompting information for the effective segmentation of new model data, a pre-trained image segmentation model, the SAM model (SegmentAnythingModel), is used. The SAM model is a general-purpose image segmentation model developed by Meta. Through the collaborative operation of its image encoder, prompt encoder, and segmentation decoder modules, it supports multimodal prompt inputs such as points, boxes, and text, and can accurately segment target objects in images based on the prompting information. Guided by the defect region prompts in the old model data, the SAM model identifies regions in the image that are similar to the prompt structure and outputs multiple candidate mask images. The system calculates the confidence and shape matching indices of multiple candidate masks and selects the final segmentation result. For the preliminary segmentation result, connected component analysis, area threshold filtering, morphological optimization, and other operations are performed to remove background noise and false responses, forming a standardized defect mask image. After the above series of operations, the new model samples are effectively segmented, and their segmented mask images are obtained.

[0014] Step 3 proposes a defect identification method based on a visual language model, breaking through the traditional domain adaptive or domain generalization techniques. It innovatively utilizes the feature similarity of defect regions between new and old product models to achieve efficient defect identification. Specifically, after segmenting the image of the new product model using the SAM model and obtaining the mask image, this mask image and a system-generated natural language prompt (e.g., "Does this mask image contain a defect region?") are input into the visual language model. The model mines the commonalities in defect features between the new and old models, performs binarization processing and feature analysis on the mask image, transforming image features into a more easily discriminative binary form. The core advantage of this approach is that when products are updated, there is no need to rebuild the dataset for model training. Only through the combined analysis of natural language prompts and the mask image can the presence of defects in the product in the image be quickly and accurately determined, significantly improving the efficiency and generalization ability of defect detection while reducing model training costs and time.

[0015] Overall, the proposed SVL model mainly consists of two components: the SAM (SegmentAnythingModel) model and the Llama3.2vision model. First, several defect samples from older models and their corresponding mask images are prepared, along with several samples from newer models. The SAM model's image encoder generates bounding boxes and point prompts from the defect samples and corresponding mask images as prompts to guide the SAM model in segmenting the new images. Finally, the Llama3.2vision model's understanding of the image is used to determine whether defects exist, forming a cross-model detection process that requires no retraining or data acquisition.

[0016] By adopting the above technical solutions, the present invention has the following advantages:

[0017] This invention proposes a cross-model defect detection method called SVL, which combines the powerful cue-guided segmentation capability of the SegmentAnythingModel (SAM model) with the semantic understanding capability of a visual language model. The method first utilizes the structural similarity of defect masks in the feature space of older models to construct cue information, guiding the SAM model to efficiently segment defect regions in images of newer models. Subsequently, the visual language model performs semantic defect discrimination on the segmentation results, achieving a closed-loop judgment from structural perception to semantic cognition. This method implements a near-zero-shot cross-model defect segmentation strategy, effectively overcoming the dependence of traditional weakly supervised methods on new model annotations and prior knowledge. It can fully reuse older model data, avoiding the need for re-collection, annotation, and training of new model images, significantly reducing system deployment and update costs. Simultaneously, this invention expands the application boundaries of cue learning mechanisms in industrial vision scenarios, providing a universal, efficient, and interpretable solution for rapid multi-model defect detection.

[0018] To facilitate the verification of the SVL model's effectiveness, this invention utilizes the Ollama platform to deploy the Llama 3.2vision model. Through interactive methods, it effectively identifies defects in both old and new model data. The entire process requires no training or fine-tuning; it effectively detects defects in new model data using only data from older models. This model can perform effective detection at the beginning of industrial product updates and iterations. Attached Figure Description

[0019] Figure 1 This is a flowchart of the overall framework of the SVL model;

[0020] Figure 2 This is a schematic diagram of an older model sample collected using a Baslerac A2500-14gc.

[0021] Figure 3 This is a schematic diagram of a new model sample collected using a Baslerac A2500-14gc.

[0022] Figure 4 It is a mask image that uses Labelme to mark the defect areas of negative samples of older models;

[0023] Figure 5 This is a schematic diagram of the segmentation of the new model sample using the SAM model combined with prompts;

[0024] Figure 6 This is a partial detection map of the SAM model segmentation map obtained by Llama 3.2vision;

[0025] Figure 7 These are partial test images from Llama 3.2vison, directly comparing the old and new models. Detailed Implementation

[0026] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings;

[0027] Appendix Figure 1 This is the overall framework diagram of the SVL cross-model detection model:

[0028] First, an industrial Basler A2500-gc camera was used to capture images of several different types of packaged cookies at a resolution of 1024 pixels * 1024 pixels. Since the images captured on an industrial production line are of moving objects, blurriness can occur. Therefore, the Basler camera's white balance function was used during the acquisition process to ensure that the objects in the captured images were clearer.

[0029] Appendix Figure 2 and attached Figure 3 These are examples of old and new model samples captured by a camera, used for SVL model detection and verification. Then, several defective samples are randomly selected from the old model samples to construct warning samples. Finally, LabelMe is used to annotate the defective areas of the old model samples with polygons, and then converted into their corresponding mask images. (Attached) Figure 4 The mask image after labeling the selected old model sample.

[0030] This invention proposes using the SAM (SegmentAnythingModel) model for segmentation processing. The SAM model mainly consists of three parts: an image encoder, a cue encoder, and a mask decoder. The image encoder can process RGB images of any resolution; the cue encoder can input multimodal cues, such as points and boxes; and the mask decoder is used to output the segmentation mask of the target object.

[0031] In order to achieve cross-model detection, this invention proposes reusing data from older models as prompting information, that is, utilizing the attached... Figure 4 The original image and its corresponding information serve as prompts to guide segmentation. The entire process can be divided into the prompt construction stage, the segmentation reasoning stage, and the post-processing stage.

[0032] During the build phase, older models In the middle, give its corresponding mask image. Information is extracted from it, including the bounding box prompt (BoxPrompt) as shown in Equation (1).

[0033] (1)

[0034] in: Similarly, define Point Prompt information, foreground point set as shown in equation (2), background point set as shown in equation (3).

[0035] (2)

[0036] (3)

[0037] Point label vector: The set of hints is then given by equation (4).

[0038] (4)

[0039] In the segmentation inference stage, for the new model image to be detected Introducing pre-trained images Its input images include and prompt information The output mask candidate set is as shown in equation (5):

[0040] (5)

[0041] The model outputs a score for each mask simultaneously. set up

[0042] (6)

[0043] Where IoU is the crossover-union ratio: (in For smoothing factor, parameters (Control the weight balance between scoring and shape consistency).

[0044] In the post-processing stage, candidate masks are... Apply morphological filtering to remove items with an area smaller than a threshold. The pseudo-region. The final mask can be expressed as equation (7).

[0045] (7)

[0046] in This indicates a connected component analysis operation. (See appendix) Figure 5 This is a schematic diagram illustrating the segmentation of the new model sample using the SAM model combined with the mask image of the old model defect sample.

[0047] In industrial production lines, it is necessary not only to identify surface defects but also to effectively remove defective products, requiring binary discrimination. This invention proposes a discrimination mechanism using the Llama 3.2vison large-scale visual language model. It is based on a multimodal alignment structure of an image encoder (typically CLIP-ViT) and a large language model (such as the LLaMA series). This model can receive a raw image and generate a semantically understood response based on a user's question in natural language. This invention utilizes the Ollama platform to build the Llama 3.2vison model, enabling interactive verification. In the actual inference process, the segmentation mask corresponding to the new model image is used as input. Subsequently, the system uses a fixed-template natural language prompt (e.g., "Does the image have defects?") as input instructions to request a judgment from the VLM. The visual language model comprehensively analyzes the correspondence between image content, regional context information, and semantic instructions to determine whether the region meets the definition of "defect."

[0048] Appendix Figure 6 This is a partial detection image of the SAM model segmentation map obtained by Llama 3.2vision. To verify whether the Llama 3.2vision model can directly distinguish objects, corresponding experiments were also conducted. (Attached) Figure 7 The Llama 3.2vison model was used to directly detect some images from both new and old models. During the experiment, it was found that directly using the model resulted in a very high error rate, while the model was more accurate and faster at distinguishing between mask images containing only black and white pixels. (See attached image.) Figure 7 During the experiment, the system provided natural language prompts ("Does the cookie have any defects? If yes, answer yes; if no, answer no.").

[0049] The SVL model framework proposed in this invention achieves near-zero-sample cross-model defect segmentation, breaking through the dependence of traditional weakly supervised models on prior knowledge of new models. Simultaneously, it utilizes the similarity of old model masks in the feature space to guide cue construction, using this to segment defect regions in new images, representing an innovative extension of existing cue learning frameworks in the industrial vision field. Furthermore, it effectively reuses existing old model data, avoiding the tedious work of re-collecting, training, or fine-tuning, ensuring a usable model is available from the outset of industrial production.

Claims

1. A method for detecting defects across different models, characterized in that, The specific steps of this method are as follows: Step 1: Use industrial cameras to collect and synthesize datasets simulating defect scenarios of multiple models, and construct visual images and datasets of different models of products; use industrial cameras to capture spatiotemporally aligned images of multiple models of products in a real production line, and construct a real dataset; among them, the old model samples contain positive and negative samples, and the new model samples to be detected contain positive and negative samples; then label the negative samples of the old models; both the old model samples and the new model samples to be detected are input into the SAM model; Step 2: For the data input into the SAM model: negative samples in the old model samples will be used to construct prompts to guide the SAM model to segment the new model samples; if there are similar defects, the SAM model will segment out the mask image of the new model sample with defects; if not, the entire object will be segmented; the segmented mask image will be input into the Llama model for further processing. Step 3: After the mask image containing only black and white pixels is input into the Llama model, the model focuses on the input mask image by combining the fixed language paradigm constraint model, and finally gives the fixed paradigm output to realize the binary discrimination of defects; To enable cross-model detection, it is proposed to reuse data from old models as prompting information, that is, to use prompting information to guide segmentation, which is divided into prompt construction stage, segmentation reasoning stage and post-processing stage; During the build phase, older models In the middle, give its corresponding mask image. Information is extracted from it, including bounding box hints as shown in equation (1). （1）； in: ,definition Point Prompt information, foreground point set as shown in formula (2), background point set as shown in formula (3); （2）； （3）； Point label vector: The set of hints is then given by equation (4); （4）； In the segmentation inference stage, for the new model image to be detected Introducing pre-trained models Its input images include and prompt information The output mask candidate set is as shown in equation (5): （5）； The model outputs a score for each mask simultaneously. set up （6）； Where IoU is the crossover-union ratio: ,in For smoothing factor, parameters Balance the weighting of scoring and shape consistency; In the post-processing stage, candidate masks are... Apply morphological filtering to remove items with an area smaller than a threshold. The pseudo-region; the final mask is expressed as equation (7): （7）； in This indicates a connected component analysis operation.

2. The cross-model defect detection method according to claim 1, characterized in that, In step 1, a Baslerac A2500-14gc industrial camera is used to collect visual images of different product models. The collected images are divided into two categories: old model samples and new model samples to be detected. During the dataset production process, a small number of old model negative samples are selected as cue samples for the SVL model. The remaining old model samples and new model samples will be used as a test set to verify the cross-model detection effect of the SVL model.

3. The cross-model defect detection method according to claim 1, characterized in that, Step 1 involves processing old model data using Labelme; using the Labelme annotation tool to annotate the defective regions in the image with polygon vertices, generating a JSON file containing contour coordinate information; converting the JSON annotation data into a binary mask image of the same size as the original image using a script, where the pixel value of the defective region is 1 and the pixel value of the background region is 0; performing data augmentation processing on the original image and the corresponding mask image, including random rotation, mirroring, and brightness adjustment; and finally generating a standardized dataset containing the original image file, the mask image file, and category labels.

4. The cross-model defect detection method according to claim 1, characterized in that, In step 2, to effectively utilize old model data as prompting information for effective segmentation of new model data, a pre-trained image segmentation model, namely the SAM model, is used. The SAM model is a general image segmentation model developed by Meta. Through the collaboration of three modules—image encoder, prompt encoder, and segmentation decoder—it supports multimodal prompt inputs of points, boxes, and text, and can accurately segment target objects in the image based on the prompting information. Guided by the defect region prompts in the old model data, the SAM model will identify regions in the image that are similar to the prompt structure and output multiple candidate mask images. The confidence and shape matching indices of multiple candidate masks are calculated, and the final segmentation result is selected. For the preliminary segmentation result, connected component analysis, area threshold filtering, and morphological optimization operations are performed to remove background noise and false responses, forming a standardized defect mask image. The new model samples are effectively segmented, and their segmented mask images are obtained.

5. The cross-model defect detection method according to claim 1, characterized in that, In step 3, after completing the region segmentation of the new product image and obtaining the mask image through the SAM model, the mask image and the natural language prompts generated by the system are input into the visual language model. The visual language model extracts commonalities in the defect features of new and old models, performs binarization and feature analysis on the mask image, and transforms the image features into an easily discriminable binary form.

6. The method for detecting defects across different models according to claim 1, characterized in that, The SVL model includes the SAM model and the Llama 3.2vision model. First, prepare several old model defect samples and corresponding mask images, and several new model samples. The image encoder of the SAM model will generate bounding boxes and point prompts from the old model defect samples and corresponding mask images as prompt information to guide the SAM model to segment the new image. Finally, the Llama 3.2vision model's understanding of the image is used to determine whether there is a defect, forming a cross-model detection process that does not require retraining or data collection.

7. The cross-model defect detection method according to claim 1, characterized in that, The discrimination mechanism utilizes the Llama 3.2vison large-scale visual language model; a multimodal alignment structure based on an image encoder and a large language model; receiving raw images and generating semantically understood responses based on user questions in natural language; building the Llama 3.2vison model using the Ollama platform, which allows for interactive verification. During the inference process, the segmentation mask corresponding to the new model image is used as input; Subsequently, using natural language prompts with a fixed template as input instructions, a judgment request is made to the VLM model; The visual language model comprehensively analyzes the correspondence between image content, regional context information, and semantic instructions to determine whether the region meets the definition of a "defect".

Images