Industrial defect segmentation method and device and storage medium

Abstract

The invention discloses an industrial defect segmentation method and device and a storage medium, and belongs to the technical field of image segmentation. The method comprises the following steps: inputting a to-be-detected image containing a target product into a preset multi-modal large model to obtain a category of the target product output by the multi-modal large model; determining a segmentation model corresponding to the category of the target product; and inputting the to-be-detected image into a segmentation model corresponding to the category of the target product to obtain a segmentation result output by the segmentation model, the segmentation result comprising a defect position of the target product in the to-be-detected image. According to the method, the to-be-detected image is processed through the multi-modal large model, the category of the target product is determined, and the corresponding segmentation model is selected according to the category of the target product, so that the appropriate segmentation model can be selected according to different product characteristics, and defect segmentation is performed by applying the specially trained segmentation model; the method can give full play to the advantages of different segmentation models in capturing different product features, and improves the accuracy of defect detection.

Description

Technical Field

[0001] The present application belongs to the field of image segmentation technology, and in particular, relates to an industrial defect segmentation method, device and storage medium. Background Art

[0002] In the field of industrial manufacturing, product quality inspection is a crucial link. With the development of automation technology, traditional manual inspection methods are gradually being replaced by industrial inspection technologies based on deep learning. Industrial inspection technologies based on deep learning can automatically identify and mark defects on the surface of products, greatly improving inspection efficiency.

[0003] In related technologies, many deep learning-based defect detection models have been applied to industrial production, such as category defect detection based on convolutional neural networks, category defect detection based on generative adversarial networks, etc. However, due to the variety of defect types in industrial products, different models have different detection effects on different defects, and defect detection using a single model results in insufficient detection accuracy. Summary of the invention

[0004] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes an industrial defect segmentation method, device and storage medium to improve the accuracy of defect detection.

[0005] In a first aspect, the present application provides an industrial defect segmentation method, comprising:

[0006] Inputting an image to be detected containing a target product into a preset multimodal large model, and obtaining the category of the target product output by the multimodal large model;

[0007] Determine a segmentation model corresponding to the category of the target product;

[0008] The image to be detected is input into a segmentation model corresponding to the category of the target product to obtain a segmentation result output by the segmentation model, wherein the segmentation result includes a defect position of the target product in the image to be detected.

[0009] According to the industrial defect segmentation method of the present application, the category of the target product output by the multimodal large model is obtained by inputting the image to be detected containing the target product into a preset multimodal large model; the segmentation model corresponding to the category of the target product is determined; the image to be detected is input into the segmentation model corresponding to the category of the target product, and the segmentation result output by the segmentation model is obtained, and the segmentation result includes the defect position of the target product in the image to be detected. The embodiment of the present application processes the image to be detected by using a multimodal large model to determine the category of the target product, and selects the corresponding segmentation model according to the category of the target product, so that the appropriate segmentation model can be selected according to different product characteristics, and the specially trained segmentation model is applied for defect segmentation, which can give full play to the advantages of different segmentation models in capturing different product characteristics and improve the accuracy of defect detection.

[0010] According to one embodiment of the present application, the method further includes:

[0011] Acquire a reference image including a product of the same category as the target product; the reference image includes a label, and the label is marked with a defect location and a defect type of the product in the reference image;

[0012] In a case where the defect type of the product in the reference image is the same as the defect type of the target product in the image to be detected, identifying the defect type of the target product in the image to be detected according to the segmentation result;

[0013] Output the defect location and defect type of the target product in the image to be inspected.

[0014] In this embodiment, by introducing a reference image and its label information, when the product defect type in the reference image matches the defect type of the target product in the image to be detected, the output results of the segmentation model can be further combined to identify the defect type of the target product in the image to be detected. Ultimately, defects that are the same as the product defect type in the reference image can be output, thereby improving the accuracy and efficiency of detecting specific defects.

[0015] According to one embodiment of the present application, the method further includes:

[0016] Cutting out a first image including the defect of the target product from the image to be detected according to the segmentation result;

[0017] cropping a second image including the defect of the target product from the reference image;

[0018] It is determined according to the first image and the second image whether the defect type of the product in the reference image is the same as the defect type of the target product in the image to be detected.

[0019] In this embodiment, by cropping a first image containing the target product defect from the image to be inspected according to the segmentation result, and cropping a second image containing the same type of defect from the reference image, the defect area can be accurately located, the extraction of defect-specific features is enhanced, and the adjacent contextual information is retained, thereby capturing more meaningful semantic features and enhancing the ability to recognize different defect types.

[0020] According to an embodiment of the present application, judging whether the defect type of the product in the reference image and the defect type of the target product in the image to be detected are the same according to the first image and the second image, includes:

[0021] In a case where the category of the target product is a target category, the first image and the second image are input into a comparison classifier corresponding to the category of the target product to obtain a first comparison result output by the comparison classifier.

[0022] In this embodiment, by judging the category of the target product, if the category of the target product is a specific category, the cropped first image containing defects and the second image in the reference image are input into the comparison classifier corresponding to the category of the target product. The comparison classifier can be pre-trained and optimized for products of a specific category, so as to be more accurate in comparing defect types.

[0023] According to an embodiment of the present application, judging whether the defect type of the product in the reference image and the defect type of the target product in the image to be detected are the same according to the first image and the second image, includes:

[0024] In a case where the category of the target product is not the target category, the first image and the second image are input into the multimodal large model to obtain a second comparison result output by the multimodal large model.

[0025] In this embodiment, by judging the category of the target product, if the category of the target product is not a specific category, the first image and the second image can be input into a multimodal large model, and the cross-modal feature extraction and fusion capabilities of the model can be used to improve the accuracy of defect type comparison through deep feature extraction and semantic understanding.

[0026] According to an embodiment of the present application, identifying the defect type of the target product in the image to be detected according to the segmentation result includes:

[0027] The first image is input into a type recognition classifier corresponding to the category of the target product to obtain the defect type of the target product output by the type recognition classifier.

[0028] In this embodiment, by cropping the first image according to the segmentation result, the defect area can be accurately located, the extraction of defect-specific features is enhanced, and the adjacent contextual information is retained, so that more meaningful semantic features can be captured. The first image is input into a type recognition classifier corresponding to the target product category, so that a suitable classifier can be selected according to different product characteristics. The specially trained classifier is used for type recognition, which can give full play to the advantages of different classifiers in capturing different product characteristics and improve the accuracy of defect type recognition.

[0029] According to an embodiment of the present application, determining the segmentation model corresponding to the category of the target product includes:

[0030] In the case where the category of the target product is the target category, the segmentation model is a Mask2Former model;

[0031] When the category of the target product is not the target category, the segmentation model is a Mask2Former model and a PatchCore model.

[0032] In this embodiment, when the target product category is the target category, the Mask2Former model is selected as the segmentation model. The advantage of this model is that it can use multi-scale features and masked self-attention mechanism to improve the accuracy of segmentation, especially when dealing with small target scenes; when the target product category is not the target category, a combination of the Mask2Former model and the PatchCore model is used. The PatchCore model can evaluate the degree of abnormality of each pixel in the image. This detailed anomaly scoring mechanism can more accurately locate the specific area where the anomaly occurs.

[0033] In a second aspect, the present application provides an industrial defect segmentation device, comprising:

[0034] An input module, used to input an image to be detected containing a target product into a preset multimodal large model, and obtain the category of the target product output by the multimodal large model;

[0035] A determination module, used to determine a segmentation model corresponding to the category of the target product;

[0036] The output module is used to input the image to be detected into a segmentation model corresponding to the category of the target product to obtain a segmentation result output by the segmentation model, wherein the segmentation result includes a defect position of the target product in the image to be detected.

[0037] According to the industrial defect segmentation device of the present application, the category of the target product output by the multimodal large model is obtained by inputting the image to be detected containing the target product into a preset multimodal large model; the segmentation model corresponding to the category of the target product is determined; the image to be detected is input into the segmentation model corresponding to the category of the target product, and the segmentation result output by the segmentation model is obtained, and the segmentation result includes the defect position of the target product in the image to be detected. The embodiment of the present application processes the image to be detected by using a multimodal large model to determine the category of the target product, and selects the corresponding segmentation model according to the category of the target product, so that the appropriate segmentation model can be selected according to different product characteristics, and the specially trained segmentation model is applied for defect segmentation, which can give full play to the advantages of different segmentation models in capturing different product characteristics and improve the accuracy of defect detection.

[0038] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the industrial defect segmentation method as described in the first aspect above is implemented.

[0039] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the industrial defect segmentation method as described in the first aspect above is implemented.

[0040] In a fifth aspect, the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the industrial defect segmentation method as described in the first aspect above.

[0041] In a sixth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the industrial defect segmentation method as described in the first aspect above.

[0042] The above one or more technical solutions in the embodiments of the present application have at least one of the following technical effects:

[0043] According to the industrial defect segmentation method of the present application, the category of the target product output by the multimodal large model is obtained by inputting the image to be detected containing the target product into a preset multimodal large model; the segmentation model corresponding to the category of the target product is determined; the image to be detected is input into the segmentation model corresponding to the category of the target product, and the segmentation result output by the segmentation model is obtained, and the segmentation result includes the defect position of the target product in the image to be detected. The embodiment of the present application processes the image to be detected by using a multimodal large model to determine the category of the target product, and selects the corresponding segmentation model according to the category of the target product, so that the appropriate segmentation model can be selected according to different product characteristics, and the specially trained segmentation model is applied for defect segmentation, which can give full play to the advantages of different segmentation models in capturing different product characteristics and improve the accuracy of defect detection.

[0044] Furthermore, in some embodiments, by introducing a reference image and its label information, when the product defect type in the reference image matches the defect type of the target product in the image to be inspected, the output results of the segmentation model can be further combined to identify the defect type of the target product in the image to be inspected, and ultimately the defect with the same defect type as the product in the reference image can be output, thereby improving the accuracy and efficiency of detecting specific defects.

[0045] Furthermore, in some embodiments, by cropping a first image containing the target product defect from the image to be inspected based on the segmentation result, and cropping a second image containing the same type of defect from the reference image, the defect area can be accurately located, the extraction of defect-specific features is enhanced, and the adjacent contextual information is retained, thereby capturing more meaningful semantic features and enhancing the ability to recognize different defect types.

[0046] Furthermore, in some embodiments, by determining the category of the target product, if the category of the target product is a specific category, the cropped first image containing defects and the second image in the reference image are input into a comparison classifier corresponding to the category of the target product. The comparison classifier can be pre-trained and optimized for products of a specific category, thereby being more accurate in comparing defect types.

[0047] Furthermore, in some embodiments, by judging the category of the target product, if the category of the target product is not a specific category, the first image and the second image can be input into a multimodal large model, and the cross-modal feature extraction and fusion capabilities of the model can be used to improve the accuracy of defect type comparison through deep feature extraction and semantic understanding.

[0048] Furthermore, in some embodiments, by cropping the first image according to the segmentation results, the defect area can be accurately located, the extraction of defect-specific features is enhanced, and the adjacent contextual information is retained, so that more meaningful semantic features can be captured. The first image is input into a type recognition classifier corresponding to the target product category, so that a suitable classifier can be selected according to different product characteristics, and the specially trained classifier is used for type recognition, which can give full play to the advantages of different classifiers in capturing different product characteristics and improve the accuracy of defect type recognition.

[0049] Furthermore, in some embodiments, when the target product category is the target category, the Mask2Former model is selected as the segmentation model. The advantage of this model is that it can utilize multi-scale features and masked self-attention mechanism to improve the accuracy of segmentation, especially when dealing with small target scenes; when the target product category is not the target category, a combination of the Mask2Former model and the PatchCore model is used. The PatchCore model can evaluate the degree of abnormality of each pixel in the image. This detailed anomaly scoring mechanism can more accurately locate the specific area where the anomaly occurs.

[0050] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0052] Figure 1 It is a flow chart of the industrial defect segmentation method provided in the embodiment of the present application;

[0053] Figure 2 is a schematic diagram of a scenario example provided in an embodiment of the present application;

[0054] Figure 3 It is a schematic diagram of the structure of the industrial defect segmentation device provided in an embodiment of the present application;

[0055] Figure 4 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0057] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0058] The industrial defect segmentation method, device and storage medium provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0059] Among them, the industrial defect segmentation method can be applied to a terminal, and can be specifically executed by hardware or software in the terminal.

[0060] The terminal includes, but is not limited to, a portable communication device such as a mobile phone or tablet computer with a touch-sensitive surface (e.g., a touch screen display and / or a touch pad). It should also be understood that in some embodiments, the terminal may not be a portable communication device, but a desktop computer with a touch-sensitive surface (e.g., a touch screen display and / or a touch pad).

[0061] In the following various embodiments, a terminal including a display and a touch-sensitive surface is described. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, a mouse and a joystick.

[0062] The industrial defect segmentation method provided in the embodiment of the present application may be executed by an electronic device or a functional module or functional entity in the electronic device that can implement the industrial defect segmentation method. The electronic devices mentioned in the embodiment of the present application include but are not limited to mobile phones, tablet computers, computers, cameras, and wearable devices. The industrial defect segmentation method provided in the embodiment of the present application is described below using an electronic device as an example of an execution subject.

[0063] like Figure 1 As shown, the industrial defect segmentation method includes: step 110, step 120, and step 130.

[0064] Step 110: input the image to be detected containing the target product into a preset multimodal large model to obtain the category of the target product output by the multimodal large model.

[0065] In the embodiment of the present application, in industrial inspection, the target products may include products of different categories, such as electronic products, auto parts, textiles, food, etc. The images to be inspected containing these products may be captured by a high-definition camera or an industrial camera.

[0066] A multimodal large language model (MLLM) is a deep learning model that can simultaneously process and understand data in multiple modalities (such as text, images, audio, etc.). The MLLM integrates information from different modalities to build a unified multimodal feature representation, and performs task learning on this representation, such as classification, recognition, generation, etc. The core advantage of the MLLM is that it can process and understand information from different modalities and fuse this information to complete complex tasks.

[0067] In an embodiment of the present application, an image to be inspected containing a target product can be input into a preset multimodal large model. The multimodal large model can use methods such as convolutional neural networks (CNN) to extract features in the image, fuse the image features with features of other modalities, such as bounding boxes and text prompts in the image, to form a unified multimodal feature representation, and classify the input image to be inspected according to the multimodal features, and output the category of the target product.

[0068] In some embodiments, the multimodal large model used can be a pre-trained multimodal large model, such as CLIP (Contrastive Language-Image Pre-training), DALL-E (Generative Pre-trained Transformer for Visual Generation), LLaMA (Large Language Model Meta-AI), etc.

[0069] Before using the multimodal big model to predict the category of the target product, the pre-trained multimodal big model can be fine-tuned on the industrial product dataset. Through various types of diverse task data, the performance of the multimodal big model in the task of identifying the category of the product and the ability of the multimodal big model to understand and obey instructions can be improved.

[0070] For the multimodal large model, the present application embodiment designs two dialogue templates:

[0071] The first dialogue template:

[0072] Question: To which of the following categories [category] does image 1 belong?

[0073] Answer: Product 1.

[0074] Question: Image 1 contains a type of defect at location Box1, what is the type of defect present in image 1?

[0075] Answer: Type 1.

[0076] Question: Compare defect Defect1 in image 1 and the defect at location Box2 in image 2 to determine if they are the same type of defects?

[0077] Answer: Yes, they are both type 1. No, image 2 is type 2.

[0078] Second dialogue template:

[0079] Question: To which of the following categories [category] does image 1 belong?

[0080] Answer: Product 1.

[0081] Question: Image 1 contains a type of defect at location Box1, what is the type of defect present in image 1?

[0082] Answer: Type 1.

[0083] Problem: Image 2 is also an image of Product 1. Determine if Image 2 is the same type of defect as Defect 1 in Image 1.

[0084] Answer: Yes, the defect in image 2 is Defect 2. No, the defect in image 2 is Defect 2. No, image 2 does not contain any defects.

[0085] Among them, Box1 represents the defect bounding box in image 1, Box2 represents the defect bounding box in image 2, Defect1 represents the combination of the defect bounding box and the defect type in image 1, and Defect2 represents the combination of the defect bounding box and the defect type in image 1.

[0086] Step 120: Determine the segmentation model corresponding to the category of the target product.

[0087] In an embodiment of the present application, the segmentation model may be a model based on deep learning, which is used to identify the locations of different objects in an image and to segment the image into multiple parts or regions according to the locations of different objects. For example, an image includes a product, a background, and one or more defects on the product. Through the segmentation model, the locations of the product, the background, and one or more defects on the product in the image can be identified respectively.

[0088] In some embodiments, the segmentation model may include different types of semantic segmentation models, anomaly detection models, etc. The semantic segmentation model can assign a category label to each pixel in the image, thereby identifying different objects and scenes in the image. Common semantic segmentation models may include U-Net, DeepLab, Mask R-CNN (Mask Region-based Convolutional Neural Network), etc. The anomaly detection model focuses on identifying abnormal areas in the image, such as scratches and dents on the surface of the product. Common anomaly detection methods include autoencoders, generative adversarial networks (GAN), etc.

[0089] In some embodiments, different product categories can use different segmentation models to identify features or defects. For example, for electronic products, it may be necessary to use a model that can identify tiny defects on circuit boards, and an anomaly detection model based on deep learning can be used. For textiles, it may be necessary to use a model that can identify blemishes or uneven colors on fabrics, such as a semantic segmentation model based on a convolutional neural network. For automotive parts, it may be necessary to use a model that can identify cracks or deformations on metal or plastic parts, such as a semantic segmentation model based on U-Net. Of course, for some product categories, more complex defects may be included, and a combination of a semantic segmentation model and an anomaly detection model can be selected as the segmentation model.

[0090] In some embodiments, a correspondence between the category of the target product and the segmentation model may be established in advance, and after the category of the target product is determined, the corresponding segmentation model is determined according to the correspondence.

[0091] In some embodiments, determining a segmentation model corresponding to a category of a target product includes:

[0092] When the category of the target product is the target category, the segmentation model is the Mask2Former model;

[0093] When the category of the target product is not the target category, the segmentation models are the Mask2Former model and the PatchCore model.

[0094] In this embodiment, when the category of the target product is the target category, the Mask2Former model can be used as the segmentation model. Mask2Former is an advanced image segmentation model that uses the Transformer architecture and can effectively process multi-scale features and improve segmentation accuracy.

[0095] In this embodiment, the defects of some categories of products are usually simple and easy to identify, and these categories of products are the target categories. Before the Mask2Former application is deployed, Mask2Former needs to be trained. This embodiment uses Mask2Former to generate a mask for a given input, connects the support image, the support image mask and the query image along the channel dimension, and creates a 7-channel input to facilitate the model to learn a one-time defect detection pattern. In order to train Mask2Former, image pairs can be randomly extracted from the training data set to form support query pairs, and a fine-tuning data set can be created so that the number of positive pairs (query images and support images have the same defects and product categories) and negative pairs (query images and support images do not have the same defects and product categories) is approximately equal. Among them, the query image is the image to be detected in the training set, and the support image is an image in the training set that includes a label, and the label is annotated with the defect location and defect type of the product in the support image.

[0096] In order to make the model pay enough attention to the support image and the mask, a strategy is designed to black out the support mask with probability p, and even in the positive match, the model generates a blank mask for the query image. This method allows the model to segment the defects of the query image according to the defects of the support image. In an example, p=0.1 is set and the Mask2Former-Swin Large model is fine-tuned.

[0097] In this embodiment, when the category of the target product is not the target category, the Mask2Former model and the PatchCore model can be used. PatchCore is an unsupervised anomaly detection algorithm that divides the image into multiple small patches, calculates the anomaly value of each patch, and then interpolates the anomaly heat map of the entire image to identify the abnormal area.

[0098] In this embodiment, defects of some categories of products are usually manifested as complex situations with obviously different characteristics, and these categories of products are non-target categories. For such categories of products, Mask2Former may not be able to detect them effectively. Therefore, the PatchCore model can be used to improve performance. PatchCore is specially designed to detect defects, even if it is only trained on defect-free samples. In addition, a patch-level repository can be built for PatchCore and defect detection can be performed through patch matching, thereby improving the recall rate of complex defects.

[0099] In this embodiment, when the category of the target product is not the target category, the category of the target product can be further refined, and a PatchCore model can be trained for each category. For each category, a training set can be compiled, which includes defect-free samples and samples of known defect types in the category, so that PatchCore can accurately detect unseen complex defects that are different from defect-free samples and previously discovered defect types.

[0100] In this embodiment, when the target product category is the target category, the Mask2Former model is selected as the segmentation model. The advantage of this model is that it can use multi-scale features and masked self-attention mechanism to improve the accuracy of segmentation, especially when dealing with small target scenes; when the target product category is not the target category, a combination of the Mask2Former model and the PatchCore model is used. The PatchCore model can evaluate the degree of abnormality of each pixel in the image. This detailed anomaly scoring mechanism can more accurately locate the specific area where the anomaly occurs.

[0101] Step 130: input the image to be inspected into a segmentation model corresponding to the category of the target product to obtain a segmentation result output by the segmentation model, where the segmentation result includes the defect position of the target product in the image to be inspected.

[0102] In an embodiment of the present application, the image to be detected can be input into a segmentation model corresponding to the category of the target product. The segmentation model can perform feature extraction, defect area location and segmentation on the image to be detected, and output a segmentation result, which may include one or more masks, and different masks identify the positions of different objects in the image to be detected, for example, some masks identify the position of the background in the image to be detected, some masks identify the position of the target product in the image to be detected, and some masks identify the position of the defects of the target product.

[0103] According to the industrial defect segmentation method of the present application, the category of the target product output by the multimodal large model is obtained by inputting the image to be detected containing the target product into a preset multimodal large model; the segmentation model corresponding to the category of the target product is determined; the image to be detected is input into the segmentation model corresponding to the category of the target product, and the segmentation result output by the segmentation model is obtained, and the segmentation result includes the defect position of the target product in the image to be detected. The embodiment of the present application processes the image to be detected by using the multimodal large model to determine the category of the target product, and selects the corresponding segmentation model according to the category of the target product, so that the appropriate segmentation model can be selected according to different product characteristics, and the specially trained segmentation model is applied for defect segmentation, which can give full play to the advantages of different segmentation models in capturing different product characteristics and improve the accuracy of defect detection.

[0104] In some embodiments, the method further comprises:

[0105] Acquire a reference image including a product of the same category as the target product; the reference image includes a label, and the label is annotated with a defect location and a defect type of the product in the reference image;

[0106] When the defect type of the product in the reference image is the same as the defect type of the target product in the image to be detected, identifying the defect type of the target product in the image to be detected according to the segmentation result;

[0107] Output the defect location and defect type of the target product in the image to be inspected.

[0108] In some inspection scenarios, for example, the products are various metal parts, such as gears, bearings, etc. These parts need to be inspected for defects to ensure product quality. Due to the complex shapes and defect types of parts, manual visual inspection is inefficient and some hidden defects are easily missed. An automated defect detection system is needed that can input a reference image (known to have a certain type of defect) and a new part image to be inspected, and then detect defect areas in the new part image that have the same defect type as the reference image. This can improve inspection efficiency and accuracy and ensure the quality of mass-produced products.

[0109] In this embodiment, the reference image is an image including products of the same category as the target product, and the defect location and defect type are marked in the reference image. For example, the defect location and defect type can be marked by a mask or a prompt box.

[0110] In this embodiment, a deep learning model can be used to extract visual features at the defect locations in the reference image and the image to be detected, such as shape, size, color and other features, and the similarity of the visual features at the defect locations in the reference image and the image to be detected can be compared. If the similarity is greater than a preset threshold, it can be considered that the defect type of the product in the reference image is the same as the defect type of the target product in the image to be detected.

[0111] In this embodiment, a model for detecting the defect type of the target product can be pre-trained. When the defect type of the product in the reference image is the same as the defect type of the target product in the image to be detected, the defect position in the image to be detected can be marked according to the segmentation result, and the image to be detected with the marked defect position can be input into the model to obtain the defect type of the target product in the image to be detected.

[0112] In this embodiment, by introducing a reference image and its label information, when the product defect type in the reference image matches the defect type of the target product in the image to be detected, the output results of the segmentation model can be further combined to identify the defect type of the target product in the image to be detected. Ultimately, defects that are the same as the product defect type in the reference image can be output, thereby improving the accuracy and efficiency of detecting specific defects.

[0113] In some embodiments, the method further comprises:

[0114] Cutting out a first image including defects of the target product from the image to be inspected according to the segmentation result;

[0115] cropping a second image including a defect of the target product from the reference image;

[0116] It is determined according to the first image and the second image whether the defect type of the product in the reference image is the same as the defect type of the target product in the image to be detected.

[0117] In this embodiment, the defect position in the image to be detected can be determined according to the segmentation result, the image to be detected is cropped according to the defect position, and the area where the defect position in the image to be detected is separated from the entire image to obtain a first image. Similarly, the reference image is marked with a defect position, and the reference image can be cropped according to the defect position, and the area where the defect position in the reference image is separated from the entire image to obtain a second image.

[0118] The features of the first image and the second image, such as shape, size, color, etc., can be extracted by using a deep learning model, and the similarity of the features of the first image and the second image can be compared. If the similarity is greater than a preset threshold, it can be considered that the defect type of the product in the reference image is the same as the defect type of the target product in the image to be detected.

[0119] In this embodiment, by cropping a first image containing the target product defect from the image to be inspected according to the segmentation result, and cropping a second image containing the same type of defect from the reference image, the defect area can be accurately located, the extraction of defect-specific features is enhanced, and the adjacent contextual information is retained, thereby capturing more meaningful semantic features and enhancing the ability to recognize different defect types.

[0120] In some embodiments, judging whether the defect type of the product in the reference image and the defect type of the target product in the image to be inspected are the same according to the first image and the second image includes:

[0121] When the category of the target product is the target category, the first image and the second image are input into a comparison classifier corresponding to the category of the target product to obtain a first comparison result output by the comparison classifier.

[0122] In this embodiment, the comparison classifier may be a pre-trained machine learning model, such as a support vector machine, K nearest neighbor, random forest, naive Bayes or deep learning model, etc. For different categories of products, a corresponding comparison classifier may be specially trained.

[0123] In one example, the comparison classifier can be a lightweight deep learning model, including convolutional layers, pooling layers, and fully connected layers. The convolutional layer uses a convolution kernel to slide on the image to perform a convolution operation, thereby extracting local features of the image, such as edges or texture details in a specific direction. Subsequently, the pooling layer reduces the data dimension by maximum pooling or average pooling while retaining important feature information. The fully connected layer further integrates these features for the final classification or regression task.

[0124] The first image and the second image are input into a comparison classifier corresponding to the category of the target product, and the comparison classifier can extract image features of the first image and the second image, compare and classify the image features of the first image and the second image, and output a first comparison result. The first comparison result can be a probability value indicating the possibility that the defects of the first image and the second image belong to the same type of defects, or the first comparison result can also be a clear classification label indicating whether the defects of the first image and the second image belong to the same type.

[0125] In this embodiment, by judging the category of the target product, if the category of the target product is a specific category, the cropped first image containing defects and the second image in the reference image are input into the comparison classifier corresponding to the category of the target product. The comparison classifier can be pre-trained and optimized for products of a specific category, so as to be more accurate in comparing defect types.

[0126] In some embodiments, judging whether the defect type of the product in the reference image and the defect type of the target product in the image to be inspected are the same according to the first image and the second image includes:

[0127] In a case where the category of the target product is not the target category, the first image and the second image are input into the multimodal large model to obtain a second comparison result output by the multimodal large model.

[0128] In this embodiment, different categories of target products may correspond to different comparison classifiers. However, in some cases, new categories of target products may appear. In this case, the multimodal large model may be used to compare the first image and the second image, thereby obtaining a second comparison result output by the multimodal large model. The second comparison result is represented by text, for example, "Yes, the defect types in the first image and the second image are both type 1".

[0129] In this embodiment, by judging the category of the target product, if the category of the target product is not a specific category, the first image and the second image can be input into a multimodal large model, and the cross-modal feature extraction and fusion capabilities of the model can be used to improve the accuracy of defect type comparison through deep feature extraction and semantic understanding.

[0130] In some embodiments, identifying the defect type of the target product in the image to be inspected according to the segmentation result includes:

[0131] The first image is input into a type recognition classifier corresponding to the category of the target product, and the defect type of the target product outputted by the type recognition classifier is obtained.

[0132] In this embodiment, the type recognition classifier can be a lightweight deep learning model, and the corresponding type recognition classifier can be pre-trained for each category of target products, so that the type recognition classifier corresponding to each category of target products can focus on the difference in defect distribution of target products within the category. This customization method helps the model converge faster and improves the accuracy of defect category identification. The lightweight design makes the model more efficient, occupies less computing resources, and is suitable for deployment in resource-constrained environments, such as embedded systems or mobile devices.

[0133] During the training phase, the type recognition classifier can receive a large amount of annotated data, which includes images of different defect types and their corresponding labels. By repeatedly learning the relationship between these image features and labels, the type recognition classifier gradually optimizes internal parameters to improve recognition accuracy.

[0134] In this embodiment, the first image can be input into a type recognition classifier corresponding to the category of the target product. The type recognition classifier can extract image features from the first image, such as image texture, edge, color distribution and other features, and then analyze these features based on the knowledge obtained through training to output a classification result. The classification result can be a probability value indicating the possibility that the defect belongs to a specific type, or it can also be a clear classification label indicating the defect type of the target product.

[0135] In this embodiment, by cropping the first image according to the segmentation result, the defect area can be accurately located, the extraction of defect-specific features is enhanced, and the adjacent contextual information is retained, so that more meaningful semantic features can be captured. The first image is input into a type recognition classifier corresponding to the target product category, so that a suitable classifier can be selected according to different product characteristics. The specially trained classifier is used for type recognition, which can give full play to the advantages of different classifiers in capturing different product characteristics and improve the accuracy of defect type recognition.

[0136] The following is an example of a scenario to introduce the industrial defect segmentation method of the embodiment of the present application. In this scenario example, the target products can include 5 categories, namely Cable, Cylinder, PCB (Printed Circuit Board), Screw and Wood, and each target product is divided into specific defect categories according to the product defect type.

[0137] like Figure 2 As shown, the Query input is the image to be detected. First, the multimodal large model is used to make a preliminary prediction (Predict) on the image to be detected, and the category of the predicted target product is matched (Match) from the current product category candidate (Current Product Class Candidate). According to the prediction results, the segmentation model corresponding to the category of the target product is determined. If the target product is Cylinder, the segmentation model is determined to be the Mask2Former model + PatchCore model. If the target product is one of the other four categories, the segmentation model is determined to be the Mask2Former model. According to the segmentation results output by the segmentation model, the defect location of the target product can be determined.

[0138] In this scenario example, the Support input is a reference image. The defects of the image to be detected can be cropped (Crop) according to the segmentation results to obtain the first image, and the defects of the reference image can be cropped to obtain the second image. Through cropping, the extraction of defect-specific features is enhanced, while the adjacent contextual information is retained, so that more meaningful semantic features can be captured.

[0139] The cropped first image and the second image can be processed differently according to the category of the target product. For example, if the category of the target product is Cylinder, since a special comparison classifier has not been trained for Cylinder, the defect types of the first image and the second image can be compared through the multimodal large model to obtain a comparison result. The prompt (Prompt) for inputting the multimodal large model can be "Carefully compare the defect at the Box1 position in the first image with the defect at the Box2 position in the second image to determine whether they belong to the same type." If the category of the target product is any of the other four categories, the comparison classifier corresponding to the category of the target product can be selected from (Model Zoo), and the first image and the second image can be input into the comparison classifier to obtain a comparison result. If the comparison result is the same defect type (Same Defect Type), the first image can be input into the type recognition classifier corresponding to the category of the target product for refined masking (RefinedMask) to identify the defect type.

[0140] The industrial defect segmentation method provided in the embodiment of the present application can be executed by an industrial defect segmentation device. In the embodiment of the present application, an industrial defect segmentation device executing the industrial defect segmentation method is taken as an example to illustrate the industrial defect segmentation device provided in the embodiment of the present application.

[0141] An embodiment of the present application also provides an industrial defect segmentation device.

[0142] like Figure 3 As shown, the industrial defect segmentation device comprises:

[0143] An input module 310 is used to input an image to be detected containing a target product into a preset multimodal large model to obtain a category of the target product output by the multimodal large model;

[0144] A determination module 320, for determining a segmentation model corresponding to the category of the target product;

[0145] The output module 330 is used to input the image to be detected into the segmentation model corresponding to the category of the target product to obtain the segmentation result output by the segmentation model, and the segmentation result includes the defect position of the target product in the image to be detected.

[0146] According to the industrial defect segmentation device of the present application, by inputting the image to be detected containing the target product into a preset multimodal large model, the category of the target product output by the multimodal large model is obtained; the segmentation model corresponding to the category of the target product is determined; the image to be detected is input into the segmentation model corresponding to the category of the target product, and the segmentation result output by the segmentation model is obtained, and the segmentation result includes the defect position of the target product in the image to be detected. The embodiment of the present application processes the image to be detected by using a multimodal large model to determine the category of the target product, and selects the corresponding segmentation model according to the category of the target product, so that the appropriate segmentation model can be selected according to different product characteristics, and the specially trained segmentation model is applied for defect segmentation, which can give full play to the advantages of different segmentation models in capturing different product characteristics and improve the accuracy of defect detection.

[0147] In some embodiments, the industrial defect segmentation device further comprises:

[0148] The recognition module is used to obtain a reference image including a product of the same category as the target product; the reference image includes a label, and the label is marked with the defect location and defect type of the product in the reference image; when the defect type of the product in the reference image is the same as the defect type of the target product in the image to be detected, the defect type of the target product in the image to be detected is identified according to the segmentation result; and the defect location and defect type of the target product in the image to be detected are output.

[0149] In some embodiments, the identification module is further configured to:

[0150] Cutting out a first image including defects of the target product from the image to be inspected according to the segmentation result;

[0151] cropping a second image including a defect of the target product from the reference image;

[0152] It is determined according to the first image and the second image whether the defect type of the product in the reference image is the same as the defect type of the target product in the image to be detected.

[0153] In some embodiments, the identification module is further configured to:

[0154] When the category of the target product is the target category, the first image and the second image are input into a comparison classifier corresponding to the category of the target product to obtain a first comparison result output by the comparison classifier.

[0155] In some embodiments, the identification module is further configured to:

[0156] In a case where the category of the target product is not the target category, the first image and the second image are input into the multimodal large model to obtain a second comparison result output by the multimodal large model.

[0157] In some embodiments, the identification module is further configured to:

[0158] The first image is input into a type recognition classifier corresponding to the category of the target product, and the defect type of the target product outputted by the type recognition classifier is obtained.

[0159] In some embodiments, the determination module 320 is further configured to:

[0160] When the category of the target product is the target category, the segmentation model is the Mask2Former model;

[0161] When the category of the target product is not the target category, the segmentation models are the Mask2Former model and the PatchCore model.

[0162] The industrial defect segmentation device in the embodiment of the present application can be an electronic device, or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or it can be other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0163] The industrial defect segmentation device in the embodiment of the present application may be a device having an operating system. The operating system may be a Microsoft (Windows) operating system, an Android (Android) operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0164] In some embodiments, Figure 4As shown, an embodiment of the present application also provides an electronic device 400, including a processor 401, a memory 402, and a computer program stored in the memory 402 and executable on the processor 401. When the program is executed by the processor 401, each process of the above-mentioned industrial defect segmentation method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0165] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0166] An embodiment of the present application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned industrial defect segmentation method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0167] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0168] An embodiment of the present application also provides a computer program product, including a computer program, which implements the above-mentioned industrial defect segmentation method when executed by a processor.

[0169] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0170] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned industrial defect segmentation method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0171] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0172] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0173] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0174] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0175] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0176] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

Claims

1. An industrial defect segmentation method, characterized in that: include: Inputting an image to be detected containing a target product into a preset multimodal large model, and obtaining the category of the target product output by the multimodal large model; Determine a segmentation model corresponding to the category of the target product; The image to be detected is input into a segmentation model corresponding to the category of the target product to obtain a segmentation result output by the segmentation model, wherein the segmentation result includes a defect position of the target product in the image to be detected.

2. The method according to claim 1, characterized in that: The method further comprises: Acquire a reference image including a product of the same category as the target product; the reference image includes a label, and the label is marked with a defect location and a defect type of the product in the reference image; In a case where the defect type of the product in the reference image is the same as the defect type of the target product in the image to be detected, identifying the defect type of the target product in the image to be detected according to the segmentation result; Output the defect location and defect type of the target product in the image to be inspected.

3. The method according to claim 2, characterized in that The method further comprises: Cutting out a first image including the defect of the target product from the image to be detected according to the segmentation result; cropping a second image including the defect of the target product from the reference image; It is determined according to the first image and the second image whether the defect type of the product in the reference image is the same as the defect type of the target product in the image to be detected.

4. The method according to claim 3, characterized in that The determining, based on the first image and the second image, whether the defect type of the product in the reference image is the same as the defect type of the target product in the image to be detected includes: In a case where the category of the target product is a target category, the first image and the second image are input into a comparison classifier corresponding to the category of the target product to obtain a first comparison result output by the comparison classifier.

5. The method according to claim 3, characterized in that: The determining, based on the first image and the second image, whether the defect type of the product in the reference image is the same as the defect type of the target product in the image to be detected includes: In a case where the category of the target product is not the target category, the first image and the second image are input into the multimodal large model to obtain a second comparison result output by the multimodal large model.

6. The method according to claim 3, characterized in that The step of identifying the defect type of the target product in the image to be detected according to the segmentation result includes: The first image is input into a type recognition classifier corresponding to the category of the target product to obtain the defect type of the target product output by the type recognition classifier.

7. The method according to any one of claims 1 to 6, characterized in that: The determining of the segmentation model corresponding to the category of the target product includes: In the case where the category of the target product is the target category, the segmentation model is a Mask2Former model; When the category of the target product is not the target category, the segmentation model is a Mask2Former model and a PatchCore model.

8. An industrial defect segmentation device, characterized in that: include: An input module, used to input an image to be detected containing a target product into a preset multimodal large model, and obtain the category of the target product output by the multimodal large model; A determination module, used to determine a segmentation model corresponding to the category of the target product; The output module is used to input the image to be detected into a segmentation model corresponding to the category of the target product to obtain a segmentation result output by the segmentation model, wherein the segmentation result includes a defect position of the target product in the image to be detected.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images