Zero-shot industrial defect segmentation method and device based on no-label image mutual scoring
By using a labelless image mutual scoring method, and leveraging multi-aggregation degree neighborhood aggregation and manifold learning to optimize classification scores, the problem of insufficient generalization in zero-shot defect detection and segmentation in industrial scenarios is solved, achieving higher accuracy in defect detection and segmentation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HUAZHONG UNIV OF SCI & TECH
- Filing Date
- 2023-12-14
- Publication Date
- 2026-05-19
AI Technical Summary
Existing zero-shot defect detection and segmentation methods lack generalization in industrial scenarios and cannot effectively understand the abstract concept of "defect," resulting in poor detection and segmentation performance.
A method based on unlabeled image mutual scoring is adopted. Image features are extracted through deep neural networks, and local regions between images are scored using a multi-aggregation degree neighborhood aggregation module and a mutual scoring module. Combining manifold learning to optimize classification scores, accurate zero-shot detection and segmentation are achieved.
It improves the accuracy of industrial defect detection and segmentation, effectively segmenting defects of various sizes and improving classification accuracy.
Smart Images
Figure CN117649395B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision technology, and more specifically, relates to a zero-sample industrial defect segmentation method and apparatus based on mutual scoring of unlabeled images. Background Technology
[0002] Defect detection and segmentation have always been crucial tasks in industrial production. During production or use, products may encounter various problems such as breakage, assembly errors, and contamination, rendering them unusable. Efficiently and automatically detecting product defects and segmenting their locations has become a significant challenge for industry. Deep learning-based methods have made considerable progress in this task. Most of these methods learn normal features from a large number of normal samples, comparing these features during testing. The greater the difference between the learned and normal features, the higher the probability that the feature location is a defect. However, in real-world industrial scenarios, factors such as lighting, machine condition, and product appearance vary significantly, resulting in substantial differences between batches of samples. It is impossible to collect a large number of normal samples beforehand for training, and collecting training samples is extremely costly in terms of manpower and resources. Consequently, these methods have limited applicability in many scenarios.
[0003] Recently, with the emergence of pre-trained large models, a number of zero-shot or few-shot defect detection and segmentation methods have gradually emerged. These methods do not require, or only require, a small number of normal samples in the training set as references, and can achieve good classification and segmentation results in the test set. Methods based on image-text alignment use pre-trained text vision models such as CLIP, relying on text prompts to match defect regions with corresponding text; methods based on data augmentation expand a small number of normal samples to simulate the distribution of normal samples.
[0004] Existing zero-shot defect detection and segmentation methods rely on prompting engineering to exploit the capabilities of pre-trained large models. However, since almost all pre-trained large models are trained on natural scene images, they suffer from insufficient generalization to industrial scene images. These models cannot well understand the relatively abstract concept of "defect," resulting in generally poor detection and segmentation performance. Summary of the Invention
[0005] To address the aforementioned shortcomings or improvement needs of existing technologies, this invention provides a zero-shot industrial defect segmentation method based on unlabeled image peer scoring. This method fully utilizes the information inherent in the unlabeled test images at both the image level and the local image region level, without relying on any prompts or auxiliary datasets, to achieve more accurate zero-shot industrial defect detection and segmentation.
[0006] To achieve the above objectives, according to one aspect of the present invention, a zero-shot industrial defect segmentation method based on unlabeled image mutual scoring is provided, the method comprising:
[0007] (1) Use a deep neural network to extract features from all unlabeled test images;
[0008] (2) Input the extracted local image region-level features into the multi-aggregation degree neighborhood aggregation module to obtain aggregated region-level features aggregated in multiple neighborhoods of different sizes;
[0009] (3) Input all aggregated region-level features of all unlabeled test images into the mutual scoring module to obtain zero-sample segmentation results;
[0010] (4) Input the extracted image-level features and the anomaly score map of the image into the classification optimization module, and use the manifold learning method to optimize the classification scores to obtain the zero-shot classification result.
[0011] In one embodiment of the present invention, step (1) specifically includes:
[0012] (1.1) Prepare a ViT network pre-trained in natural scene images as a feature extractor. Any existing ViT network architecture and pre-trained network parameters can be used.
[0013] (1.2) Input a batch of unlabeled test images of industrial scenes to be inspected, D = {I1, I2, ..., I...} N}, where I i Let be the i-th image, and N be the total number of images. The unlabeled test images contain an unknown number of normal images and abnormal images, where the abnormal images contain defects.
[0014] (1.3) Fix the network parameters of the feature extractor, divide the multi-layer ViT backbone network into 4 stages evenly by layer, and extract the local image region-level features output by each stage. Where i represents image I i Let l be the ViT stage number, C be the feature vector length, and m be the local image region number. Define the total number of local image regions as M. Extract the features of the cls token from the last layer of ViT after linear mapping as the image-level feature F. i .
[0015] In one embodiment of the present invention, step (2) specifically includes:
[0016] (2.1) For each local image region of each image, obtain the local image region-level neighborhood features of its three different neighborhoods in space: 1×1, 3×3, and 5×5, in order to detect and segment defects of different sizes.
[0017] (2.2) Adaptive average pooling is used to aggregate the neighborhood features of the local image region m with three different aggregation degrees, respectively, to obtain the aggregated region-level features at that location. Where r is the aggregation degree, representing the aggregation of neighborhood features of size r×r.
[0018] In one embodiment of the present invention, step (3) specifically includes:
[0019] (3.1) Unlabeled test images are mutually scored for local image regions, using the minimum L2 distance as the scoring criterion. Image I j For image I i The scoring formula for the m-th local image region is as follows:
[0020]
[0021] in This represents image I under the same conditions of r and l. j Give image I i The abnormal score assigned to the m-th local image region. Normal local image regions are assigned smaller abnormal scores by more images, while defective local image regions are assigned larger abnormal scores. The larger the abnormal score, the greater the probability that the location is defective.
[0022] (3.2) To reduce false positives, interval averaging is used to dynamically select the lowest 30% of scores based on different l and r values to obtain the comprehensive score. Image I i Image I is obtained by averaging the comprehensive scores at different l and r. i The final anomaly score of the m-th local image region as follows:
[0023]
[0024] (3.3) Unlabeled test image I i Abnormal score plot The abnormal fraction map is restored to its original shape and upsampled to the original image size using bilinear interpolation. A threshold is set to classify all pixels in the image into two categories: normal and defect-free, and defective, thus obtaining the defect segmentation result.
[0025] In one embodiment of the present invention, step (4) specifically includes:
[0026] (4.1) For image I i The anomaly score map is used, and the maximum value is taken as the classification anomaly score c of the image. iA higher classification anomaly score indicates a greater probability of an anomaly being present in the image. C is defined as C = [c1, c2, ..., c...]. N ] T , represents the anomaly score before optimization, which is optimized by leveraging the similarity between different images;
[0027] (4.2) Define the similarity matrix W between all images, and group the images I... i With Image I j The similarity is defined as the similarity between images I i Image-level features F i With Image I j Image-level features F j The dot product of vectors, i.e., W i,j =F i ·F j ;
[0028] (4.3) Use multi-window masking operation on W to limit the range of mutual optimization, and use a binarized window mask M. k as follows:
[0029]
[0030] Where N k Indicates with I i The k closest samples are used to form a multi-window mask using multiple k values. Where k∈{k1,k2,…,k K}, where K is the number of window masks;
[0031] (4.4) Based on the idea of manifold learning, the classification anomaly score is optimized using the following formula:
[0032]
[0033] Where M k For multi-window masks In the context of a single-window mask, C represents the unoptimized classification anomaly score. The optimized classification anomaly score, D is a diagonal matrix. ⊙ indicates element-wise multiplication. In Image I i The probability of defects existing in it, according to A threshold is set to classify all images into two categories: normal, defect-free images and images with defects, thus obtaining the defect image classification results.
[0034] In one embodiment of the present invention, in step (1.1), large-ViT pre-trained on the WIT-400M dataset using the CLIP method is used.
[0035] In one embodiment of the present invention, in the large-ViT, the size of the local image region is 14×14 and the resolution is 336×336.
[0036] In one embodiment of the present invention, in step (4.3), K = 2, k ∈ {2, 3}.
[0037] According to another aspect of the present invention, a zero-sample industrial defect segmentation device based on unlabeled image mutual scoring is also provided, characterized in that it includes at least one processor and a memory, the at least one processor and the memory are connected through a data bus, the memory stores instructions that can be executed by the at least one processor, and the instructions, after being executed by the processor, are used to complete the zero-sample industrial defect segmentation method based on unlabeled image mutual scoring.
[0038] In summary, the technical solutions conceived by this invention have the following beneficial effects compared with the prior art:
[0039] This invention employs a multi-degree-of-aggregation neighborhood aggregation module to effectively segment defects of various sizes in industrial scenarios. In the mutual scoring module, the rich prior information in unlabeled test images is fully utilized through scoring between unlabeled test images, resulting in significantly higher detection and segmentation accuracy than current zero-sample industrial defect detection and segmentation methods. Furthermore, the classification optimization module in this invention can be used as a general-purpose classification optimization module for defect detection in industrial scenarios, and can improve the classification accuracy of most industrial defect detection methods. Attached Figure Description
[0040] Figure 1 This is a flowchart illustrating the zero-sample industrial defect segmentation method based on unlabeled image mutual scoring according to the present invention. Detailed Implementation
[0041] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.
[0042] In order to solve the problems existing in the current technology, such as Figure 1As shown, this invention provides a zero-shot industrial defect segmentation method based on unlabeled image mutual scoring, comprising the following steps:
[0043] (1) Use a deep neural network to extract image-level features from all unlabeled test images. The specific steps include:
[0044] (1.1) Prepare a ViT network pre-trained in natural scene images as a feature extractor. Any existing ViT network architecture and pre-trained network parameters can be used.
[0045] This invention uses large-ViT pre-trained on the WIT-400M dataset using the CLIP method, with a local image region (patch) size of 14×14 and a resolution of 336×336;
[0046] (1.2) Input a batch of unlabeled test images of industrial scenes to be inspected, D = {I1, I2, ..., I...} N}, where I i Let N be the i-th image and N be the total number of images. The unlabeled test images contain an unknown number of normal images and abnormal images, among which the abnormal images contain defects. The present invention needs to detect the abnormal images with defects and segment the regions where the defects are located.
[0047] (1.3) Fix the network parameters of the feature extractor, divide the multi-layer ViT backbone network into 4 stages evenly by layer, and extract the local image region-level features output by each stage. Where i represents image I i Let l be the ViT stage number, C be the feature vector length, and m be the local image region number. Define the total number of local image regions as M. Extract the features of the cls token from the last layer of ViT after linear mapping as the image-level feature F. i .
[0048] (2) Extract local image region-level features The input is processed by the multi-aggregation degree neighborhood aggregation module to obtain aggregation region-level features aggregated within multiple neighborhoods of different sizes. r represents the degree of polymerization, which includes the following sub-steps:
[0049] (2.1) For each local image region of each image, obtain the local image region-level neighborhood features of its three different neighborhoods in space: 1×1, 3×3, and 5×5, in order to detect and segment defects of different sizes.
[0050] (2.2) Adaptive average pooling is used to aggregate the neighborhood features of the local image region m with three different aggregation degrees, respectively, to obtain the aggregated region-level features at that location. Where r is the aggregation degree, representing the aggregation of neighborhood features of size r×r.
[0051] (3) Input all aggregated region-level features of all unlabeled test images into the mutual scoring module to obtain zero-sample segmentation results, including the following sub-steps:
[0052] (3.1) Unlabeled test images are mutually scored for local image regions, using the minimum L2 distance as the scoring criterion. Image I j For image I i The scoring formula for the m-th local image region is as follows:
[0053]
[0054] in This represents image I under the same conditions of r and l. j Give image I i The abnormal score is assigned to the m-th local image region. In this way, normal local image regions will be assigned smaller abnormal scores to more images, while defective local image regions will be assigned larger abnormal scores. The larger the abnormal score, the greater the probability that the location is defective.
[0055] (3.2) To reduce false positives, interval averaging is used to dynamically select the lowest 30% of scores based on different l and r values to obtain the comprehensive score. Image I i Image I is obtained by averaging the comprehensive scores at different l and r. i The final anomaly score of the m-th local image region as follows:
[0056]
[0057] (3.3) Unlabeled test image I i Abnormal score plot The abnormal fraction map is restored to its original shape and upsampled to the original image size using bilinear interpolation. An appropriate threshold is set to classify all pixels in the image into two categories: normal and defect-free, and defective, thus obtaining the defect segmentation result.
[0058] (4) Extract the image-level features F i Anomaly score plot A of the image i The input is fed into the classification optimization module, where a manifold-based learning method is used to optimize the classification scores to obtain zero-shot classification results. This process includes the following sub-steps:
[0059] (4.1) For image I i The anomaly score map is used, and the maximum value is taken as the classification anomaly score c of the image.i A higher classification anomaly score indicates a greater probability of an anomaly being present in the image. The score represents the anomaly score before optimization. The anomaly score C is then optimized by leveraging the similarity between different images.
[0060] (4.2) Define the similarity matrix W between all images, and group the images I... i With Image I j The similarity is defined as the similarity between images I i Image-level features F i With Image I j Image-level features F j The dot product of vectors, i.e., W i,j =F i ·F j ;
[0061] (4.3) Use multi-window masking operation on W to limit the range of mutual optimization, and use a binarized window mask M. k as follows:
[0062]
[0063] Where N k Indicates with I i The k closest samples are used to form a multi-window mask using multiple k values. Where k∈{k1,k2,…,k K}, where K is the number of window masks, and in this invention K = 2, k ∈ {2, 3};
[0064] (4.4) Based on the idea of manifold learning, the classification anomaly score is optimized using the following formula:
[0065]
[0066] Where M k For multi-window masks In the context of a single-window mask, C represents the unoptimized classification anomaly score. The optimized classification anomaly score, D is a diagonal matrix. ⊙ indicates element-wise multiplication. In Image I i The probability of defects existing in it, according to By setting an appropriate threshold, all images are divided into two categories: normal, defect-free images and images with defects, thus obtaining the defect image classification results.
[0067] Furthermore, the present invention also provides a zero-sample industrial defect segmentation device based on unlabeled image mutual scoring, characterized in that it includes at least one processor and a memory, the at least one processor and the memory are connected through a data bus, the memory stores instructions that can be executed by the at least one processor, and the instructions, after being executed by the processor, are used to complete the zero-sample industrial defect segmentation method based on unlabeled image mutual scoring.
[0068] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A zero-shot industrial defect segmentation method based on unlabeled image cross-scoring, characterized in that, The method includes: (1) Use a deep neural network to extract features from all unlabeled test images; (2) Input the extracted local image region-level features into the multi-aggregation degree neighborhood aggregation module to obtain aggregated region-level features aggregated in multiple neighborhoods of different sizes; specifically including: (2.1) For each local image region of each image, obtain its local image region-level neighborhood features in three different neighborhoods of 1×1, 3×3, and 5×5 in space to detect and segment defects of different sizes; (2.2) Use adaptive average pooling operation to process local image regions These three neighborhood features with different degrees of aggregation are aggregated separately to obtain the aggregated region-level features at that location. ,in The degree of aggregation indicates the degree of aggregation. Neighborhood features of size; (3) Input all aggregated region-level features of all unlabeled test images into the mutual scoring module to obtain zero-sample segmentation results; specifically including: (3.1) Mutual scoring of local image regions between unlabeled test images, using the minimum L2 distance as the scoring criterion, image For images The The scoring formula for a local image region is as follows: in Indicates that in the same and Image in case Give image The An abnormal score is assigned to each local image region. Normal local image regions are assigned smaller abnormal scores to more images, while defective local image regions are assigned larger abnormal scores. The larger the abnormal score, the greater the probability that the location is defective. (3.2) To reduce false positives, interval averaging is used based on different and The system dynamically selects the bottom 30% of scores to obtain the overall score. , to image Different and The image is obtained by averaging the comprehensive scores over time. The The final anomaly score for each local image region ,as follows: ； (3.3) Unlabeled test images Abnormal score plot The abnormal fraction map is restored to its original shape and upsampled to the original size using bilinear interpolation. A threshold is set to classify all pixels in the image into two categories: normal and defect-free, and defective, thus obtaining the defect segmentation result. (4) Input the extracted image-level features and the anomaly score map of the image into the classification optimization module, and use the manifold learning method to optimize the classification scores to obtain the zero-shot classification result.
2. The zero-shot industrial defect segmentation method based on unlabeled image mutual scoring as described in claim 1, characterized in that, Step (1) specifically includes: (1.1) Prepare a ViT network pre-trained in natural scene images as a feature extractor; (1.2) Input a batch of unlabeled test images of industrial scenes to be inspected. ,in For the first Zhang Image The total number of images is represented by the unlabeled test images, which contain an unknown number of normal and abnormal images, with the abnormal images containing defects. (1.3) Fix the network parameters of the feature extractor, divide the multi-layer ViT backbone network into 4 stages evenly by layer, and extract the local image region-level features output by each stage. ,in Representing an image , This refers to the sequence number of the ViT phase. The length of the feature vector. Let be the local image region index, and define the total number of local image regions as . The features of the cls tokens in the last layer of ViT, after linear mapping, are extracted as image-level features. .
3. The zero-sample industrial defect segmentation method based on unlabeled image mutual scoring as described in claim 1 or 2, characterized in that, Step (4) specifically includes: (4.1) For images The anomaly score map is used, and the maximum value is taken as the classification anomaly score of the image. A higher classification anomaly score indicates a greater probability of an anomaly being present in the image. This represents the anomaly score before optimization, which is improved by leveraging the similarity between different images. Optimize; (4.2) Define the similarity matrix between all images. , to image With images The similarity is defined as the image Image-level features With images Image-level features The dot product of vectors, i.e. ; (4.3) to Using multi-window masking operations to limit the scope of mutual optimization, binarized window masks. as follows: in Indicates and closest One sample, using multiple To form a multi-window mask ,in , The number of window masks; (4.4) Based on the idea of manifold learning, the classification anomaly score is optimized using the following formula: in For multi-window masks A single-window mask, The outlier score before optimization. , The optimized classification anomaly score, D is a diagonal matrix. , This indicates element-wise multiplication. In Representing an image The probability of defects existing in it, according to A threshold is set to classify all images into two categories: normal, defect-free images and images with defects, thus obtaining the defect image classification results.
4. The zero-shot industrial defect segmentation method based on unlabeled image mutual scoring as described in claim 2, characterized in that, In step (1.1), large-ViT, which is pre-trained on the WIT-400M dataset using the CLIP method, is used.
5. The zero-shot industrial defect segmentation method based on unlabeled image mutual scoring as described in claim 4, characterized in that, In the large-ViT, the local image region size is 14×14, and the resolution is 336×336.
6. The zero-shot industrial defect segmentation method based on unlabeled image mutual scoring as described in claim 3, characterized in that, In step (4.3), , .
7. A zero-sample industrial defect segmentation device based on unlabeled image mutual scoring, characterized in that, It includes at least one processor and a memory, which are connected via a data bus. The memory stores instructions that can be executed by the at least one processor. After being executed by the processor, the instructions are used to complete the zero-sample industrial defect segmentation method based on unlabeled image mutual scoring as described in any one of claims 1-6.