Fresnel lens defect segmentation method based on dynamic similarity mutual scoring
Through dynamic similarity mutual scoring and manifold learning, the pixel-level exception classification score is optimized, and the problem of failure to utilize test sample information in the unsupervised learning method is solved, and efficient segmentation and detection of unknown irregular defects is achieved.
Patent Information
- Application Number
- CN202411375064.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-30
- Publication Date
- 2025-01-14
AI Technical Summary
The existing unsupervised learning methods rely on normal and abnormal images comparison learning in the training set in the defect detection, and fail to make full use of the unlabeled image information in the test sample, resulting in insufficient detection ability of unknown irregular defects.
Using Fresnel lens defect segmentation method based on dynamic similarity mutual score, the unlabeled test image set is obtained, patch features are extracted and adaptive average pooling is performed in a multi-scale sliding window, and pixel-level is optimized by combining manifold learning method. Anomaly classification scores reduce local noise sensitivity and achieve high-quality defect segmentation.
Make full use of normal information in unlabeled test images, improve the segmentation ability of unknown defect images, improve the accuracy and generalization ability of defect detection, and reduce the overlap area between abnormal and normal patch scores.
Smart Images
Figure CN119313902A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the technical field of target segmentation, and in particular to a Fresnel lens defect segmentation method based on dynamic similarity mutual scoring. Background Art
[0002] Industrial anomaly segmentation (AS) is a basic task in the field of computer vision. With the continuous development of target segmentation technology, automated anomaly segmentation systems have been widely used in the manufacturing industry. In modern manufacturing, product quality inspection is an important link to ensure product qualification rate. However, in the face of the challenge of detecting unknown irregular abnormal defects, traditional supervised learning often relies on a large amount of labeled data, resulting in high training and reasoning costs, and insufficient generalization ability. The emergence of unsupervised learning has effectively solved a series of problems such as dependence on labeled data and insufficient generalization ability, and at the same time provided a theoretical basis and technical support for the development of this project solution.
[0003] The defect detection methods based on unsupervised learning that have been proposed can be roughly divided into three categories: full-shot, few-shot, and zero-shot. For example, the twin-space reprojection network and the prototype residual network for anomaly detection and localization, these methods usually rely on the entire normal data set for training, so they are called full-shot methods. In contrast, RegAD and GraphCore have achieved good defect detection results using only a few normal images, which are called few-shot methods. Recently, WinCLIP and APRIL-GAN have implemented a zero-shot learning method, which has achieved new breakthroughs by using text prompts for anomaly measurement. However, the normal information of a large number of unlabeled images in the test samples has not been fully utilized. The existing unsupervised learning still relies on the comparative learning of normal and abnormal images in the training set to implement the so-called "zero-shot" defect segmentation algorithm. Summary of the invention
[0004] In order to overcome the shortcomings of the above technologies, the present invention provides a Fresnel lens defect segmentation method based on dynamic similarity mutual scoring, which makes full use of a large amount of potential normal information in unlabeled test images and improves the segmentation capability of unknown irregular defect images.
[0005] The technical solution adopted by the present invention to overcome the technical problems is:
[0006] A Fresnel lens defect segmentation method based on dynamic similarity mutual scoring, comprising:
[0007] a) Obtain N normal Fresnel lens images and defective and abnormal Fresnel lens images to obtain a Fresnel lens unlabeled test image set I u ={T i,i=1,...,N}, where T i is the i-th Fresnel lens image; b) extract the i-th Fresnel lens image T i M patch features;
[0008] c) For the i-th Fresnel lens image T i The M patch features are adaptively averaged in three sliding windows to obtain the aggregated patch token.
[0009] d) By aggregating patch tokens Get the pixel-level anomaly classification score θ of the i-th Fresnel lens image i ;
[0010] e) Pixel-level anomaly classification score θ of the i-th Fresnel lens image i Update to get the optimized pixel-level anomaly classification score θ i ′;
[0011] f) According to the optimized pixel-level anomaly classification score θ i ′Get the defect shape and position and get the i-th Fresnel lens image T i Multi-classification.
[0012] Furthermore, the defective abnormal Fresnel lens image in step a) includes a Fresnel lens image with scratches on the surface, a Fresnel lens image with fluff on the surface, a Fresnel lens image with black spots on the surface, a Fresnel lens image with partial distortion, and a Fresnel lens image with deformation.
[0013] Furthermore, in step b), the OpenAI pre-trained dinov2_vitl14 model is used as the network backbone to extract the i-th Fresnel lens image T i There are M patch features in .
[0014] Furthermore, in step b), the i-th Fresnel lens image T i Scaled to 518×518 resolution.
[0015] Further, step c) comprises the following steps:
[0016] c-1) Take the i-th Fresnel lens image T i The M patch features are defined as F i , is the real number space, C is the dimension;
[0017] c-2) Use the numpy.reshape function in python to reshape F i Reshape c-3) Set the scale of the first sliding window to 1, the stride to 1, and the dimension to 1024. Use adaptive average pooling operation in the first sliding window to get the patch token
[0018] c-4) Set the scale of the second sliding window to 2, the stride to 1, and the dimension to 1024. Use adaptive average pooling operation in the second sliding window to get the patch token
[0019] c-5) Set the scale of the third sliding window to 3, the stride to 1, and the dimension to 1024. Use adaptive average pooling operation in the third sliding window to get the patch token
[0020] c-6) Get the aggregated patch token of different sliding windows s=1,2,3.
[0021] Further, step d) comprises the following steps:
[0022] d-1) Aggregate patch token Divided into l stages, through the formula Calculate the i-th Fresnel lens image T i The abnormal score of the pth stage p∈{1,...,l}, W is the dynamic weighted average weight, is the i-th Fresnel lens image T i The mth patch token of the pth stage, m∈{1,...,M}, is the jth Fresnel lens image T j The nth patch token of the pth stage, j∈{1,...,N}, n∈{1,...,M}, ||·||2 is the L2 norm, and the abnormal scores of the pth stage of N-1 normal Fresnel lens images and defective abnormal Fresnel lens images constitute the scoring vector:
[0023]
[0024] Rating vector The average of the abnormal scores in the ith Fresnel lens image T is calculated. If the average is less than 0.3, then the ith Fresnel lens image T i is an abnormal patch. If the average value is greater than or equal to 0.3, then the i-th Fresnel lens image T i It is a normal patch;
[0025] d-2) Through the formula Calculate the score vector after the compressed interval mean operation d-3) Through the formula Calculate the i-th Fresnel lens image T i The patch-level anomaly score of the combination of l stages and 3 sliding windows The i-th Fresnel lens image T i The patch-level anomaly score vector is c i , T is transposed; d-4) Use the numpy.reshape function in Python to transform the i-th Fresnel lens image T i The patch-level anomaly score of the combination of l stages and 3 sliding windows Reshape The i-th Fresnel lens image T i The patch-level anomaly score vector Reshape into patch structure c i ′, Patch structure c i ' Perform upsampling operation to obtain the image T of the Fresnel lens of the i-th image i The anomaly segmentation results are the same as the original resolution By formula Calculate the pixel-level anomaly classification score θ of the i-th Fresnel lens image i .
[0026] Further, step e) comprises the following steps:
[0027] e-1) Define a weighted graph G, G = (V, W), where V is the unlabeled test image set I of the Fresnel lens u The Fresnel lens image in , W is the edge weight, W={W 12 ,...,W 1j ,...,W 21 ,...,W 2j ,...,W ij ,...,W N-1N},W ij =F i·F j , F j is the i-th Fresnel lens image T i There are M patch features in it, and · is the dot product operation;
[0028] e-2) Use the manifold learning method SSO to smooth the edge weight W to classify the pixel-level anomaly score θ i Optimize and obtain the optimized pixel-level anomaly classification score θ i ′.
[0029] Further, step f) comprises the following steps:
[0030] f-1) Use OpenCV's findContours function to extract the optimized pixel-level anomaly classification score θ i The defect shape and location of each abnormal patch in ′;
[0031] f-2) Using the DBSCAN density clustering function on the i-th Fresnel lens image, a multi-classification of unlabeled Fresnel lens image defects is obtained.
[0032] The beneficial effects of the present invention are as follows: the dinov2_vitl14 visual large model is used as the backbone for extracting the features of the Fresnel lens image; a multi-scale sliding window aggregation module is designed to realize the effective representation of cross-scale abnormal regions, the patch structure is redefined, and the multi-scale patch is obtained by using the adaptive average pooling operation in windows of different sizes; a high-quality patch-level abnormal score is obtained based on the dynamic mutual scoring mechanism DSMS, and the overlapping area of the abnormal patch and the normal patch score is reduced by the compression interval mean operation; the edge weighted graph is reshaped and the classification score is optimized by using the relationship between pixel-level features to reduce the sensitivity of the pixel-level abnormal segmentation score to local noise during the mutual scoring process; the abnormal defect probability heat map and the original Fresnel lens image are superimposed and the abnormal region category is marked to realize the visualization of the defect segmentation of the Fresnel lens. The present invention makes full use of a large amount of potential normal information in the unlabeled test image to improve the segmentation capability for unknown irregular defect images. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION
[0034] The following is combined with Figure 1 The present invention is further described.
[0035] A Fresnel lens defect segmentation method based on dynamic similarity mutual scoring, comprising:
[0036] a) Obtain N normal Fresnel lens images and defective and abnormal Fresnel lens images to obtain a Fresnel lens unlabeled test image set I u ={T i ,i=1,...,N}, where T i is the i-th Fresnel lens image. b) Extract the i-th Fresnel lens image T i There are M patch features in .
[0037] c) For the i-th Fresnel lens image T i The M patch features in the image are respectively subjected to adaptive average pooling in three sliding windows to improve the anomaly modeling capability of cross-scale patches. The feature structure is redefined and the aggregated patch token is obtained by using adaptive average pooling in windows of different sizes.
[0038] d) By aggregating patch tokens Get the pixel-level anomaly classification score θ of the i-th Fresnel lens image i A dynamic similarity mutual scoring mechanism is proposed, which only needs to use the unlabeled test image set I of the Fresnel lens. u High-quality patch-level anomaly classification scores can be obtained from unlabeled test images. Higher scores are assigned to abnormal patches and lower scores are assigned to normal patches. The overlap between abnormal and normal patch scores is reduced by compressing the interval mean operation.
[0039] e) Pixel-level anomaly classification score θ of the i-th Fresnel lens image i Update to get the optimized pixel-level anomaly classification score θ i ′.
[0040] f) According to the optimized pixel-level anomaly classification score θ i ′Get the defect shape and position and get the i-th Fresnel lens image T i Multi-classification.
[0041] In one embodiment of the present invention, the defective abnormal Fresnel lens image in step a) includes surface defects and structural defects of the object. The surface defects of the object include: Fresnel lens image with scratches on the surface, Fresnel lens image with fluff on the surface, and Fresnel lens image with black spots on the surface. The structural defects include: Fresnel lens image with partial distortion of the lens and Fresnel lens image with deformation of the lens.
[0042] In one embodiment of the present invention, in step b), the OpenAI pre-trained dinov2_vitl14 model is used as the network backbone to extract the i-th Fresnel lens image T i There are M patch features in .
[0043] In one embodiment of the present invention, in step b), the i-th Fresnel lens image T i Scaled to 518×518 resolution.
[0044] In one embodiment of the present invention, step c) comprises the following steps:
[0045] c-1) Take the i-th Fresnel lens image T i The M patch features are defined as F i , is the real number space and C is the dimension.
[0046] c-2) Use the numpy.reshape function in python to reshape F i Reshape c-3) Set the scale of the first sliding window to 1, the stride to 1, and the dimension to 1024. Use adaptive average pooling operation in the first sliding window to get the patch token
[0047] c-4) Set the scale of the second sliding window to 2, the stride to 1, and the dimension to 1024. Use adaptive average pooling operation in the second sliding window to get the patch token
[0048] c-5) Set the scale of the third sliding window to 3, the stride to 1, and the dimension to 1024. Use adaptive average pooling operation in the third sliding window to get the patch token
[0049] c-6) Get the aggregated patch token of different sliding windows In one embodiment of the present invention, step d) comprises the following steps:
[0050] d-1) Aggregate patch token Divided into l stages, through the formula Calculate the i-th Fresnel lens image T i The abnormal score of the pth stage W is the dynamic weighted average weight, is the i-th Fresnel lens image T i The mth patch token of the pth stage, m∈{1,...,M}, is the jth Fresnel lens image T j The nth patchtoken of the pth stage, j∈{1,...,N}, n∈{1,...,M}, ·2 is the L2 norm, and the abnormal scores of the pth stage of N-1 normal Fresnel lens images and defective abnormal Fresnel lens images constitute the scoring vector:
[0051]
[0052] Rating vector The average of the abnormal scores in the ith Fresnel lens image T is calculated. If the average is less than 0.3, then the ith Fresnel lens image T i is an abnormal patch. If the average value is greater than or equal to 0.3, then the i-th Fresnel lens image T i This is a normal patch.
[0053] d-2) Through the formula Calculate the score vector after the compressed interval mean operation By using the partial Fresnel lens image after the compression interval to score each patch, the problem of excessive overlap between normal patches and abnormal patches can be effectively suppressed. d-3) By formula Calculate the i-th Fresnel lens image T i The patch-level anomaly score of the combination of l stages and 3 sliding windows The i-th Fresnel lens image T i The patch-level anomaly score vector is c i , T is transpose.
[0054] d-4) Use the numpy.reshape function in python to transform the i-th Fresnel lens image T i The patch-level anomaly score of the combination of l stages and 3 sliding windows Reshape The i-th Fresnel lens image T i The patch-level anomaly score vector Reshape into patch structure c i ′, Patch structure c i ' Perform upsampling operation to obtain the image T of the Fresnel lens of the i-th image i The anomaly segmentation results are the same as the original resolution By formula Calculate the pixel-level anomaly classification score θ of the i-th Fresnel lens image i .
[0055] In one embodiment of the present invention, step e) comprises the following steps:
[0056] e-1) Define a weighted graph G, G = (V, W), where V is the unlabeled test image set I of the Fresnel lens u The Fresnel lens image in , W is the edge weight, W={W 12 ,...,W 1j ,...,W 21 ,...,W 2j ,…,W ij ,...,W N-1N},W ij =F i ·F j , F j is the i-th Fresnel lens image T i There are M patch features in it, and · is the dot product operation.
[0057] e-2) Use the manifold learning method SSO to smooth the edge weight W to classify the pixel-level anomaly score θ i Optimize and obtain the optimized pixel-level anomaly classification score θ i ′. Reshaping the edge weighted graph reduces the sensitivity of the pixel-level anomaly segmentation score to local noise during the mutual scoring process, and optimizes the similarity scoring mechanism by utilizing the patch similarity weight. The manifold learning method SSO is an existing technology, see: Jiayan Jiang, Bo Wang, and Zhuowen Tu. Unsupervised metric learning by self-smoothing operator. In 2011 International Conference on Computer Vision, pp. 794–801. IEEE, 2011.
[0058] In one embodiment of the present invention, step f) comprises the following steps:
[0059] f-1) Finally, all optimized pixel-level anomaly classification scores are output to form an anomaly probability map. Each pixel value represents the probability that this position is an anomaly. Therefore, the optimized pixel-level anomaly classification score θ is extracted using the findContours function of OpenCV. i f-2) Use the DBSCAN density clustering function on the i-th Fresnel lens image to obtain the multi-classification of unlabeled Fresnel lens image defects.
[0060] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A Fresnel lens defect segmentation method based on dynamic similarity mutual scoring, characterized in that: include: a) Obtain N normal Fresnel lens images and defective and abnormal Fresnel lens images to obtain a Fresnel lens unlabeled test image set I u ={T i ,i=1,...,N}, where T i is the i-th Fresnel lens image; b) Extract the i-th Fresnel lens image T i M patch features; c) For the i-th Fresnel lens image T i The M patch features are adaptively averaged in three sliding windows to obtain the aggregated patch token. d) By aggregating patch tokens Get the pixel-level anomaly classification score θ of the i-th Fresnel lens image i ; e) Pixel-level anomaly classification score θ of the i-th Fresnel lens image i Update to get the optimized pixel-level anomaly classification score θ i ′; f) According to the optimized pixel-level anomaly classification score θ i ′Get the defect shape and position and get the i-th Fresnel lens image T i Multi-classification.
2. The Fresnel lens defect segmentation method based on dynamic similarity mutual scoring according to claim 1 is characterized in that: The defective abnormal Fresnel lens image in step a) includes a Fresnel lens image with scratches on the surface, a Fresnel lens image with fluff on the surface, a Fresnel lens image with black spots on the surface, a Fresnel lens image with partial distortion, and a Fresnel lens image with deformation.
3. The Fresnel lens defect segmentation method based on dynamic similarity mutual scoring according to claim 1, characterized in that: In step b), the OpenAI pre-trained dinov2_vitl14 model is used as the network backbone to extract the i-th Fresnel lens image T i There are M patch features in .
4. The Fresnel lens defect segmentation method based on dynamic similarity mutual scoring according to claim 1, characterized in that: In step b), the i-th Fresnel lens image T i Scaled to 518×518 resolution.
5. The Fresnel lens defect segmentation method based on dynamic similarity mutual scoring according to claim 1, characterized in that: Step c) comprises the following steps: c-1) Take the i-th Fresnel lens image T i The M patch features are defined as F i , is the real number space, C is the dimension; c-2) Use the numpy.reshape function in python to reshape F i Reshape c-3) Set the scale of the first sliding window to 1, the stride to 1, and the dimension to 1024. Use adaptive average pooling operation in the first sliding window to get the patch token c-4) Set the scale of the second sliding window to 2, the stride to 1, and the dimension to 1024. Use adaptive average pooling operation in the second sliding window to get the patch token c-5) Set the scale of the third sliding window to 3, the stride to 1, and the dimension to 1024. Use adaptive average pooling operation in the third sliding window to get the patch token c-6) Get the aggregated patch token of different sliding windows s=1,2,3.
6. The Fresnel lens defect segmentation method based on dynamic similarity mutual scoring according to claim 5, characterized in that: Step d) comprises the following steps: d-1) Aggregate patch token Divided into l stages, through the formula Calculate the i-th Fresnel lens image T i The abnormal score of the pth stage p∈{1,…,l}, W is the dynamic weighted average weight, is the i-th Fresnel lens image T i The mth patch token of the pth stage, m∈{1,…,M}, is the jth Fresnel lens image T j The nth patch token of the pth stage, j∈{1,…,N}, n∈{1,…,M}, ||·||2 is the L2 norm, and the abnormal scores of the pth stage of N-1 normal Fresnel lens images and defective abnormal Fresnel lens images constitute the scoring vector: Rating vector The average of the abnormal scores in the ith Fresnel lens image T is calculated. If the average is less than 0.3, then the ith Fresnel lens image T i is an abnormal patch. If the average value is greater than or equal to 0.3, then the i-th Fresnel lens image T i It is a normal patch; d-2) Through the formula Calculate the score vector after the compressed interval mean operation d-3) Through the formula Calculate the i-th Fresnel lens image T i The patch-level anomaly score of the combination of l stages and 3 sliding windows The i-th Fresnel lens image T i The patch-level anomaly score vector is c i , T is transposed; d-4) Use the numpy.reshape function in Python to transform the i-th Fresnel lens image T i The patch-level anomaly score of the combination of l stages and 3 sliding windows Reshape The i-th Fresnel lens image T i The patch-level anomaly score vector Reshape into patch structure c i ′, Patch structure c i ' Perform upsampling operation to obtain the image T of the Fresnel lens of the i-th image i The anomaly segmentation results are the same as the original resolution By formula Calculate the pixel-level anomaly classification score θ of the i-th Fresnel lens image i .
7. The Fresnel lens defect segmentation method based on dynamic similarity mutual scoring according to claim 5, characterized in that: Step e) comprises the following steps: e-1) Define a weighted graph G, G = (V, W), where V is the unlabeled test image set I of the Fresnel lens u The Fresnel lens image in , W is the edge weight, W={W 12 ,…,W 1j ,…,W 21 ,…,W 2j ,...,W ij ,...,W N-1N },W ij =F i ·F j , F j is the i-th Fresnel lens image T i There are M patch features in it, and · is the dot product operation; e-2) Use the manifold learning method SSO to smooth the edge weight W to classify the pixel-level anomaly score θ i Optimize and obtain the optimized pixel-level anomaly classification score θ i ′.
8. The Fresnel lens defect segmentation method based on dynamic similarity mutual scoring according to claim 6, characterized in that: Step f) comprises the following steps: f-1) Use OpenCV's findContours function to extract the optimized pixel-level anomaly classification score θ i The defect shape and location of each abnormal patch in ′; f-2) Using the DBSCAN density clustering function on the i-th Fresnel lens image, a multi-classification of unlabeled Fresnel lens image defects is obtained.