A machine vision-based intelligent detection method for product surface anomalies
By combining convolutional neural networks and SVM classifiers, the problem of detecting unknown abnormalities on the surface of products in machine vision systems has been solved, enabling fast and real-time detection and localization of product surface anomalies and reducing detection costs.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG UNIV
- Filing Date
- 2022-11-17
- Publication Date
- 2026-05-26
AI Technical Summary
Existing machine vision systems struggle to effectively detect unknown types of product surface defects in industrial production, and require a large number of known abnormal samples for training, making it difficult to obtain abnormal datasets.
A convolutional neural network-based approach was adopted, combining MSE, Cosine, and SSIM loss functions to train a model to identify surface anomalies on products. An abnormal region was located using an SVM classifier, and detection was performed using a normal image training dataset.
It enables rapid, real-time detection of surface anomalies on products without the need to construct anomaly image datasets, and can locate the anomaly positions, reducing detection costs and improving detection efficiency.
Smart Images
Figure CN115908323B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the fields of machine vision and anomaly detection technology, specifically to an intelligent anomaly detection method for product surfaces based on machine vision. Background Technology
[0002] With the advent of Industry 4.0, machine vision is playing an increasingly important role in intelligent manufacturing. Currently, in industrial automation, machine vision systems typically include hardware such as light sources, lenses, industrial cameras, and image acquisition cards, as well as vision processing software. Performing non-contact inspection of objects, improving processing accuracy, detecting product anomalies, and making automated analysis and decisions are crucial components of advanced manufacturing.
[0003] Products manufactured on industrial assembly lines undergo multiple processing steps to complete the entire production process. Each step can introduce defects, leading to various surface anomalies, including dents, bumps, cracks, scratches, and stains. To ensure product quality, surface anomaly detection is crucial. Traditional manual inspection is inefficient, slow, inaccurate, and unreliable. Machine vision-based surface anomaly detection systems for industrial products on assembly lines not only reduce labor and material management costs but also improve detection efficiency.
[0004] Most existing technologies rely on known anomaly samples, requiring an understanding of existing anomaly types, collection of sufficient anomaly samples, and learning of known anomaly types and characteristics. This approach can achieve good results for specific known anomaly types, but anomaly types are infinitely varied and cannot be exhaustively listed, inevitably leading to the encounter of new anomalies that were not predicted beforehand. Common methods for training and testing neural network models using anomaly datasets are essentially supervised training, which has high requirements for anomaly data. However, anomaly samples exist in only a small number of all products, making it very difficult to obtain large-scale anomaly datasets. Summary of the Invention
[0005] To address the aforementioned technical problems, this invention provides an intelligent detection method for product surface anomalies based on machine vision, which can determine whether there are anomalies on the product surface and automatically segment out the areas where anomalies exist.
[0006] The surface anomaly detection system includes a light source, a camera, a convolutional neural network, and an anomaly feature extraction and analysis module.
[0007] Technical solution
[0008] To achieve the above objectives, the present invention provides a machine vision-based intelligent detection method for product surface anomalies, comprising the following steps:
[0009] Step 1: Image dataset collection and screening to comprehensively cover the surface of the industrial product to be tested.
[0010] Step 2: Use a neural network to process the image, select an appropriate network structure, and use a loss function that combines MSE (Mean Square Error), Cosine (Cosine Similarity), and SSIM (Structure Similarity Index Measure) to train and save the model parameters.
[0011] Step 3: Post-processing and testing. Compare the heatmaps of abnormal images with those of normal images to determine the threshold. Perform preprocessing, network computation, and post-processing operations on the test images to determine if there are any anomalies and their locations.
[0012] Furthermore, step 1 specifically includes:
[0013] Step 1.1: Specify the light source and shooting angle. The dataset acquisition scene should be consistent with the actual working scene of the anomaly detection system. The captured images need to have sufficient resolution, and the shooting background needs to remain unchanged and be as pure black as possible.
[0014] Step 1.2: Divide the surface of the industrial products on the production line into regions. For larger areas, divide them into multiple smaller regions.
[0015] Step 1.3: Preprocess the collected data and extract the target foreground and remove the background. Position, crop, and unify the input size of the original image to ensure the subject is centered. Use a solid color background, preferably black, to minimize unnecessary background. Extract the foreground mask image, matching its size to the original image, with the background area set to 0 and the foreground area set to 1. If there are breaks or holes within the foreground, use a closing operation to fill them.
[0016] Since the product position and camera position on the production line are relatively fixed, one sample can be selected to determine the position of the product in the image and the segmentation rules of the product area, and other samples in the same batch are the same.
[0017] Step 1.4: Filter the dataset, remove unclear and low-quality images, use normal images for the training dataset, and keep the abnormal images as the validation dataset or test dataset.
[0018] The Mask image is determined based on the region of interest in a sample image. The corresponding region is then extracted from other product images captured by the camera using the given Mask image, while the background and regions of no interest are masked out.
[0019] The image processing using a neural network described in step 2, which involves training on a preprocessed training dataset, includes:
[0020] Step 2.1: Design a convolutional neural network model. The main body of the network consists of an encoder and a decoder structure, with the encoder and decoder maintaining basic symmetry.
[0021] Step 2.2: Construct the loss function. Combining MSE, Cosine, and SSIM, construct the loss function:
[0022]
[0023] Where α > 0, β > 0, x is the network input, Y is the network output, and h, w, and c are the height, width, and number of channels of the input image, respectively. To avoid the overall loss L being negative, a parameter θ is added for adjustment, where θ > log2 + β.
[0024] Step 2.3: Set appropriate training parameters, train the model using the images processed in Step 1, and save the model parameters.
[0025] After training, the test set reserved in step 1 can be used, or images can be acquired in real time for testing. Images acquired in real time need to undergo the same preprocessing operation as in step 1.4.
[0026] Step 3, the post-processing and testing process, includes the following parts:
[0027] Step 3.1: Use the score to represent the corresponding similarity score: score = 0.5 × [1 - SSIM(X, Y)]. Select a normal image as a sample and input it into the trained model. The model outputs the predicted image. Use the SSIM method to compare the similarity between the input and output images, calculate the similarity score, and obtain the sample image's popularity. Figure 1 .
[0028] Step 3.2: Input the verification image into the trained model for testing to obtain the verification image popularity. Figure 1 .
[0029] Step 3.3: Heat in Steps 3.1 and 3.2 Figure 1 By comparing the results, a quantitative comparison score is obtained, and the verification image heat is obtained. Figure 2 The similarity comparison here can be performed using either the SSIM method or the MSE method.
[0030] Note the need to verify image heat. Figure 2 The test is highly dependent on sample images; the selection and quality of these images directly impact the results. Therefore, it's required that the images being compared have the same placement and shooting angle, or that the two images used for comparison have been adjusted in size, position, etc., without considering positional changes in this task. After the second similarity comparison, common non-abnormal regions in all images are suppressed, while abnormal regions become prominent.
[0031] Step 3.4: Classify the heatmap using SVM (Support Vector Machine). Classify the heatmaps from the validation set images using the heatmaps obtained in Step 3.3. Figure 2 Used for training SVM. A trained SVM classifier can directly classify new test samples into two categories: present or absent.
[0032] Step 3.5: Use a mask to exclude irrelevant areas such as the background, and adjust the foreground area based on the heat value obtained in Step 3.4. Figure 2 Identify the abnormal location, set an appropriate threshold, and identify the location of the abnormality if the threshold is exceeded.
[0033] Working principle
[0034] Production scenarios on assembly lines are relatively fixed and easy to control, resulting in many similarities between normal and abnormal image samples. However, anomalies are diverse, and abnormal regions are often small and inconspicuous. This invention uses normal samples as training data, aiming to identify subtle differences between abnormal and normal samples through network training and post-processing comparison, making it suitable for use in industrial production scenarios.
[0035] Beneficial effects
[0036] The intelligent detection method for product surface anomalies based on machine vision provided by this invention has the following advantages compared with the prior art:
[0037] 1. No abnormal image dataset needs to be constructed; all training images are non-abnormal images. Obtaining a large number of training images is simple and convenient.
[0038] 2. It can not only identify products with surface abnormalities, but also locate the abnormal locations, providing convenience for secondary manual review.
[0039] 3. Perform multiple similarity comparisons to eliminate differences caused by noise, and further eliminate non-abnormal areas.
[0040] 4. Using a lightweight network, the training and testing processes are relatively fast, and real-time detection can be achieved in the testing phase. Attached Figure Description
[0041] Figure 1 This is a flowchart of the process for detecting abnormalities in industrial products on an assembly line according to the present invention;
[0042] Figure 2 This is a schematic diagram of an encoding / decoding network structure according to an embodiment of the present invention;
[0043] Figure 3 This is a foreground extraction mask image according to an embodiment of the present invention;
[0044] Figure 4This is a diagram illustrating the processing effect of an anomaly-free sample according to an embodiment of the present invention.
[0045] Figure 5 This is an image showing the effect of abnormal image processing according to an embodiment of the present invention;
[0046] Figure 6 This is a flowchart of the post-processing operation of the present invention. Detailed Implementation
[0047] The methods and systems of the present invention will now be described more clearly and completely with reference to the accompanying drawings.
[0048] like Figure 1 The figure shows an intelligent anomaly detection method for product surfaces based on machine vision provided by the present invention, which includes the following steps:
[0049] Step 1: Dataset Acquisition and Screening. The acquired images of industrial products from the production line need to fully cover the surface of the product under test.
[0050] Step 1.1: Specify the light source and shooting angle. The dataset collection scene should be consistent with the actual industrial scene where the anomaly detection system works. The captured images need to have sufficient resolution, and the shooting background needs to remain unchanged and be as pure black as possible.
[0051] The DPI (Dots Per Inch) used in the embodiments ranges from 300 to 350. To collect a certain number of product images, taking a cup or kettle as an example, at least three cameras are needed to cover the entire outer surface of the cup or kettle, with the angle between the three cameras and the product being 120 degrees. To obtain clearer images and more photos directly facing the surface of the cup or kettle, more cameras are needed. One camera can be placed above the top of the cup or kettle to primarily capture images of the lid, while three cameras at 120-degree angles are used to capture images of the cup body. Additionally, for capturing images of the inside of the cup or kettle, since cups and kettles are generally open on one side, an additional light source is required, and the camera needs to be close to the rim or inserted into the cup.
[0052] Step 1.2: Divide the surface of the industrial products on the production line into regions. For larger areas, divide them into multiple smaller regions.
[0053] Taking cups and kettles as an example, the lid is considered as a separate area; the connection between the lid and the body is considered as another area, where the surface normal is usually not horizontal or vertical, but rather at a certain angle for transition. For a 360-degree circular area, it can be divided into three or more smaller areas; the outer surface of the cup body is also approximately a circular area, which can be divided into three or more smaller areas; photographing the bottom of the cup requires moving the cup, so we can invert the cup body so that the bottom side is facing up, and treat the bottom area as one area; the junction between the bottom and the body, similar to the junction between the lid and the body, is a circular area, which can be divided into three or more smaller areas.
[0054] Step 1.3: Preprocess the collected data and extract the target foreground and remove the background. Position, crop, and unify the input size of the original image to ensure the subject is centered and reduce unnecessary background. Use a solid color background, preferably black, to facilitate image segmentation and target extraction. Extract the foreground to generate a mask image, the same size as the original image, with the background area set to 0 and the foreground area set to 1. Fill any breaks or holes within the foreground using a closing operation.
[0055] With the product and camera positions relatively fixed on the production line, we can select one sample to determine the product's position in the image and the segmentation rules for the product area, and then repeat this process for other samples in the same batch. Taking a cup or pitcher as an example, the captured image of the cup body can include the connection between the lid and the body, as well as the connection between the bottom and the body. The captured image of the cup body can be divided into three parts: the main body of the cup, the connection between the lid and the body, and the connection between the bottom and the body.
[0056] Step 1.4: Filter the dataset, remove unclear and low-quality images, use normal images for the training dataset, and keep the abnormal images as the validation dataset or test dataset.
[0057] In the product region segmentation process, mask images can be used. A mask image is determined based on the region of interest in a sample image. Using this mask image, the corresponding region is extracted from other product images captured by the camera, masking out the background and areas of no interest. Before training, the acquired images are manually inspected to ensure they are high-quality, complete, and anomaly-free. Any blurry images are discarded and reacquired. Any images with anomalies are retained as validation or test sets.
[0058] Step 2: Process the image using a neural network.
[0059] Step 2.1: Design a convolutional neural network model, such as... Figure 2 As shown, the network consists of an encoder and a decoder structure, with the encoder and decoder maintaining basic symmetry.
[0060] In this example, the input image is 1024×1024, the encoder output is 8×8×512, and the decoder output is the same size as the input image. Figure 2 In the network structure shown, ReLU activation is performed after each convolutional layer computation. The stride of the convolutional layer is 2, and the kernel size is 4. Each time this operation is performed, the length and height of the feature map are halved. The decoder does the opposite; each upsampling operation doubles the length and height of the feature map. The network output size is 1024×1024, consistent with the input image.
[0061] Both the encoder and decoder are non-linear computations, and we aim for the network output image to be as close as possible to a normal image. During training, the input image is a normal image, which can be directly used as ground truth for constraints. After the input image is processed by the encoder, some key features are retained, and the decoder restores the complete image. The network weights retain the common feature information of normal images. The set of normal images is considered as the original space, and the set of network output images is its subspace. Network training continuously brings its subspace closer to the original space.
[0062] Step 2.2: Construct the loss function.
[0063] SSIM is a structural similarity metric used to determine the similarity between two images. Compared to mean squared error loss (MSE) and cosine loss, SSIM more closely resembles subjective human evaluation. The SSIM function can be expressed as:
[0064]
[0065] Where X is the network input image, and Y is the network output predicted image. μ X It is the average value of the input image X, μ Y It is the average value of the predicted image Y, σ X σ is the variance of the input image X. Y It is the variance of the predicted image Y, σ XY It is the covariance between the input image X and the predicted image Y.
[0066] Mean SSIM (MSSIM, Average Structural Similarity) is used in the examples:
[0067]
[0068] Where M is the window size. Based on experience, the parameters are selected as follows: window size of 11x11, and standard deviation of Gaussian weighted formula of 1.5.
[0069] Combining mean squared error loss (MSE), cosine loss, and SSIM loss, the loss function is constructed as follows:
[0070]
[0071] Where α > 0, β > 0, X is the network input, Y is the network output, and h, w, and c are the height, width, and number of channels of the input image, respectively. To avoid the overall loss L being negative, a parameter θ is added for adjustment, where θ > log2 + β.
[0072] In this example, α = 1, β = 0.5, and θ = 1. MSE loss, Cosine loss, and SSIM loss are equally important. Based on the SSIM function's range of [-1, 1], the theoretical minimum overall loss L is 1.5 - log2.
[0073] Step 2.3: Set appropriate training parameters, train the model using the images processed in Step 1, and save the model parameters.
[0074] Step 3: Post-processing and testing.
[0075] The heatmaps of test images are compared with those of normal images to score the samples and determine if any anomalies exist. A threshold for the heatmap of anomaly images is determined; for anomaly samples, the heatmap is used to pinpoint the location of the anomaly. The validation set includes both normal and anomaly images, used to determine parameters such as the threshold. After completing the above settings, anomaly detection can be directly performed on newly acquired test samples.
[0076] Step 3.1: Select a normal image as a sample and input it into the trained model. The model outputs the predicted image. Use the SSIM method to compare the similarity between the input and output images, calculate the similarity score, and obtain the sample image's popularity. Figure 1 The SSIM function has a range of [-1, 1]. A value of 1 indicates that the two images are completely identical, and the smaller the value, the greater the difference between the two images. In order to make the similarity value positively correlated with the degree of similarity and fix the value range between [0, 1], the corresponding similarity score is used: score = 0.5 × [1 - SSIM(X, Y)].
[0077] Implementation, for example Figure 4 As shown, the left image is a normal sample image, and the right image is the heat of a normal image sample. Figure 1 (Represented as a grayscale image). Compared to the normal sample input image, the output predicted image is prone to differences at the background boundary and in areas with complex structures, but it can show consistency in most areas of the main body of the product.
[0078] Step 3.2: Input the verification image into the trained model for testing to obtain the verification image popularity. Figure 1 .
[0079] This part of the operation is similar to step 3.1, except that the objects being processed are different. The images selected in the verification set here include both normal and abnormal images.
[0080] Implementation, for example Figure 5 As shown, for ease of explanation, abnormal images are displayed to illustrate the detection results. The first column shows the verification images with anomalies, and the second column shows the popularity of the verification images. Figure 1 Verify image heat at the locations where anomalies exist. Figure 1 A clearly highlighted area will appear. Verify image heat. Figure 1 Popularity of other regions and examples Figure 1 They are basically the same. The brightness in the heatmap roughly reflects the probability of an anomaly, with each point ranging from [0,1]. The closer the value is to 1, the greater the probability of an anomaly, and the brighter the color in the heatmap; the closer the value is to 0, the lower the probability of an anomaly, and the closer the color in the heatmap is to black.
[0081] Step 3.3: Perform a similarity comparison again, comparing the popularity from steps 3.1 and 3.2. Figure 1 By comparing the results, a quantitative comparison score is obtained, and the verification image heat is obtained. Figure 2 The similarity comparison here can use the SSIM method or the MSE method, and is not limited to these.
[0082] Note the need to verify image heat. Figure 2 The test is highly dependent on sample images; the selection and quality of these images directly impact the results. Therefore, it's required that the images being compared have the same placement and shooting angle, or that the two images used for comparison have been adjusted in size, position, etc., without considering positional changes in this task. After the second similarity comparison, common non-abnormal regions in all images are suppressed, while abnormal regions become prominent.
[0083] Implementation, for example Figure 5 As shown, the SSIM method is used again for similarity comparison, and the third column is the verification image popularity. Figure 2 The edges of the product and areas with complex structures are subject to heat. Figure 1 There may be slight indications, and abnormal areas may appear in the heat index. Figure 1 Highlighting is used. However, in terms of popularity... Figure 2 In the image, product edges and areas with complex structures are almost not displayed; only abnormal areas are clearly displayed.
[0084] Step 3.4: Classify the heatmap using SVM. The images in the validation set are then processed using the heatmap obtained in Step 3.3. Figure 2 Used for training SVM. A trained SVM classifier can directly classify new test samples into two categories: present or absent.
[0085] heat Figure 2 The system can already highlight abnormal areas. We can simply set a threshold; if the popularity... Figure 2 Points exceeding a threshold are considered anomalies. However, it cannot be ruled out that a few points may exceed the threshold. For this binary classification task, higher accuracy and reliability are required; therefore, SVM is used as an auxiliary method. True anomaly regions exist in clusters and continuously, while false anomaly regions are small and discrete, resembling noise.
[0086] Step 3.5: Determine the region where the abnormality was identified in step 3.4.
[0087] Use a mask to exclude irrelevant areas such as the background, and then use the heat value obtained in step 3.4 for the foreground. Figure 2 Identify the abnormal area, set an appropriate threshold, and identify the location of the abnormality if the point exceeds the threshold.
[0088] For example, the threshold value for the cup lid portion can be set to 0.95, heat... Figure 2 The portion exceeding the threshold is marked as an abnormal region. For example... Figure 5 The image shown is the result of abnormal image processing. The first column is the original test image with anomalies, and the second column is the heat value of the test image. Figure 1 (Represented as a grayscale image), the third column shows the heat of the test image. Figure 2 The fourth column shows the finally segmented abnormal regions.
[0089] For example, the screw threshold can be set to 0.95, and the heat... Figure 2 The portion exceeding the threshold is marked as an abnormal region. For example... Figure 5 The image shown is the result of abnormal image processing. The first column is the original test image with anomalies, and the second column is the heat value of the test image. Figure 1 (Represented as a grayscale image), the third column shows the heat of the test image. Figure 2 .
[0090] After setting the threshold, the test set is collected using the method described in step 1, and the same preprocessing operation is performed. The trained network is used to obtain the prediction map and perform post-processing calculations to obtain the classification results of the two classes: with and without anomalies. For images with anomalies, an anomaly region segmentation image is obtained.
Claims
1. A machine vision-based intelligent detection method for product surface anomalies, characterized in that, Includes the following steps: Step 1: Image dataset collection and filtering to comprehensively cover the surface of the product to be tested; Step 1.1: Specify the light source and shooting angle. The dataset acquisition scene should be consistent with the actual working scene of the anomaly detection system. The captured images should have sufficient resolution, and the shooting background should remain unchanged and be pure black. Step 1.2: Divide the industrial products on the assembly line into areas. For larger areas, divide them into multiple smaller areas. Step 1.3: Preprocess the collected data and extract the target foreground and remove the background; Step 1.4: Filter the dataset, remove unclear and low-quality images, use normal images for the training dataset, and keep the abnormal images as the validation dataset or test dataset; The Mask image is determined based on the region of interest in a sample image. The corresponding region is extracted from the image captured by the camera using the given Mask image, and the background and regions of no interest are masked out. Step 2: Use a neural network to process the image, select the network structure, use a loss function combining MSE, Cosine, and SSIM, train and save the model parameters; Step 2.1: Design a convolutional neural network model. The main body of the network consists of an encoder and a decoder structure, with the encoder and decoder maintaining symmetry. Step 2.2: Construct the loss function by combining MSE, Cosine, and SSIM; Step 2.3: Set the training parameters, train the model using the images processed in Step 1, and save the model parameters; Step 3: Post-processing and testing; compare the heat map of the verification image with the heat map of the normal surface image to determine the threshold; determine whether there are any anomalies and the location of the anomalies; Step 3.1: Select a normal image as a sample, input it into the trained model, and the model outputs the predicted image; use the SSIM method to compare the similarity between the input and output images, calculate the similarity score, and obtain the sample image heatmap 1; Step 3.2: Input the verification image into the trained model for testing, and obtain the verification image heatmap 1; Step 3.3: Perform a similarity comparison on heatmap 1 from steps 3.1 and 3.2 to obtain a quantitative comparison score and obtain verification image heatmap 2; the similarity comparison here uses the SSIM method or the MSE method; Step 3.4: Use SVM to classify the heatmap 2 of the verification image, dividing the test samples into two categories: those with and without anomalies; Step 3.5: Use the Mask to exclude irrelevant areas, set a threshold, and determine the abnormal parts based on the heat map 2 of the verification image.
2. The intelligent detection method for product surface anomalies based on machine vision according to claim 1, characterized in that, In step 1, the image acquisition scene remains consistent; the original image is positioned, cropped, and its input size is unified to ensure that the subject is in the center of the image, and a solid black background is used to reduce unnecessary background; the foreground mask image is extracted, which is the same size as the original image, with the background area set to 0 and the foreground area set to 1; If there are breaks or voids inside the foreground, use the closing operation to fill them; The image is divided into small regions as needed. The training dataset uses images of normal surfaces. In the application of defect detection for production line products, the product position and camera position are relatively fixed. A sample is selected to determine the position of the product in the image and the segmentation rules of the product region. Samples in the same batch all use the same product position and segmentation rules.
3. The intelligent detection method for product surface anomalies based on machine vision according to claim 1, characterized in that, The loss function constructed in step 2, combining MSE, Cosine, and SSIM, is as follows: （1） in , , For network input, For network output, , , These are the height, width, and number of channels of the input image, respectively; to avoid overall loss If the value is negative, add Adjust the parameters. .
4. The intelligent detection method for product surface anomalies based on machine vision according to claim 1, characterized in that, In step 3, the two similarity comparisons, where the SSIM method is used for the similarity comparison, correspond to the following similarity scores: .
5. The intelligent detection method for product surface anomalies based on machine vision as described in claim 1, characterized in that, In the post-processing and testing methods in step 3, it is important to note that the verification image heatmap 2 is highly dependent on the sample image. The selection and quality of the sample image will directly affect the test results. Therefore, it is required that the verification image and the sample image have the same placement and shooting angle, or that the size and position of the verification image and the sample image have been adjusted. After the second similarity comparison, the common non-abnormal areas in all images are suppressed, and the abnormal areas are highlighted.