Complex electronic product surface paint defect detection method, storage medium and computing equipment

Abstract

The invention discloses a complex electronic product surface paint defect detection method, which comprises the steps of obtaining an original sample set for training, carrying out image preprocessing on the original sample set, distinguishing normal samples and abnormal samples of all cut images, carrying out unsupervised anomaly detection network training by utilizing a small image sample set obtained by preprocessing, obtaining an image anomaly detection sub-model; performing classification and labeling of normal images and various defect images on a small image sample set obtained after preprocessing, performing semantic segmentation and labeling of defect positions of the defect images, and then putting the labeled sample set into an image classification and segmentation network with supervised learning for training to obtain an image defect detection sub-model; and taking the trained image anomaly detection sub-model and image defect detection sub-model as a complex electronic product surface paint defect detection model. The method not only realizes a high defect detection rate, but also can well detect unknown defects.

Description

Technical Field

[0001] The present invention belongs to the field of surface defect detection, and in particular relates to a method, storage medium and computing device for detecting paint defects on the surface of complex electronic products. Background Art

[0002] Surface painting is a key process in the inspection of complex electronic products. Currently, the inspection of surface paint defects of complex electronic products is mainly based on manual visual inspection. However, manual inspection consumes a lot of manpower, material resources and time. With the development of deep learning technology and machine vision technology, defect detection has entered the era of automation. For example, Patent 202311724313.7 discloses a paint surface defect detection model training method. This patent combines unsupervised learning with supervised learning to realize paint surface defect detection, which not only avoids the omission of paint surface defects, but also improves the accuracy of paint surface defect detection. However, the above patent can only detect whether there are defects in the image and identify the type of defects, but cannot accurately mark and display the location of the defects in the image. In actual applications, it is often necessary to know the specific location of the defect so that the staff can re-check and take treatment measures. In response to the above-mentioned shortcomings, the technical problem to be solved by the present invention is to realize a method for detecting paint defects on the surface of complex electronic products, which can judge whether there are defects in the paint image of the surface of complex electronic products, and further classify the types of defects and accurately locate the locations of the defects. The present invention not only improves the classification and positioning accuracy of the model for various defects, but also achieves a higher defect detection rate and a lower false detection rate. Summary of the Invention

[0003] The main purpose of the present invention is to provide a method for detecting paint defects on the surface of complex electronic products. This method combines unsupervised learning with supervised learning to realize the detection of paint defects on the surface of complex electronic products. It not only achieves a high defect detection rate but also can detect unknown defects well. It has wide application value for surface paint defect detection.

[0004] To this end, the present invention provides a method for detecting paint defects on the surface of complex electronic products, comprising: Obtaining an original sample set for training, wherein the original sample set includes a large number of original sample images without paint defects and a small number of original sample images with paint defects; Perform image preprocessing on the original sample set First, semantic segmentation and annotation are performed on the original sample set to distinguish normal samples from defective samples and generate label files corresponding to the original sample images. Then, all original samples are amplified, image size is cropped, and image background is removed to obtain a small sample set. The small sample set is matched with the label file that has also been cropped to determine whether there is a masked area. If the masked area exists in the cropped image, it is an anomaly sample, thus completing the distinction between normal and abnormal samples of all cropped images. The small sample set is then divided into a training set, a validation set, and a test set according to a certain ratio. These sets are then trained on an unsupervised anomaly detection network to obtain an image anomaly detection sub-model. Defective samples are not included in the model training. The small sample set obtained after preprocessing is classified and labeled into normal images and images with various defects, and the defect locations of the defective images are annotated through semantic segmentation. The labeled sample set is then trained in a supervised learning image classification and segmentation network to obtain an image defect detection sub-model. The trained image anomaly detection sub-model and image defect detection sub-model are used as the paint defect detection model for the surface of complex electronic products.

[0005] Specifically, an image cropping test is first performed. Each time the image cropping test is performed, the rectangular sliding window size and / or the moving step size are changed, and the step value is in units of L pixels. Then, the obtained test data is fitted to obtain the changing relationship between the detection accuracy and the rectangular sliding window size and the moving step size, so as to determine the optimal rectangular sliding window size and moving step size. Then, the optimal rectangular sliding window is made to slide and scan from the upper left corner of the sample image to the lower right corner of the sample image with the optimal moving step size as the unit, so as to obtain a series of detection sub-images of consistent size.

[0006] Specifically, through multiple image cropping tests, the curve variation law between the defect detection accuracy and the rectangular sliding window size and the moving step size is obtained, so as to determine the optimal cropping size and moving step size.

[0007] Specifically, the sample set is expanded by performing up-down mirror flipping and left-right mirror flipping on the sample image.

[0008] Specifically, the specific process of removing the image background is as follows: first, using the image segmentation method, according to the gray value difference between the detection object and the background, a suitable segmentation threshold is selected to determine whether each pixel of the image belongs to the detection object or the background. Finally, the detection object is obtained while eliminating the interference caused by the background. Then, the residual background is further removed through image morphological processing operators such as dilation and erosion.

[0009] Specifically, during model training in step three, after setting the parameters of the unsupervised anomaly detection network, the sample sets with over-detection and false positives in each training result are further manually screened. The manually screened training sets are then put into training, and the model is continuously iterated and optimized. The validation set and test set are used to judge the model detection performance until the training results meet the actual requirements.

[0010] Specifically, when classifying images, the sample set is divided into training set, validation set, and test set in a certain ratio, the network model training parameters are set, and the image is input into the pre-trained classification model in the supervised image classification network to obtain the classification prediction value. The predicted value is compared with the true value, and the loss function is calculated. Samples with over-detection errors are returned to training, and the sample set and training model parameters are updated. The training is continuously iterated and optimized until the loss function meets the convergence conditions. The trained image classification model can be used to distinguish the defect types of abnormal images obtained by the image defect detection sub-model.

[0011] Specifically, during image segmentation, the network model training parameters are set by inputting the sample set file and the label file generated by marking the defect location, so that the model gradually learns the relationship between the input image and the corresponding label. The model detection performance is judged based on the validation set and test set, and the model is continuously iterated and optimized until the defect segmentation result meets the actual requirements.

[0012] Specifically, users set parameters for the industrial camera through the designed main interface and use the industrial camera to collect images in real time as a training sample library.

[0013] First, the trained image anomaly detection sub-model is used to realize the binary classification of the presence or absence of defects in the detection image. Then, the image defect detection sub-model is used to distinguish the defect types and accurately locate the defect positions of the defect anomaly images obtained by the image anomaly detection sub-model.

[0014] A storage medium stores a computer program, which, when executed by a processor, causes the processor to execute the method for detecting paint defects on the surface of complex electronic products.

[0015] A computing device includes a processor and a memory for storing a program executable by the processor. When the processor executes the program stored in the memory, the method for detecting paint defects on the surface of complex electronic products is implemented.

[0016] Compared with the prior art, the present invention has the following beneficial effects: 1. This method combines unsupervised learning with supervised learning to implement defect detection in a two-stage network. In the first stage, the paint image of the complex electronic product surface is preprocessed and then trained on an anomaly detection network for deep learning. The trained model can finally achieve binary classification of the detected image into the presence or absence of defects. In the second stage, in order to further obtain detailed information about the defects, a supervised learning model is trained to classify the defects and accurately locate their locations.

[0017] 2. This invention designs an adjustable sliding window for image cropping, dividing the inspection image into multiple sub-images. This not only unifies the image input size but also reduces the computational effort, speeding up model training. Furthermore, for defects as small as a few dozen pixels, it can amplify the defect feature ratio, facilitating model learning and training of defect features. Furthermore, a curve is constructed that shows the relationship between defect detection accuracy, the size of the rectangular sliding window, and the step size of the movement, thereby obtaining the optimal cropping size and step size, and rapidly improving model detection accuracy.

[0018] 3. Before model training, the present invention uses the image segmentation method to remove the image background of the sample set, which not only reduces the interference of background noise on the defect detection algorithm, but also makes the contrast between the defect and the surrounding more obvious, which is beneficial to the detection of the defect algorithm and greatly reduces the computational complexity of the model.

[0019] 4. The present invention amplifies the original samples by flipping the upper and lower mirrors and flipping the left and right mirrors. Since the number of samples of paint defects on the surface of complex electronic products is small, in order to enhance the robustness of the model, the diversity of the data is increased and the sensitivity of the model to the defect position and angle is reduced.

[0020] 5. The present invention matches the preprocessed sample set with the label file that has also been cropped to determine whether there is a mask area. If a mask area exists in the small image, it is an abnormal sample. Using the label file to distinguish the small images as abnormal can avoid people misjudging the small image flaws as defects and causing detection errors, thereby improving the model detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 This is a flow chart of a method for detecting paint defects on the surface of complex electronic products according to the present invention; Figure 2 Schematic diagram of the steps for extracting sub-images from a paint image of a complex electronic product surface according to the present invention. Figure (a) is the first sub-image, Figure (b) is the second sub-image, and Figure (c) is the i-th sub-image. Figure 3 The figure is a schematic diagram of the structure of the paint defect detection system for complex electronic products according to the present invention. DETAILED DESCRIPTION

[0023] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0024] See also Figure 1 A method for detecting paint defects on the surface of complex electronic products comprises the following steps: Obtain the original sample set for training. The original sample set includes a large number of original sample images without paint defects and a small number of original sample images with paint defects. For the convenience of description, the original sample images are named large-image samples and the original images are named large-size images, referred to as large images.

[0025] Perform image preprocessing on the sample library. First, semantic segmentation and annotation are performed on large image samples to distinguish normal samples from defective samples and generate label files corresponding to large-scale images. Due to the small number of samples with paint defects on the surface of complex electronic products, in order to enhance the robustness of the model, improve the model training effect, and reduce the interference of background noise on the defect detection algorithm, it is necessary to expand the sample set, crop the image size, and remove the image background. The preprocessed sample set is matched to the cropped label file to determine whether a masked region exists. If a masked region exists in a small image, it is considered an anomaly. This distinguishes normal from abnormal samples for all small images. Using the label file to distinguish small images is because small images tend to magnify defects, which can lead to false positives. Furthermore, images with a small target area are removed, as this can affect the model's detection accuracy. These small image samples are then divided into training, validation, and test sets according to a specific ratio and fed into an unsupervised anomaly detection network. Defective samples are not included in the model training. The parameters of the anomaly detection network are set, and samples with over-detection errors are further manually screened to improve the classification quality. The manually screened training set is then used for training again, and the model is iteratively optimized. The model's detection performance is evaluated using the validation and test sets until the training results meet practical requirements. This results in an unsupervised learning sub-model for anomaly detection in complex electronic product surface paint images. The anomaly detection sub-model can be used to perform binary classification of whether an image has defects and roughly mark the defect locations; Setting the anomaly detection network parameters mainly includes minimum training error, complexity, regularization noise, proportion, normalization factor, etc. 1) The minimum training error parameter determines when the model reaches the training completion requirement during training.

[0026] 2) The complexity parameter controls the model's ability to handle more complex applications, which can include changes in color, brightness, size, shape, etc.

[0027] 3) Regularizing the noise parameters can improve the problem that training is ineffective due to small changes in image data.

[0028] 4) The ratio parameter is determined from the pixel ratio in the preprocessed image.

[0029] In order to further obtain specific defect information, the preprocessed sample set is classified and labeled into normal images and images with various defects, and the defect locations of the defective images are semantically segmented and labeled. The labeled sample set is then put into the supervised learning image classification and image segmentation network model for training: 1) Image classification: Divide the sample set into training, validation, and test sets according to a certain ratio, and set the network model training parameters. Input the image into the pre-trained classification model included in the supervised image classification network to obtain the classification prediction value. The prediction value is compared with the true value, and the loss function is calculated. Samples with over-detection errors are returned to training, and the sample set and training model parameters are updated. Continuous iterative optimization training is carried out until the loss function meets the convergence conditions. The trained image classification model can be used to distinguish the defect types of abnormal images obtained by the anomaly detection sub-model; 2) Image Segmentation: Image segmentation is a pixel-level detection process. By inputting a sample set file and a label file generated by annotating defect locations, network model training parameters are set, allowing the model to gradually learn the relationship between the input image and the corresponding label. The model's detection performance is judged based on a validation test set, and the model is continuously iterated and optimized until the defect segmentation results meet practical requirements. The image segmentation network can obtain the precise location and geometric information of abnormal image defects.

[0030] The method of the present invention combines unsupervised and supervised learning to implement defect detection in a two-stage network. In the first stage, after preprocessing the paint image of the complex electronic product surface, the image is trained with an anomaly detection network for deep learning. The trained model can achieve a binary classification of the detected image, i.e., the presence or absence of defects. In the second stage, to further obtain detailed information about the defects, a supervised learning model is trained to classify the defects and accurately locate their locations. The proposed method of combining unsupervised and supervised learning to detect paint defects on the surface of complex electronic products not only achieves a high defect detection rate but also effectively detects unknown defects, thus having broad application value for surface paint defect detection.

[0031] Specifically, the process of sample set expansion is as follows: when the number of sample sets is too small, training is likely to lead to overfitting of the training results. Therefore, the sample set can be expanded through data augmentation technology, mainly through up and down mirror flipping and left and right mirror flipping. Sample set expansion can increase data diversity and reduce the model's sensitivity to defect location and angle.

[0032] In some embodiments, the specific process of image size cropping is as follows: since the defect area is small, large-size image input training is not convenient for the model to learn defect features. Size cropping of large images can amplify the defect feature ratio and ultimately improve the defect detection rate. Figure 2As shown in Figure 1, size cropping primarily involves dividing the inspection image into multiple inspection sub-images, or cropping them into smaller images (referred to as thumbnails). By setting the rectangular sliding window size (W*H), the image can be cropped to the corresponding window size, reducing computational complexity and unifying the input image size. Furthermore, due to the uncertainty of defect size and location, a step size of L pixels is required. This causes the rectangular sliding window to slide from the upper left corner of the image to the lower right corner, generating a series of inspection sub-images. Different rectangular sliding window sizes and step sizes can affect defect detection accuracy. A window size that is too large can lead to high memory usage and poor pixel recognition of millimeter-level defects. A step size that is too small can cause large defects to be cut into multiple smaller ones, making them undetectable. Therefore, multiple experiments are needed to construct a curve showing the relationship between defect detection accuracy, rectangular sliding window size, and step size, in order to determine the optimal cropping size and step size.

[0033] In this embodiment, when cropping an image, an image cropping test is first performed. During each image cropping test, the rectangular sliding window size and / or the moving step size are changed, with the step value being in units of L pixels. The obtained test data is then fitted to obtain the relationship between the detection accuracy and the rectangular sliding window size and the moving step size, thereby determining the optimal rectangular sliding window size and moving step size. The optimal rectangular sliding window is then slid and scanned from the upper left corner of the sample image to the lower right corner of the sample image with the optimal moving step size as a unit, thereby obtaining a series of detection sub-images of consistent size.

[0034] This embodiment uses an adjustable sliding window to crop the image size, dividing the inspection image into multiple sub-images. This not only unifies the image input size but also reduces the computational effort, speeding up model training. Furthermore, for defects measuring only a few dozen pixels, the defect feature ratio can be magnified, facilitating model learning and training. Furthermore, a curve is constructed that shows the relationship between defect detection accuracy, the size of the rectangular sliding window, and the step size. This allows for optimal cropping and step sizes, resulting in a rapid improvement in model detection accuracy.

[0035] It can be understood that in practical applications, the process of removing the image background is: using the image segmentation method, according to the grayscale value difference between the detection object and the background, a suitable segmentation threshold is selected to judge whether each pixel of the image belongs to the detection object or the background. Finally, the detection object is obtained while eliminating the interference caused by the background. However, image segmentation to remove the background will have the problem of incomplete removal, such as burrs, black holes, etc. Therefore, it is necessary to further use image morphological processing operators such as dilation and corrosion to further clean the background.

[0036] Before model training, this embodiment uses image segmentation methods to remove the image background of the sample set, which not only reduces the interference of background noise on the defect detection algorithm, but also makes the contrast between the defect and the surrounding more obvious, which is beneficial to the detection of the defect algorithm and greatly reduces the computational complexity of the model.

[0037] The present invention also discloses a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the method for detecting paint defects on the surface of complex electronic products.

[0038] The present invention also discloses a computing device, comprising a processor and a memory for storing a program executable by the processor. When the processor executes the program stored in the memory, the method for detecting paint defects on the surface of complex electronic products is implemented.

[0039] In this embodiment, see Figure 3 The present invention also provides a complex electronic product surface paint defect detection system, which includes a complex electronic product surface paint image preprocessing module, a complex electronic product surface paint image anomaly detection module based on unsupervised learning, a complex electronic product surface paint defect information acquisition module based on supervised learning, and a complex electronic product surface paint defect detection main interface design module; Users can set parameters for industrial cameras through the designed main interface. Camera parameters include frame rate, exposure time, trigger mode, etc. Images are collected in real time as the original sample library for training. The collected images include a large number of normal images and a small number of defective samples. The function of the preprocessing module for complex electronic product surface paint images is to use preprocessing methods to amplify the image set, crop the image size, and remove the image background to reduce the impact of background noise, unify the input images of consistent size, and reduce the model calculation amount; The anomaly detection module for paint images on the surface of complex electronic products based on unsupervised learning is based on a convolutional autoencoder network, which includes an encoder and a decoder. It encodes the input data into a low-dimensional representation, and then decodes the low-dimensional representation to restore it to the original input data. The anomaly detection module only needs to input normal sample images. By training the autoencoder, it can be enabled to effectively reconstruct normal samples, that is, the reconstruction error is small, and the maximum reconstruction error of all normal samples is taken as the anomaly classification threshold. Then, when a new sample is given, by calculating its reconstruction error, if it is greater than the anomaly classification threshold, the sample can be judged as an abnormal sample. In addition, the anomaly detection model can only mark the approximate location of the defect and cannot locate it more accurately; The module for acquiring paint defect information on complex electronic product surfaces based on supervised learning is based on image classification and image segmentation networks. Based on the anomaly detection results, it further subdivides the detected abnormal images into defect categories to determine the type of defect. The image segmentation network model is then used to obtain the precise location and geometric information of the defect. The design module for the main interface of complex electronic product surface paint defect detection controls the camera's start and stop, sets camera parameters, displays captured images in real time, visualizes test results, and saves them. This module imports trained anomaly detection networks, along with image classification and segmentation network models, inputs the image to be tested, and uses the network model to obtain test results, which are then visualized on the main interface.

[0040] The above embodiments are merely examples to clearly illustrate the present invention and are not intended to limit its implementation. Those skilled in the art will readily appreciate that other variations or modifications based on the above descriptions are possible. It is not necessary and impossible to provide an exhaustive list of all embodiments. Obvious variations or modifications arising therefrom remain within the scope of protection of the present invention.

Claims

1. A method for detecting paint defects on the surface of complex electronic products, characterized in that: include: Obtaining an original sample set for training, wherein the original sample set includes a large number of original sample images without paint defects and a small number of original sample images with paint defects; Perform image preprocessing on the original sample set First, semantic segmentation and annotation are performed on the original sample set to distinguish normal samples from defective samples and generate label files corresponding to the original sample images. Then, all the original sample sets are amplified, image size is cropped, and image background is removed to obtain a small sample set. The small sample set is matched with the label file that has also been cropped to determine whether there is a masked area. If the masked area exists in the cropped image, it is an anomaly sample, thus completing the distinction between normal and abnormal samples of all cropped images. The small sample set is then divided into a training set, a validation set, and a test set according to a certain ratio. These sets are then trained on an unsupervised anomaly detection network to obtain an image anomaly detection sub-model. Defective samples are not included in the model training. The small sample set obtained after preprocessing is classified and labeled into normal images and images with various defects, and the defect locations of the defective images are annotated through semantic segmentation. The labeled sample set is then trained in a supervised learning image classification and segmentation network to obtain an image defect detection sub-model. The trained image anomaly detection sub-model and image defect detection sub-model are used as the paint defect detection model for the surface of complex electronic products.

2. The method for detecting paint defects on the surface of complex electronic products according to claim 1, characterized in that: The image size cropping process is as follows: first, an image cropping test is performed. Each time an image cropping test is performed, the rectangular sliding window size and / or the moving step size are changed, and the step value is in units of L pixels; then, the obtained test data is fitted to obtain the relationship between the detection accuracy and the rectangular sliding window size and the moving step size, so as to determine the optimal rectangular sliding window size and moving step size; then, the optimal rectangular sliding window is made to slide and scan from the upper left corner of the sample image to the lower right corner of the sample image with the optimal moving step size as the unit, so as to obtain a series of detection sub-images of consistent size.

3. The method for detecting paint defects on the surface of complex electronic products according to claim 1, characterized in that: The sample set is expanded by flipping the sample images up and down and left and right.

4. The method for detecting paint defects on the surface of complex electronic products according to claim 1, characterized in that: The specific process of removing the image background is as follows: first, using the image segmentation method, according to the gray value difference between the detection object and the background, a suitable segmentation threshold is selected to determine whether each pixel of the image belongs to the detection object or the background. Finally, the detection object is obtained while eliminating the interference caused by the background. Then, the residual background is further removed through image morphological processing operators such as dilation and erosion.

5. The method for detecting paint defects on the surface of complex electronic products according to claim 1, characterized in that: During the model training in step three, after setting the parameters of the unsupervised anomaly detection network, the sample sets with over-detection and false positives in each training result are further manually screened. The manually screened training sets are then put into training, and the model is continuously iterated and optimized. The validation set and test set are used to judge the model detection performance until the training results meet the actual requirements.

6. The method for detecting paint defects on the surface of complex electronic products according to claim 1, characterized in that: When classifying images, the sample set is divided into training set, validation set, and test set in a certain ratio. The network model training parameters are set, and the image is input into the pre-trained classification model in the supervised image classification network. The classification prediction value will be obtained, and the predicted value will be compared with the true value. The loss function is calculated. Samples with over-detection errors must be returned to training, and the sample set and training model parameters are updated. The training is continuously iterated and optimized until the loss function meets the convergence conditions. The trained image classification model can be used to distinguish the types of defects in abnormal images obtained by the image defect detection sub-model.

7. The method for detecting paint defects on the surface of complex electronic products according to claim 1, characterized in that: During image segmentation, the network model training parameters are set by inputting the sample set file and the label file generated by marking the defect locations, so that the model gradually learns the relationship between the input image and the corresponding label. The model detection performance is judged based on the validation set and test set, and the model is continuously iterated and optimized until the defect segmentation results meet the actual requirements.

8. The method for detecting paint defects on the surface of complex electronic products according to claim 1, characterized in that: Users set the parameters of the industrial camera through the designed main interface and use the industrial camera to collect images in real time as a training sample library.

9. A storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, enables the processor to execute the method for detecting paint defects on the surface of complex electronic products according to any one of claims 1 to 8.

10. A computing device comprising a processor and a memory for storing a program executable by the processor, characterized in that When the processor executes the program stored in the memory, it implements the method for detecting paint defects on the surface of complex electronic products as described in any one of claims 1 to 8.

Images