Neural network model for defect detection and training method, system and device thereof

Abstract

The application discloses a neural network model for defect detection and a training method, system and device thereof, and belongs to the technical field of industrial product image defect detection. The neural network model for defect detection comprises a segmentation module and a classification module; the segmentation module is used for segmentation prediction on an original picture, and the classification module is used for defect or normal classification on the picture predicted by the segmentation module. The training of the neural network model comprises the following steps: extracting defect components from different defect pictures; randomly pasting the defect components on different normal pictures to form synthetic pictures; copying the defect pictures and normal pictures corresponding to the synthetic pictures to form training set data; and training the deep neural network model for defect detection by using the training set data. The training method can generate a large number of synthetic pictures containing defect image features by using a small amount of defect pictures, so as to effectively train the deep neural network model for defect detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of industrial product image defect detection, and in particular to a neural network model for defect detection, a training method, system and device thereof. BACKGROUND

[0002] In the production process of industrial products, product quality detection is an important procedure. The most commonly used method for previous industrial product surface quality detection is manual detection. However, manual detection is easily affected by subjective factors and working experience of the detection personnel, and the efficiency and accuracy of the detection are not very high. For example, some relatively subtle abnormalities are not easy to observe with the naked eye. In recent years, as a non-contact automatic detection technology, industrial product surface defect detection based on visual perception is gradually replacing manual defect detection in defect detection tasks such as ceramic tiles, textiles, steel plates and PCBs, and has become an indispensable part of industrial product surface defect detection due to its high precision, high speed and long-time operation in complex environments.

[0003] Although early defect detection based on traditional methods has initially realized the automation of defect detection, they often cannot adapt to different types of industrial products at the same time because of the need for manual feature design. The deep learning method automatically extracts features from data and has strong adaptability to the environment, but its data-driven property makes it highly dependent on training data, which requires a large number of industrial product sample images with surface defects and their annotations for training. If it is a segmentation task, it also needs laborious pixel-level annotation. However, in a real industrial environment, due to the low probability of product defects, it is difficult to collect defect samples of various situations, and it is not in line with the interests of industrial production to build a large data set that meets the performance needs of the model in actual production. Therefore, it is crucial to train a robust defect detection model based on deep learning with a small amount of samples. In terms of model design, many current works use generative deep learning frameworks for defect detection. This model mainly uses normal pictures to train the network, so that the network generates normal pictures and detects defects by comparing them with the pictures to be detected. Although this greatly reduces the dependence on defect pictures, it still requires a large number of normal pictures. In terms of data augmentation, current works are limited to basic deep learning data expansion measures such as image rotation, image flipping and image cropping, and lack targeted data expansion measures. SUMMARY

[0004] In view of the above problems, the present application aims to provide a neural network model for defect detection, a training method, system and device thereof, which generates a large number of synthetic images containing defect image features from a small amount of defect images to effectively train a defect detection deep neural network model.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A neural network model for defect detection is characterized by comprising a segmentation module and a classification module; the segmentation module comprises four blocks, each block comprising different groups of convolutional operation layers, and a two-dimensional max pooling operation layer at the end of each block; the output of the last block is connected to a segmentation prediction layer, which performs segmentation prediction through convolutional operations to obtain a single-channel prediction map.

[0007] The classification module includes a concatenation layer, a convolutional layer, a pooling layer, a fully connected layer, an activation function layer, and a loss function layer. The concatenation layer concatenates the output of the fourth block of the segmentation module and the output of the segmentation prediction layer along the channel dimension. After passing through the convolutional and pooling layers, the number of feature channels and resolution are reduced. Finally, the max pooling operation of the pooling layer transforms the 3D convolutional features into a vector. After passing through the fully connected layer, a scalar is output. This scalar is processed by the activation function layer and then fed into the loss function layer for calculation. During testing, a final result greater than 0.5 is considered a defective image, while a result less than 0.5 is considered a normal image.

[0008] Furthermore, each convolutional operation layer includes a two-dimensional convolution, a two-dimensional BatchNormal corresponding to the two-dimensional convolution, and an activation function ReLU. The kernel size parameter of the two-dimensional convolution is set differently for each layer.

[0009] Furthermore, for the segmentation module, its loss function is:

[0010]

[0011] In the formula, L seg For the segmentation loss function, P i and M i Here, H and W represent the segmentation prediction and corresponding label at pixel i, respectively, where H and W represent the size of the predicted image, and w i It is the weight at position i for each pixel, and its value is... When pixel i is located in the defect area, w i The value of C is a constant greater than 1, otherwise w i The value is 1.

[0012] Furthermore, for the classification module, its loss function is:

[0013] L dec =||DG|| 2 ,

[0014] In the formula, L decFor the classification loss function, D and G are the final classification result prediction and the corresponding label respectively, both of which are scalars, G is 0 indicating that the picture is normal, and G is 1 indicating that the picture is a defect picture.

[0015] Further, the training method of the neural network model for defect detection comprises the following steps:

[0016] S1: extracting a defect component from different defect pictures;

[0017] S2: randomly pasting the defect component extracted in step S1 to different normal pictures to form a synthetic picture;

[0018] S3: copying the corresponding defect picture and normal picture of the synthetic picture to form a training set data;

[0019] S4: training the deep neural network model for defect detection using the training set data.

[0020] Further, the specific operation of step S1 comprises the following steps,

[0021] Given a defect picture and its pixel-level label, first, the label patch of the defect region is cropped from the label, and then the image patch of the defect region is cropped at the same position of the defect image, and the image patch and the label patch are the required defect component.

[0022] Further, the specific operation of step S2 is: for each given normal picture, randomly select a position on it, and according to the defect pixel position identifier of the label patch in the defect component, gradually paste the defect pixels in the image patch in the defect component onto the normal picture to form a synthetic defect picture, and similarly, a pixel-level label corresponding to the synthetic defect picture can also be formed.

[0023] Further, a system for defect detection is characterized in that: the system is embedded with a neural network model for defect detection.

[0024] Further, a device for defect detection comprises at least one processor and a memory in communication with the processor, and the memory stores a neural network model for defect detection.

[0025] The beneficial effects of the present application are:

[0026] 1. The neural network model for defect detection disclosed in the application comprises a segmentation module and a classification module; the model can realize image-level judgment of defects and pixel-level segmentation of defect regions at the same time, and the segmentation module can provide defect region suggestions for the classification module, further improving the performance of the classification module. The segmentation module comprises four subblocks, each subblock comprises different groups of convolution operation layers, and a two-dimensional maximum pooling operation layer is arranged at the end of each subblock; the output of the last subblock is connected with a segmentation prediction layer, the segmentation prediction layer performs segmentation prediction through convolution operation to obtain a single-channel prediction map; the classification module comprises a series connection layer, a convolution layer, a pooling layer, a full connection layer, an activation function layer and a loss function layer, the series connection layer connects the output of the fourth subblock of the segmentation module and the output of the segmentation prediction layer in the channel dimension, the feature channel number and the resolution are reduced through the convolution layer and the pooling layer, finally, the 3D convolution feature is changed into a vector through the maximum pooling operation of the pooling layer, a scalar is output through the full connection layer, the scalar is processed through the activation function layer and then sent to the loss function layer for calculation, when testing, the final result greater than 0.5 is regarded as that the input picture is a defect picture, otherwise, it is regarded as a normal picture. The neural network model can reduce the influence of too large positive and negative sample ratio through pixel-level weighting of the loss function of segmentation.

[0027] 2. The training method of the neural network model for defect detection provided in the application can generate a large amount of synthetic data to train the neural network model for defect detection only by using a small amount of original defect pictures and normal pictures, so as to perform defect detection, and the performance obtained is close to that of the network trained by using all original defect pictures and normal pictures, so that the problem that a large number of defect samples cannot be provided for training in industrial production can be greatly alleviated. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 FIG. 1 is a structural schematic diagram of the neural network model for defect detection in the first embodiment of the application.

[0029] Figure 2 FIG. 2 is a flowchart of the implementation of the synthetic picture in the second embodiment of the application.

[0030] Figure 3 FIG. 3 is a schematic diagram of the generation of a single synthetic picture in the second embodiment of the application. DETAILED DESCRIPTION

[0031] In order for those skilled in the art to better understand the technical solutions of the application, the technical solutions of the application are further described below in combination with the drawings and embodiments.

[0032] Embodiment I:

[0033] Embodiment I provides a neural network model for defect detection, as shown in FIG. 1.Figure 1 As shown, it comprises a segmentation module and a classification module; the segmentation module comprises four sub-blocks, each of which comprises a different group of convolution operation layers, each of which comprises a two-dimensional convolution, a two-dimensional Batch Normal (BN) corresponding to the two-dimensional convolution, and an activation function ReLu, wherein the parameters of the two-dimensional convolution have different settings in each sub-block, as shown in Table 1 below, and other parameters such as stride and padding are all set to 1 and 2 by default, respectively, and are not presented in the table. In order to reduce the amount of calculation, a two-dimensional maximum pooling operation layer is provided at the end of each sub-block to reduce the resolution of the features by half through two-dimensional maximum pooling operation (Max Pooling). The output of the last sub-block is connected to a segmentation prediction layer, which performs segmentation prediction through simple convolution operation to obtain a single-channel prediction map.

[0034] The classification module comprises a concatenation layer, a convolution layer, a pooling layer, a fully connected layer, an activation function layer and a loss function layer, the concatenation layer concatenates the output of the fourth sub-block of the segmentation module and the output of the segmentation prediction layer in the channel dimension, the feature channel number and the resolution are reduced through the convolution layer and the pooling layer, finally the 3-dimensional convolution features are changed into a vector through the maximum pooling operation of the pooling layer, a scalar is output through the fully connected layer, the scalar is processed through the activation function layer and then sent to the loss function layer for calculation, and when testing, the final result greater than 0.5 is regarded as a defective picture, and vice versa.

[0035] Table 1 detailed configuration of neural network model for defect detection

[0036]

[0037] In the present application, the MSE function is used as the loss function for segmentation and classification. Specifically, for the segmentation module, the loss function is

[0038]

[0039] In the formula, L seg is the segmentation loss function, P i and M i are the segmentation prediction and the corresponding label of pixel i position respectively, H and W represent the size of the prediction image, w i is the weight of each pixel i position, in particular, in order to balance the positive and negative example sample imbalance problem during segmentation training, the present embodiment performs a weighting operation on the segmentation loss function, that is, w i in the formula, which takes a value of When the pixel i position belongs to the defect area, w i takes a constant C greater than 1, so as to increase the influence of the defect area supervision; otherwise, w i takes a value of 1.

[0040] For the classification module, its loss function is

[0041] L dec =||D-G|| 2 ,

[0042] wherein, L dec is the classification loss function, D and G are the final classification result prediction and the corresponding label respectively, both of which are scalars, and G is 0 indicating that the picture is normal, and G is 1 indicating that the picture is a defect picture.

[0043] Embodiment two:

[0044] Embodiment two provides a training method of a neural network model for defect detection, the neural network model for defect detection adopts the network model structure in embodiment one, a large number of synthetic pictures containing defect images are generated by using a small number of picture samples to train the deep neural network, and the specific implementation process of the synthetic pictures is shown in FIG. 2, which includes the following steps: Figure 2

[0045] S1: extracting defect components from different defect pictures;

[0046] Specifically, given a defect picture and its pixel-level labeled label, first, the label patch of the defect region is cropped from the label, and then the image patch of the defect region is cropped at the same position of the defect image. The image patch and the label patch are the required defect components.

[0047] S2: randomly paste the defect components extracted in step S1 to different normal pictures to form synthetic pictures;

[0048] Specifically, for each given normal picture, a position is randomly selected thereon, and according to the defect pixel position identifier of the label patch in the defect component, the defect pixels in the image patch in the defect component are gradually pasted onto the normal picture to form a synthetic defect picture. Similarly, a pixel-level label corresponding to the synthetic defect picture can also be formed.

[0049] S3: copying the defect picture and the normal picture corresponding to the synthetic picture to form a training set data;

[0050] S4: training the deep neural network model for defect detection by using the training set data.

[0051] Simulation experiment:

[0052] ​The original database used in the simulation experiment is the Kolektor SDD dataset, which is an electronic commutator defect detection database provided by the Kolektor Group. The surface defects are fine cracks on the surface of plastic packaging. The database contains 50 real objects, each with 8 faces, and images are collected for each face. The official website provides a total of 399 pictures, of which 52 have visual defects and 347 are normal. The picture size is 500 in width and 1240 to 1270 in height, and the annotation form is pixel-level mask annotation. The database is divided into a training set of 259 pictures and a test set of 140 pictures according to the conventional setting.

[0053] Six defect pictures and 15 normal pictures are selected from the training set to synthesize defect pictures. First, the defect region is cropped from the defect picture as a defect component using the annotation of the defect picture. Then, for each defect component, it is sequentially pasted onto 15 normal pictures. For each normal picture, 4 positions can be randomly selected for 4 pastes. The generation process of each synthesized defect picture is shown in FIG. 2, and the total number of synthesized pictures is 360. In order to ensure the balance of positive and negative sample quantities, the original normal picture and the original defect picture are copied once for each synthesized defect picture. Finally, the total number of training pictures is 1080. Figure 3

[0054] The neural network model in Example 1 is trained using the training pictures. The hardware configuration is based on Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz CPU, NVIDIA GTX 1080Ti 12GB GPU, Ubuntu 16.04.6 operating system, Python programming language, and deep neural network built under the Pytorch framework to complete the training and testing of defect detection segmentation and classification.

[0055] The Batch Size in the training process is set to 2, the Adam optimizer is used, the initial learning rate is 10 -3 , β 1 and β 2 are 0.5 and 0.99 respectively. The segmentation network is trained for 20 epochs first, and then the classification network is trained for 12 epochs to obtain the converged result.

[0056] ​Finally, in order to verify the effectiveness of the sample synthesis and model training method in the embodiment, the simulation experiment makes the following contrast experiment as shown in Table 2. First, the network is trained directly using the complete training set of the KolektorSDD database (the training set and test set of KolektorSDD are divided into 259 and 140 respectively), and finally the classification result with a precision of 0.99 is obtained. Only 20 pictures in the training set are used for training, and when the data is expanded in the conventional way such as flipping, rotating, and cropping, the performance of the final model decreases by 12%. However, using the model training method provided by the present application, the final performance only decreases by 2%, and the amount of data used is reduced from 259 in the complete training set to 20, with a reduction of about 92% in training data.

[0057] Table 2 Comparison of results of different training data

[0058]

[0059] Embodiment three:

[0060] Embodiment three provides a system for defect detection, wherein the neural network model for defect detection described in embodiment one is embedded.

[0061] Embodiment four:

[0062] Embodiment four provides a device for defect detection, comprising at least one processor and a memory connected in communication with the processor, wherein the memory stores the neural network model for defect detection described in embodiment one.

[0063] The basic principles, main features and advantages of the present application are shown and described above. It should be understood by those skilled in the art that the present application is not limited by the above embodiments, and the above embodiments and descriptions in the specification are only to illustrate the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. A system for defect detection, wherein a neural network model configured for defect detection is embedded in the system, characterized in that: The neural network model comprises a segmentation module and a classification module; the segmentation module comprises four sub-blocks, each of which comprises a different group of convolution operation layers, and a two-dimensional maximum pooling operation layer is arranged at the end of each sub-block; the output of the last sub-block is connected with a segmentation prediction layer, which performs segmentation prediction through convolution operation to obtain a single-channel prediction map; The classification module comprises a concatenation layer, a convolution layer, a pooling layer, a full connection layer, an activation function layer and a loss function layer, the concatenation layer concatenates the output of the fourth sub-block of the segmentation module and the output of the segmentation prediction layer in the channel dimension, the feature channel number and the resolution are reduced through the convolution layer and the pooling layer, finally the three-dimensional convolution features are changed into a vector through the maximum pooling operation of the pooling layer, a scalar is output through the full connection layer, the scalar is processed through the activation function layer and then sent to the loss function layer for calculation, when testing, the final result greater than 0.5 is regarded as that the input picture is a defective picture, otherwise, it is regarded as a normal picture; Each convolution operation layer comprises a two-dimensional convolution, a two-dimensional BatchNormal corresponding to the two-dimensional convolution and an activation function ReLu, and the corresponding two-dimensional convolution parameter kernel size of each layer is set to be different; For the segmentation module, the loss function is ， wherein, is a segmentation loss function, and are the segmentation prediction and the corresponding label for each pixel i location, H and W represent the size of the predicted image, is the weight for each pixel i location, which takes the value when the pixel i location belongs to the defect region, takes a constant C greater than 1, otherwise takes the value 1; For the classification module, the loss function is ， In the formula, D and G are the final classification result prediction and the corresponding label, respectively, both of which are scalars, and G is 0 indicates that the picture is normal, and G is 1 indicates that the picture is a defective picture.

2. The method of claim 1, wherein the method further comprises: The method comprises the following steps: S1: extracting a defect component from different defect pictures; S2: randomly pasting the defect component extracted in step S1 to different normal pictures to form a synthetic picture; S3: copying the defect picture and the normal picture corresponding to the synthetic picture, and the synthetic picture and the copied picture together form a training set data; S4: training the deep neural network model for defect detection by using the training set data.

3. The training method of claim 2, wherein, The specific operation of step S1 comprises the following steps, Given a defect picture and its pixel-level label, first, the label patch of the defect region is cropped from the label, and then the image patch of the defect region is cropped at the same position of the defect image, and the image patch and the label patch are the required defect component.

4. The training method of claim 3, wherein, The specific operation of step S2 is: for each given normal picture, randomly select a position on it, and according to the defect pixel position identifier of the label patch in the defect component, gradually paste the defect pixels in the image patch in the defect component onto the normal picture to form a synthetic defect picture, and similarly, a pixel-level label corresponding to the synthetic defect picture can also be formed.

5. An apparatus for defect detection, characterized by: The system comprises at least one processor and a memory connected with the processor in communication, and the memory stores the neural network model in the system for defect detection of claim 1.

Images