Optical element surface defect data set manufacturing method based on deep learning

Abstract

The invention discloses an optical element surface defect data set manufacturing method based on deep learning. The method comprises the following steps: acquiring an optical element surface defect image; inputting the obtained defect image into a pre-trained defect detection model to carry out defect automatic detection; based on the defect detection result, drawing the outer contour of each defect by using a contour retrieval method, counting the corresponding defect type and defect total number, and then generating a contour retrieval result; carrying out rapid manual correction on the obtained contour retrieval result; and generating a label of the optical element surface defect image. The defect detection model is a semantic segmentation network, can not only judge whether defects exist or not and perform defect classification, but also can realize defect positioning and extraction at a pixel level. According to the method, the probability of false detection and missing detection under a complex background when the defect pixel-level label is manufactured is reduced, and the manufacturing efficiency and the manufacturing precision of the surface defect data set of the optical element are greatly improved.

Description

Technical Field

[0001] The present invention belongs to the field of optical element defect detection and deep learning, and in particular relates to a method for preparing an optical element surface defect data set based on deep learning. Background Art

[0002] Optical components are widely used in many fields such as optical communications, laser technology, photography and videography. In recent years, with the continuous development of advanced optical manufacturing technology, optical components have been increasingly used in various high-precision optical instruments and equipment, and have become a key component. The quality of the surface of optical components is usually related to the normal operation of the entire optical system. Especially in precision optical systems, even some tiny defects on the surface of optical components can cause immeasurable consequences to the entire system. However, due to the existence of various uncertain factors, even modern precision processing technology cannot completely avoid the occurrence of surface defects of optical components, so it is particularly important to accurately detect surface defects of optical components.

[0003] Deep learning is a new branch in the field of machine learning, specifically referring to machine learning based on deep neural network models and methods. It is developed on the basis of algorithmic models such as statistical machine learning and artificial neural networks, combined with the development of contemporary big data and large computing power. It has powerful high-dimensional data processing capabilities and the ability to automatically extract features. At present, deep learning is being increasingly used in the field of target recognition and target detection. Compared with traditional manual visual methods and machine vision methods, deep learning can automatically learn a variety of data features from raw data, thus not only reducing labor costs, but also greatly improving the accuracy of data processing.

[0004] With the development of deep learning, the field of defect detection has also begun to use deep learning more and more to improve detection accuracy and efficiency. However, as a data-driven method, deep learning has always faced the problem of scarce training samples in the field of optical component surface defect detection, which restricts the further development of deep learning in optical component surface defect detection. In the production process, the yield rate of optical components is much higher than the defect rate, which makes it difficult to obtain defect samples. In addition, it is also extremely difficult to label the acquired defect samples with high precision. The labeling accuracy of the sample has a very important impact on the detection performance of deep learning. The current mainstream sample labeling method, namely the manual labeling method, mainly relies on domain experts to manually label the defect information on the acquired optical component surface defect image. Therefore, the establishment of the optical component surface defect dataset is not only time-consuming and labor-intensive, with high labor costs, but also has great subjective uncertainty.

[0005] In order to promote the automation and intelligent development of optical component surface defect detection, it is necessary to design an efficient and low-cost pixel-level defect sample annotation method to quickly generate a high-precision optical component surface defect dataset. Summary of the invention

[0006] In order to solve the above technical problems, the present invention provides a method for producing an optical element surface defect dataset based on deep learning, which realizes high-efficiency, low-cost and high-quality production of an optical element surface defect dataset.

[0007] To achieve the above object, the technical solution adopted by the present invention is:

[0008] A method for preparing a surface defect dataset of an optical component based on deep learning, the specific steps of which are:

[0009] S1: Acquire the surface defect image of the optical element;

[0010] S2: Input the acquired defect image into a pre-trained defect detection model to perform intelligent recognition of optical component surface defects; the defect detection model needs to be trained in advance, and the training process includes the following steps:

[0011] S2.1: constructing a semantic segmentation network, wherein the semantic segmentation network is used to classify, locate and extract surface defects of optical components, i.e. scratches and pits, at the pixel level;

[0012] S2.2: Pre-train the encoder of the semantic segmentation network built in step S2.1 based on the open source classification sample set ImageNet ILSVRC2012;

[0013] S2.3: Collect a small number of optical component surface defect images and perform pixel-level labeling to identify the location, type and size of the defects, and obtain training samples for training the semantic segmentation network;

[0014] S2.4: Import the training samples into the pre-trained semantic segmentation network, combine them with the manually labeled information, and use the optimizer to continuously reduce the network loss to train the semantic segmentation network to obtain a trained defect detection model.

[0015] S3: Based on the defect detection results obtained in step S2, use the contour retrieval method to draw the outer contour of each defect, count the corresponding defect type and the total number of defects, and generate a contour retrieval result; the contour retrieval method has the following specific steps:

[0016] S3.1: Slice the intelligent recognition results of optical component surface defects in the channel direction, perform binarization on different slices, and obtain the binary detection results of scratches and pits and ;

[0017] S3.2: Binary detection results of scratches Filter and draw the outer contour of the scratch , the filtering formula is as follows:

[0018] (1)

[0019] (2)

[0020] in, represents the binary detection result of scratches, and Represent the row and column coordinates of the pixel respectively, and Respectively represent the width and height of the defect image input into the pre-trained defect detection model, represents the filter coefficient, and Represented in pixels The row and column coordinates of the pixel in the current filter window centered at .

[0021] S3.3: Scratch-based external contour , from left to right and from top to bottom, find the first point in the contour with a pixel value of 1 , record the contour number , current contour value ; Then set the point The value of , the contour point set , remember is the contour point set The starting point of Starting from the left pixel, point As the center, find the point in the counterclockwise direction The first pixel value of 1 in the eight-neighborhood , place the point The value of , update the contour point set ; with a point As the center, point to the contour point set The previous point in is the direction, and the point is found in the counterclockwise direction. The first pixel value of 1 in the eight-neighborhood , place the point The value of , update the contour point set ; Repeat the above steps until the contour point set Find the starting point again ;

[0022] S3.4: From the contour point set Zhongli Starting from the right side of the farthest point, repeat step S3.3, and each time a new contour starting point is found, let the contour number , current contour value , until the outer contour of the entire scratch obtained in step S3.1 is retrieved ;

[0023] S3.5: Eliminate the contour point sets with the number of points ≤ 2 in step S3.4, and each time a contour point set is deleted, the contour number ;

[0024] S3.6: Based on the set obtained in step S3.5, obtain the final scratch profile and the total number of scratches ;

[0025] S3.7: Binary detection results for pits , according to step S3.2 to step S3.6, the final pitting profile and the total number of pitting are obtained;

[0026] S3.8: Fusing the final scratch and pit contour retrieval results obtained in steps S3.6 and S3.7 to generate the final contour retrieval result;

[0027] S4: Combined with the acquired defect image, the contour retrieval result is quickly corrected manually;

[0028] S5: Generate pixel-level labels for optical component surface defect images.

[0029] In step S2.2, in addition to using the ImageNet classification dataset, other datasets can also be used to pre-train the encoder of the semantic segmentation network. By utilizing the feature richness of the classification dataset, the pre-trained semantic segmentation network can effectively reduce the risk of overfitting introduced by small samples during the training process, and has better representation information extraction capabilities.

[0030] Among them, in step S2.4, only a small amount of optical component surface defect data is selected to train the defect detection capability of the semantic segmentation network. The purpose is to cope with the situation in actual production where the number of defective optical component products is small and it is difficult to collect and produce samples.

[0031] Among them, the defect detection model described in step S2 can accept images of any size as input, and the output result is an image with the same size as the input image, but the number of channels is equal to the total number of categories (if there are only two categories, the model output result is a binary image).

[0032] Among them, the defect detection model described in step S2 is a semantic segmentation network, which can not only determine whether a defect exists, but also classify, locate and extract the defect at the pixel level.

[0033] Among them, the contour retrieval method described in step S3 can draw the outer contour of each defect, count the corresponding defect type and the total number of defects; the area of ​​the defect can be calculated by calculating the number of pixels enclosed by the closed area formed by each set in step S3.8; the length of the defect can be calculated by calculating the contour circumference of each defect, and then the width of the defect can be obtained.

[0034] Among them, the contour retrieval result obtained in step S3 is in a .json format that can be directly processed by labelme, that is, the contour retrieval result can be quickly manually corrected through labelme.

[0035] The beneficial effects of the present invention are:

[0036] 1. The present invention only uses a small number of optical component surface defect samples when training the defect detection model, and does not need to collect a large number of defect images to train the model, thereby avoiding the contradiction between the large amount of defect data required but the small actual amount when collecting optical component surface defect data.

[0037] 2. The defect detection model described in the present invention can accept images of any size as input, and automatically classify, locate and extract defects in the image at the pixel level, which not only avoids the limitations of some deep learning models on image size, but also eliminates image preprocessing operations such as cropping and scaling, and also eliminates the need to train multiple models for different defects.

[0038] 3. The present invention requires fewer computing resources to implement defect detection. Other deep learning methods usually require the use of high-performance GPUs when performing defect detection, but the defect detection model built by the present invention is a lightweight deep learning network, that is, the model can only rely on the CPU when performing defect detection.

[0039] 4. The present invention improves the production efficiency and accuracy of defect samples through deep learning, effectively making up for the problems of low labeling efficiency, interference of labeling accuracy by subjective factors, and high labeling cost in traditional manual labeling methods. Compared with other defect dataset production methods based on machine vision, the method proposed in the present invention can not only provide pixel-level labeling of multiple defect samples and multiple defects, but also has more accurate standard results. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A flow chart of a method for preparing a surface defect dataset of an optical element based on deep learning according to the present invention;

[0041] Figure 2It is the structural diagram of the defect detection model;

[0042] Figure 3 Schematic diagram of the training process of the defect detection model;

[0043] Figure 4 Schematic diagram of the detection result of the defect detection model in the embodiment, wherein (a) is the surface defect image of the optical element, and (b) is the output of the defect detection model;

[0044] Figure 5 Schematic diagram of contour retrieval results in the embodiment, wherein (a) is the contour retrieval result, and (b) is an enlarged view of a local area of ​​the contour retrieval result;

[0045] FIG6 (a) is a diagram showing a quick manual correction to delete misidentified pits in an embodiment;

[0046] FIG6( b ) is a quick manual correction of the scratches that interrupt the manual connection detection result in the embodiment;

[0047] Figure 7 Schematic diagram of defect image labels finally generated in the embodiment. DETAILED DESCRIPTION

[0048] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. It is necessary to point out that the embodiments are only used to further describe the present invention and do not mean any limitation on the scope of protection of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0049] The present invention provides a method for preparing a surface defect dataset of an optical element based on deep learning, such as Figure 1 As shown, the specific implementation includes the following steps:

[0050] Step 1: Use a microscope lens to fix vertically above the optical component, then use a high-resolution camera to obtain surface defect imaging of the optical component and delete blurred images.

[0051] Step 2: Use the pre-trained defect detection model to perform intelligent defect recognition on the image obtained in step 1.

[0052] Preferably, step 2 takes the following steps:

[0053] Step 2.1, such as Figure 2As shown, a semantic segmentation network is built. The network is based on DeepLabV3+, and a ResNet-34 classification network that is pre-trained on the ImageNet ILSVRC2012 dataset and then removes the fully connected layer is used as the encoder of the semantic segmentation network. Usually, the low-level feature map of the network contains more position and spatial information, while the high-level features of the network contain more semantic information. Therefore, in order to further improve the contextual information extraction capability of the network, the network proposed in the present invention also adds a multi-scale feature map fusion module to the decoder on the basis of DeepLabV3+, that is, a jump connection is introduced after each block of ResNet to transfer the multi-scale feature map to the decoder for fusion. In order to reduce the computational overhead, the network proposed in the present invention uses 1×1 convolution in the jump connection to reduce the dimension of the feature map and a bilinear upsampling operation to unify the feature map resolution.

[0054] In step 2.2, a small number of defect samples are input into the pre-trained semantic segmentation network, and then the entire network is trained. The specific training settings are as follows: the network is trained using the stochastic gradient descent algorithm. The network is trained for a total of 50 epochs, with a batch size of 8. The initial learning rate of the training is 0.01. As the number of training epochs increases, the learning rate decays polynomially with a decay power of 0.9. Cross entropy is used as the loss function during training.

[0055] Step 2.3, during training, augment the image data in the training set to increase the diversity of the data. The specific augmentation transformation methods include: random contrast adjustment, random addition of Gaussian noise, random angle rotation, random cropping, random horizontal flipping, random vertical flipping, and random displacement. When performing augmentation transformation, in addition to the first two methods, it is necessary to ensure that the image and its corresponding label are transformed in the same way.

[0056] In step 2.4, the constructed semantic segmentation network is trained using the 3-fold method, and then the network with the highest segmentation accuracy on the test set is selected as the final defect detection model.

[0057] The model training process is as follows Figure 3 As shown in Figure 2, the defect detection results obtained by the model are as follows: Figure 4 As shown in (a)-(b).

[0058] Step 3: Based on the intelligent recognition results of surface defects of optical components, use the contour retrieval method to draw the external contour of each defect, count the corresponding defect types and the total number of defects, and generate the contour retrieval results.

[0059] Preferably, step 3 takes the following steps:

[0060] Step 3.1: Slice the intelligent recognition results of optical component surface defects in the channel direction, perform binarization on different slices, and obtain the binary detection results of scratches and pits. and .

[0061] In this embodiment, the defect detection result obtained in step 2 marks scratches as green and pits as red, and their RGB values ​​are (0, 255, 0) and (255, 0, 0), respectively. Therefore, the pits are the first channel slice obtained in step 3.1, and the scratches are the second slice. The binarization thresholds of these two slices are both 255.

[0062] Step 3.2, binary detection results of scratches Filter and draw the outer contour of the scratch , the filtering formula is as follows:

[0063] (1)

[0064] (2)

[0065] in, represents the binary detection result of scratches, and Represent the row and column coordinates of the pixel respectively, and Respectively represent the width and height of the defect image input into the pre-trained defect detection model, represents the filter coefficient, and Represented in pixels The row and column coordinates of the pixel in the current filter window centered at .

[0066] Step 3.3, External contour based on scratches , from left to right and from top to bottom, find the first point in the contour with a pixel value of 1 , record the contour number , current contour value ; Then set the point The value of , the contour point set , remember is the contour point set The starting point of Starting from the left pixel, point As the center, find the point in the counterclockwise direction The first pixel value of 1 in the eight-neighborhood , place the point The value of , update the contour point set ; with a point As the center, point to the contour point set The previous point in is the direction, and the point is found in the counterclockwise direction. The first pixel value of 1 in the eight-neighborhood , place the point The value of , update the contour point set ; Repeat the above steps until the contour point set Find the starting point again ;

[0067] Step 3.4, from the contour point set Zhongli Starting from the right side of the farthest point, repeat step S3.3, and each time a new contour starting point is found, let the contour number , current contour value , until the outer contour of the entire scratch obtained in step S3.1 is retrieved ;

[0068] Step 3.5, remove the contour point sets with the number of points ≤ 2 in step S3.4, and each time a contour point set is deleted, the contour number is ;

[0069] Step 3.6, based on the set obtained in step 3.5, obtain the final scratch profile and total number of scratches ;

[0070] Step 3.7, binary detection results for pitting , according to step S3.2 to step S3.6, the final pitting profile and the total number of pitting are obtained;

[0071] Step 3.8, the final scratch and pit contour retrieval results obtained in steps S3.6 and S3.7 are integrated to generate the final contour retrieval results, such as Figure 5 As shown in (a)-(b).

[0072] Step 4, as shown in FIG6(a)-FIG6(b), performs a quick manual correction on the contour retrieval result obtained in step 3.

[0073] Step 5, such as Figure 7 As shown, the final pixel-level label is generated based on the contour retrieval result obtained in step 4.

[0074] Through the above embodiments, it is shown that the method for preparing optical component surface defect dataset based on deep learning in the present invention realizes pixel-level annotation of optical component surface defect images through deep learning network. Compared with the existing method for preparing optical component surface defect dataset, the method of the present invention is not only lower in cost and higher in efficiency, but also promotes the intelligent development of optical component surface defect detection.

[0075] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for preparing a surface defect dataset of an optical element based on deep learning, characterized in that: The steps include: S1: Acquire the surface defect image of the optical element; S2: Input the acquired defect image into the pre-trained defect detection model to perform intelligent recognition of optical component surface defects; S3: Based on the intelligent recognition results of surface defects of optical components, the contour retrieval method is used to draw the external contour of each defect, count the corresponding defect types and the total number of defects, and generate contour retrieval results; S4: Combined with the acquired defect image, the contour retrieval result is quickly corrected manually; S5: Generate pixel-level labels for optical component surface defect images.

2. The method for preparing a surface defect dataset of an optical element based on deep learning according to claim 1, characterized in that: The structure and training process of the defect detection model include: S2.1: constructing a semantic segmentation network, wherein the semantic segmentation network is used to classify, locate and extract surface defects of optical components, i.e. scratches and pits, at the pixel level; S2.2: Pre-train the constructed semantic segmentation network based on the open source classification sample set; S2.3: Collect defect images on the surface of optical components and perform pixel-level defect marking to identify the location, type and size of the defects, and obtain training samples for training the semantic segmentation network; S2.4: Import the training samples into the pre-trained semantic segmentation network, combine them with the manually labeled information, and use the optimizer to continuously reduce the network loss to train the semantic segmentation network to obtain a trained defect detection model.

3. The method for preparing a surface defect dataset of an optical element based on deep learning according to claim 1, characterized in that: The contour retrieval method comprises: S3.1: Slice the intelligent recognition results of optical component surface defects in the channel direction, perform binarization on different slices, and obtain the binary detection results of scratches and pits and ; S3.2: Binary detection results of scratches Filter and draw the outer contour of the scratch , the filtering formula is as follows: (1) (2) in, represents the binary detection result of scratches, and Represent the row and column coordinates of the pixel respectively, and Respectively represent the width and height of the defect image input into the pre-trained defect detection model, represents the filter coefficient, and Represented in pixels The row and column coordinates of the pixels in the current filter window centered at ; S3.3: Scratch-based external contour , from left to right and from top to bottom, find the first point in the contour with a pixel value of 1 , record the contour number , current contour value ; Then set the point The value of , the contour point set , remember is the contour point set The starting point of Starting from the left pixel, point As the center, find the point in the counterclockwise direction The first pixel value of 1 in the eight-neighborhood , place the point The value of , update the contour point set ; with a point As the center, point to the contour point set The previous point in is the direction, and the point is found in the counterclockwise direction. The first pixel value of 1 in the eight-neighborhood , place the point The value of , update the contour point set ; Repeat the above steps until the contour point set Find the starting point again ; S3.4: From the contour point set Zhongli Starting from the right side of the farthest point, repeat step S3.3, and each time a new contour starting point is found, let the contour number , current contour value , until the outer contour of the entire scratch obtained in step S3.1 is retrieved ; S3.5: Eliminate the contour point sets with the number of points ≤ 2 in step S3.4, and each time a contour point set is deleted, the contour number ; S3.6: Based on the set obtained in step S3.5, obtain the final scratch profile and the total number of scratches ; S3.7: Binary detection results for pits , according to step S3.2 to step S3.6, the final pitting profile and the total number of pitting are obtained; S3.8: The final scratch and pit contour retrieval results obtained in steps S3.6 and S3.7 are integrated to generate the final contour retrieval results.

4. The method for preparing a data set of optical element surface defects based on deep learning according to claim 3, characterized in that: The area of ​​the defect is calculated by calculating the area of ​​the closed region formed by each set in step S3.8; the length of the defect is calculated by calculating the maximum distance between any two points in each set, and then the width of the defect is obtained.

5. The method for preparing a surface defect dataset of an optical element based on deep learning according to claim 1, characterized in that: The contour retrieval result obtained in step S3 is in a .json format that can be directly processed by labelme.

Images