An optical glass AOI detection method based on deep learning

Abstract

The application discloses and provides an optical glass AOI detection method based on deep learning. The application innovatively introduces a two-fold deep learning target detection mechanism for image splicing, can accurately identify the product area and key structural features in each local image, and calculates the splicing offset between images by matching with a standard template. In the case that the product size has no large tolerance, the splicing accuracy can reach 100%; in the actual production environment containing tolerance, the overall splicing accuracy reaches 99.8%. For a small number of images with structural deviation, compensation can be carried out by setting a threshold or using a splicing strategy of key structure normalized position, and finally a high splicing accuracy can still be realized, meeting the strict requirements of the production line on splicing quality and stability. The application is applied to the technical field of optical glass AOI detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of optical glass AOI detection, and in particular to an optical glass AOI detection method based on deep learning. BACKGROUND

[0002] Optical glass AOI detection is a process of detecting surface and internal defects of optical glass by using automatic optical inspection technology (AOI). In the field of optical glass manufacturing, the automatic detection of surface defects (such as scratches, bubbles, stains, delamination, edge collapse, foreign matter, etc.) is a key link to ensure product quality, improve production efficiency and reduce labor costs. Traditional image processing methods (such as particle swarm optimization combined with threshold segmentation, edge extraction, morphological processing, etc.) rely on manual design of feature parameters, have poor robustness, and lack generalization ability for different types of defects, especially in the case of changing light conditions, more background interference, and dense and small defects, which cannot meet the high precision and real-time requirements of actual industrial production lines. In recent years, as an important development direction of artificial intelligence, deep learning technology has made certain achievements in image semantic understanding and target detection tasks. For example, some domestic research based on improved UNet structure carries out lens defect detection, although certain effects have been achieved, but most of them only focus on single defect types such as scratches, and have not been expanded to more extensive defect categories. In addition, two-stage detection models such as Faster R-CNN and Mask R-CNN have strong detection ability, but the inference speed is slow, which is difficult to meet the real-time requirements of online detection. One-stage detection models such as the YOLO series have been applied in optical glass surface defect detection and have shown high recall rate and detection speed, but since their task targets are mainly “detection boxes”, they cannot provide defect contour information, and cannot accurately judge the length, width, area, shape and other geometric features of the defects, which limits their application effect in high-precision quality evaluation and subsequent decision-making systems. SUMMARY

[0003] The technical problem to be solved by the present application is to overcome the shortcomings of the prior art, and to provide a deep learning-based optical glass AOI detection method that can effectively expand the defect type and scene diversity, and improve the clarity and contrast of image acquisition, can be combined with data enhancement technology, and can construct a high-quality optical lens defect data set that can be used for semantic segmentation tasks. This method can realize fine detection and segmentation identification of optical lens defects, while maintaining high detection speed, greatly improving the accuracy of defect contour positioning and the richness of class identification, and providing an efficient and feasible technical solution for industrial-grade optical glass intelligent detection.

[0004] The technical scheme adopted by the present application is: the deep learning-based optical glass AOI detection method of the present application includes a data acquisition stage:

[0005] In the data acquisition stage, it is acquired by continuously shooting multiple local images with overlapping areas; for the structure features of the correct class product, an automatic splicing method based on two deep learning target detection is adopted; the automatic splicing method includes the following steps:

[0006] Firstly, the YOLOv11 network is used to detect the overall product region in each image, extract the height and Y direction coordinate information of the detection frame, calculate the vertical offset of the product position in the adjacent image, and complete the preliminary alignment of the image in the Y direction according to the vertical offset;

[0007] Subsequently, the image after vertical adjustment is executed for the second time target detection, the key structure is extracted, and the standard template is matched, the affine transformation relationship between the images is calculated, and the splicing value between the two images is obtained;

[0008] Finally, the splicing value is compared with the set X and Y direction splicing threshold value, if the threshold value condition is met, the splicing is directly completed; if it is not met, the degree of change of the key structure in the image position is judged: when the change is small, the splicing can still be completed according to the splicing threshold value; when the change is large, the key structure and the overall product position are normalized first, and the splicing operation is performed according to the normalized relative position.

[0009] Further, before entering the data acquisition stage, the following steps are included:

[0010] ① Fine manual annotation is performed on the optical glass defect image data;

[0011] ② The labeled data obtained in step ① is subjected to data enhancement processing;

[0012] ③ The enhanced sample set is divided into a training set and a validation set according to a 7:3 ratio, ensuring that the model can learn sample features sufficiently and monitor overfitting risk through the validation set during the training process; in terms of model selection, the YOLOv11-Seg model is adopted, which is a model after the segmentation module is fused with the YOLOv11 algorithm;

[0013] ④ After the model training is completed, the effect is systematically evaluated by using multiple key performance indicators;

[0014] ⑤ After the evaluation is passed, the trained model is imported into the deployment platform to provide core inference ability for the subsequent detection process

[0015] Further, after entering the data acquisition stage, the following steps are included:

[0016] ⑦In the actual detection stage, the system acquires high-resolution glass optical glass images in real time through a high-speed image acquisition device, and according to the detection requirements of defects in different regions, the key regions are reduced by 2 times and the other regions are reduced by 4 times;

[0017] ⑧The scaled images are further processed by sliding window clipping with a size of 1024x1024, and the model inference throughput is significantly improved through multi-batch and multi-thread parallel computing. Each sub-image is sent as an independent input into the YOLOv11-Seg model for high-precision defect prediction, ensuring accurate identification of tiny defects in large-size images;

[0018] ⑨To realize spatial positioning and screening of defects, a basic template matching mechanism is combined with a coarse positioning and fine positioning strategy. A weighted least squares method is used to fit a straight line, and affine transformation and other geometric calibration methods are used to accurately obtain the actual coordinates of the defects in the original image. Through accurate calculation of the defect size and combining with the preset screening conditions, the detection results are post-processed to eliminate non-key defects that do not meet the requirements, realizing high-reliability screening output

[0019] Further, the manual labeling categories in step ① include but are not limited to tiny point defects, scratches, structural defects, color abnormalities, edge collapse, foreign matter residues, surface dirt, and embossing. A multi-label and multi-instance method is used to improve label quality and data diversity.

[0020] Further, the YOLOv11-Seg model in step ② is a lightweight network structure that adds P2 / 4 to the original YOLOv11-n-seg, outputs four scales of feature maps, and increases a neck module on the PAN-FPN layer and the bottom-up path fusion, which can obtain lower layer features 160x160. Based on the original network layer, the feature map contains more context information and semantic information, enhancing the model's detection ability for different scale targets.

[0021] Further, the data enhancement processing method in step ② includes but is not limited to image layer mixing, random cropping, random flipping, color disturbance, and mosaic data enhancement.

[0022] Further, in step ④, the key performance indicators include the precision and recall of bounding box prediction, the precision and recall of mask segmentation, and the average precision mAP50 based on IoU ≥ 0.5. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 is an improved YOLOv11-Seg network structure;

[0024] Figure 2is a flow chart of an automatic splicing method based on two deep learning target detection;

[0025] Figure 3 is a coarse-fine positioning flow chart;

[0026] Figure 4 is an algorithm application implementation flow chart;

[0027] Figure 5 is a detection effect diagram;

[0028] Figure 6 is a small point defect detection effect diagram. DETAILED DESCRIPTION

[0029] In this embodiment, the present application relates to defect detection technology in the glass production process, aiming at the deficiencies of the prior art in precision and efficiency, a YOLOv11-seg optical glass AOI detection method is proposed, which can realize fast and efficient multi-class defect recognition for large resolution collected glass images, and is particularly suitable for recognizing subtle defects that are difficult to distinguish. The optical glass AOI detection method adopts an image acquisition module, a model training module, a real-time detection module, a defect analysis module, and a defect information transmission and interaction module.

[0030] The present application proposes a YOLOv11-seg optical glass chip defect automatic detection process, which covers a complete closed-loop system from data preparation, model training to deployment application, specifically including the following eight steps:

[0031] 1) Use professional annotation software to finely manually annotate the optical glass defect image data. The annotation categories are comprehensive, covering multiple defect types commonly seen in the optical glass manufacturing process, including but not limited to small point defects, scratches, structural defects, color abnormalities (color abnormalities), edge collapse, foreign matter residues, surface dirt, and multiple dimensions such as embossing, etc. Ensure that the subsequent model can accurately identify and classify defects. At the same time, use multi-label and multi-instance methods to improve label quality and data diversity;

[0032] 2) After obtaining high-quality labeled data, perform systematic data enhancement operations on sample data to improve the generalization ability of the model. Enhancement methods include layer mixing (such as MixUp, CutMix, etc.), random cropping (to improve the perception of local features), random flipping (to enhance direction invariance), color perturbation, and mosaic data enhancement techniques, effectively improving the complexity and diversity of training samples, thereby enhancing the model's ability to adapt to actual complex scenarios;

[0033] 3) The enhanced sample set is divided into training set and validation set according to the ratio of 7:3, ensuring that the model can learn the sample features sufficiently and monitor the overfitting risk through the validation set during the training process. In terms of model selection, the YOLOv11-n-seg lightweight network structure adopts C3K2 module, C3K2 module combines variable convolution kernel and channel separation strategy, adapts to more complex tasks and multi-scale detection, and C2PSA is added in the SPPF module, C2PSA contains spatial attention mechanism, enhances feature extraction, and the output of the traditional YOLO mainly includes P3 / 8, P4 / 16 and P5 / 32, as shown in the Figure 1 , the improved YOLOv11-n-seg lightweight network structure is adopted, P2 / 4 is added, and four scales of feature maps are output, and a module is sequentially added on the PAN-FPN layer and the bottom-up path fusion, as shown in Figure 1 , the added neck module can obtain lower layer features 160x160, and based on the original network layer, the feature map contains more rich context information and semantic information, and the detection ability of the model to different scale targets is enhanced, the number of detection boxes is increased from 8400 to 34000, and in the output of the seg, the semantic of the mask is increased from 160 to 320, thereby greatly increasing the output ability, and the detection effect is greatly improved, especially for smaller targets, the detection rate is significantly improved;

[0034] 4) After the model training is completed, the effect is evaluated by using multiple key performance indicators, including the precision and recall of the boundary box prediction, the precision and recall of the mask segmentation, and the average precision (mAP50) based on IoU ≥ 0.5 and other indicators. These evaluation indicators can comprehensively reflect the comprehensive performance of the model in defect positioning, classification and contour segmentation, etc., and ensure that the model has good practical application ability;

[0035] 5) After the evaluation, the trained model is imported into the deployment platform to provide core inference ability for the subsequent detection process. The model format, inference efficiency, platform compatibility and the like are optimized and adapted during the model import process;

[0036] 6) During the data acquisition phase, this invention employs a method for automatic stitching based on dual deep learning object detection. This method involves continuously capturing multiple local images with overlapping areas, targeting the structural features of square-shaped products. First, a YOLOv11 network is used to detect the overall product region in each image, extracting the height and Y-coordinate information of the detection box. The vertical offset of the product position in adjacent images is calculated, and preliminary alignment of the images in the Y-direction is achieved. Then, a second object detection is performed on the vertically adjusted images to extract key structures (product feature points, etc.) and match them with a standard template. The affine transformation relationship between the images is calculated to obtain the stitching value between the two images. The stitching value is compared with set X and Y-direction stitching thresholds. If the threshold conditions are met, stitching is completed directly. If not, the degree of change in the position of the key structure in the image is used for judgment: when the change is small (product sizes are basically consistent), stitching can still be completed based on the stitching threshold; when the change is large (overall scaling exists), the positions of the key structure and the overall product are first normalized, and the stitching operation is performed based on the normalized relative positions. The test results show that about 99% of the product images meet the stitching threshold requirements and can be stitched directly, about 0.9% of the images need to be stitched after normalization adjustment, and about 0.1% of the images require special processing due to large structural changes. The whole image process is shown in Figure 2.

[0037] 7) During the actual inspection phase, the system acquires high-resolution images of the glass optical glass in real time using a high-speed image acquisition device, with an image resolution of up to 36604×4221 pixels. To reduce computational resource consumption and improve processing efficiency, this invention reduces the size of critical areas by a factor of 2 and other areas by a factor of 4, based on the inspection requirements of defects in different regions, thus balancing detail preservation and processing speed.

[0038] 8) The scaled images are further cropped using a sliding window to a size of 1024×1024. Multi-batch and multi-threaded parallel computation significantly improves the model's inference throughput. Each sub-image is fed as independent input into the YOLOv11-Seg model for high-precision defect prediction, ensuring accurate identification of minute defects in large images.

[0039] Finally, to achieve spatial localization and screening of defects, this invention combines a basic template matching mechanism with a coarse-to-fine localization strategy. It uses weighted least squares to fit a straight line and employs geometric calibration methods such as affine transformation. Figure 3 As shown, the actual coordinates of the defect in the original image are accurately obtained. By accurately calculating the defect size (such as area, aspect ratio, main axis direction, etc.) and combining it with preset screening conditions, the detection results are post-processed to remove non-critical defects that do not meet the requirements, thus achieving highly reliable screening output.

[0040] The flow integrates a deep learning segmentation model, a traditional image processing method and an efficient deployment strategy, significantly improves the accuracy, recall rate and real-time performance of optical glass defect detection, and is suitable for intelligent defect detection tasks under large-scale high-resolution images, and provides a practical solution for the semiconductor industry and precision manufacturing.

[0041] In the embodiment, the algorithm application implementation flow is as shown in the figure. Figure 4 To solve the problems of insufficient defect type coverage, difficult to balance detection accuracy and efficiency in current optical glass defect automatic detection, a set of optical glass AOI detection method is designed in this paper, which effectively expands the defect type and scene diversity, and improves the clarity and contrast of image acquisition;At the same time, combined with data enhancement technology (such as random rotation, flip, color disturbance, mixed enhancement, etc.), a high-quality optical lens defect data set that can be used for semantic segmentation task is constructed;Finally, the new generation YOLOv11 algorithm and segmentation module are fused, and the YOLOv11-Seg method is proposed, which realizes the fine detection and segmentation identification of optical lens defects, while maintaining high detection speed, greatly improves the accuracy of defect contour positioning and the richness of class identification, and provides an efficient and feasible solution for industrial optical glass intelligent detection.

[0042] The beneficial effects of the present application are as follows:

[0043] Table 1 yolo-seg and improved yolo-seg comparison table

[0044]

[0045] The core innovation of the present application is to introduce the improved YOLOv11-Seg target detection network, which effectively enhances the detection ability of small targets, especially point defects. Through the optimization of model structure and the enhancement of feature fusion strategy, the overall detection performance is significantly improved on the basis of maintaining the detection speed. The experimental results show that the recall rate of point defects is improved by 23% compared with the baseline model, and the corresponding boundary box detection accuracy (mAP@0.5) is improved by 25%;At the same time, due to the increase of more low feature fusion, the overall detection effect is comprehensively enhanced, and the mask (mask) detection performance is improved by about 8% on average. The specific detection accuracy of each type of defect is shown in Table 1, which further verifies the effectiveness and superiority of the algorithm in complex defect detection scenarios.

[0046] The application innovatively introduces a dual deep learning target detection mechanism for image stitching, which can accurately identify the product area and its key structural features in each local image, and calculate the stitching offset between images by matching with a standard template. In the case of no large tolerance in product size, the stitching accuracy can reach 100%; in the actual production environment containing tolerance, the overall stitching accuracy reaches 99.8%. For a small number of images with structural deviation, compensation can be made by setting a threshold or using a key structure normalization position stitching strategy, and ultimately a high stitching accuracy can still be achieved to meet the strict requirements of the production line on stitching quality and stability.

[0047] The application realizes accurate detection of multi-modal defects of optical glass, and through cooperative analysis of preliminary positioning and precise positioning, a straight line is fitted using a weighted least squares method, and a spatial calibration technology of affine transformation is combined to classify and filter the feature defects in different regions. While maintaining an ultra-high detection rate of 99.7%, the method effectively distinguishes between heterogeneous defects such as scratches, imprints, and foreign matter, and its detection accuracy and robustness are significantly better than the current industrial detection standard, fully meeting the mass production quality inspection requirements of high-precision optical devices.

[0048] The detection effect in the application example is as shown in Figure 5 .

[0049] Specific application examples:

[0050] 1. Data acquisition and sample preparation

[0051] On the optical glass chip production line, a large number of high-resolution defect image data are collected using an industrial camera, covering typical defect types such as point defects, scratches, structural defects, color differences, foreign matter, and dirt.

[0052] The collected images are preprocessed by image cropping, and the various defect regions are finely labeled with the help of professional labeling software, finally forming a high-quality labeled sample data set for subsequent model training.

[0053] 2. Model training parameter configuration and training process

[0054] Based on a well-configured hardware platform (such as a deep learning workstation equipped with an NVIDIA RTX 3090 GPU), the training parameters are set as follows: Batch size = 32, Epoch: 1000, initial learning rate (lr0): 0.01, learning rate scheduling strategy: cosine annealing, and data augmentation methods: random cropping, layer mixing (such as MixUp, CutMix), color perturbation, flipping, and mosaic enhancement. Using the above parameters, the improved YOLOv11 segmentation model (YOLOv11-Seg) is trained end-to-end, and the detection capability and segmentation accuracy of the model on different defect types are gradually improved.

[0055] 3. Model evaluation and test deployment

[0056] After training, the optimal model is selected from multiple training weights based on the set evaluation criteria (such as mAP50, Precision, Recall, Mask IoU, etc.).

[0057] The model is deployed in the detection software for inference testing of actual production images to verify the recognition ability and response efficiency of the model for multiple defect types.

[0058] 4. System operation and defect detection process

[0059] After the detection system is started, the optical glass chip image data sent by the image acquisition module is received in real time according to the set process, the image is preprocessed (such as scaling and sliding window cropping), and each sub-image is sent to the YOLOv11-Seg model for inference analysis.

[0060] The system automatically completes defect detection, segmentation, spatial positioning, and screening, and finally outputs key information such as defect category, location, and area, and displays the results to the operator or uploads them to the backend quality control platform through the interactive module.

[0061] Although the embodiments of the present application are described in actual schemes, they do not constitute a limitation on the meaning of the present application. For those skilled in the art, modifications of the embodiments and combinations with other schemes according to the present specification are obvious.

Claims

1. A deep learning-based optical glass AOI detection method, characterized in that: The optical glass AOI detection method based on deep learning comprises a data acquisition stage: In the data acquisition stage, a plurality of local images with overlapping areas are continuously photographed for acquisition; for the structure characteristics of the aligned product, an automatic splicing method based on two deep learning target detections is adopted; the automatic splicing method comprises the following steps: Firstly, the YOLOv11 network is used to detect the product whole region in each image, the height and Y direction coordinate information of the detection frame are extracted, the vertical offset of the product position in the adjacent image is calculated, and the preliminary alignment of the images in the Y direction is completed according to the offset; Subsequently, the second target detection is performed on the vertically adjusted image, the key structure is extracted, and the key structure is matched with the standard template to calculate the affine transformation relationship between the images, so that the splicing value between the two images is obtained; Finally, the splicing value is compared with the set X and Y direction splicing threshold value, if the threshold value condition is met, the splicing is directly completed, if the threshold value condition is not met, the degree of change of the key structure in the image position is judged: when the product size is basically consistent, the splicing can still be completed according to the splicing threshold value; when there is overall scaling, the key structure and the overall product position are normalized, and the splicing operation is performed according to the normalized relative position.

2. The optical glass AOI detection method based on deep learning according to claim 1, further comprising the following steps before entering the data acquisition stage: ① fine manual labeling of optical glass defect image data; ② data enhancement processing of the labeled data obtained in step ①; ③ dividing the enhanced sample set into a training set and a validation set according to a 7:3 ratio to ensure that the model can learn the sample features and monitor the overfitting risk in the training process; in terms of model selection, a YOLOv11-Seg model is adopted, which is a model obtained by fusing a segmentation module to a YOLOv11 algorithm; ④ after the model training is completed, the effect of the model is systematically evaluated by using multiple key performance indicators; ⑤ after the evaluation is passed, the trained model is imported into a deployment platform to provide core reasoning capability for the subsequent detection process.

3. The optical glass AOI detection method based on deep learning according to claim 1, further comprising the following steps after entering the data acquisition stage: ⑦ entering the actual detection stage, the system acquires high-resolution glass optical glass images in real time through a high-speed image acquisition device, and according to the detection requirements of defects in different regions, the key region is reduced by 2 times and the other regions are reduced by 4 times; ⑧ the scaled images are further processed by windowing and cropping according to the size of 1024x1024, and the model reasoning throughput is significantly improved by using the multi-batch and multi-thread parallel computing mode; each sub-image is sent as an independent input into the YOLOv11-Seg model for high-precision defect prediction, ensuring accurate identification of small defects in large-size images; ​ ​ 9. To realize the spatial positioning and screening of defects, a coarse positioning and fine positioning strategy is adopted in combination with a basic template matching mechanism, a straight line is fitted using a weighted least squares method, and the actual coordinates of the defects in the original image are accurately obtained with the help of an affine transformation geometric calibration method; the size of the defects is accurately calculated, the detection results are post-processed in combination with preset screening conditions, non-key defects that do not meet the requirements are removed, and high-reliability screening output is realized. 4.The optical glass AOI detection method based on deep learning of claim 2, characterized in that: The artificial marking categories in step 1 include at least one of a tiny point defect, a scratch, a structure defect, a color anomaly, a collapsed edge, a foreign matter residue, a surface dirt, and an embossing. 5.The optical glass AOI detection method based on deep learning of claim 2, characterized in that: The YOLOv11-Seg model in step 2 is a lightweight network structure that adds P2 / 4 to the original YOLOv11-n-seg, outputs four scales of feature maps, and sequentially adds a neck module to the sampling and bottom-up path fusion on the PAN-FPN layer, so as to obtain lower layer features 160x160, and make the feature maps contain more context information and semantic information on the basis of the original network layer, thereby enhancing the detection capability of the model for different scale targets. 6.The optical glass AOI detection method based on deep learning of claim 2, characterized in that: The data enhancement processing method in step 2 includes at least one of layer mixing, random cropping, random flipping, color disturbance, and mosaic data enhancement. 7.The deep learning-based optical glass AOI detection method of claim 2, wherein: In step 4, the key performance indicators include the precision and recall of the bounding box prediction, the precision and recall of the mask segmentation, and the average precision mAP50 based on IoU ≥ 0.5.

Images