Defect detection method, device and equipment

Abstract

The invention relates to the technical field of computer vision, in particular to a defect detection method, device and equipment, and the method comprises the steps: obtaining a historical defect sample set and an expansion set to generate a first defect sample set, covering more defect conditions, and improving the adaptability of a model to complex defects; secondly, a deep convolutional neural network model and the like are used for preprocessing, a Micro LED defect detection model is generated, the detection precision is improved, similar defects are accurately marked and classified, and it is guaranteed that the detection result is reliable; furthermore, automatic sample set generation and model optimization save time and energy, and the model can quickly and accurately assist in production decision making; and finally, the model is beneficial to improving production decision quality, promoting scientific and technological innovation, assisting in solving production problems and guaranteeing industrial sustainable development.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a defect detection method, device and equipment. Background Art

[0002] Artificial intelligence faces complex and ever-changing defect detection challenges in the fields of industry and manufacturing, especially in AOI inspection such as Micro LED chips. Defects are of various types and shapes, with low contrast and small size, which increases the difficulty of identification. Manual labeling faces many problems: the concealment and diversity of defects make it difficult for labelers to accurately capture information, and it is easy to make incorrect labeling and omissions, affecting the training effect and detection accuracy of deep learning models. It also takes a lot of time and energy, and may cause blurred details to be lost due to insufficient image brightness, reflective areas to cover up distorted content, and some areas to be difficult to identify with the naked eye, resulting in difficulty in labeling and poor accuracy. The performance of deep learning models is related to the quality of industrial production decisions and scientific and technological innovation. Labeling defects will lead to poor model detection results, misleading production processes, and hindering technological innovation and industrial upgrading. For example, in links such as Micro LED mass transfer, multiple defects need to be detected, and some similar defects are difficult to label with deep learning pre-training models, and labeling accuracy needs to be urgently addressed. Summary of the invention

[0003] In order to solve the above-mentioned shortcomings in the prior art, the present invention proposes a defect detection method, device and equipment. The technical solution adopted by the present invention is as follows:

[0004] A defect detection method includes: obtaining a historical defect sample set from a database, and using a data enhancement method to perform data expansion on the historical defect sample set to obtain a first expanded sample set; generating a first defect sample set according to the historical defect sample set and the first expanded sample set; preprocessing the first defect sample set through a preset deep convolutional neural network model to obtain a second defect sample set and a third defect sample set; constructing a Micro LED defect detection model according to the first defect sample set, the second defect sample set, the third defect sample set and preset conditions; obtaining a real-time defect sample set, and predicting the real-time defect sample set according to the Micro LED defect detection model to obtain a prediction result. In view of the complexity of defect detection, a rich and diverse first defect sample set is generated by obtaining a historical defect sample set and a first expanded sample set. The expanded sample set can cover more different types of defect situations, providing a more comprehensive data basis for subsequent model training, thereby improving the model's adaptability to various complex defects; the preset deep convolutional neural network model and the first defect sample set are used for preprocessing to obtain the second defect sample set and the third defect sample set, and the Micro LED defect detection model is generated through preset conditions. This process greatly improves the detection accuracy of the model. This solution can more accurately label and classify some similar defects, providing more reliable detection results for industrial production; in terms of efficiency, it greatly saves time and energy. Through automated sample set generation and model optimization, accurate detection models can be obtained quickly and efficiently, providing timely support for industrial production decisions; finally, an accurate Micro LED defect detection model is of great significance to the quality of industrial production decisions and scientific and technological innovation; it can reduce the occurrence of poor model detection results caused by labeling defects, thereby misleading the production process; at the same time, it helps to timely discover and solve problems in production, promote technological innovation and industrial upgrading, and provide strong guarantees for the sustainable development of the industrial field.

[0005] Furthermore, the preprocessing of the first defect sample set by the preset deep convolutional neural network model to obtain the second defect sample set and the third defect sample set includes: constructing an original deep learning model according to the preset deep convolutional neural network model; training the original deep learning model according to the first defect sample set to obtain the first deep learning model; annotating and segmenting the first defect sample set by the first deep learning model to obtain the second defect sample set; and annotating the second defect sample set for microscopic defects to obtain the third defect sample set. Constructing the original deep learning model based on the preset deep convolutional neural network model provides a reliable infrastructure for subsequent processing; training it with the first defect sample set can continuously optimize the model and improve its accuracy; annotating and segmenting the sample set by the first deep learning model to obtain the second defect sample set realizes automated data processing; and then annotating the second defect sample set for microscopic defects to obtain the third defect sample set, which provides support for more sophisticated defect analysis and helps improve the accuracy and efficiency of Micro LED defect detection.

[0006] Furthermore, the method of training a preset deep convolutional neural network model according to the first defect sample set to obtain a first deep learning model includes: obtaining a first data set to be labeled from the first defect sample set according to a preset first ratio; performing defect partition labeling on the first data set to obtain a first labeled data set and label data; and training the deep convolutional neural network model based on the first labeled data set and label data to obtain the first deep learning model. Obtaining the data set to be labeled from the first defect sample set according to a preset ratio, performing defect partition labeling, and using it to train the deep convolutional neural network model can improve the pertinence and efficiency of model training, and help to more accurately detect Micro LED defects.

[0007] Further, the labeling and segmenting of the first defect sample set by the first deep learning model to obtain the second defect sample set includes: obtaining a second data set to be labeled from the first defect sample set according to a preset second ratio; labeling the second data set to be labeled based on the first deep learning model to obtain a second labeled data set; obtaining a second ROI area according to the second labeled data set; obtaining the first ROI area according to the first labeled data set; and segmenting the first defect sample set based on the first deep learning model, the first ROI area, and the second ROI area to obtain the second defect sample set. The second data set to be labeled is obtained from the first defect sample set by a preset ratio, and the existing first deep learning model is used for labeling to efficiently obtain the second labeled data set, thereby obtaining different ROI areas, and segmenting the first defect sample set based on these areas and the deep learning model to obtain a more targeted second defect sample set. This method improves the efficiency and accuracy of data processing, provides more refined data support for Micro LED defect detection, and helps to improve the accuracy and reliability of defect detection.

[0008] Furthermore, the method of annotating the second data set to be annotated based on the first deep learning model to obtain a second annotated data set includes: predicting the second data set to be annotated based on the first deep learning model to obtain a first prediction result; analyzing the first prediction result to obtain a defect category, and comparing the defect category with a preset real defect category to obtain a comparison result; determining whether the comparison result is consistent; when the comparison result is consistent, obtaining a third annotated data set according to the prediction result; when the comparison result is inconsistent, correcting the prediction result according to a preset quantitative standard to obtain a third annotated data set; removing low confidence and merging adjacent bounding boxes from the third annotated data set to obtain a corrected annotated data set; and annotating the corrected annotated data set by defect partitioning to obtain a second annotated data set. The second dataset to be labeled is predicted based on the first deep learning model, providing preliminary analysis results for subsequent processing; by comparing with the preset real defect categories, the accuracy and reliability of the labeling are ensured; when the comparison results are consistent, the third labeled dataset is directly obtained based on the prediction results, which improves the processing efficiency; when inconsistent, the preset quantitative standards are used for correction to ensure the quality of the data; the third labeled dataset is removed from the low confidence and adjacent bounding boxes are merged to further optimize the data and reduce interference factors; finally, the defect partitioning annotation of the corrected labeled dataset is performed to obtain the second labeled dataset, which provides more accurate and targeted data support for Micro LED defect detection, and helps to improve the accuracy and performance of defect detection.

[0009] Furthermore, the Micro LED defect detection model is constructed according to the first defect sample set, the second defect sample set, the third defect sample set and the preset conditions, including: performing nonlinear interpolation operation on the third defect sample set based on a preset interpolation algorithm to obtain an interpolated defect sample set; performing upsampling network structure operation on the interpolated defect sample set based on a preset deep convolution generative adversarial network to obtain a second extended sample set; generating a corrected defect sample set according to the first defect sample set, the second defect sample set, the third defect sample set and the second extended sample set; adjusting and configuring the first deep learning model based on the preset first parameter to obtain the second deep learning model; training the second deep learning model according to the corrected defect sample set to obtain the third deep learning model; adjusting and configuring the third deep learning model based on the preset convolution void method and the preset second parameter to obtain a fourth deep learning model; iteratively optimizing and training the fourth deep learning model based on the first labeled data set, the second labeled data set and the preset conditions to obtain the MicroLED defect detection model. Through the preset interpolation algorithm and deep convolution generative adversarial network, the third defect sample set is processed to obtain the second expanded sample set, enriching the sample data; combining multiple sample sets to generate a corrected defect sample set provides a more comprehensive data basis for model training. The first deep learning model is gradually adjusted and configured, and the fourth deep learning model is obtained after multiple rounds of training, which improves the accuracy and robustness of the model; the fourth deep learning model is iteratively optimized and trained using different annotated data sets and preset conditions, and the final Micro LED defect detection model can detect defects more accurately, providing strong guarantees for Micro LED production quality control, and helping to improve product reliability and market competitiveness.

[0010] Furthermore, the iterative optimization training of the fourth deep learning model based on the first annotated data set, the second annotated data set and the preset conditions to obtain the Micro LED defect detection model includes: constructing a test set and a training set based on the first annotated data set and the second annotated data set; training the fourth deep learning model based on the training set and the preset conditions to obtain a first training result; testing the first training result according to the test set and the preset conditions to obtain a test result; analyzing the test results; when the test result is a test success, obtaining the Micro LED defect detection model according to the first training result; otherwise, training the fourth deep learning model again based on the training set and the preset conditions. By constructing a training set and a test set for the first labeled data set and the second labeled data set, a more representative data division is provided for the training of the fourth deep learning model, which effectively improves the generalization and robustness of the model prediction; the fourth deep learning model is trained based on the training set and preset conditions, while reducing the model calculation amount and storage cost, the data set integrity and model robustness are continuously strengthened; the first training result is tested according to the test set and preset conditions, which can greatly improve the recall rate and precision of the model detection, reduce the missed detection rate and the false positive rate, and optimize the fourth deep learning model through multiple iterations. The final Micro LED defect detection model has higher detection accuracy, providing reliable technical support for Micro LED industrial defect detection.

[0011] Furthermore, the operations of removing low-confidence objects and merging adjacent bounding boxes on the third annotated data set to obtain a revised annotated data set include: determining whether the prediction confidence of the third annotated data set is greater than a preset first threshold; when the prediction confidence of the third annotated data set is greater than the first threshold, calculating the overlap of all adjacent bounding boxes of the third annotated data set; determining whether the overlap is greater than a preset second threshold; when the overlap is greater than the second threshold, merging all adjacent bounding boxes to obtain a revised annotated data set; when the prediction confidence of the third annotated data set is less than a preset threshold, removing low-confidence objects in the third annotated data set according to the preset third threshold to obtain a revised annotated data set. By judging the relationship between the prediction confidence of the third annotated data set and the preset first threshold, data can be accurately screened. When the confidence meets the standard, the overlap of adjacent bounding boxes is further considered and merged when it is greater than the second threshold, which can optimize the data structure and reduce redundancy. When the confidence is less than the threshold, low-confidence targets are removed based on the third threshold to improve data quality. In this way, the final corrected annotated data set is more accurate and reliable, providing a high-quality data foundation for subsequent Micro LED defect detection and other related work, helping to improve overall work results.

[0012] Furthermore, a defect detection device includes: an expanded sample set acquisition module, which is used to acquire a historical defect sample set from a database, and use a data enhancement method to perform data expansion on the historical defect sample set to obtain a first expanded sample set; a first defect sample set generation module, which is used to generate a first defect sample set based on the historical defect sample set and the first expanded sample set; a sample set convolution preprocessing module, which is used to perform preprocessing based on the first defect sample set and a preset deep convolutional neural network model to obtain a second defect sample set and a third defect sample set; a defect sample set fusion modeling module, which is used to construct a Micro LED defect detection model based on the first defect sample set, the second defect sample set, the third defect sample set and preset conditions; a prediction result generation module, which is used to acquire a real-time defect sample set, and predict the real-time defect sample set according to the Micro LED defect detection model to obtain a prediction result. The expanded sample set acquisition module expands the historical defect sample set through data enhancement to provide a rich data source for the subsequent process; the first defect sample set generation module generates a first defect sample set covering more defect situations based on the historical defect sample set to strengthen the data foundation. The sample set convolution preprocessing module further processes with the help of a deep convolutional neural network model to obtain a key sample set. The defect sample set fusion modeling module uses this to build a high-precision Micro LED defect detection model. The prediction result generation module uses this model to predict the real-time sample set and quickly output accurate results. Overall, it not only improves the model's adaptability and detection accuracy to complex defects, but also saves time and energy, providing strong support for industrial production decision-making and sustainable development.

[0013] Furthermore, a defect detection device is provided, characterized in that the defect detection device comprises: a memory and at least one processor, wherein instructions are stored in the memory; at least one of the processors calls the instructions in the memory so that the defect detection device executes each step of a defect detection method as described in any one of the above items.

[0014] The beneficial effects of a defect detection method of the present invention are:

[0015] In view of the complexity of defect detection, a rich and diverse first defect sample set is generated by obtaining a historical defect sample set and a first expanded sample set. The expanded sample set can cover more different types of defect situations, providing a more comprehensive data basis for subsequent model training, thereby improving the model's adaptability to various complex defects; the preset deep convolutional neural network model and the first defect sample set are used for preprocessing to obtain the second defect sample set and the third defect sample set, and the Micro LED defect detection model is generated through preset conditions. This process greatly improves the detection accuracy of the model. This solution can more accurately label and classify some similar defects, providing more reliable detection results for industrial production; in terms of efficiency, it greatly saves time and energy. Through automated sample set generation and model optimization, accurate detection models can be obtained quickly and efficiently, providing timely support for industrial production decisions; finally, an accurate Micro LED defect detection model is of great significance to the quality of industrial production decisions and scientific and technological innovation; it can reduce the occurrence of poor model detection results caused by labeling defects, thereby misleading the production process; at the same time, it helps to timely discover and solve problems in production, promote technological innovation and industrial upgrading, and provide strong guarantees for the sustainable development of the industrial field. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A first flow chart of a defect detection method provided by an embodiment of the present invention;

[0017] Figure 2 A second flow chart of the defect detection method provided by an embodiment of the present invention;

[0018] Figure 3 A third flow chart of the defect detection method provided by an embodiment of the present invention;

[0019] Figure 4 A fourth flow chart of the defect detection method provided by an embodiment of the present invention;

[0020] Figure 5 A fifth flow chart of the defect detection method provided by an embodiment of the present invention;

[0021] Figure 6 A sixth flow chart of the defect detection method provided by an embodiment of the present invention;

[0022] Figure 7 A seventh flow chart of the defect detection method provided by the embodiment of the present invention;

[0023] Figure 8 An eighth flow chart of the defect detection method provided by an embodiment of the present invention;

[0024] Fig. 9A schematic diagram of the structure of a defect detection device provided by an embodiment of the present invention;

[0025] Fig.10 A schematic diagram of the structure of a defect detection device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 , the number of Micro LED chips placed on a 6-inch wafer is approximately between 1.2 million and 1.6 million. Assuming a 10x lens is used for detection, the number of chips in each field of view is about 80, the camera will take about 20,000 shots, and each chip will be detected for about 5-6 hours, which means the detection time is relatively long. Assuming a 5x lens is used for detection, the number of chips in each field of view is about 320, the camera will take about 5,000 shots, and each chip will be detected for about 1.4 hours, which shortens the detection time. However, the pixel accuracy of the 5x camera is approximately 0.7μm, and the maximum defect size that can be detected is 2-3μm. In order to ensure the efficiency of Micro LED defect detection, improve the detection accuracy of tiny defects, and increase the traceability of defects in the Micro LED production process, an embodiment of a defect detection method in an embodiment of the present invention includes:

[0029] 101. Obtain a historical defect sample set from a database, and use a data enhancement method to perform data expansion on the historical defect sample set to obtain a first expanded sample set;

[0030] 102. Generate a first defect sample set according to the historical defect sample set and the first expanded sample set;

[0031] 103. Preprocess the first defect sample set by using a preset deep convolutional neural network model to obtain a second defect sample set and a third defect sample set;

[0032] 104. Constructing a Micro LED defect detection model according to the first defect sample set, the second defect sample set, the third defect sample set and preset conditions;

[0033] 105. Obtain a real-time defect sample set, and predict the real-time defect sample set according to the Micro LED defect detection model to obtain a prediction result.

[0034] In this embodiment, in step 105, the prediction result includes defect type information, defect location information, defect degree information, etc., and an image acquisition device (generally an industrial CCD or CMOS camera with an industrial lens or a telecentric lens, and a suitable system layout scheme is selected for different defect types) is used to obtain a real-time defect sample set, wherein the historical defect sample set in the database is also obtained from the image acquisition device; in order to overcome the difficulty of insufficient number of historical defect sample sets, different lighting methods are used, including light sources of different angles, different distances, and different colors, etc., to increase the data volume of the same defect sample in the historical defect sample set; and then data enhancement methods are used to expand the defect data set, such as image fusion, flip transformation, color jitter, translation transformation, scale transformation, contrast transformation, noise disturbance and rotation transformation.

[0035] In view of the complexity of defect detection, a rich and diverse first defect sample set is generated by obtaining a historical defect sample set and a first expanded sample set. The expanded sample set can cover more different types of defect situations, providing a more comprehensive data basis for subsequent model training, thereby improving the model's adaptability to various complex defects; the preset deep convolutional neural network model and the first defect sample set are used for preprocessing to obtain the second defect sample set and the third defect sample set, and the Micro LED defect detection model is generated through preset conditions. This process greatly improves the detection accuracy of the model. This solution can more accurately label and classify some similar defects, providing more reliable detection results for industrial production; in terms of efficiency, it greatly saves time and energy. Through automated sample set generation and model optimization, accurate detection models can be obtained quickly and efficiently, providing timely support for industrial production decisions; finally, an accurate Micro LED defect detection model is of great significance to the quality of industrial production decisions and scientific and technological innovation; it can reduce the occurrence of poor model detection results caused by labeling defects, thereby misleading the production process; at the same time, it helps to timely discover and solve problems in production, promote technological innovation and industrial upgrading, and provide strong guarantees for the sustainable development of the industrial field.

[0036] See also Figure 2 A second embodiment of a defect detection method in an embodiment of the present invention includes:

[0037] 201. Constructing an original deep learning model according to a preset deep convolutional neural network model;

[0038] 202. Train the original deep learning model according to the first defect sample set to obtain a first deep learning model;

[0039] 203. Perform labeling and segmentation processing on the first defect sample set by using the first deep learning model to obtain a second defect sample set;

[0040] 204. Perform microscopic defect annotation on the second defect sample set to obtain a third defect sample set.

[0041] In this embodiment, in step 204, microscopic defect annotation is performed on the generated second defect sample set, and the types of microscopic defect annotation include dirt, damage, black spots, residual glue, burrs, electrode scratches, electrode detachment, etc.; when performing microscopic defect annotation, in order to ensure that the annotation standards between different annotators and the same annotator in different time periods are consistent, for uncertain defect categories, lengths, widths and other information, it is necessary to establish quantitative standards according to indicator requirements, and develop intelligent quantitative review tools so that the annotators can label according to the determined standards; for defects with longer lengths and complex shapes, the idea of ​​segmentation can be used; larger defects are segmented into multiple bounding boxes, and the defects are segmented and regionally annotated; an original deep learning model is constructed based on a preset deep convolutional neural network model, providing a reliable infrastructure for subsequent processing; the first defect sample set is used to train it, so that the model can be continuously optimized and the accuracy can be improved; the sample set is annotated and segmented by the first deep learning model to obtain a second defect sample set, thereby realizing automated data processing; the second defect sample set is then annotated with microscopic defects to obtain a third defect sample set, providing support for more sophisticated defect analysis, which helps to improve Micro Accuracy and efficiency in LED defect detection.

[0042] See also Figure 3 A third embodiment of a defect detection method in an embodiment of the present invention includes:

[0043] 301. Obtain a first to-be-annotated data set from a first defect sample set according to a preset first ratio;

[0044] 302. Perform defect partition annotation on the first to-be-annotated data set to obtain a first annotated data set and label data;

[0045] 303. Train a deep convolutional neural network model based on the first annotated data set and the label data to obtain a first deep learning model.

[0046] In this embodiment, in step 301: a first data set to be labeled is obtained from a first defect sample set according to a first ratio (for example, the first data set to be labeled accounts for 2 / 5, 3 / 10, etc.); in step 302, defect partitioning and labeling of the first data set to be labeled is to divide the first data set to be labeled into regions and label them macroscopically, for example, the detected upper left corner coordinates of the rectangular bounding box, the length and width of the rectangular bounding box (or the lower right corner coordinates of the rectangular bounding box), the defect category name, etc. are recorded.

[0047] <object>< / object> The tag form is written in the XML (eXtensible Markup Language) file, where the category name is <name> class name< / name> , the bounding box is <bndbox>

[0048] <xmin>< / xmin> <ymin>< / ymin> <xmax>< / xmax> <ymax>< / ymax>

[0049] < / bndbox>; or written into a JSON (JavaScript Object Notation) file in the form of key "bbox", "label" and value "[x_min, y_min, width, height]", "class name"; or written into a TXT (Text File) file in the form of category ID: 0, 1... and bounding box coordinates [x_center, y_center, width, height], so as to provide label data for subsequent verification of prediction results, labeling of unlabeled dataset defect samples, and training of deep convolutional neural networks so that the model can learn defect features. This is the first rough defect partitioning and labeling work carried out for the first dataset to be labeled; from a macro perspective, defects are divided into epitaxial aisle defects, chip epitaxial defects, and chip electrode defects according to their location distribution on the chip; defects can also be divided according to the manufacturing process of the Micro LED chip, for example: divided into epitaxial wafer related defects and chip manufacturing process defects and other process defects; in step 303, after the labeling is completed, a deep convolutional neural network model is used for training.

[0050] The data set to be labeled is obtained from the first defect sample set according to a preset ratio, and the defect partitions are labeled and used to train the deep convolutional neural network model, which can improve the pertinence and efficiency of model training and help to detect MicroLED defects more accurately.

[0051] See also Figure 4 A fourth embodiment of a defect detection method in an embodiment of the present invention includes:

[0052] 401. Obtain a second to-be-annotated data set from the first defect sample set according to a preset second ratio;

[0053] 402. Annotate the second to-be-annotated data set based on the first deep learning model to obtain a second annotated data set;

[0054] 403. Acquire a second ROI region according to a second labeled data set;

[0055] 404. Acquire a first ROI region according to the first labeled data set;

[0056] 405 . Perform a segmentation operation on the first defect sample set based on the first deep learning model, the first ROI region, and the second ROI region to obtain a second defect sample set.

[0057] In this embodiment, in steps 403 to 405, the first defect sample set is segmented according to the extracted defect ROI area, that is, the original image in the first defect sample set is segmented to generate a Micro LED second defect sample set; the second data set to be labeled is obtained from the first defect sample set according to a preset ratio, and the existing first deep learning model is used for labeling to efficiently obtain the second labeled data set, and then different ROI areas are obtained, and the first defect sample set is segmented based on these areas and the deep learning model to obtain a more targeted second defect sample set. This method improves the efficiency and accuracy of data processing, provides more refined data support for Micro LED defect detection, and helps to improve the accuracy and reliability of defect detection.

[0058] See also Figure 5 A fifth embodiment of a defect detection method in an embodiment of the present invention includes:

[0059] 501. Predicting a second to-be-annotated data set based on the first deep learning model to obtain a first prediction result;

[0060] 502. Analyze the first prediction result to obtain a defect category, and compare the defect category with a preset real defect category to obtain a comparison result;

[0061] 503. Determine whether the comparison results are consistent;

[0062] 504. When the comparison results are consistent, a third labeled data set is obtained according to the prediction results;

[0063] 505. When the comparison results are inconsistent, the prediction results are corrected using a preset quantitative standard to obtain a third annotated data set;

[0064] 506. Perform operations of removing low confidence and merging adjacent bounding boxes on the third annotated data set to obtain a corrected annotated data set;

[0065] 507 . Perform defect partition annotation on the corrected annotated data set to obtain a second annotated data set.

[0066] In this embodiment, the prediction results of step 501 are post-processed, such as removing low-confidence predictions and merging similar prediction boxes, and the post-processed prediction results are output as pre-labeling results; in step 505, the quantization standard is the correct label category and rectangular box, and the correct label category and rectangular box are used to replace the defect data with incorrect predictions, that is, the prediction results are corrected. The second data set to be labeled is predicted based on the first deep learning model, providing preliminary analysis results for subsequent processing; by comparing with the preset real defect category, the accuracy and reliability of the labeling are ensured; when the comparison results are consistent, the third labeled data set is directly obtained according to the prediction results, which improves the processing efficiency; when inconsistent, the preset quantization standard is used for correction to ensure the quality of the data; the third labeled data set is removed from the low-confidence and adjacent boundary boxes are merged, which further optimizes the data and reduces interference factors; finally, the defect partitioning annotation of the corrected labeled data set is performed to obtain the second labeled data set, which provides more accurate and targeted data support for Micro LED defect detection, and helps to improve the accuracy and performance of defect detection.

[0067] See also Figure 6 A sixth embodiment of a defect detection method in an embodiment of the present invention includes:

[0068] 601. Perform a nonlinear interpolation operation on the third defect sample set based on a preset interpolation algorithm to obtain an interpolated defect sample set;

[0069] 602. Perform an upsampling network structure operation on the interpolation defect sample set based on a deep convolutional generative adversarial network to obtain a second expanded sample set;

[0070] 603. Generate a corrected defect sample set according to the first defect sample set, the second defect sample set, the third defect sample set, and the second expanded sample set;

[0071] 604. Adjust and configure the first deep learning model based on the preset first parameter to obtain a second deep learning model;

[0072] 605. Training the second deep learning model according to the corrected defect sample set to obtain a third deep learning model;

[0073] 606. Adjust and configure the third deep learning model based on a preset convolutional dilation method and a preset second parameter to obtain a fourth deep learning model;

[0074] 607. Perform iterative optimization training on the fourth deep learning model based on the first labeled data set, the second labeled data set and preset conditions to obtain a Micro LED defect detection model.

[0075] In this embodiment, in step 601, nonlinear interpolation operations and upsampling network structure operations are performed on the third defect sample set to improve the resolution of the micro defect image; in step 603, the first defect sample set, the second defect sample set, the third defect sample set and the second expanded sample set are added to the second deep learning model for training, thereby improving the multi-scale defect detection capability of the model; in step 604, the configuration of the first deep learning model includes configuring the S2-MLPv2 attention mechanism, increasing the number of channels of the model to 3*channel, and adjusting the first deep learning model includes performing different shift operations on each channel to increase the weight of the micro defect features to obtain the second deep learning model, which can increase the learning ability of the second deep learning model and improve the detection accuracy of the second deep learning model; in step 606, within a camera field of view, an incomplete chip may appear at the edge of the field of view, causing the model to detect the incomplete chip as unqualified, increasing the misjudgment rate. Solution: Iteratively optimize and train the fourth deep learning model by combining the second parameter and the convolution hole method with the first and second labeled data sets, including the following adjustments and configuration operations: determine the number of pixels occupied according to the size of the complete chip (the second parameter), pad the original image, and then complete the incomplete chip according to the number of missing pixels; increase the model's learning ability for incomplete chips by increasing the receptive field of the model, such as adjusting the size of the convolution kernel based on the second parameter again or using methods such as hole convolution to adjust the size of the convolution kernel; in addition, the loss function can also be adjusted to increase the weight of chip edge detection, thereby improving the model's ability to distinguish incomplete chips. The third defect sample set is processed by the preset interpolation algorithm and deep convolution generative adversarial network to obtain the second expanded sample set, enriching the sample data; combining multiple sample sets to generate a corrected defect sample set, providing a more comprehensive data basis for model training. The first deep learning model was gradually adjusted and configured, and the fourth deep learning model was obtained after multiple rounds of training, which improved the accuracy and robustness of the model. The fourth deep learning model was iteratively optimized and trained using different labeled data sets and preset conditions. The final Micro LED defect detection model can detect defects more accurately, provide strong guarantees for Micro LED production quality control, and help improve product reliability and market competitiveness.

[0076] See also Figure 7 A seventh embodiment of a defect detection method in an embodiment of the present invention includes:

[0077] 701. Construct a test set and a training set according to the first annotated data set and the second annotated data set;

[0078] 702. Train a fourth deep learning model based on the training set and preset conditions to obtain a first training result;

[0079] 703. Testing the first training result according to the test set and preset conditions to obtain a test result;

[0080] 704. Analyze the test results;

[0081] 705. When the test result is that the test is successful, a Micro LED defect detection model is obtained according to the first training result;

[0082] 706. On the contrary, the fourth deep learning model is trained again based on the training set and the preset conditions.

[0083] In this embodiment, in steps 701 to 702, in order to improve the generalization and robustness of model prediction, a cross-validation method is used to randomly shuffle the first labeled data set and the second labeled data set to construct a training set and a test set, and then train the fourth deep learning model; in steps 702 and 703, the preset conditions include optimizing and compressing the model by transfer learning and knowledge distillation, reducing the model calculation amount and storage cost, and manually correcting the defect data of the test set according to the model training results (manual intervention annotation); so that the integrity of the data set and the robustness of the model are continuously strengthened, and the recall rate and precision rate of the model detection can be greatly improved for the detection of Micro LED microscopic defects of various shapes that are prone to errors and easy to ignore, so that industrial defect detection has a lower missed detection rate and false positive rate; by using a cross-validation method, the fourth deep learning model is iteratively optimized for multiple times, and finally a Micro LED defect detection model is obtained, thereby improving the detection accuracy of the model. By constructing a training set and a test set for the first labeled data set and the second labeled data set, a more representative data division is provided for the training of the fourth deep learning model, which effectively improves the generalization and robustness of the model prediction; the fourth deep learning model is trained based on the training set and preset conditions, while reducing the model calculation amount and storage cost, the data set integrity and model robustness are continuously strengthened; the first training result is tested according to the test set and preset conditions, which can greatly improve the recall rate and precision of the model detection, reduce the missed detection rate and the false positive rate, and optimize the fourth deep learning model through multiple iterations. The final Micro LED defect detection model has higher detection accuracy, providing reliable technical support for MicroLED industrial defect detection.

[0084] See also Figure 8 An eighth embodiment of a defect detection method in an embodiment of the present invention includes:

[0085] 801. Determine whether the prediction confidence of the third labeled data set is greater than a preset first threshold;

[0086] 802. When the prediction confidence of the third annotated data set is greater than a preset first threshold, calculating the overlap of all adjacent bounding boxes of the third annotated data set;

[0087] 803. Determine whether the degree of overlap is greater than a preset second threshold;

[0088] 804. When the overlap is greater than a second threshold, merging all adjacent bounding boxes to obtain a corrected annotation data set;

[0089] 805. When the prediction confidence of the third labeled data set is less than the first threshold, the targets with low confidence in the third labeled data set are removed according to a preset third threshold to obtain a corrected labeled data set.

[0090] In this embodiment, in step 805, for the bounding boxes with low confidence predicted by the model, they can be eliminated according to the preset third threshold, and the defect categories and bounding box coordinate information with high confidence are retained. By judging the relationship between the prediction confidence of the third annotated data set and the preset first threshold, the data can be accurately screened; for the case where the confidence meets the standard, the overlap of adjacent bounding boxes is further considered, and when it is greater than the second threshold, they are merged, which can optimize the data structure and reduce redundancy; and when the confidence is less than the threshold, the low-confidence targets are removed according to the third threshold to improve the data quality; in this way, the final corrected annotated data set is more accurate and reliable, providing a high-quality data foundation for subsequent Micro LED defect detection and other related work, helping to improve the overall work results.

[0091] A defect detection method in an embodiment of the present invention is described above. A defect detection device in an embodiment of the present invention is described below. Fig. 9 In one embodiment of the present invention, a defect detection device includes:

[0092] The expanded sample set acquisition module 1 is used to acquire a historical defect sample set from a database, and perform data expansion on the historical defect sample set using a data enhancement method to obtain a first expanded sample set;

[0093] A first defect sample set generating module 2, configured to generate a first defect sample set according to a historical defect sample set and a first expanded sample set;

[0094] A sample set convolution preprocessing module 3, used for preprocessing according to the first defect sample set and a preset deep convolutional neural network model to obtain a second defect sample set and a third defect sample set;

[0095] A defect sample set fusion modeling module 4 is used to construct a Micro LED defect detection model according to the first defect sample set, the second defect sample set, the third defect sample set and preset conditions;

[0096] The prediction result generation module 5 is used to obtain a real-time defect sample set and predict the real-time defect sample set according to the Micro LED defect detection model to obtain a prediction result.

[0097] The expanded sample set acquisition module 1 expands the historical defect sample set through data enhancement to provide a rich data source for the follow-up; the first defect sample set generation module 2 generates a first defect sample set covering more defect situations based on it to strengthen the data foundation. The sample set convolution preprocessing module 3 further processes with the help of a deep convolutional neural network model to obtain a key sample set. The defect sample set fusion modeling module 4 uses this to construct a high-precision Micro LED defect detection model. The prediction result generation module 5 uses the model to predict the real-time sample set and quickly output accurate results. The overall model not only improves the adaptability and detection accuracy of complex defects, but also saves time and energy, providing strong support for industrial production decision-making and sustainable development.

[0098] Fig.10 is a schematic diagram of the structure of a defect detection device provided by an embodiment of the present invention. The defect detection device 600 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 613 (for example, one or more processors) and a memory 620, and one or more storage media 630 (for example, one or more massive storage devices) storing application programs 633 or data 632. Among them, the memory 620 and the storage medium 630 may be short-term storage or permanent storage. The program stored in the storage medium 630 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the defect detection device 600. Furthermore, the processor 610 may be configured to communicate with the storage medium 630, and execute a series of instruction operations in the storage medium 630 on the defect detection device 600 to implement the steps of the defect detection method provided in the above-mentioned method embodiments.

[0099] The defect detection device 600 may further include one or more power supplies 640, one or more wired or wireless network interfaces 650, one or more input and output interfaces 660, and / or one or more operating systems 631, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. It will be appreciated by those skilled in the art that Fig.10 The defect detection device structure shown does not constitute a limitation on the defect detection device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0100] Finally, it should be noted that the above description is only a preferred example of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A defect detection method, characterized in that: include: Acquire a historical defect sample set from a database, and perform data augmentation on the historical defect sample set using a data augmentation method to obtain a first augmented sample set; Generate a first defect sample set according to the historical defect sample set and the first expanded sample set; Preprocessing the first defect sample set by using a preset deep convolutional neural network model to obtain a second defect sample set and a third defect sample set; A MicroLED defect detection model is constructed according to the first defect sample set, the second defect sample set, the third defect sample set and preset conditions; A real-time defect sample set is obtained, and the real-time defect sample set is predicted according to the Micro LED defect detection model to obtain a prediction result.

2. A defect detection method according to claim 1, characterized in that: The preprocessing of the first defect sample set by a preset deep convolutional neural network model to obtain the second defect sample set and the third defect sample set includes: The original deep learning model is constructed according to the preset deep convolutional neural network model; Training the original deep learning model according to the first defect sample set to obtain a first deep learning model; Performing labeling and segmentation processing on the first defect sample set by using the first deep learning model to obtain a second defect sample set; The second defect sample set is annotated with microscopic defects to obtain a third defect sample set.

3. A defect detection method according to claim 2, characterized in that: The step of training a preset deep convolutional neural network model according to the first defect sample set to obtain a first deep learning model includes: Acquire a first to-be-annotated data set from a first defect sample set according to a preset first ratio; Perform defect partition annotation on the first data set to be annotated to obtain a first annotated data set and label data; A deep convolutional neural network model is trained based on the first annotated data set and the label data to obtain a first deep learning model.

4. A defect detection method according to claim 3, characterized in that: The labeling and segmenting process of the first defect sample set by the first deep learning model to obtain the second defect sample set includes: Acquire a second to-be-annotated data set from the first defect sample set according to a preset second ratio; Annotating the second to-be-annotated data set based on the first deep learning model to obtain a second annotated data set; Acquire a second ROI region according to a second labeled data set; Acquire a first ROI region according to the first labeled data set; A segmentation operation is performed on the first defect sample set based on the first deep learning model, the first ROI region, and the second ROI region to obtain a second defect sample set.

5. A defect detection method according to claim 4, characterized in that: The step of labeling the second to-be-labeled data set based on the first deep learning model to obtain a second labeled data set includes: Predicting the second to-be-annotated data set based on the first deep learning model to obtain a first prediction result; Analyzing the first prediction result to obtain a defect category, and comparing the defect category with a preset actual defect category to obtain a comparison result; Determine whether the comparison results are consistent; When the comparison results are consistent, a third labeled data set is obtained according to the prediction results; When the comparison results are inconsistent, the prediction results are corrected using a preset quantitative standard to obtain a third annotated data set; Performing operations of removing low confidence and merging adjacent bounding boxes on the third labeled data set to obtain a corrected labeled data set; The defect partitions are annotated on the corrected annotated data set to obtain a second annotated data set.

6. A defect detection method according to claim 4, characterized in that: The Micro LED defect detection model is constructed according to the first defect sample set, the second defect sample set, the third defect sample set and preset conditions, including: Performing a nonlinear interpolation operation on the third defect sample set based on a preset interpolation algorithm to obtain an interpolated defect sample set; Based on a preset deep convolutional generative adversarial network, an upsampling network structure operation is performed on the interpolation defect sample set to obtain a second expanded sample set; Generate a corrected defect sample set according to the first defect sample set, the second defect sample set, the third defect sample set, and the second expanded sample set; Adjusting and configuring the first deep learning model based on the preset first parameters to obtain a second deep learning model; Training the second deep learning model according to the corrected defect sample set to obtain a third deep learning model; Adjusting and configuring the third deep learning model based on a preset convolutional dilation method and a preset second parameter to obtain a fourth deep learning model; The fourth deep learning model is iteratively optimized and trained based on the first labeled data set, the second labeled data set and preset conditions to obtain a Micro LED defect detection model.

7. A defect detection method according to claim 6, characterized in that: The iterative optimization training of the fourth deep learning model based on the first labeled data set, the second labeled data set and the preset conditions to obtain the Micro LED defect detection model includes: Construct a test set and a training set according to the first annotated data set and the second annotated data set; Training the fourth deep learning model based on the training set and the preset conditions to obtain a first training result; Testing the first training result according to the test set and preset conditions to obtain a test result; Analyze the test results; When the test result is that the test is successful, a Micro LED defect detection model is obtained according to the first training result; On the contrary, the fourth deep learning model is trained again based on the training set and preset conditions.

8. A defect detection method according to claim 5, characterized in that: The step of removing low confidence boxes and merging adjacent bounding boxes from the third labeled data set to obtain a corrected labeled data set includes: Determine whether the prediction confidence of the third labeled data set is greater than a preset first threshold; When the prediction confidence of the third labeled data set is greater than the first threshold, calculating the overlap of all adjacent bounding boxes of the third labeled data set; Determining whether the degree of overlap is greater than a preset second threshold; When the overlap is greater than a second threshold, all adjacent bounding boxes are merged to obtain a corrected annotation dataset; When the prediction confidence of the third labeled data set is less than a preset threshold, the targets with low confidence in the third labeled data set are removed according to the preset third threshold to obtain a corrected labeled data set.

9. A defect detection device, characterized in that: include: An expanded sample set acquisition module is used to acquire a historical defect sample set from a database, and perform data expansion on the historical defect sample set using a data enhancement method to obtain a first expanded sample set; A first defect sample set generating module, used to generate a first defect sample set according to the historical defect sample set and the first expanded sample set; A sample set convolution preprocessing module, used for preprocessing according to the first defect sample set and a preset deep convolutional neural network model to obtain a second defect sample set and a third defect sample set; A defect sample set fusion modeling module, used to construct a Micro LED defect detection model according to the first defect sample set, the second defect sample set, the third defect sample set and preset conditions; The prediction result generation module is used to obtain a real-time defect sample set and predict the real-time defect sample set according to the Micro LED defect detection model to obtain a prediction result.

10. A defect detection device, characterized in that: The defect detection device comprises: a memory and at least one processor, wherein the memory stores instructions; At least one of the processors calls the instructions in the memory to enable the defect detection device to perform each step of a defect detection method as described in any one of claims 1-8.

Images