A Micro LED Chip Defect Detection Method Based on MLCT-YOLO

Abstract

The present invention relates to the technical field of computer vision image processing. More specifically, it relates to a method for detecting Micro LED chip defects based on MLCT-YOLO. The key points of its technical solution are as follows: S1. Construct a multi-scale Micro LED data set; S2. Establish a deep neural network MLCT-YOLO for chip defect localization and classification; S3. Deploy the deep neural network MLCT-YOLO to edge devices. The present invention designs an end-to-end deep neural network MLCT-YOLO model, and its model size, number of parameters, and computational complexity are greatly optimized compared with other algorithms. Design the bottleneck block MA-Bottleneck to achieve better localization and classification effects while saving computational resources. Design the class balance loss CB-BCE Loss to make the training cost more reasonably allocated and improve the model's ability to extract important features at the same time.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision image processing, and more specifically, it relates to a method for detecting Micro LED chip defects based on MLCT-YOLO. Background Art

[0002] Chip defect detection is an important quality control technology in the LED display industry. During the production process, different Micro LED chip defects will occur due to technological deficiencies, resulting in a decrease in the yield rate of later products. In previous production, manual visual inspection methods were often used. However, this method is not only inefficient but also very prone to misdetection.

[0003] With the continuous development of neural networks and the improvement of hardware computing capabilities, a series of constructive major breakthroughs have been made in the field of object detection. Among them, the single-stage object detection algorithm YOLO series has achieved a balance between inspection speed and accuracy. Glenn Jocher et al. proposed the YOLOv5 algorithm in 2020, achieving state-of-the-art performance with higher accuracy and faster speed. Its powerful real-time processing ability and low hardware requirements make it easy to be transplanted into mobile devices.

[0004] However, the YOLOv5 algorithm still has problems such as a bloated network model and a large number of parameters. It is a challenge to directly use object detection algorithms such as YOLOv5 to detect chip defects in the Micro LED chip dataset. First, the size of Micro LED chips is small. An image of 1920×1080 contains 182 chips. Before feeding the image into YOLOv5, the image needs to be reduced to 640×640. The number of pixels occupied by one chip is too small, and key feature information on the chip surface is easily lost, thus affecting the detection performance. Second, due to experimental conditions, the sample quantity distribution of various defects of the obtained Micro LED chips is extremely uneven, and the training cost cannot be reasonably allocated. Summary of the Invention

[0005] Aiming at the deficiencies of the existing technology, the purpose of the present invention is to provide a method for detecting MicroLED chip defects based on MLCT-YOLO, which has the advantages of high detection efficiency and accurate accuracy.

[0006] The above technical objective of the present invention is achieved through the following technical solutions: A method for detecting Micro LED chip defects based on MLCT-YOLO includes the following steps:

[0007] S1. Construct a multi-scale Micro LED dataset;

[0008] S2. Establish a deep neural network MLCT-YOLO for chip defect location and classification;

[0009] S3. Deploy the deep neural network MLCT-YOLO to edge devices.

[0010] In one embodiment, the step S1 includes the following steps:

[0011] S11. Take images of the chip and manually annotate them as the initial dataset;

[0012] S12. Randomly crop the images in the initial dataset to expand the initial dataset;

[0013] S13. Divide the expanded initial dataset into a training set and a test set, and perform augmentation operations on the images in the training set.

[0014] In one embodiment, the augmentation method adopted in the step S13 is to select image flipping, add random noise, and change image brightness.

[0015] In one embodiment, the ratio of the number of the training set to the test set is 7:1.

[0016] In one embodiment, the step S2 includes the following steps:

[0017] S21. Use CSPDarkNet53 as the backbone network;

[0018] S22. Adopt the form of a feature pyramid connection path aggregation network as the neck network;

[0019] S23. Use CIOU as the bounding box loss function, and use a weighted non-maximum suppression model to determine the ideal target frame.

[0020] In one embodiment, the CSPDarkNet53 includes multiple CSP residual structures.

[0021] In one embodiment, the feature pyramid connection path aggregation network is composed of a combination of multiple convolutional layers and splicing layers, and after passing through the path aggregation network, the low-level features are mapped to high-dimensional and then three feature maps are output.

[0022] In one embodiment, the step S22 further includes the following steps:

[0023] S221. Based on the Transformer model, propose the MA-Bottleneck module to replace the C3 module at the output end of the neck network;

[0024] The bottleneck block MA-Bottleneck constitutes a new convolutional bottleneck layer BO-C3 module, and the BO-C3 module includes a first branch and a second branch;

[0025] The image of the chip input to the first branch passes through a 1×1 convolutional layer, is fed into the MA-Bottleneck module to extract global features, and then is output;

[0026] The image of the chip input to the second branch passes through a 1×1 convolutional layer and then is output, and is concatenated with the image output from the first branch to form a concatenated image;

[0027] The concatenated image is output from the BO-C3 module after being processed by a 1×1 convolution.

[0028] In one embodiment, the CIOU is the class balance loss CB-BCELoss, and the number of effective samples is calculated in a model-agnostic and loss-agnostic manner, and a class balance weight inversely proportional to the number of effective samples is introduced into the original loss function;

[0029] Regarding the number of effective samples as E n , and the total volume of samples as N, then there is:

[0030]

[0031] In the above formula: the hyperparameter β ∈ [0, 1);

[0032] For the input sample X, let its label y ∈ {1, 2, 3..., M}, where M is the total number of all classes, and let the estimated class probability corresponding to the deep neural network MLCT-YOLO model be p = [p1, p2, p3,..., p M T , where p i ∈ [0, 1]. By introducing the number of effective samples E n and the smoothing coefficient β, due to the independence of the class balance term, the model is made agnostic and the loss is made agnostic. The weight factor is used to endow the original loss function BCEWithLogitsLoss of MLCT-YOLO to obtain a new class balance loss CB-BCE Loss, denoted as CB(p, y), and the formula is as follows:

[0033]

[0034] In the above formula: L(p, y) represents any loss function, and n y represents the number of samples of the true class y.

[0035] In one embodiment, the edge device is the NVIDIA Jetson Xavier NX platform.​

[0036] The above-mentioned Micro LED chip defect detection method based on MLCT-YOLO has the following beneficial effects:

[0037] First, a multi-scale Micro LED dataset is constructed to ensure the diversity of the input image size and the richness of the experimental data;

[0038] Second, an end-to-end deep neural network MLCT-YOLO model is designed, and its model size, number of parameters, and computational complexity are greatly optimized compared with other algorithms. The bottleneck block MA-Bottleneck is designed to achieve better localization and classification effects while saving computational resources. The class balance loss CB-BCE Loss is designed to make the training cost more reasonably allocated and improve the model's ability to extract important features at the same time;

[0039] Third, the network model is deployed to the edge device Nvidia Jetson Xavier NX, which has good detection speed and detection performance in the embedded scenario. Description of the Drawings

[0040] Figure 1 is the schematic diagram of the process of this embodiment;

[0041] Figure 2 is the schematic diagram of eight defect categories in this embodiment;

[0042] Figure 3 is the flowchart of the deep neural network MLCT-YOLO in this embodiment;

[0043] Figure 4 is the structural diagram of the convolutional bottleneck layer in this embodiment;

[0044] Figure 5 is the heat map of the three-scale detection layers before and after adding the MA-Bottleneck module in this embodiment;

[0045] Figure 6 is the detection effect diagram of the multi-scale Micro LED dataset in this embodiment;

[0046] Figure 7 is the detection effect diagram after changing the illumination in this embodiment;

[0047] Figure 8 is the real-time display of the detection screen on the external display in this embodiment. Detailed Embodiment

[0048] To enable those skilled in the art to better understand the solution of this application, the following will clearly and completely describe the technical solution in the embodiments of this application in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of this application.

[0049] It should be noted that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances for the embodiments of this application described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0050] A Micro LED chip defect detection method based on MLCT - YOLO, as Figure 1 shown, includes the following steps:

[0051] S1. Construct a multi-scale Micro LED data set;

[0052] Specifically, S11. Take images of the chips and manually label them as the initial data set;

[0053] Take 128 chip images as the initial data set, with a resolution of 1920×1080 for all. Input the 128 chip images into the annotation software, manually label the defects existing in the chips in the chip images. The annotation software automatically generates relevant defect information and saves it as a document. The defect information document is in the txt format file.

[0054] Among them, there are eight types of Micro LEDs to be detected: normal, black chip, electrode missing, continuous crystal, empty chip, foreign object, offset, and deflection. The eight defect categories are as Figure 2 shown. There is an uneven distribution of samples in the eight categories. Among them, the three categories of normal, offset, and deflection account for the majority, far exceeding the number of samples in the other five categories.

[0055] S12. Randomly crop the images in the initial data set to expand the initial data set;

[0056] Due to the need for multi-scale detection and to ensure the diversity of input image sizes, the images in the initial dataset are randomly cropped, and the number of images in the dataset is expanded to 826.

[0057] S13. Divide the expanded initial dataset into a training set and a test set, and perform augmentation operations on the images in the training set.

[0058] By using data augmentation methods such as image flipping, adding random noise, and changing image brightness, the images in the training set are augmented to 1827, so that the number of images in the training set is 7:1 compared to the number of images in the test set.

[0059] S2. Build a deep neural network MLCT-YOLO for chip defect localization and classification.

[0060] Specifically, S21. Use CSPDarkNet53 as the backbone network.

[0061] CSPDarkNet53 includes multiple CSP residual structures, which can strengthen the feature fusion of the deep neural network MLCT-YOLO and prevent gradient disappearance.

[0062] S22. Use the form of a feature pyramid connection path aggregation network as the neck network.

[0063] The feature pyramid connection path aggregation network is composed of multiple convolutional layers and splicing layers. After passing through the path aggregation network, the low-level features are mapped to high-dimensional and then three feature maps are output.

[0064] S221. Based on the Transformer model, a MA-Bottleneck module is proposed to replace the C3 module at the output end of the neck network.

[0065] The bottleneck block MA-Bottleneck forms a new convolutional bottleneck layer BO-C3 module, and the BO-C3 module includes a first branch and a second branch.

[0066] The image of the chip input to the first branch passes through a 1×1 convolutional layer, is sent into the MA-Bottleneck module to extract global features, and then is output.

[0067] The image of the chip input to the second branch passes through a 1×1 convolutional layer and then is output, and is spliced with the image output by the first branch to form a spliced image.

[0068] The spliced image is output from the BO-C3 module after being processed by a 1×1 convolution.

[0069] Introduce a multi-scale receptive field to enhance the generalization ability of the network for object recognition. The heatmaps of the three-scale detection layers before and after adding the MA-Bottleneck module to MLCT-YOLO are introduced, as Figure 5 shown. After improving the module, most of the highlighted areas are concentrated inside the chip, indicating that the improved module increases the probability of the detection layer extracting the chip area features, pays more attention to the information on the chip surface rather than the edge information, and can achieve better positioning and classification effects while saving computing resources.

[0070] S23. Adopt CIOU as the bounding box loss function and use the weighted non-maximum suppression model to determine the ideal target frame.

[0071] CIOU is the class balance loss CB-BCELoss, and the effective number of samples is calculated in a model-agnostic and loss-agnostic manner. A class balance weight inversely proportional to the effective number of samples is introduced into the original loss function;

[0072] Regarding the effective number of samples as E n , and the total volume of samples as N, then there is:

[0073]

[0074] In the above formula: the hyperparameter β ∈ [0, 1);

[0075] For the input sample X, let its label y ∈ {1, 2, 3..., M}, where M is the total number of all classes. Let the estimated class probability corresponding to the deep neural network MLCT-YOLO model be p = [p1, p2, p3, …, p M T , where p i ∈ [0, 1]. By introducing the effective number of samples E n and the smoothing coefficient β, due to the independence of the class balance term, the model is made agnostic and the loss is made agnostic. The weight factor is used to endow the original loss function BCEWithLogitsLoss of MLCT-YOLO to obtain a new class balance loss CB-BCE Loss, denoted as CB(p, y), and the formula is as follows:

[0076]

[0077] In the above formula: L(p, y) represents any loss function, and n y represents the number of samples of the true class y.

[0078] Using the CB-BCE Loss function makes the class frequency transition smoother, the training cost is more reasonably allocated, and at the same time, the ability of the model to extract important features is improved.

[0079] ​S3. Deploy the deep neural network MLCT - YOLO to edge devices.

[0080] Among them, the edge device is the NVIDIA Jetson Xavier NX platform, and an external display is used to display the detection screen in real time. The results are as Figure 8 shown. The model can detect chip defects of all categories, without missing or misdetecting, and the frame rate basically maintains above 25 frames, meeting the speed and accuracy requirements of real - time detection.

[0081] The average precision (mAP) values are tested for eight types of Micro LED chip types respectively. The results are shown in Table 1. Except for the blank chip defect, four types of fewer - sample categories have obtained higher mAP values. The mAP for the black chip is 0.941, for the electrode missing is 0.955, for the foreign object is 0.428, and for the crystal connection is 0.995. The total mAP is 0.899. Therefore, the ability of the model to discriminate fewer - sample defects during the training process has been significantly improved.

[0082] Table 1 Comparison of mAP values for eight types of Micro LED chip types

[0083]

[0084]

[0085] Calculate various evaluation indicators of the end - to - end deep neural network MLCT - YOLO model for chip defect localization and classification. The results are shown in Table 2.

[0086] Table 2 Evaluation indicators of different network models

[0087]

[0088] According to the data in Table 2, the mAP of the proposed MLCT - YOLO is 88.9%, and the precision is 93.0%. Compared with the standard YOLO, it has increased by 3.0% and 1.3% respectively. In addition, the model size, the number of parameters, and the computational complexity of MLCT - YOLO have been greatly optimized compared with other algorithms, and the detection speed has also increased from 78 frames of the standard YOLO to 93 frames, indicating its obvious superiority.

[0089] Apply the model to the constructed multi - scale Micro LED dataset. The results are as Figure 6 shown. The MLCT - YOLO model proposed in the present invention can accurately identify all targets and classify them correctly, and the confidence level basically remains above 0.95.

[0090] By changing the types and angles of annular illumination to simulate the impact of changes in illumination conditions on chip detection during the actual production process, the results are asFigure 7 As shown. The end-to-end deep neural network MLCT-YOLO proposed by the present invention can well identify the defects existing in the chip, and the recognition confidence is generally above 0.95. Therefore, the proposed end-to-end deep neural network MLCT-YOLO has strong generalization ability in the defect classification of Micro LED chips.

[0091] The above embodiments only represent several implementation manners of the present invention, and the description thereof is relatively specific and detailed, but it should not be construed as a limitation to the scope of the patent of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several deformations and improvements can still be made, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the appended claims.

Claims

1. A defect detection method for Micro LED chips based on MLCT-YOLO, characterized in that, It includes the following steps: S1. Construct a multi-scale Micro LED dataset; S2. Establish a deep neural network MLCT-YOLO for chip defect localization and classification; S3. Deploy the deep neural network MLCT-YOLO to an edge device; The step S2 includes the following steps: S21. Adopt CSPDarkNet53 as the backbone network; S22. Adopt the form of a feature pyramid connection path aggregation network as the neck network; S23. Adopt CIOU as the bounding box loss function and use a weighted non-maximum suppression model to determine the ideal target frame; The CSPDarkNet53 includes multiple CSP residual structures; The feature pyramid connection path aggregation network is composed of a combination of multiple convolutional and splicing layers. After passing through the path aggregation network, the low-level feature maps are mapped to high-dimensional ones and then three feature maps are output; The step S22 further includes the following steps: S221. Based on the Transformer model, propose the MA-Bottleneck module to replace the C3 module at the output end of the neck network; The bottleneck block MA-Bottleneck constitutes a new convolutional bottleneck layer BO-C3 module, and the BO-C3 module includes a first branch and a second branch; The image of the chip input to the first branch passes through a 1×1 convolutional layer, is fed into the MA-Bottleneck module to extract global features, and then is output; The image of the chip input to the second branch passes through a 1×1 convolutional layer and then is output, and is spliced with the image output by the first branch to form a spliced image; The spliced image is output by the BO-C3 module after being processed by a 1×1 convolution; The CIOU is the class balance loss CB-BCELoss, and the effective number of samples is calculated in a model-agnostic and loss-agnostic manner, and a class balance weight inversely proportional to the effective number of samples is introduced into the original loss function; Regarding the effective number of samples as and the total volume of the samples as we have: ， In the above formula: hyperparameter ; For the input sample , let its label , be the total number of all classes. Let the estimated class probability corresponding to the deep neural network MLCT-YOLO model be , where . By introducing the sample valid number and the hyperparameter , due to the independence of the class balance term, the model is agnostic and the loss is agnostic. The weight factor is used to endow the original loss function BCEWithLogitsLoss of MLCT-YOLO to obtain a new class balance loss CB-BCE Loss, which is denoted as. The formula is as follows: ， In the above formula: represents any loss function, represents the true value category the number of samples.

2. The method for detecting defects of Micro LED chips based on MLCT-YOLO according to claim 1, wherein, The step S1 includes the following steps: S11. Take images of the chip and manually annotate them as the initial dataset; S12. Randomly crop the images of the initial dataset to expand the initial dataset; S13. Divide the expanded initial dataset into a training set and a test set, and perform augmentation operations on the images of the training set.

3. The method for defect detection of Micro LED chips based on MLCT-YOLO according to claim 2, wherein: The augmentation method adopted in the step S13 is to select image flipping, add random noise, and change the image brightness.

4. A Micro LED chip defect detection method based on MLCT-YOLO according to claim 2, characterized in that: The ratio of the number of the training set to the test set is 7:

1.

5. A method for detecting defects of Micro LED chips based on MLCT-YOLO according to claim 1, characterized in that: The edge device is the NVIDIA Jetson Xavier NX platform.

Images