Defect detection model training method and electronic equipment

Abstract

The invention relates to the technical field of deep learning, provides a training method of a defect detection model and electronic equipment, and aims to improve the detection accuracy and reduce the sample labeling work at the same time. According to the method, self-supervision training of a model is carried out by using a large amount of spliced images of unlabeled chips on a production line, and local pixel values of the unlabeled spliced images are processed by taking image blocks with preset sizes as units, so that the purpose of constructing own labels by using data is achieved, the characteristics of learning the chips by fully utilizing a large amount of sample data, and the self-supervision training efficiency is improved. The feature learning richness is improved to improve the accuracy of model defect detection, meanwhile, the convergence speed of model training can be increased, and the preliminarily trained model has good chip feature learning ability, so that the learning of the model on defect features can be guided by using a small number of labeled spliced images, and the defect detection accuracy is improved. And the manual labeling workload and the labeling time are reduced.

Description

Background Art

[0002] LED chips are limited by the production process, and damaged or defective chips are inevitable. Therefore, defect detection is required to improve the delivery quality of chips.

[0003] Generally, a single wafer is about 6-8 inches in size, and contains tens of thousands to hundreds of thousands of LED chips, up to millions. These chips are of the same size and are neatly arranged in rows and columns on the wafer, such as Figure 1 As shown, the size of a single LED chip on the wafer is extremely small.

[0004] Initially, defect detection of industrial products mainly relied on manual visual inspection, but since chips are tiny and large in number, they need to be seen under a microscope, which makes human eyes easily fatigued and inefficient. With the development of computer vision (CV) and deep learning technology, defect detection models based on deep learning are gradually being used in industry.

[0005] LED chip defect detection is usually performed on a wafer basis. Each chip on the wafer, as a separate individual, needs to give a specific defect type and location, but there is no need to give information such as the specific location and size of the defect on each chip. Since there are hundreds of thousands of chips on a wafer, the size of a single chip on the entire wafer is too small, and the detection effect of conventional models is poor. For some chips where defects are not obvious, a large number of labeled samples are often required for the model to learn such defects. In addition, the location and shape of defects on each chip vary greatly, and it is difficult for the model to effectively learn defect characteristics under limited sample conditions, resulting in a large missed detection rate and over-detection rate. Manual review is required before delivery, and collecting a large number of labeled samples for the model to learn will consume a lot of manpower and time costs, which is not very friendly to small and medium-sized enterprises.

[0006] Therefore, improving model detection performance while reducing the workload required for model training is an urgent problem to be solved in chip defect detection technology. Summary of the invention

[0007] The embodiments of the present application provide a defect detection model training method and electronic device for improving the defect detectability of the model while reducing the labeling workload.

[0008] In a first aspect, an embodiment of the present application provides a method for training a defect detection model, comprising:

[0009] Acquire an initial image set, wherein the initial image set includes multiple image subsets, each image subset corresponds to a wafer model, each image subset includes respective stitched images of multiple LED chips on the wafer of the corresponding model, and each stitched image is generated based on chip images of a single chip under multiple light sources;

[0010] Processing local pixel values ​​of the first stitched image in units of image blocks of a preset size to generate a first training sample set; wherein the first stitched image is any stitched image in the initial image set;

[0011] Iteratively training the defect detection model according to the first training sample set to obtain an initial defect detection model;

[0012] Acquire a second training sample set, where the second training sample set is generated by annotating defects on the spliced ​​images selected from the initial image set, and the second training sample set covers spliced ​​images of all defect types and defect-free LED chips;

[0013] The initial defect detection model is iteratively trained according to the second training sample set to obtain a target defect detection model.

[0014] The beneficial effects of the above technical solution are as follows: a self-supervised algorithm is used to train the model using an unlabeled initial image set generated by stitching images of LED chips on wafers of various models on the production line under multiple light sources. Since each stitched image in the first training sample set is processed with local pixel values ​​in units of image blocks of preset sizes, the purpose of constructing its own label with unlabeled data is achieved, thereby making full use of the large amount of sample data produced daily on the production line for preliminary chip feature learning, improving the richness of feature learning to improve the accuracy of model defect detection, and accelerating the convergence speed of model training. In addition, a supervised algorithm is used to train the model again using a second labeled training sample set generated by stitching images of selected LED chips covering all defect types and defect-free chips, thereby guiding the model to learn defect features through a small amount of labeled sample data, thereby reducing the workload and time of manual labeling while achieving high detection accuracy.

[0015] Optionally, the processing of local pixel values ​​of the first spliced ​​image in units of image blocks of a preset size to generate a first training sample set includes:

[0016] Dividing the first stitched image into a plurality of p*p image blocks, wherein p is preset according to the number of downsampling layers and the downsampling multiple of each layer in the defect detection model;

[0017] Setting pixel values ​​in some image blocks of the plurality of image blocks to preset values ​​to generate a first training sample;

[0018] The first training sample is stored in the first training sample set.

[0019] The beneficial effect of the above technical solution is: by setting the pixel values ​​in some image blocks among the multiple image blocks in the first spliced ​​image to preset values, the purpose of constructing its own label for the unlabeled first spliced ​​image is achieved, thereby realizing a self-supervised training method, and because the size of the image block is pre-set according to the number of downsampling layers in the model and the downsampling multiples of each layer, in this way, after the first spliced ​​image is downsampled in the feature extraction process, the information is retained in the last layer, thereby improving the accuracy of model feature learning.

[0020] Optionally, each stitched image is a tiled stitching of chip images of a single chip under multiple light sources, and the method further includes:

[0021] The tiling mode of chip images under multiple light sources in the target stitched image is converted into a channel-based stitching mode to reduce the size of the model input; wherein the target stitched image is the first stitched image or any second stitched image in the second training sample set.

[0022] The beneficial effects of the above technical solution are as follows: compared with the tiling splicing method, the channel-based splicing method can reduce the size of the input image, thereby reducing the number of convolutions of the model and improving the speed of model training. Moreover, the increase in the number of channels, thanks to the optimized acceleration of the convolution operation in the GPU, will hardly increase the calculation time, further improving the speed of model training.

[0023] Optionally, the iterative training of the defect detection model according to the first training sample set to obtain an initial defect detection model includes:

[0024] Multiple first training samples in the first training sample set are batch-inputted into the defect detection model for iterative training, and a defect detection model that meets a preset convergence condition is used as an initial defect detection model, wherein each iterative training process is as follows:

[0025] Using a sparse convolution kernel to perform a convolution operation on a first portion of data in the first training sample to extract chip features of the first training sample; wherein the first portion of data corresponds to an image block in the first training sample whose pixel values ​​are not set to preset values;

[0026] Determine the predicted pixel value of the second part of the data in the first training sample according to the chip feature, and determine the first loss value of the defect detection model in combination with the real pixel value of the second part of the data; wherein the second part of the data corresponds to the image block in which the pixel value in the first training sample is set to a preset value;

[0027] The network parameters of the defect detection model are adjusted according to the first loss value.

[0028] The beneficial effect of the above technical solution is: since the first training sample contains two parts, one is a pixel value that is not set to a preset value and the other is set to a preset value, based on the feature learning of the part that is not set to a preset value, the pixel value of the part that is set to a preset value is predicted, so that the model can better extract chip features, thereby improving the accuracy of defect detection. At the same time, the use of sparse convolution kernels for feature extraction can speed up feature learning, thereby speeding up model convergence.

[0029] Optionally, the iteratively training the initial defect detection model according to the second training sample set to obtain a target defect detection model includes:

[0030] Multiple second training samples in the second training sample set are batch-inputted into the initial defect detection model for iterative training, and the initial defect detection model that meets the preset convergence condition is used as the target defect detection model, wherein each iterative training process is as follows:

[0031] Using a common convolution kernel to perform a convolution operation on the second training sample to extract chip features of the second training sample;

[0032] Determine a predicted label of the second training sample according to the chip feature, and determine a second loss value of the initial defect detection model in combination with a true label of the second training sample;

[0033] The network parameters of the initial defect detection model are adjusted according to the second loss value.

[0034] The beneficial effect of the above technical solution is as follows: since the initial defect detection model has been preliminarily trained based on a large number of unlabeled spliced ​​images of LED chips, the initial defect detection model can fully extract chip features from the images. In this way, only a small number of labeled spliced ​​images are needed to guide the further learning of the initial defect detection model, so that the model can better learn the characteristics of various defect types and defect-free LED chips, accelerate the convergence speed of model training and improve the defect detection effect, while reducing the workload and labeling time of manual labeling.

[0035] Optionally, the initial defect detection model includes at least two feature processing stages, and the network parameters of the first feature processing stage are frozen during the training of the initial defect detection model;

[0036] The iterative training of the initial defect detection model according to the second training sample set to obtain a target defect detection model includes:

[0037] For any second training sample in the second training sample set, the second training sample and the third training sample form a training sample pair; wherein the third training sample is a spliced ​​image of a defect-free LED chip;

[0038] The other feature processing stages of the initial defect detection model are iteratively trained according to a plurality of training samples to obtain the target defect detection model.

[0039] The beneficial effects of the above technical solution are: in order to address the problem that defects of some LED chips are difficult to identify due to low differentiation, small proportion and other reasons, the second training sample in the second training sample set and the third training sample of the defect are combined into a training sample pair for model training, wherein the third training sample serves as prior knowledge for model defect learning, which reduces the difficulty of model learning and thus improves the detection effect of the model. In addition, since the initial defect detection model of the preliminary training can effectively learn the characteristics of the chip, by freezing the network parameters of the first feature processing stage in the initial defect detection model, the good chip feature representation capability can be retained while accelerating the module training speed.

[0040] Optionally, the iterative training of other feature processing stages of the initial defect detection model according to a plurality of training samples to obtain the target defect detection model includes:

[0041] The plurality of training sample pairs are batch-inputted into the initial defect detection model for iterative training, and the initial defect detection model that meets the preset convergence condition is used as the target defect detection model, wherein each iterative training process is as follows:

[0042] Extracting original chip features of the second training sample and reference chip features of the third training sample in the training sample pair based on the first feature processing stage;

[0043] Fusion the original chip features with the reference chip features to obtain enhanced chip features;

[0044] After processing the original chip feature of the second training sample in the training sample pair based on other feature processing stages, the original chip feature is stacked with the enhanced chip feature to obtain the target chip feature;

[0045] Determine a predicted label of the second training sample according to the target chip feature, and determine a third loss value in combination with a true label of the second training sample;

[0046] The network parameters of the other feature processing stages are adjusted according to the third loss value.

[0047] The beneficial effect of the above technical solution is: by combining the original chip features of the second training sample and the reference chip features of the third training sample, the significance of the defects in the second training sample can be improved. In this way, on the basis of retaining the original chip features, the defect features can be enhanced to improve the defect recognition ability, thereby reducing the number of labeled samples required for model training, and further reducing the labeling workload and labeling time.

[0048] Optionally, the stitched image is generated by:

[0049] For any LED chip, the chip images of the LED chip under multiple light sources are tiled at set intervals in a preset order, and each preset interval is filled to generate the spliced ​​image.

[0050] The beneficial effects of the above technical solution are as follows: since different defect types present different states under different light sources, the significance of chip defects is improved by splicing chip images under multiple light sources, and compared with using chip images under different light sources alone, the model training process is simplified and the model training speed is improved.

[0051] In a second aspect, an embodiment of the present application provides a method for detecting defects in an LED chip, comprising:

[0052] Acquire partial wafer images taken from multiple viewing angles under different light sources, wherein the partial wafer images taken from the multiple viewing angles constitute a complete wafer image, and each partial wafer image contains multiple LED chips;

[0053] For any single viewing angle among the multiple viewing angles, local wafer images under different light sources corresponding to the single viewing angle are stitched together to generate a multi-light source stitched image.

[0054] Classifying the multi-light source stitched image to output the position coordinates of each chip to be detected, and cutting out the chip image of each chip to be detected from the multi-light source stitched image according to the position coordinates;

[0055] splicing chip images corresponding to the chip to be detected under different light sources to obtain an image of the chip to be detected;

[0056] Defect detection is performed on the image of the chip to be inspected using a target defect detection model, wherein the target defect detection model is trained based on any one of the methods described in the first aspect.

[0057] In a third aspect, an embodiment of the present application provides a training device for a defect detection model, comprising:

[0058] An acquisition module is used to acquire an initial image set and a second training sample set; wherein the initial image set includes multiple image subsets, each image subset corresponds to a wafer model, each image subset includes respective stitched images of multiple LED chips on the wafer of the corresponding model, each stitched image is generated based on chip images of a single chip under multiple light sources, the second training sample set is generated by marking defects on the stitched images selected from the initial image set, and the second training sample set covers stitched images of all defect types and defect-free LED chips

[0059] An image processing module, used for processing local pixel values ​​of the first stitched image in units of image blocks of a preset size to generate a first training sample set; wherein the first stitched image is any stitched image in the initial image set;

[0060] A model training module is used to iteratively train the defect detection model according to the first training sample set to obtain an initial defect detection model; and iteratively train the initial defect detection model according to the second training sample set to obtain a target defect detection model.

[0061] In a fourth aspect, an embodiment of the present application provides an electronic device, including a processor, a memory, and a communication interface, wherein the communication interface, the memory, and the processor are connected via a bus;

[0062] The communication interface is configured to send and receive data;

[0063] The memory stores a computer program, and the processor executes the steps of the method described in any one of the first aspect and the second aspect according to the computer program.

[0064] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed, the steps of any chip defect detection method provided in the first aspect can be implemented.

[0065] The technical effects brought about by any one of the implementation methods in the second to fifth aspects can refer to the technical effects brought about by the corresponding implementation method in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 A schematic diagram of a wafer provided in an embodiment of the present application;

[0067] Figure 2 The main process of LED chip appearance defect detection provided by the embodiment of the present application;

[0068] Figure 3A system architecture diagram of LED chip defect detection provided in an embodiment of the present application;

[0069] Figure 4 A flow chart for constructing training samples provided in the embodiment of the present application;

[0070] Figure 5 Schematic diagram of the shooting process of a local area image;

[0071] Figure 6 It is a schematic diagram of a local area image taken from multiple viewing angles under a single light source;

[0072] Figure 7 and Figure 8 A schematic diagram of a stitched image of a single chip;

[0073] Fig. 9 A training method flow of a defect detection model provided in an embodiment of the present application;

[0074] Fig.10 This is a schematic diagram of splicing format conversion;

[0075] Fig.11 The training process of the defect detection model provided in the embodiment of the present application;

[0076] Fig.12 The network structure of the defect detection model provided in the embodiment of the present application;

[0077] Fig.13 The training process of the initial defect detection model provided in the embodiment of the present application;

[0078] Fig.14 A training process of another initial defect detection model provided in an embodiment of the present application;

[0079] Fig.15 Schematic diagram of the training process of the initial defect detection model;

[0080] Fig.16 It is a defect detection method process based on a target defect detection model;

[0081] Fig.17 Another defect detection method flow based on the target defect detection model;

[0082] Fig.18A A complete wafer image recovery process after defect detection provided in an embodiment of the present application;

[0083] Fig.18B A schematic diagram of the restoration effect of a complete wafer image is provided for the embodiment of the present application;

[0084] Fig.19A structural diagram of a defect detection model training device provided in an embodiment of the present application;

[0085] Fig. 20 A structural diagram of a defect detection device provided in an embodiment of the present application;

[0086] Fig.21 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0087] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0088] Based on the exemplary embodiments shown in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application. In addition, although the disclosure in this application is introduced according to one or several exemplary examples, it should be understood that each aspect of these disclosures can also constitute a complete technical solution separately.

[0089] The following is an overview of the design concept of the embodiments of the present application in conjunction with application scenarios.

[0090] Generally, the production cycle of an LED chip production line is about 2-4 minutes per wafer. The entire process of taking pictures, testing, filing, and outputting results must be completed. Defect detection is required to be completed within 1-3 minutes, so there are great challenges in terms of detection accuracy and speed.

[0091] like Figure 2 As shown in the figure, the main process of using a deep learning-based target detection model to detect appearance defects in LED chips is to fix the entire wafer and illuminate it with different light sources. An industrial camera is used to capture wafer images under different light sources. The LED chips are then detected and positioned based on the wafer images under different light sources. Sample data is collected and manually labeled based on the detection and positioning results to train the model. Finally, the trained defect detection model is deployed on the quality inspection equipment to identify the defect types of LED chips and their locations on the wafer.

[0092] Since defects are mainly targeted at chips, a single chip may have multiple defects, and there are hundreds of thousands of chips on each wafer. If the target detection model based on deep learning is used to identify chip defects on the entire wafer image, the defect of each chip is too small compared to the wafer, resulting in a defect miss detection rate of about 300ppm and a pass detection rate greater than 0.2%, which cannot meet the delivery quality requirements of LED chips. If defect detection is performed separately on each chip image, the detection time of hundreds of thousands of chips is unacceptable for the production line. Among them, a low miss detection rate can ensure that there are almost no defects in the output chip products, ensuring the quality of product delivery. A low pass detection rate is beneficial to reduce the losses caused by false detection and improve product yield. In order to achieve a higher miss detection index to achieve high-accuracy defect detection, it is necessary to use the powerful feature extraction and learning capabilities of deep learning. However, deep learning models require a large amount of labeled data for training, and labeled data often consumes a lot of manpower and time costs. In addition, since the chips are too small, a large number of annotations will cause visual fatigue to cause annotation errors, thereby affecting the accuracy of the model.

[0093] In view of this, in order to address the problem that a large amount of labeled data is required when a deep learning model performs high-precision, fully automatic appearance defect detection on LED chips, an embodiment of the present application provides a training method for a defect detection model. The method is based on a self-supervised learning algorithm. The model constructs a large amount of unlabeled data based on LED chips on wafers of different models on the production line for feature learning, and then uses a small amount of labeled data to perform further feature learning on the defect type. This reduces the demand for labeled data, saves labor costs, and speeds up the model launch while achieving high detection accuracy.

[0094] See also Figure 3, is a system architecture diagram for LED chip defect detection provided in an embodiment of the present application, including four parts: training sample generation, self-supervised feature learning, defect detection model training, and defect detection model deployment. The training sample generation part includes an original chip database and a defect annotation database, wherein the original chip database stores a large number of spliced ​​images of LED chips on wafers of different models on the production line under multiple light sources, and the defect annotation database stores spliced ​​images generated after manually annotating a small amount of data in the original chip database. By converting the format of the splicing method of the chip images corresponding to multiple light sources in the spliced ​​image, a first unlabeled training sample set and a second labeled training sample set can be obtained. The self-supervised feature learning part constructs self-supervised label data based on the first training sample set, and realizes the prediction of chip features by feature learning of the self-supervised label data. The defect detection model training part is based on the training sample set, and the model after self-supervised learning is trained again to learn various defect types, thereby obtaining a defect detection model that can detect various defects. The defect detection model deployment part is used to deploy the trained defect detection model to the production line equipment to perform defect detection on LED chips.

[0095] Before training the defect detection model, training samples need to be prepared. Due to the large amount of data on the production line, the training samples of the defect detection model are constructed based on the LED chips on different models of wafers on the production line, such as Figure 4 As shown, the training sample construction process provided in the embodiment of the present application mainly includes the following steps:

[0096] S401: For any type of wafer, local area images of the wafer are captured from multiple viewing angles under different light sources.

[0097] Since there are various types of defects in the appearance of chips, such as dirt, scratches, needle marks, off-cut, damage to various components, etc., and different types of defects appear differently under different light sources, for any type of wafer, the complete wafer is used as input, and local area images of the wafer are taken from multiple perspectives under different light sources. Among them, the type and number of light sources can be determined according to the model of the wafer, usually 2-4 light sources.

[0098] In some embodiments, after determining the light source corresponding to the wafer, the camera is controlled to shoot different areas of the wafer in a set order (such as a Z-order). The shooting area covers the entire wafer, and each area generates a local area image under a different light source. Each wafer area shot corresponds to a field of view of the camera, that is, a viewing angle. There is an overlapping part of the chip between two adjacent viewing angles. In this way, the local area image shot at each viewing angle contains multiple LED chips on a wafer, and the local area images shot at multiple viewing angles can form a complete wafer image. There are overlapping areas in the local area images between adjacent viewing angles.

[0099] Take two light sources as an example. Figure 5 As shown, it is a schematic diagram of the process of shooting local area images. After the camera shoots a local area image of the wafer at a first perspective under a first light source, the position of the camera is kept unchanged, that is, the first perspective of the camera is kept unchanged, and the first light source is switched to the second light source. The camera shoots another local area image of the wafer at the first perspective under the second light source. The two local area images correspond to the same wafer area. After completing the shooting of the two light sources at the first perspective, the position of the camera is moved so that the camera shoots a local area image of the wafer at the second perspective under the first light source, wherein the edge areas of the local area image at the first perspective and the local area image at the second perspective overlap. As shown in FIG. Figure 6 Shown are local area images of a single wafer taken from multiple viewing angles under one light source.

[0100] In some embodiments, since wafers of different models have different sizes, the required shooting angles are also different, and the number of shooting angles is positively correlated with the size of the wafer, that is, the larger the size of the wafer, the more shooting angles there are.

[0101] In some embodiments, after shooting is completed, the local area image under each light source is temporarily stored in the memory separately in a lossless format (such as RAW format).

[0102] In some embodiments, since there are a large number of LED chips on the wafer and their sizes are small, a microscope camera with higher precision is used when photographing the wafer.

[0103] In some embodiments, the size of the chip overlap between adjacent viewing angles is greater than or equal to a complete chip.

[0104] S402: For any single viewing angle among the multiple viewing angles, local area images under different light sources corresponding to the single viewing angle are stitched together to generate a multi-light source stitched image.

[0105] In some embodiments, for any single perspective among multiple perspectives, local area images taken under all light sources corresponding to the perspective are read from the memory in sequence, and the local area images under different light sources are converted into single-channel images, and then the single-channel images of different light sources are spliced ​​by channel to obtain a multi-light source spliced ​​image.

[0106] For example, assuming that the width and height of a local area image are H*W, a single-channel image can be represented by a single-channel matrix, and the final multi-light source stitching image can be represented by a matrix of (H, W, L), where L is the number of light sources used.

[0107] S403: Classify the multi-light source stitched image to output the position coordinates of each chip in the first category and the position coordinates of each chip in the second category.

[0108] In some embodiments, for any single viewing angle among multiple viewing angles, the matrix of the multi-light source stitched image corresponding to the viewing angle is input into a target detection model based on deep learning for feature extraction, and classification is performed based on the extracted features to output the position coordinates of each chip in the first category and the position coordinates of each chip in the second category. The chips in the first category are conventional chips among the multiple LED chips contained in the local area image corresponding to the viewing angle, such as Figure 1 The normal chips in this part are completely photographed. The chips in the second category have obvious appearance differences from the chips in the first category. They are some unconventional chips among the multiple LED chips contained in the local wafer image corresponding to this viewing angle, including standard chips for positioning, half chips at the edge of the image due to incomplete shooting, etc. Figure 1 Other chips are shown in the figure.

[0109] It should be noted that the embodiments of the present application do not impose restrictive requirements on target detection models based on deep learning, including but not limited to YOLOv8 networks, SSD networks, and ResNet networks.

[0110] In some embodiments, since the positioning and rough inspection of chips are not sensitive to chip defect features, and the difference between the first category of chips and the second category of chips is relatively obvious, when positioning and rough inspection of chips under a single viewing angle, the multi-light source spliced ​​image can be replaced with a local area image taken under a single viewing angle under a single light source, so that the positioning of LED chips and the classification of chips with obvious appearance differences can be achieved, thereby improving the efficiency of chip positioning and rough inspection. At this time, the single light source can be selected to make the chip outer contour features obvious. Specialized light source for chip positioning.

[0111] In some embodiments, since the positioning and rough inspection of the chip are the basis of specific defect detection, the chip that is not positioned will directly lead to the missed detection of the chip, so the missed detection rate of chip positioning is extremely high, and it is as close to zero missed detection as possible. Therefore, when the local area image under a single light source is inaccurate for the positioning and rough inspection results of the chip, an additional local area image taken at the same angle under another light source can be added, that is, a multi-light source stitched image (L=2) is used for chip positioning and rough inspection to ensure that each chip can be accurately detected. In other words, the positioning and rough inspection of the chip under a single angle of view can use a single light source image, or a multi-light source stitched image with a number of light sources less than or equal to the total number of light sources.

[0112] For example, four light sources are used to photograph the entire wafer, and a stitched image of two light sources can be used for classification using the target detection model.

[0113] S404: According to the position coordinates of each chip in the first category and the position coordinates of each chip in the second category, a chip image of a single chip is cropped from the multi-light source stitched image.

[0114] In some embodiments, the position coordinates of the chip are the center point coordinates of the chip. In this case, the chip image cropping process is as follows:

[0115] First, the size of the cropping frame of a single chip is determined according to the size of the single chip and the intervals between adjacent chips in the horizontal and vertical directions.

[0116] Considering that some defects occur on the periphery of the chip, it is necessary to expand some areas when cutting the chip, that is, to make the size of the cutting frame larger than the size of a single chip. The formula is expressed as:

[0117] h=height+ω·Intern h Formula 1

[0118] w=width+ω·Intern w Formula 2

[0119] Where height and width are the height and width of the chip respectively, Intern h and Intern w are the vertical and horizontal intervals between adjacent chips, respectively; ω is the expansion coefficient, and its value range is [0,1]; h and w are the height and width of the cropping box.

[0120] Optionally, ω=0.7, which can be adjusted according to actual needs, and the embodiment of the present application does not make any restrictive requirements.

[0121] Then, the center point coordinates of each chip are used as the center point coordinates of the corresponding cropping frame, and the chip image of a single chip is cropped from the multi-light source stitching image in combination with the size of the corresponding cropping frame.

[0122] After the size of the cropping frame is obtained, the center point coordinates of each chip are used as the center point coordinates of the corresponding cropping frame, so that the position of each cropping frame in the multi-light source stitching image is fixed, thereby cropping multiple chip images of a single chip. After cropping, each chip obtains a chip image of size (h, w) under L light sources, where L is the total number of light sources used when photographing the wafer.

[0123] S405: For any LED chip in the first category, stitching chip images of the LED chip under multiple light sources to obtain a stitched image.

[0124] In some embodiments, chip images of the same LED chip under the first category under multiple light sources are tiled at set intervals in a preset order, and each preset interval is filled to generate a spliced ​​image.

[0125] like Figure 7 The figure shows a schematic diagram of a spliced ​​image. Taking L=4 as an example, the chip images of the same chip under 4 light sources are tiled at intervals of θ in the horizontal and vertical directions, and the intervals are filled with 0.

[0126] It should be noted that the present application embodiment does not impose any restrictions on the chip image stitching method. Figure 7 In addition to the splicing method shown, you can also use Figure 8 Other splicing methods shown.

[0127] After the stitching image of a single chip is obtained, the stitching image is stored in the original chip database, so that the original chip database contains a large number of stitching images of conventional single chips on wafers of different models.

[0128] In an embodiment of the present application, chip images under multiple light sources are spliced ​​with gaps and filled in between to facilitate the distinction between chip images under different light sources. Since different defect types present different states under different light sources, the significance of chip defects is improved by splicing the chip images under multiple light sources, so that the chip images under multiple light sources can be simultaneously observed during subsequent manual annotation to accurately identify the defect type of the chip. Furthermore, compared with using a separate model for chip images under multiple light sources, by splicing the chip images under multiple light sources, the chip images under multiple light sources are merged and processed, thereby simplifying the model training process and only requiring training of a single model, which has low complexity and higher efficiency.

[0129] In some embodiments, in order to allow the defect detection model to accurately identify various defects of LED chips, training samples containing defect types can be generated through manual annotation. In specific implementation, spliced ​​images of different defect types are selected from the original chip database for manual annotation, and the annotated spliced ​​images are stored in the defect annotation database, so that the defect annotation database contains spliced ​​images of annotated LED chips on wafers of different models. Among them, the annotation only needs to give the defect type of each chip, and the selected spliced ​​images must contain all defect types and defect-free LED chips.

[0130] Based on the training samples in the original chip database and the defect annotation database, the training method flow of the defect detection model provided in the embodiment of the present application is as follows: Fig. 9 As shown, it mainly includes the following steps:

[0131] S901: Acquire an initial image set.

[0132] In some embodiments, an initial image set is obtained from an original chip database, wherein the initial image set includes multiple image subsets, each image subset corresponds to a model of wafer, each image subset includes respective stitched images of multiple LED chips on the corresponding model of wafer, and each stitched image is generated based on chip images of a single chip under multiple light sources.

[0133] In some embodiments, each stitched image in the initial image set is a tiled stitching of chip images of a single chip under multiple light sources. Therefore, the stitching method of the stitched images can also be converted. In specific implementation, for any first stitched image in the initial image set, the tiled method of the chip images under multiple light sources in the first stitched image is converted to a channel-based stitching method, such as Fig.10 The figure shows a schematic diagram of the splicing format conversion. Compared with the tiling splicing method, the channel-based splicing method can reduce the size of the model input. Since the number of convolutions is proportional to the size of the input, the splicing format conversion can reduce the number of convolutions in subsequent model training, thereby increasing the speed of model training. In addition, the increase in the number of channels, thanks to the optimized acceleration of the convolution operation in the GPU, will hardly increase the calculation time, further increasing the speed of model training.

[0134] S902: For any first stitched image in the initial image set, perform local pixel value processing on the first stitched image in units of image blocks of a preset size to generate a first training sample set.

[0135] Since the initial image set is unlabeled, a self-supervised algorithm can be sampled for feature learning. The self-supervised feature learning process uses the initial image set as the original input, randomly deletes part of the data in the original input to construct new data as the target input for feature learning, abstracts the target input into high-dimensional features in the feature space, and predicts the deleted part of the data based on the high-dimensional features to achieve the prediction of the original input, thereby achieving the learning of chip features.

[0136] In some embodiments, for any first stitched image in the initial image set, a partial data deletion process may be performed on the first stitched image to construct self-supervised training data. The specific construction process includes:

[0137] First, the first stitched image is divided into a plurality of p*p image blocks.

[0138] Taking the first stitched image stitched by channels as an example, assuming that the width and height of each stitched image are w / h, the first stitched image is divided into w / p*h / p image blocks according to the width and height, and the size of each image block is (p, p, L).

[0139] Among them, p is pre-set according to the number of down-sampling layers and the down-sampling multiples of each layer in the defect detection model.

[0140] For example, when the number of downsampling layers of the defect detection model is 3 and the downsampling multiple of each layer is 2, p=8.

[0141] In an embodiment of the present application, since the size of the image block is pre-set according to the number of downsampling layers in the model and the downsampling multiples of each layer, after the first spliced ​​image is downsampled in the feature extraction process, the information is retained in the last layer, thereby improving the accuracy of model feature learning.

[0142] Then, pixel values ​​in some image blocks among the multiple image blocks are set to preset values ​​to generate first training samples.

[0143] For example, after dividing the first stitched image into w / p*h / p image blocks according to width and height, δ% of image blocks are randomly selected from the w / p*h / p image blocks, and the pixel values ​​in the selected δ% of image blocks are set to 0, thereby obtaining a first training sample.

[0144] Finally, the first training sample is stored in the first training sample set.

[0145] Since the initial image set includes stitched images of LED chips on wafers of different models, the features of LED chips of different models can be learned based on the first training sample set.

[0146] In an embodiment of the present application, by setting the pixel values ​​in some image blocks among the multiple image blocks in the first stitched image to preset values, the purpose of constructing a self-label for the unlabeled first stitched image is achieved, that is, the first stitched image itself can be used as its own label to facilitate self-supervised feature learning.

[0147] S903: Iteratively train the defect detection model according to the first training sample set to obtain an initial defect detection model.

[0148] Since the initial image set does not require manual annotation and is directly generated based on the wafers on the production line, the amount of data is huge. Therefore, the first training sample set will contain a large amount of unlabeled data. In this way, the defect detection model is trained based on the first training sample set to learn chip features, thereby making full use of the unlabeled data of LED chips on a large number of wafers on the production line, improving the learning effect of the defect detection model on chip features, and at the same time accelerating the convergence speed of LED chip defect detection model training, thereby accelerating the online process.

[0149] In some embodiments, when the defect detection model is self-supervisedly trained based on the first training sample set, a plurality of first training samples in the first training sample set are batch-inputted into the defect detection model for iterative training, and the defect detection model that meets the preset convergence condition is used as the initial defect detection model, wherein each iterative training process is as follows: Fig.11 As shown, it mainly includes the following steps:

[0150] S9031: Use a sparse convolution kernel to perform a convolution operation on a first portion of data in a first training sample to extract chip features of the first training sample.

[0151] The first part of data corresponds to an image block in the first training sample whose pixel values ​​are not set to preset values.

[0152] S9032: Determine the predicted pixel value of the second part of the data in the first training sample according to the chip characteristics, and determine the first loss value of the defect detection model in combination with the actual pixel value of the second part of the data.

[0153] The second part of data corresponds to an image block in which pixel values ​​in the first training sample are set to preset values.

[0154] S9033: Adjust the network parameters of the defect detection model according to the first loss value.

[0155] In some embodiments, the defect detection model mainly includes a feature extraction network and a feature prediction network. These two networks mainly use convolutional networks, such as Fig.12As shown in the figure, the feature extraction network uses sparse convolution kernels to operate on the non-zero data in the first spliced ​​image to extract high-dimensional chip features. The first layer is a downsampling layer, which uses a convolution kernel with a size of 2 and a stride of 2; the second layer contains two blocks, each of which is composed of a convolution kernel with a size of 5 and a channel of 64, a convolution kernel with a size of 1 and a channel of 192, and a convolution kernel with a size of 1 and a channel of 64 in sequence; the third layer is a downsampling layer, which uses a convolution kernel with a size of 2 and a stride of 2. The fourth layer contains 7 blocks, each of which is composed of 1 convolution kernel of size 7 and 128 channels, 1 convolution kernel of size 1 and 384 channels, and 1 convolution kernel of size 1 and 128 channels in sequence; the fifth layer is a downsampling layer, using a convolution kernel of size 2 and stride 2; the sixth layer contains 3 blocks, each of which is composed of 1 convolution kernel of size 7 and 256 channels, 1 convolution kernel of size 1 and 768 channels, and 1 convolution kernel of size 1 and 256 channels in sequence. The feature prediction network contains 2 blocks, each of which is composed of 1 convolution kernel of size 7 and 256 channels, 1 convolution kernel of size 1 and 768 channels, and 1 convolution kernel of size 1 and 256 channels in sequence, and is mainly used to predict the data set to 0 in the first spliced ​​image.

[0156] In an embodiment of the present application, since the first training sample contains two parts, one in which the pixel value is not set to the preset value and the other in which the pixel value is set to the preset value, the pixel value of the part set to the preset value is predicted based on the feature learning of the part in which the preset value is not set, so that the model can better learn the chip features and thus improve the accuracy of defect detection. At the same time, the use of sparse convolution kernels for feature extraction can speed up the feature learning and thus speed up the model convergence.

[0157] S904: Obtain a second training sample set.

[0158] In some embodiments, a second training sample set is obtained from a defect annotation database, wherein the second training sample set is generated by annotating defects on spliced ​​images selected from the initial image set, and the second training sample set covers spliced ​​images of all defect types and non-defective LED chips.

[0159] In some embodiments, each stitched image in the second training sample set is a tiled stitching of chip images of a single chip under multiple light sources. Therefore, the stitching method of the stitched images can also be converted. In specific implementation, for any second stitched image in the second training sample set, the tiled method of the chip images under multiple light sources in the second stitched image is converted to a channel-based stitching method, such as Fig.10The figure shows a schematic diagram of the splicing format conversion. Compared with the tiling splicing method, the channel-based splicing method can reduce the size of the model input. Since the number of convolutions is proportional to the size of the input, the splicing format conversion can reduce the number of convolutions in subsequent model training, thereby increasing the speed of model training. In addition, the increase in the number of channels, thanks to the optimized acceleration of the convolution operation in the GPU, will hardly increase the calculation time, further increasing the speed of model training.

[0160] S905: Iteratively train the initial defect detection model according to the second training sample set to obtain a target defect detection model.

[0161] Among them, the network structure of the initial defect detection model is Fig.12 The network structure shown remains basically unchanged. During the supervised training using the second training sample set, only the sparse convolution kernels used in the self-supervised training need to be replaced with ordinary convolution kernels.

[0162] Since the initial defect detection model has been preliminarily trained based on a large number of unlabeled stitched images of LED chips, the initial defect detection model can fully extract chip features from the images. In this way, only a small number of labeled stitched images are needed to guide the further learning of the initial defect detection model, so that the model can better learn the characteristics of various defect types and defect-free LED chips, accelerate the convergence speed of model training and improve the defect detection effect, while reducing the workload and time of manual labeling.

[0163] In some embodiments, when the initial defect detection model is supervisedly trained based on the second training sample set, a plurality of second training samples in the second training sample set are batch-inputted into the initial defect detection model for iterative training, and the initial defect detection model that meets the preset convergence condition is used as the target defect detection model, wherein each iterative training process is as follows: Fig.13 As shown, it mainly includes the following steps:

[0164] S9051: Perform a convolution operation on the second training sample using a common convolution kernel to extract chip features of the second training sample.

[0165] S9052: Determine a predicted label of the second training sample according to the chip feature, and determine a second loss value of the initial defect detection model in combination with the true label of the second training sample.

[0166] S9053: Adjust network parameters of the initial defect detection model according to the second loss value.

[0167] The model training method provided by the embodiment of the present application combines self-supervision and supervised training methods, wherein the self-supervision algorithm uses the unlabeled initial image set generated by the spliced ​​images of LED chips on wafers of various models on the production line under multiple light sources for model training. Since each spliced ​​image in the first training sample set is processed with local pixel values ​​in units of image blocks of preset sizes, the purpose of constructing its own label with unlabeled data is achieved, thereby making full use of the large amount of sample data produced daily on the production line for preliminary chip feature learning, improving the richness of feature learning to improve the accuracy of model defect detection, and at the same time accelerating the convergence speed of model training. In addition, the supervised algorithm uses the selected second training sample set generated by the spliced ​​images covering all defect types and defect-free LED chips to perform model training again, thereby guiding the model to learn defect features through a small amount of labeled sample data, thereby reducing the workload and time of manual labeling while achieving high detection accuracy.

[0168] In some embodiments, considering the problem that chip defects are difficult to identify due to low differentiation and small proportion, when training the initial defect detection model, spliced ​​images of defect-free LED chips can be introduced to enhance the defect characteristics of the second training sample, thereby further reducing the number of labeled samples required for the model and reducing the workload and working time of manual labeling.

[0169] In some embodiments, the initial defect detection model includes at least two feature processing stages, and when the initial defect detection model is trained, the network parameters of the first feature processing stage are frozen, and only the network parameters of other feature processing stages are trained.

[0170] See also Fig.14 , is a flow chart of another training method of an initial defect detection model provided in an embodiment of the present application, which mainly includes the following steps:

[0171] S1401: For any second training sample in the second training sample set, the second training sample and the third training sample are combined into a training sample pair.

[0172] The third training sample is a spliced ​​image of a defect-free LED chip.

[0173] S1402: Iteratively train other feature processing stages of the initial defect detection model according to multiple training samples to obtain a target defect detection model.

[0174] In an embodiment of the present application, model training is performed by combining the second training sample in the second training sample set and the third training sample of the defect into a training sample pair. Since the third training sample can be used as a priori knowledge of the defect for the model to learn, the difficulty of model learning is reduced, thereby improving the detection effect of the model. In addition, since the initial defect detection model of the preliminary training can effectively learn the characteristics of the chip, by freezing the network parameters of the first feature processing stage in the initial defect detection model, a good chip feature representation capability can be retained while accelerating the speed of module training.

[0175] In some embodiments, when training is performed based on multiple training samples, multiple training sample pairs are batch-inputted into the initial defect detection model for iterative training, and the initial defect detection model that meets the preset convergence condition is used as the target defect detection model, wherein each iterative training process is as follows:

[0176] S1402_1: extracting original chip features of the second training sample and reference chip features of the third training sample in the training sample pair based on the first feature processing stage.

[0177] like Fig.15 The following is a schematic diagram of the training process of the initial defect detection model, where the first feature processing stage is Fig.12 The second layer in the feature extraction network shown.

[0178] S1402_2: Fuse the original chip features with the reference chip features to obtain enhanced chip features.

[0179] In some embodiments, the feature fusion method is expressed as:

[0180]

[0181] Among them, F 1 represents the original chip features of the second training sample, F 2 represents the reference chip feature of the third training sample, F n To enhance chip features.

[0182] S1402_3: After processing the original chip features of the second training sample in the training sample pair based on other feature processing stages, stack them with the enhanced chip features to obtain target chip features.

[0183] Take two other feature processing stages as an example, such as Fig.15 As shown, the second feature processing stage is Fig.12 The fourth layer in the feature extraction network shown, the third feature processing stage is Fig.12In the sixth layer of the feature extraction network shown in the figure, the dimension of the original chip features will increase through the processing of the second feature processing stage and the third feature processing stage, thereby obtaining a deeper feature representation, and the enhanced chip features after fusion are shallow feature representations. After transforming their sizes to be consistent with the sizes of the original chip features output by the third feature processing stage through sampling, they are stacked in the channel layer to obtain the target chip features after defect enhancement.

[0184] S1402_4: Determine the predicted label of the second training sample according to the target chip characteristics, and determine the third loss value in combination with the true label of the second training sample.

[0185] Since the second training sample is pre-labeled, the training loss value can be calculated by label prediction.

[0186] S1402_5: Adjust network parameters of other feature processing stages according to the third loss value.

[0187] In an embodiment of the present application, by combining the original chip features of the second training sample and the reference chip features of the third training sample, the significance of the defects in the second training sample can be improved. In this way, the defect features can be enhanced while retaining the original chip features to improve the defect recognition capability, thereby reducing the number of labeled samples required for model training, and further reducing the labeling workload and labeling time.

[0188] In some embodiments, the trained target defect detection model can be deployed on the quality inspection equipment of the production line to realize fully automatic and high-precision LED chip defect detection, thereby improving the delivery quality and efficiency of LED chips.

[0189] See also Fig.16 , is a flow chart of the defect detection method based on the target defect detection model, which mainly includes the following steps:

[0190] S1601: Acquire local wafer images taken from multiple viewing angles under different light sources.

[0191] Among them, the local wafer images taken from multiple perspectives constitute a complete wafer image, and each local wafer image contains multiple LED chips.

[0192] S1602: For any single viewing angle among the multiple viewing angles, local wafer images under different light sources corresponding to the single viewing angle are stitched together to generate a multi-light source stitched image.

[0193] The method for generating the multi-light source stitching image is described in the training sample construction process and will not be described in detail here.

[0194] S1603: Classify the multi-light source stitched image to output the position coordinates of each chip to be detected, and cut out the chip image of each chip to be detected from the multi-light source stitched image according to the position coordinates.

[0195] The classification and cropping process of the multi-light source stitching image refers to the training sample construction process and will not be described in detail here.

[0196] S1604: splicing chip images corresponding to the chip to be detected under different light sources to obtain the image of the chip to be detected.

[0197] The splicing process of the chip image to be detected refers to the training sample construction process, which will not be described in detail here.

[0198] S1605: Perform defect detection on the image of the chip to be inspected using the target defect detection model.

[0199] See also Fig.17 , is a flow chart of another defect detection method based on the target defect detection model, which mainly includes the following steps:

[0200] S1701: Acquire local wafer images taken from multiple viewing angles under different light sources.

[0201] Among them, the local wafer images taken from multiple perspectives constitute a complete wafer image, and each local wafer image contains multiple LED chips.

[0202] S1702: For any single viewing angle among the multiple viewing angles, local wafer images under different light sources corresponding to the single viewing angle are stitched together to generate a multi-light source stitched image.

[0203] The method for generating the multi-light source stitching image is described in the training sample construction process and will not be described in detail here.

[0204] S1703: Classify the multi-light source stitched image to output the position coordinates of each chip to be detected, and cut out the chip image of each chip to be detected from the multi-light source stitched image according to the position coordinates.

[0205] The classification and cropping process of the multi-light source stitching image refers to the training sample construction process and will not be described in detail here.

[0206] S1704: splicing chip images corresponding to the chip to be detected under different light sources to obtain the chip image to be detected.

[0207] The splicing process of the chip image to be detected refers to the training sample construction process, which will not be described in detail here.

[0208] S1705: Input the chip image to be inspected and the non-defective chip image into the target defect detection model to perform defect detection on the chip image to be inspected.

[0209] In some embodiments, when classifying the multi-light source stitched image, in addition to outputting the position coordinates of each chip to be inspected, the position coordinates of each detection-free chip are also output. Therefore, when the multi-light source stitched image is cropped, the chip image of each detection-free chip is also generated. At this time, after completing the defect detection of the chip to be inspected, combined with the chip image of the detection-free chip, the complete wafer image after defect detection can be restored.

[0210] See also Fig.18A , which is a flow chart of a method for restoring a complete wafer image after defect detection provided in an embodiment of the present application, mainly includes the following steps:

[0211] S1801: Summarize chip images of the chips to be inspected and chip images of the chips exempted from inspection in a single viewing angle to obtain a chip set in a single viewing angle.

[0212] Among them, each chip image in the chip set is associated with the chip's location coordinates and target type. The target type of the chip to be inspected is a specific defect type (such as dirt, electrode scratches, etc.), and the target type of the chip to be exempted from inspection is the original type, which can be represented by the original name of the chip to be exempted from inspection (such as positioning chip).

[0213] S1802: Draw a complete wafer image with defects marked based on the chip sets corresponding to the multiple perspectives.

[0214] Since the chip sets of multiple viewpoints contain all LED chips on a wafer, and the positions and types of LED chips in each chip set are known, the chips in each chip set can be globally stitched to restore the complete wafer image after defect detection. The specific process is as follows:

[0215] S1802_1: For any chip in the plurality of chip sets, determine the global position of the chip on the wafer according to the viewing angle and position coordinates of the chip.

[0216] Since LED chips are of uniform size and are neatly arranged in rows and columns on the wafer, and there is a certain order (such as Z-order) when photographing the wafer from multiple perspectives, the global position of each chip on the wafer can be determined based on the perspective and position coordinates of the chip.

[0217] S1802_2: Draw rectangles of the same size at multiple global locations to output a complete wafer image with defects marked.

[0218] Each rectangle corresponds to a chip, and rectangles corresponding to chips of different target types are displayed differently.

[0219] In some embodiments, different colors may be used to represent chips of different target types, such as dirt defects represented by red rectangles, scratch defects represented by green rectangles, needle marks represented by yellow rectangles, and so on.

[0220] It should be noted that the embodiment of the present application does not impose any restrictive requirements on the manner in which the rectangles are displayed. In addition to being distinguished by color, they can also be distinguished by different filling methods, or by using boxes with different lines, etc.

[0221] like Fig.18B The figure shows the restored complete wafer image, where each rectangle represents a chip. Defect-free chips, chips with three types of defects, and inspection-free chips are represented by rectangles with different fillings.

[0222] In an embodiment of the present application, each chip is represented by a rectangle, and a complete wafer image is drawn according to the global position of each chip on the wafer. Since the target type has been determined for each chip, the rectangles corresponding to chips of different target types are displayed separately, which can more intuitively highlight the results of defect detection and provide a stronger visual effect.

[0223] Based on the same technical concept, an embodiment of the present application provides a training device for a defect detection model, which can implement the steps of the above-mentioned chip defect detection model training method and achieve the same technical effect.

[0224] See also Fig.19 The training device includes an acquisition module 1901, an image processing module 1902, and a model training module 1903, wherein:

[0225] The acquisition module 1901 is used to acquire an initial image set and a second training sample set; wherein the initial image set includes multiple image subsets, each image subset corresponds to a wafer model, each image subset includes stitched images of multiple LED chips on the wafer of the corresponding model, each stitched image is generated based on chip images of a single chip under multiple light sources, and the second training sample set is generated by marking defects on the stitched images selected from the initial image set, and the second training sample set covers stitched images of all defect types and defect-free LED chips.

[0226] An image processing module 1902 is used to process local pixel values ​​of the first stitched image in units of image blocks of a preset size to generate a first training sample set; wherein the first stitched image is any stitched image in the initial image set;

[0227] The model training module 1903 is used to iteratively train the defect detection model according to the first training sample set to obtain an initial defect detection model; and iteratively train the initial defect detection model according to the second training sample set to obtain a target defect detection model.

[0228] Optionally, the image processing module 1902 is specifically used for:

[0229] Divide the first stitched image into a plurality of p*p image blocks, wherein p is preset according to the number of downsampling layers and the downsampling multiple of each layer in the defect detection model;

[0230] Setting pixel values ​​in some image blocks among the plurality of image blocks to preset values ​​to generate a first training sample;

[0231] The first training sample is stored in a first training sample set.

[0232] Optionally, each stitched image is a tiled stitching of chip images of a single chip under multiple light sources. The image processing module 1902 is further used for:

[0233] The tiling mode of chip images under multiple light sources in the target stitched image is converted into a channel-based stitching mode to reduce the size of the model input; wherein the target stitched image is the first stitched image or any second stitched image in the second training sample set.

[0234] Optionally, the model training module 1903 is specifically used for:

[0235] Multiple first training samples in the first training sample set are batch-inputted into the defect detection model for iterative training, and the defect detection model that meets the preset convergence condition is used as the initial defect detection model, wherein each iterative training process is as follows:

[0236] Using a sparse convolution kernel to perform a convolution operation on a first portion of data in a first training sample to extract chip features of the first training sample; wherein the first portion of data corresponds to an image block in the first training sample whose pixel values ​​are not set to a preset value;

[0237] Determine the predicted pixel value of the second part of the data in the first training sample according to the chip characteristics, and determine the first loss value of the defect detection model in combination with the real pixel value of the second part of the data; wherein the second part of the data corresponds to the image block in which the pixel value in the first training sample is set to a preset value;

[0238] The network parameters of the defect detection model are adjusted according to the first loss value.

[0239] Optionally, the model training module 1903 is specifically used for:

[0240] Multiple second training samples in the second training sample set are batch-inputted into the initial defect detection model for iterative training, and the initial defect detection model that meets the preset convergence condition is used as the target defect detection model, wherein each iterative training process is as follows:

[0241] Performing a convolution operation on the second training sample using a common convolution kernel to extract chip features of the second training sample;

[0242] Determine a predicted label of a second training sample according to the chip feature, and determine a second loss value of the initial defect detection model in combination with the true label of the second training sample;

[0243] The network parameters of the initial defect detection model are adjusted according to the second loss value.

[0244] Optionally, the initial defect detection model includes at least two feature processing stages, and the network parameters of the first feature processing stage are frozen during the training of the initial defect detection model;

[0245] The model training module 1903 is specifically used for:

[0246] For any second training sample in the second training sample set, the second training sample and the third training sample are combined into a training sample pair; wherein the third training sample is a spliced ​​image of a defect-free LED chip;

[0247] The other feature processing stages of the initial defect detection model are iteratively trained according to multiple training samples to obtain a target defect detection model.

[0248] Optionally, the model training module 1903 is specifically used for:

[0249] Multiple training sample pairs are batch-inputted into the initial defect detection model for iterative training, and the initial defect detection model that meets the preset convergence conditions is used as the target defect detection model, wherein each iterative training process is as follows:

[0250] Extracting original chip features of the second training sample and reference chip features of the third training sample in the training sample pair based on the first feature processing stage;

[0251] The original chip features and the reference chip features are merged to obtain enhanced chip features;

[0252] After processing the original chip features of the second training sample in the training sample pair based on other feature processing stages, the original chip features are stacked with the enhanced chip features to obtain the target chip features;

[0253] Determine a predicted label of the second training sample according to the target chip feature, and determine a third loss value in combination with the true label of the second training sample;

[0254] The network parameters of other feature processing stages are adjusted according to the third loss value.

[0255] Optionally, the image processing module 1902 is further configured to:

[0256] For any LED chip, the chip images of the LED chip under multiple light sources are tiled at set intervals in a preset order, and each preset interval is filled to generate a spliced ​​image.

[0257] Based on the same technical concept, an embodiment of the present application provides a defect detection device that can implement the steps of the above-mentioned chip defect method and achieve the same technical effect.

[0258] See also Fig. 20 The defect detection device includes an acquisition module 2001, an image stitching module 2002, a chip positioning and cutting module 2003, and a defect detection module 2004, wherein:

[0259] An acquisition module 2001 is used to acquire partial wafer images taken from multiple viewing angles under different light sources, wherein the partial wafer images taken from multiple viewing angles constitute a complete wafer image, and each partial wafer image contains multiple LED chips;

[0260] The image stitching module 2002 is used to stitch the local wafer images under different light sources corresponding to any single viewing angle among the multiple viewing angles to generate a multi-light source stitching image.

[0261] The chip positioning and cropping module 2003 is used to classify the multi-light source stitched image to output the position coordinates of each chip to be detected, and to crop the chip image of each chip to be detected from the multi-light source stitched image according to the position coordinates;

[0262] The image stitching module 2002 is further used to stitch chip images corresponding to the chip to be detected under different light sources to obtain the chip image to be detected;

[0263] The defect detection module 2004 is used to perform defect detection on the image of the chip to be detected using the target defect detection model.

[0264] For the convenience of description, the above parts are divided into modules (or units) according to their functions and described separately. Of course, when implementing this application, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.

[0265] After introducing the defect detection model training method, defect detection method and device according to the exemplary embodiment of the present application, next, an electronic device according to another exemplary embodiment of the present application is introduced.

[0266] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0267] Based on the same inventive concept as the above method embodiment, the electronic device provided in the embodiment of the present application can be a server or a terminal device, and its structure can be as follows: Fig.21 As shown, it includes a processor 2101, a memory 2102 and a communication interface 2103;

[0268] The communication interface 2103 is used to send and receive data;

[0269] The memory 2102 stores a computer program, and the processor 2101 executes the steps of any one of the defect detection model training methods and defect detection methods in the above embodiments according to the computer program.

[0270] In the embodiment of the present application, the memory 2102 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, and programs required to run the instant messaging function, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc. The memory 2102 may be a volatile memory (volatile memory), such as a random-access memory (RAM); the memory may also be a non-volatile memory (non-volatile memory), such as a read-only memory, a flash memory (flash memory), a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 2102 may be any other medium that can be used to carry or store a desired computer program in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 2102 may be a combination of the above memories.

[0271] The processor 2101 may include one or more central processing units (CPU), GPU or a digital processing unit, etc.

[0272] In the embodiment of the present application, the specific connection medium between the communication interface 2103, the memory 2102 and the processor 2101 is not limited. In the embodiment of the present application, the bus 2104 between the communication interface 2103, the memory 2102 and the processor 2101 is Fig.21The connections between the other components are only for illustration and are not intended to be limiting. The bus 2104 can be divided into an address bus, a data bus, a control bus, etc. For ease of description, Fig.21 The diagram shows that only one thick line is used, but this does not mean that there is only one bus or only one type of bus.

[0273] It should be noted that Fig.21 It is only the equipment necessary for the electronic device to implement the training and detection methods in the embodiments of the present application. Optionally, the electronic device may also include hardware of conventional electronic devices such as a display screen, a power supply, and buttons.

[0274] An embodiment of the present application also provides a computer-readable storage medium for storing some instructions, which, when executed, can complete the steps of any one of the defect detection model training methods and defect detection methods in the aforementioned embodiments.

[0275] An embodiment of the present application also provides a computer program product for storing a computer program, wherein the computer program is used to execute the steps of any one of the defect detection model training methods and defect detection methods in the aforementioned embodiments.

[0276] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0277] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

Claims

1. A chip defect detection model training method, characterized in that: include: Acquire an initial image set, wherein the initial image set includes multiple image subsets, each image subset corresponds to a wafer model, each image subset includes respective stitched images of multiple LED chips on the wafer of the corresponding model, and each stitched image is generated based on chip images of a single chip under multiple light sources; Processing local pixel values ​​of the first stitched image in units of image blocks of a preset size to generate a first training sample set; wherein the first stitched image is any stitched image in the initial image set; Iteratively training the defect detection model according to the first training sample set to obtain an initial defect detection model; Acquire a second training sample set, where the second training sample set is generated by annotating defects on the spliced ​​images selected from the initial image set, and the second training sample set covers spliced ​​images of all defect types and defect-free LED chips; The initial defect detection model is iteratively trained according to the second training sample set to obtain a target defect detection model.

2. The method according to claim 1, characterized in that The processing of local pixel values ​​of the first spliced ​​image in units of image blocks of a preset size to generate a first training sample set includes: Dividing the first stitched image into a plurality of p*p image blocks, wherein p is preset according to the number of downsampling layers and the downsampling multiple of each layer in the defect detection model; Setting pixel values ​​in some image blocks among the plurality of image blocks to preset values ​​to generate a first training sample; The first training sample is stored in the first training sample set.

3. The method according to claim 1, characterized in that Each stitched image is a tiled stitching of chip images of a single chip under multiple light sources. The method further includes: The tiling mode of chip images under multiple light sources in the target stitched image is converted into a channel-based stitching mode to reduce the size of the model input; wherein the target stitched image is the first stitched image or any second stitched image in the second training sample set.

4. The method according to claim 1, characterized in that The iterative training of the defect detection model according to the first training sample set to obtain an initial defect detection model includes: Multiple first training samples in the first training sample set are batch-inputted into the defect detection model for iterative training, and a defect detection model that meets a preset convergence condition is used as an initial defect detection model, wherein each iterative training process is as follows: Using a sparse convolution kernel to perform a convolution operation on a first portion of data in the first training sample to extract chip features of the first training sample; wherein the first portion of data corresponds to an image block in the first training sample whose pixel values ​​are not set to preset values; Determine the predicted pixel value of the second part of the data in the first training sample according to the chip feature, and determine the first loss value of the defect detection model in combination with the real pixel value of the second part of the data; wherein the second part of the data corresponds to the image block in which the pixel value in the first training sample is set to a preset value; The network parameters of the defect detection model are adjusted according to the first loss value.

5. The method according to claim 1, characterized in that The iterative training of the initial defect detection model according to the second training sample set to obtain a target defect detection model includes: Multiple second training samples in the second training sample set are batch-inputted into the initial defect detection model for iterative training, and the initial defect detection model that meets the preset convergence condition is used as the target defect detection model, wherein each iterative training process is as follows: Using a common convolution kernel to perform a convolution operation on the second training sample to extract chip features of the second training sample; Determine a predicted label of the second training sample according to the chip feature, and determine a second loss value of the initial defect detection model in combination with a true label of the second training sample; The network parameters of the initial defect detection model are adjusted according to the second loss value.

6. The method according to claim 1, characterized in that The initial defect detection model includes at least two feature processing stages, and the network parameters of the first feature processing stage are frozen during the training of the initial defect detection model; The iterative training of the initial defect detection model according to the second training sample set to obtain a target defect detection model includes: For any second training sample in the second training sample set, the second training sample and the third training sample form a training sample pair; wherein the third training sample is a spliced ​​image of a defect-free LED chip; The other feature processing stages of the initial defect detection model are iteratively trained according to a plurality of training samples to obtain the target defect detection model.

7. The method according to claim 6, characterized in that The iterative training of other feature processing stages of the initial defect detection model according to a plurality of training samples to obtain the target defect detection model includes: The plurality of training sample pairs are batch-inputted into the initial defect detection model for iterative training, and the initial defect detection model that meets the preset convergence condition is used as the target defect detection model, wherein each iterative training process is as follows: Extracting original chip features of the second training sample and reference chip features of the third training sample in the training sample pair based on the first feature processing stage; Fusion the original chip features with the reference chip features to obtain enhanced chip features; After processing the original chip feature of the second training sample in the training sample pair based on other feature processing stages, the original chip feature is stacked with the enhanced chip feature to obtain the target chip feature; Determine a predicted label of the second training sample according to the target chip feature, and determine a third loss value in combination with a true label of the second training sample; The network parameters of the other feature processing stages are adjusted according to the third loss value.

8. The method according to any one of claims 1 to 7, characterized in that The stitched image is generated by: For any LED chip, chip images of the LED chip under multiple light sources are tiled at set intervals in a preset order, and each preset interval is filled to generate the spliced ​​image.

9. A method for detecting defects in LED chips, characterized in that: include: Acquire partial wafer images taken from multiple viewing angles under different light sources, wherein the partial wafer images taken from the multiple viewing angles constitute a complete wafer image, and each partial wafer image contains multiple LED chips; For any single viewing angle among the multiple viewing angles, local wafer images under different light sources corresponding to the single viewing angle are stitched together to generate a multi-light source stitched image. Classifying the multi-light source stitched image to output the position coordinates of each chip to be detected, and cutting out the chip image of each chip to be detected from the multi-light source stitched image according to the position coordinates; splicing chip images corresponding to the chip to be detected under different light sources to obtain an image of the chip to be detected; Defect detection is performed on the image of the chip to be detected using a target defect detection model, wherein the target defect detection model is trained based on any one of the methods described in claims 1 to 8.

10. An electronic device, characterized in that: It includes a processor, a memory and a communication interface, wherein the communication interface, the memory and the processor are connected via a bus; The communication interface is configured to send and receive data; The memory stores a computer program, and the processor executes the steps of the method according to any one of claims 1 to 9 according to the computer program.

Images