Mini-LED repair method and device based on automatic optical detection

Abstract

The invention relates to the technical field of Mini-LED chip detection, and discloses a Mini-LED repair method and device based on automatic optical detection, and the method comprises the steps: carrying out the Gaussian filtering of an optical chip image, obtaining a filtered chip image, training a neural network based on an original chip image set, obtaining a defect region detection model, and carrying out the detection of a defect region. Detecting the filter chip image according to the defect area detection model to obtain a defect image area group, if the defect image area group is not a null set, marking a to-be-repaired product as a repairable product, carrying out local contrast enhancement on the filter chip image to obtain a target chip image, carrying out single chip identification on the target chip image, and if the defect image area group is not a null set, marking the to-be-repaired product as a repairable product; and obtaining a single chip area set, identifying a to-be-maintained chip area group according to the single chip area set, and generating a repair opinion report based on the to-be-maintained chip area group. According to the invention, the precision of Mini-LED product defect detection can be improved, and the manpower consumption of Mini-LED products can be reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Mini-LED chip detection, and particularly relates to a Mini-LED repair method and device based on automatic optical detection. BACKGROUND

[0002] With the rapid popularization of Mini-LED in the high-end display field, thousands of 50-200 micron Micro-level chips are integrated on a single panel, and the fracture, electrode oxidation or fluorescent powder falling of any chip will cause pixel-level bad points, directly reducing the picture quality and reliability of the terminal product. Therefore, how to efficiently and accurately complete Mini-LED repair has become a key link to determine product yield and manufacturing cost.

[0003] The traditional technical implementation mode is a combination of manual detection or simple optical detection and manual repair, such as relying on manual visual judgment and experience to identify defects, combining simple optical equipment for auxiliary detection, and then manually repairing through manual operation. However, this method highly depends on subjective judgment of manual work in defect identification, consumes a large amount of manpower, and also reduces the repair efficiency. SUMMARY

[0004] The present application provides a Mini-LED repair method and device based on automatic optical detection, which mainly aims to improve the accuracy of Mini-LED product defect detection and reduce the labor consumption of Mini-LED products.

[0005] To achieve the above purpose, the present application provides a Mini-LED repair method based on automatic optical detection, comprising:

[0006] receiving an optical detection instruction, determining a set of products to be repaired based on the optical detection instruction, wherein the set of products to be repaired includes a plurality of products to be repaired, and the product to be repaired is a Mini-LED product, and the product to be repaired includes a plurality of Mini-LED chips;

[0007] extracting the product to be repaired in the set of products to be repaired in turn, placing the product to be repaired on a preset background stage to obtain a product to be detected, wherein the background stage includes a stage, a high-definition camera and a light source;

[0008] photographing the product to be detected by using the high-definition camera and the light source to obtain an optical chip image, and performing Gaussian filtering on the optical chip image to obtain a filtered chip image;

[0009] obtaining an original chip image set, and training a pre-obtained neural network based on the original chip image set to obtain a defect area detection model, wherein the original chip image set includes a plurality of original chip images, and the original chip image is a defective chip image or a normal chip image;

[0010] detecting the filtered chip image according to the defect area detection model to obtain a defect image region set, wherein the defect image region set includes a plurality of defect image regions or the defect image region set is an empty set;

[0011] If the defect image region set is not an empty set, the product to be repaired is recorded as a repairable product, and local contrast enhancement is performed on the filtered chip image to obtain a target chip image;

[0012] performing single chip recognition on the target chip image to obtain a single chip region set, identifying a chip region set to be repaired in the defect image region set according to the single chip region set, and generating a repair opinion report of the repairable product based on the chip region set to be repaired.

[0013] The repair opinion report is summarized to obtain a plurality of repair opinion reports, and the Mini-LED repair based on automatic optical detection is completed based on the plurality of repair opinion reports.

[0014] Optionally, the pre-acquired neural network is trained based on the original chip image set to obtain the defect area detection model, including:

[0015] extracting the original chip image in the original chip image set in sequence, and obtaining an image label of the original chip image, wherein the image label includes a normal label or a defect label;

[0016] dividing the original chip image according to a preset region specification to obtain a plurality of original image regions;

[0017] extracting the original image region in the plurality of original image regions in sequence, and standardizing the original image region to obtain a target image region;

[0018] performing feature extraction on the target image region to obtain a target region feature vector;

[0019] labeling the target region feature vector based on the image label to obtain a labeled region feature vector;

[0020] summarizing the labeled region feature vector corresponding to each original image region in the plurality of original image regions to obtain a labeled region feature vector group, and merging the labeled region feature vector group corresponding to each original chip image in the original chip image set to obtain a labeled region feature vector set;

[0021] setting a loss function, training the neural network by using the loss function and the labeled region feature vector set, and obtaining the defect area detection model.

[0022] Optionally, the setting of the loss function includes:

[0023] Confirming a training data set of the neural network in a training process, wherein the training data set comprises a plurality of training data, and each training data comprises a defect probability value and a true label value;

[0024] Constructing a distribution overlap loss term according to the training data set, wherein the distribution overlap loss term is expressed as:

[0025]

[0026] wherein F1 represents the distribution overlap loss term, n represents the number of training data in the training data set, r j represents the true label value of the jth training data in the training data set, r j ' represents the defect probability value of the jth training data in the training data set, and ω represents a preset minimum value.

[0027] Weighted sum of the distribution overlap loss term and a preset classification loss term to obtain a loss function, wherein the classification loss term is a binary cross-entropy loss.

[0028] Optionally, the defect image region group is obtained by detecting the filter chip image according to the defect region detection model, comprising:

[0029] Regionally dividing the filter chip image according to the region specification to obtain a plurality of divided image regions;

[0030] Extracting the divided image regions in the plurality of divided image regions in turn, and standardizing the divided image regions to obtain standard image regions;

[0031] Feature extraction is performed on the standard image regions to obtain divided region feature vectors;

[0032] The divided region feature vectors are input into the defect region detection model to obtain region defect probability values;

[0033] If the region defect probability value is greater than a preset standard probability value, the divided image region is recorded as a defect image region;

[0034] The defect image regions are summarized to obtain a defect image region group.

[0035] Optionally, the feature extraction on the standard image regions to obtain the divided region feature vectors comprises:

[0036] Color feature groups in the standard image regions are identified;

[0037] generate a gray level co-occurrence matrix of the standard image region, and calculate a texture feature group of the standard image region based on the gray level co-occurrence matrix, wherein the texture feature group comprises contrast of the gray level co-occurrence matrix, entropy of the gray level co-occurrence matrix, and angular second moment of the gray level co-occurrence matrix;

[0038] perform morphological detection on the standard image region to obtain a morphological feature group, wherein the morphological feature group comprises edge density, number of connected domains, and average area of connected domains;

[0039] merge the color feature group, the texture feature group, and the morphological feature group to obtain a divided region feature group, and construct a divided region feature vector based on the divided region feature group.

[0040] Optionally, the local contrast enhancement on the filtered chip image to obtain the target chip image comprises:

[0041] setting a low-frequency traversal template, and traversing the filtered chip image by using the low-frequency traversal template to obtain a plurality of traversal regions, wherein each traversal region comprises a plurality of region pixel points, and each region pixel point corresponds to a region gray value;

[0042] extracting the traversal regions in the plurality of traversal regions in sequence, and calculating average gray values of the plurality of region pixel points in the traversal regions;

[0043] if the average gray value is not greater than a preset standard gray value, the traversal region is recorded as a low-frequency region;

[0044] summarizing the low-frequency regions to obtain a plurality of low-frequency regions;

[0045] if the average gray value is greater than the standard gray value, the traversal region is recorded as a high-frequency region, and the high-frequency region is subjected to contrast enhancement to obtain an enhanced region;

[0046] summarizing the low-frequency regions and the enhanced regions respectively to obtain a plurality of low-frequency regions and a plurality of enhanced regions;

[0047] combining the plurality of low-frequency regions and the plurality of enhanced regions to obtain the target chip image.

[0048] Optionally, the contrast enhancement on the high-frequency region to obtain the enhanced region comprises:

[0049] calculating a gray standard deviation of the high-frequency region based on the average gray value, wherein the high-frequency region comprises a plurality of high-frequency pixel points, and each high-frequency pixel point corresponds to a high-frequency gray value;

[0050] setting a background gray value of the filtered chip image;

[0051] The high-frequency pixels are extracted in sequence from the plurality of high-frequency pixels, and the high-frequency pixels are subjected to contrast gain according to a preset gain formula, a gray scale standard deviation and a background gray scale value, to obtain enhanced pixels, wherein the gain formula is expressed as:

[0052]

[0053] wherein H' represents a gray scale value of the enhanced pixel, H represents a high-frequency gray scale value corresponding to the high-frequency pixel, H represents a background gray scale value, β represents a preset adjustment constant, σ represents a gray scale standard deviation, and H represents the high-frequency gray scale value corresponding to the high-frequency pixel; back

[0054] The enhanced pixels corresponding to each high-frequency pixel in the plurality of high-frequency pixels are summarized to obtain an enhanced pixel set, and the high-frequency region is updated based on the enhanced pixel set to obtain an enhanced region.

[0055] Optionally, the single-chip recognition on the target chip image is performed to obtain a single-chip region set, which comprises:

[0056] The contour extraction is performed on the target chip image to obtain a connected domain set, wherein the connected domain set comprises a plurality of connected domains, and the contour extraction is performed by using a Canny operator;

[0057] The total area of the chips in the target chip image is recognized, and the noise area is set based on the total area of the chips;

[0058] The connected domain area of each connected domain in the connected domain set is confirmed to obtain a connected domain area set, and the connected domain area set is filtered based on the total area of the chips and the noise area to obtain an effective connected domain area set, wherein the effective connected domain area set comprises a plurality of effective connected domain areas, and the effective connected domain area is greater than the noise area and less than the total area of the chips;

[0059] The effective connected domain set corresponding to the effective connected domain area set is confirmed in the connected domain set;

[0060] The effective connected domains are extracted in sequence from the effective connected domain set, and the connected domain center in the effective connected domain is recognized, wherein the connected domain center is the geometric center of the effective connected domain;

[0061] The image center in the target chip image is recognized, and the geometric distance between the image center and the connected domain center is calculated, wherein the image center is the geometric center of the target chip image;

[0062] The geometric distances are summarized to obtain a geometric distance set;

[0063] The minimum geometric distance in the geometric distance set is determined, the target connected domain center corresponding to the minimum geometric distance is confirmed, and the target connected domain center is taken as a target seed point;

[0064] ​Perform a preset region growing algorithm based on the target seed point to obtain a single-chip region set.

[0065] Optionally, the identifying a chip region group to be repaired from the single-chip region set in the defect image region group comprises:

[0066] The following operations are performed on each defect image region in the defect image region group:

[0067] Determine a defect chip region group contained in the defect image region in the single-chip region set, wherein the defect chip region group comprises a plurality of defect chip regions, and the defect chip regions are in the defect image region;

[0068] Extract the defect chip regions in the defect chip region group in sequence;

[0069] Identify a light-emitting region and an electrode region in the defect chip region according to a preset binarization method;

[0070] Obtain an electrode region template, compare the electrode region with the electrode region template, and obtain an electrode template matching degree;

[0071] If the electrode template matching degree is not greater than a preset standard matching degree, mark the defect chip region as a chip region to be repaired;

[0072] If the electrode template matching degree is greater than the standard matching degree, perform long strip connected domain identification on the light-emitting region to obtain a light-emitting region connected domain, and determine a connected domain length of the light-emitting region connected domain;

[0073] If the connected domain length is greater than a preset standard scratch length, mark the defect chip region as a chip region to be repaired;

[0074] Summarize the chip regions to be repaired to obtain a chip region group to be repaired.

[0075] To achieve the above-mentioned purpose, the application further provides a Mini-LED repair device based on automatic optical detection, comprising:

[0076] A detection instruction receiving module is configured to receive an optical detection instruction and determine a product set to be repaired based on the optical detection instruction, wherein the product set to be repaired comprises a plurality of products to be repaired, the product to be repaired is a Mini-LED product, the product to be repaired comprises a plurality of Mini-LED chips, the product to be repaired is extracted in sequence from the product set to be repaired, and the product to be repaired is placed in a preset background stage to obtain a product to be detected, wherein the background stage comprises a stage, a high-definition camera and a light source.

[0077] The chip image shooting module is configured to shoot the product to be detected by using a high-definition camera and a light source, obtain an optical chip image, perform Gaussian filtering on the optical chip image to obtain a filtered chip image, acquire an original chip image set, and train a pre-acquired neural network based on the original chip image set to obtain a defect region detection model, wherein the original chip image set includes a plurality of original chip images, and the original chip images are defect chip images or normal chip images.

[0078] The filtered image enhancement module is configured to detect the filtered chip image based on the defect region detection model to obtain a defect image region group, wherein the defect image region group includes a plurality of defect image regions or is an empty set, if the defect image region group is not an empty set, the product to be repaired is recorded as a repairable product, the filtered chip image is subjected to local contrast enhancement to obtain a target chip image.

[0079] The repair report generation module is configured to perform single chip recognition on the target chip image to obtain a single chip region set, identify a chip region set to be repaired in the defect image region group based on the single chip region set, generate a repair opinion report of the repairable product based on the chip region set to be repaired, and aggregate the repair opinion reports to obtain a plurality of repair opinion reports.

[0080] To solve the above problems, the present application further provides an electronic device, which comprises:

[0081] a memory configured to store at least one instruction; and

[0082] a processor configured to execute the instruction stored in the memory to implement the Mini-LED repair method based on automatic optical detection.

[0083] To solve the above problems, the present application further provides a computer readable storage medium, which stores at least one instruction, and the at least one instruction is executed by a processor in an electronic device to implement the Mini-LED repair method based on automatic optical detection.

[0084] The present application is to solve the problems described in the background art. First, an original chip image set is obtained, and a neural network is trained based on the original chip image set to obtain a defect region detection model. This step solves the imbalance problem of positive and negative samples in Mini-LED defect detection by constructing an original chip image set containing defect and normal chip images and using a neural network training strategy combined with a distribution overlap loss term. The defect region detection model can still maintain high sensitivity on minority class samples (such as rare defects such as fracture and electrode oxidation), thereby significantly reducing the miss rate. Then, the defect image region group is obtained by detecting the filtered chip image according to the defect region detection model. The trained defect region detection model is used to evaluate the region probability of the filtered chip image. This step realizes the pixel-level accurate positioning of the defect region through dynamic comparison of the region defect probability value and the standard probability value, avoiding false positives or false negatives caused by uneven illumination in traditional threshold segmentation methods, thereby improving the defect detection accuracy. Further, after confirming that the defect image region group is not empty, the local contrast enhancement technique is used to adaptively adjust the gain of the high-frequency region, significantly amplifying the gray difference between the defect region and the normal region, solving the problem of difficult identification of Mini-LED chip surface reflection or low-contrast defects, and providing a high-contrast image basis for subsequent chip-level defect positioning. Then, the single chip region set is obtained by recognizing the target chip image. The repairable chip region group is identified in the defect image region group according to the single chip region set, and the repair opinion report of the repairable product is generated based on the repairable chip region group. This step realizes the accurate segmentation of a single chip region in a densely arranged Mini-LED array through a single chip recognition method combining the Canny operator contour extraction and the region growing algorithm, and effectively distinguishes different types of defects such as electrode oxidation and chip scratches through a double verification mechanism combining electrode template matching and light-emitting area connected component analysis, avoiding misjudgment caused by chip adhesion or false connected components in traditional methods. Finally, the defect distribution map covering the entire Mini-LED batch is constructed through multiple repair opinion reports, so that the operator can directly locate the defect position of the specific chip without full-surface manual re-inspection, thereby greatly reducing the labor consumption and improving the efficiency of Mini-LED repair. Therefore, the present application can improve the accuracy of Mini-LED product defect detection and reduce the labor consumption of Mini-LED products. BRIEF DESCRIPTION OF DRAWINGS

[0085] Figure 1 The flowchart of the Mini-LED repair method based on automatic optical detection provided by an embodiment of the present application is shown.

[0086] Figure 2 The functional module diagram of the Mini-LED repair device based on automatic optical detection provided by an embodiment of the present application is shown.

[0087] Figure 3 A structural schematic diagram of an electronic device for implementing the Mini-LED repair method based on automatic optical detection is provided in an embodiment of the present application.

[0088] Legend of reference signs:

[0089] 1, electronic device; 10, processor; 11, memory; 12, bus.

[0090] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0091] It should be understood that the specific embodiments described herein are merely intended to explain the present application and not to limit the present application.

[0092] Embodiments of the present application provide a Mini-LED repair method based on automatic optical detection. The execution subject of the Mini-LED repair method based on automatic optical detection includes but is not limited to at least one of electronic devices such as a server, a terminal, etc. that can be configured to execute the method provided by the embodiments of the present application. In other words, the Mini-LED repair method based on automatic optical detection can be executed by software or hardware installed in a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to a single server, a server cluster, a cloud server, or a cloud server cluster, etc.

[0093] Reference Figure 1 As shown in the figure, a flowchart of a Mini-LED repair method based on automatic optical detection is provided in an embodiment of the present application. In the embodiment, the Mini-LED repair method based on automatic optical detection includes:

[0094] S1, receiving an optical detection instruction, and determining a set of products to be repaired based on the optical detection instruction, wherein the set of products to be repaired includes a plurality of products to be repaired, and the product to be repaired is a Mini-LED product, and the product to be repaired includes a plurality of Mini-LED chips.

[0095] As can be understood, the optical detection instruction refers to an instruction for detecting a specific Mini-LED product initiated by a person. The set of products to be repaired refers to a collection of a plurality of products to be repaired, and the product to be repaired refers to a specific Mini-LED product indicated by the optical detection instruction. The product to be repaired is, for example, a Mini-LED television panel, a Mini-LED vehicle display screen, etc. The product to be repaired is composed of a plurality of Mini-LED chips, and the Mini-LED chip refers to a LED light emitting unit with a size of 50-200 microns.

[0096] S2, sequentially extract the products to be repaired from the set of products to be repaired, and place the products to be repaired on a preset background table to obtain products to be detected, wherein the background table comprises: a carrier table, a high-definition camera and a light source.

[0097] It can be understood that the background table refers to a device for photographing the products to be repaired, which comprises: a carrier table, a high-definition camera and a light source, wherein the carrier table refers to a carrier for carrying the products to be repaired, the high-definition camera refers to an optical sensor for capturing high-resolution images, such as a 50 million pixel industrial CCD camera, and the light source refers to an optical assembly for providing uniform illumination, such as a ring-shaped LED white light source. The product to be detected refers to the product to be repaired placed in the background table.

[0098] S3, photograph the product to be detected using the high-definition camera and the light source to obtain an optical chip image, and perform Gaussian filtering on the optical chip image to obtain a filtered chip image.

[0099] It can be understood that the optical chip image refers to the front image of the product to be detected obtained after photographing, which includes all Mini-LED chips in the product to be detected. The filtered chip image refers to the image after Gaussian filtering of the optical chip image.

[0100] Illustratively, the detailed steps of photographing the product to be detected using the high-definition camera and the light source are as follows: an operator Zhang fixes the product to be detected at the center of the carrier table, then Zhang adjusts the light source to an incident angle of 45° (as specified by the operation standard), ensures uniform illumination, sets the camera parameters: sets the focal length and exposure time, triggers the camera to take pictures and saves the images.

[0101] S4, obtain an original chip image set, and train a pre-obtained neural network based on the original chip image set to obtain a defect area detection model, wherein the original chip image set comprises a plurality of original chip images, and the original chip images are defect chip images or normal chip images.

[0102] It needs to be explained that the original chip image set refers to a set comprising a plurality of original chip images, wherein the original chip image refers to an image of a product of the same model as the product to be repaired obtained artificially in advance, which includes: a defect chip image and a normal chip image, wherein the defect chip image refers to an image of a product having a defect (such as chip fracture, electrode oxidation, and fluorescent powder falling off) and needing to be repaired, and the normal chip image refers to an image of a product not needing to be repaired. The original chip images in the above original chip image set are all from the repaired products and normally produced products in the past period.

[0103] Further, the neural network can be selected as a convolutional neural network (CNN) or a Vision Transformer (ViT). The defect area detection model refers to a model capable of determining whether a defect exists in a product through a product image (an image of a product to be repaired). The input of the defect area detection model is a vector, and the output is a probability value (corresponding to a subsequent area defect probability value). When the probability value is greater than a preset standard probability value (corresponding to a subsequent standard probability value), it indicates that the product corresponding to the input image has a defect and needs to be repaired.

[0104] In detail, the pre-acquired neural network is trained based on the original chip image set to obtain a defect area detection model, including:

[0105] Original chip images are sequentially extracted from the original chip image set to obtain image labels of the original chip images, wherein the image labels include normal labels or defect labels;

[0106] The original chip images are divided according to a preset area specification to obtain a plurality of original image areas;

[0107] Original image areas are sequentially extracted from the plurality of original image areas, and the original image areas are standardized to obtain target image areas;

[0108] The target image areas are subjected to feature extraction to obtain target area feature vectors;

[0109] The target area feature vectors are labeled based on the image labels to obtain labeled area feature vectors;

[0110] The labeled area feature vectors corresponding to each of the plurality of original image areas are summarized to obtain a labeled area feature vector group, and the labeled area feature vector groups corresponding to each of the original chip images in the original chip image set are combined to obtain a labeled area feature vector set;

[0111] A loss function is set, and the neural network is trained using the loss function and the labeled area feature vector set to obtain the defect area detection model.

[0112] It needs to be explained that the area specification refers to a rectangular size set by man, which is used to divide the original chip image into multiple rectangular areas, and the size of the rectangular area is the area specification. The division refers to dividing the original chip image into multiple rectangular areas with the area specification, and there is no overlap between all rectangular areas. The plurality of original image areas refers to the plurality of areas in the original chip image after division, wherein the size of the original image area is the area specification. The target image area refers to the original image area after standardization, wherein standardization refers to normalization processing of pixel values (such as scaling pixel values to the [0, 1] range using z-score standardization), and the purpose of standardization is to eliminate the interference of light difference on feature extraction.

[0113] It can be understood that the target area feature vector refers to a vector quantifying the features of the target image area, and the acquisition steps of the target area feature vector and the subsequent acquisition method of the divided area feature vector are the same, and the acquisition method of the divided area feature vector will be given in detail later. The image label refers to the label of the state (normal and defect) of the product corresponding to the original chip image, wherein the normal label index value is 0, and the defect label index value is 1. The normal label and the defect label are set by man. The marked area feature vector refers to the target area feature vector after marking, wherein marking refers to associating the target area feature vector with the corresponding image label, and the marking step is to add a label field (i.e. value 0 or value 1) to each target area feature vector. The merging of the marked area feature vector group corresponding to each original chip image in the original chip image set refers to placing all the marked area feature vectors in the marked area feature vector group into the same set, and the set into which the marked area feature vector is placed is the marked area feature vector set. The loss function refers to a function for measuring the prediction error of the model. The above process of training the neural network is supervised training, and the training method is prior art, which will not be described here.

[0114] In detail, the setting of the loss function includes:

[0115] Confirming the training data set of the neural network in the training process, wherein the training data set includes a plurality of training data, and each training data includes a defect probability value and a true label value;

[0116] Constructing a distribution overlap loss term according to the training data set, wherein the distribution overlap loss term is expressed as:

[0117]

[0118] Wherein, F1 represents the distribution overlap loss term, n represents the number of training data in the training data set, r j represents the true label value of the jth training data in the training data set, rj ω represents a preset minimum value;

[0119] The distribution overlap loss term and the preset classification loss term are weighted and summed to obtain a loss function, wherein the classification loss term is a binary cross-entropy loss.

[0120] It can be understood that the training data set refers to a set of multiple training data, wherein the training data includes a defect probability value and a true label value, the defect probability value refers to a probability value output by the neural network after a certain training in the training process of the neural network, the true label value and the defect probability value come from the same iteration process, and the true label value refers to a value represented by an image label in a marked region feature vector in the iteration process. The distribution overlap loss term refers to a loss term for constraining the consistency of the predicted probability distribution and the true distribution. The reason for introducing the distribution overlap loss term is to alleviate the problem that the number of defect chip images and the number of normal chip images differ greatly (i.e., positive and negative samples are imbalanced). The higher the distribution overlap loss term, the higher the overlap degree of the predicted distribution and the true distribution, that is, the smaller the loss.

[0121] Further, the minimum value refers to a constant artificially set, and the minimum value functions to prevent the denominator in the formula from being 0. The classification loss term refers to a binary cross-entropy loss. Through the combined action of the classification loss term and the distribution overlap loss term, the distribution alignment and the classification accuracy of the model can be simultaneously optimized. The weighted sum is represented as F = α1 x F1 + α2 x F2, wherein F represents the loss function, α1 and α2 respectively represent an overlap loss weight artificially set and a classification loss weight artificially set, F2 represents the classification loss term, and the overlap loss weight and the classification loss weight respectively refer to weight coefficients of the distribution loss and the classification loss. Alternatively, the overlap loss weight and the classification loss weight are respectively set to 0.6 and 0.4.

[0122] S5, detecting the filtered chip image according to the defect region detection model to obtain a defect image region group, wherein the defect image region group includes multiple defect image regions or the defect image region group is an empty set.

[0123] It can be understood that the defect image region group refers to a combination of regions in which defects occur in the filtered chip image. Since there can be no defects in the filtered chip image, there is no defect image region in the filtered chip image at this time, that is, the defect image region group is an empty set.

[0124] In detail, the detecting the filtered chip image according to the defect region detection model to obtain the defect image region group includes:

[0125] performing region division on the filtered chip image according to the region specification to obtain multiple divided image regions;

[0126] extracting the divided image region in sequence, normalizing the divided image region to obtain a standard image region;

[0127] extracting features of the standard image region to obtain a divided region feature vector;

[0128] inputting the divided region feature vector into the defect region detection model to obtain a region defect probability value;

[0129] if the region defect probability value is greater than a preset standard probability value, recording the divided image region as a defect image region;

[0130] summarizing the defect image region to obtain a defect image region group.

[0131] It can be understood that the divided image region refers to a part of the filter chip image after region division. The step of region division is the same as the step of dividing the original chip image described above, and will not be repeated here. The standard image region refers to the divided image region after standardization, and the standardization is the same as the step of standardizing the original image region described above. The divided region feature vector refers to a vector quantifying the features of the standard image region. The region defect probability value refers to the output value of the defect region detection model. The higher the region defect probability value, the higher the probability of defects in the standard image region. The standard probability value refers to a probability constant set by a person.

[0132] In detail, the extracting features of the standard image region to obtain a divided region feature vector includes:

[0133] identifying a color feature group in the standard image region;

[0134] generating a gray level co-occurrence matrix of the standard image region, and calculating a texture feature group of the standard image region based on the gray level co-occurrence matrix, wherein the texture feature group includes: a contrast of the gray level co-occurrence matrix, an entropy of the gray level co-occurrence matrix, and an angular second moment of the gray level co-occurrence matrix;

[0135] performing morphological detection on the standard image region to obtain a morphological feature group, wherein the morphological feature group includes: edge density, number of connected domains, and average area of connected domains;

[0136] merging the color feature group, the texture feature group, and the morphological feature group to obtain a divided region feature group, and constructing a divided region feature vector based on the divided region feature group.

[0137] The color feature group is a combination of parameters representing color statistical characteristics of the image, and the color feature group includes RGB three-channel mean value, HSV space saturation variance, etc. The gray level co-occurrence matrix is a probability matrix describing the spatial relationship of the texture, and the contrast of the gray level co-occurrence matrix, the entropy of the gray level co-occurrence matrix, and the angular second moment of the gray level co-occurrence matrix in the texture feature group can be calculated by using existing calculation formulas. The morphological feature group is a combination of features describing the geometric structure in the standard image region, wherein the edge density is the number of edge pixels in the standard image region divided by the total number of pixels in the standard image region, the number of connected domains is the total number of connected domains obtained after binarization of the standard image region, and the mean value of the area of the connected domains is the average value of the area of all connected domains in the standard image region after binarization.

[0138] For example, the divided region feature group is (A1, A2, and A3), and the divided region feature vector corresponding to the divided region feature group is (A1 A2 A3).

[0139] S6, if the defect image region group is not empty, the product to be repaired is recorded as a repairable product, and the filter chip image is subjected to local contrast enhancement to obtain a target chip image.

[0140] It can be understood that if the defect image region group is not empty, it means that a defect has occurred in the product to be repaired, i.e., the product to be repaired needs to be repaired. In order to ensure the accuracy of subsequent repair, after confirming that the product to be repaired is a repairable product, it is necessary to determine the specific chip in the repairable product that needs to be repaired.

[0141] Further, the purpose of the local contrast enhancement is to improve the contrast between the defect region and the background. The target chip image refers to the filter chip image subjected to local contrast enhancement.

[0142] In detail, the local contrast enhancement of the filter chip image to obtain the target chip image includes:

[0143] A low-frequency traversal template is set, and the low-frequency traversal template is used to traverse the filter chip image to obtain a plurality of traversal regions, wherein the traversal region includes a plurality of region pixel points, and the region pixel points correspond to a region gray value;

[0144] The average gray value of the plurality of region pixel points in the traversal region is calculated by sequentially extracting the traversal region from the plurality of traversal regions;

[0145] If the average gray value is not greater than a preset standard gray value, the traversal region is recorded as a low-frequency region;

[0146] The low-frequency regions are summarized to obtain a plurality of low-frequency regions;

[0147] If the average gray value is greater than the standard gray value, the traversal region is recorded as a high-frequency region, and the high-frequency region is subjected to contrast enhancement to obtain an enhanced region;

[0148] The low-frequency regions and the enhanced regions are respectively aggregated to obtain a plurality of low-frequency regions and a plurality of enhanced regions;

[0149] The plurality of low-frequency regions and the plurality of enhanced regions are combined to obtain a target chip image.

[0150] It should be explained that the low-frequency traversal template refers to a sliding window (such as 15x15 pixels) artificially set, and the size of the low-frequency traversal template is set by human beings. The plurality of traversal regions refer to a plurality of regions passed through by the low-frequency traversal template in the traversal process in the filtered chip image. The region pixel point refers to a pixel point in the traversal region, and the region gray value refers to a gray value of the region pixel point. The average gray value refers to an average value of a plurality of region gray values of a plurality of region pixel points in the traversal region. The standard gray value refers to a gray value constant set by human beings. When the average gray value is not greater than the standard gray value, it indicates that the region is insufficiently illuminated, that is, the traversal region is recorded as a low-frequency region. If the average gray value is greater than the standard gray value, it indicates that the region has a risk of overexposure, that is, the traversal region is recorded as a high-frequency region. Since the high-frequency region is easy to mask the subtle defects, it is necessary to enhance the contrast of the high-frequency region. The enhanced region refers to the high-frequency region after the contrast enhancement. The combination of the plurality of low-frequency regions and the plurality of enhanced regions refers to recombining the processed regions according to the original coordinates.

[0151] In detail, the contrast of the high-frequency region is enhanced to obtain an enhanced region, including:

[0152] The gray standard deviation of the high-frequency region is calculated based on the average gray value, wherein the high-frequency region includes a plurality of high-frequency pixel points, and each high-frequency pixel point corresponds to a high-frequency gray value;

[0153] The background gray value of the filtered chip image is set;

[0154] The high-frequency pixel points are sequentially extracted from the plurality of high-frequency pixel points, and the contrast gain of the high-frequency pixel points is performed according to a preset gain formula, the gray standard deviation and the background gray value to obtain enhanced pixel points, wherein the gain formula is expressed as:

[0155]

[0156] wherein H' represents the gray value of the enhanced pixel point, H represents the high-frequency gray value corresponding to the high-frequency pixel point, H represents the background gray value, β represents a preset adjustment constant, σ represents the gray standard deviation, and H represents the high-frequency gray value corresponding to the high-frequency pixel point. back

[0157] ​The high-frequency pixel points in the plurality of high-frequency pixel points are aggregated to obtain a set of enhanced pixel points, and the high-frequency region is updated based on the set of enhanced pixel points to obtain an enhanced region.

[0158] It can be understood that the gray scale standard deviation refers to the standard deviation of a plurality of high-frequency gray scale values corresponding to a plurality of high-frequency pixel points in the high-frequency region. The background gray scale value refers to the gray scale value of the background part in the high-frequency region, which is set by manually detecting the background part, wherein the background part refers to the black encapsulating adhesive region between the Mini-LED chips. The gain formula refers to a formula for calculating the gray scale value of the enhanced pixel point, wherein the enhanced pixel point refers to the high-frequency pixel point after contrast gain. The adjustment constant refers to a constant set by human, which is dimensionless. The adjustment constant controls the enhancement amplitude. Optionally, the adjustment constant is set to 1.8.

[0159] Further, updating the high-frequency region based on the set of enhanced pixel points refers to replacing the plurality of high-frequency pixel points in the high-frequency region with the set of enhanced pixel points. The high-frequency region after replacement is the enhanced region.

[0160] S7, performing single chip recognition on the target chip image to obtain a set of single chip regions, identifying a set of chip regions to be repaired in the defect image region group according to the set of single chip regions, and generating a repair opinion report of the repairable product based on the set of chip regions to be repaired.

[0161] It can be understood that the set of single chip regions refers to a set including a plurality of single chip regions, wherein a single chip region refers to a region containing only one Mini-LED chip. The set of chip regions to be repaired includes a plurality of chip regions to be repaired, and a chip region to be repaired refers to a single chip region containing a Mini-LED chip with defects (such as large-area scratches, etc.). The repair opinion report refers to a report comprehensively describing the repairable product, which contains the product number of the repairable product and the position of each chip region to be repaired in the set of chip regions to be repaired corresponding to the repairable product.

[0162] In detail, the single chip recognition on the target chip image to obtain a set of single chip regions includes:

[0163] contour extraction is performed on the target chip image to obtain a set of connected domains, wherein the set of connected domains includes a plurality of connected domains, and the contour extraction is performed by using a Canny operator;

[0164] The total area of the chips in the target chip image is identified, and the noise area is set based on the total area of the chips.

[0165] Confirm the connected domain area of each connected domain in the connected domain set to obtain a connected domain area set, and screen the connected domain area set based on the total chip area and the noise area to obtain an effective connected domain area set, wherein the effective connected domain area set includes a plurality of effective connected domain areas, and the effective connected domain area is greater than the noise area and less than the total chip area;

[0166] Confirm the effective connected domain set corresponding to the effective connected domain area set in the connected domain set;

[0167] Extract the effective connected domain in the effective connected domain set in sequence, and identify the connected domain center in the effective connected domain, wherein the connected domain center is the geometric center of the effective connected domain;

[0168] Identify the image center in the target chip image, and calculate the geometric distance between the image center and the connected domain center, wherein the image center is the geometric center of the target chip image;

[0169] Summarize the geometric distances to obtain a geometric distance set;

[0170] Determine the minimum geometric distance in the geometric distance set, confirm the target connected domain center corresponding to the minimum geometric distance, and take the target connected domain center as a target seed point;

[0171] Execute a preset region growing algorithm based on the target seed point to obtain a single-chip region set.

[0172] It can be understood that the total chip area refers to the total area of the target chip image. The connected domain area refers to the area of the connected domain. The noise area refers to the connected domain area generated in the target chip image due to a small noise, and the noise area is calculated in the following manner: total chip area x 1%. The effective connected domain area set refers to the connected domain area set after screening. The above step of screening the connected domain area set based on the total chip area to obtain the effective connected domain area set aims to remove pseudo-connected domains with excessively large (tray edge) or small (noise spot) areas, and only keep the real chip region. The effective connected domain set refers to the set of a plurality of connected domains corresponding to each effective connected domain area in the effective connected domain area set. The geometric distance refers to the Euclidean geometric distance between the image center and the connected domain center. The minimum geometric distance refers to the geometric distance with the smallest value in the geometric distance set. The target connected domain center refers to the connected domain center corresponding to the minimum geometric distance. The target seed point refers to the connected domain center point closest (in distance) to the image center of the target chip image. The reason for determining the minimum geometric distance and the target connected domain center and taking the target connected domain center as the target seed point is that the center region is least affected by optical distortion, and by this method, it can be ensured that the region growing starts from the most complete chip. The above region growing algorithm is a prior art, which will not be described here.

[0173] In detail, the identifying the chip region group to be repaired from the single-chip region set in the defect image region group comprises:

[0174] The following operations are performed on each defect image region in the defect image region group:

[0175] The defect chip region group contained in the defect image region is determined in the single-chip region set, wherein the defect chip region group comprises a plurality of defect chip regions, and the defect chip regions are in the defect image region;

[0176] The defect chip regions are extracted in the defect chip region group in sequence;

[0177] The light-emitting region and the electrode region in the defect chip region are identified according to a preset binarization method;

[0178] An electrode region template is obtained, the electrode region is compared by using the electrode region template, and an electrode template matching degree is obtained;

[0179] If the electrode template matching degree is not greater than a preset standard matching degree, the defect chip region is recorded as a chip region to be repaired;

[0180] If the electrode template matching degree is greater than the standard matching degree, the light-emitting region is identified as a long strip-shaped connected domain, a light-emitting region connected domain is obtained, and a connected domain length of the light-emitting region connected domain is determined;

[0181] If the connected domain length is greater than a preset standard scratch length, the defect chip region is recorded as the chip region to be repaired;

[0182] The chip regions to be repaired are summarized, and a chip region group to be repaired is obtained.

[0183] It can be understood that the defective chip region refers to the region of a single Mini-LED chip contained in the defective image region. The binarization method refers to: first, the gray histogram of the defective chip region is counted, and the defective chip region is binarized according to the gray histogram, so as to obtain the light-emitting region and the electrode region. The counting of the above-mentioned gray histogram and the binarization of the image are both prior art, and will not be described here. The light-emitting region refers to the light-emitting grain part of the Mini-LED chip contained in the Mini-LED chip in the defective chip region, and the electrode region refers to the metal structure of the gold wire or the pad. The electrode region template refers to the binary image of the standard defect-free electrode, and the electrode region template is obtained by: collecting 100 groups of normal electrode images and taking the average to generate. The electrode template matching degree refers to a numerical value quantifying the similarity between the electrode region template and the electrode region. The greater the electrode template matching degree, the higher the similarity between the electrode region template and the electrode region. The calculation method of the electrode template matching degree is: calculating the structural similarity index (SSIM) between the electrode region template and the electrode region. The standard matching degree refers to a matching degree constant set by humans. When the electrode template matching degree is not greater than the standard matching degree, it means that the electrode region and the electrode region template are quite different, that is, the electrode region has defects.

[0184] Further, the light-emitting region connected domain refers to the rectangular connected domain in the light-emitting region. The identification method of the light-emitting region connected domain is: using hough straight line detection method for identification. The connected domain length refers to the length of the longer side of the light-emitting region connected domain. The standard scratch length refers to a length set by humans. If the connected domain length is greater than the standard scratch length, it means that the light-emitting region in the defective chip region has scratches, and needs to be repaired.

[0185] S8, aggregate the repair opinion report to obtain a plurality of repair opinion reports, and complete the Mini-LED repair based on automatic optical detection based on the plurality of repair opinion reports.

[0186] It needs to be explained that each repair opinion report in the plurality of repair opinion reports corresponds to a repairable product, and each repair opinion report gives the position of the Mini-LED chip that may have defects in the repairable product, thereby greatly reducing the manual inspection of all chips of the repairable product.

[0187] The present application is to solve the problems described in the background art. First, an original chip image set is obtained, and a neural network is trained based on the original chip image set to obtain a defect region detection model. This step solves the imbalance problem of positive and negative samples in Mini-LED defect detection by constructing an original chip image set containing defect and normal chip images and using a neural network training strategy combined with a distribution overlap loss term. The defect region detection model can still maintain high sensitivity on minority class samples (such as rare defects such as fracture and electrode oxidation), thereby significantly reducing the miss rate. Then, the defect image region group is obtained by detecting the filtered chip image according to the defect region detection model. The trained defect region detection model is used to evaluate the region probability of the filtered chip image. This step realizes the pixel-level accurate positioning of the defect region through dynamic comparison of the region defect probability value and the standard probability value, avoiding false positives or false negatives caused by uneven illumination in traditional threshold segmentation methods, thereby improving the defect detection accuracy. Further, after confirming that the defect image region group is not empty, the local contrast enhancement technique is used to adaptively adjust the gain of the high-frequency region, significantly amplifying the gray difference between the defect region and the normal region, solving the problem of difficult identification of Mini-LED chip surface reflection or low-contrast defects, and providing a high-contrast image basis for subsequent chip-level defect positioning. Then, the single chip region set is obtained by recognizing the target chip image. The repairable chip region group is identified in the defect image region group according to the single chip region set, and the repair opinion report of the repairable product is generated based on the repairable chip region group. This step realizes the accurate segmentation of a single chip region in a densely arranged Mini-LED array through a single chip recognition method combining Canny operator contour extraction and region growing algorithm, and effectively distinguishes different types of defects such as electrode oxidation and chip scratches through a double verification mechanism combining electrode template matching and light-emitting area connected component analysis, avoiding misjudgment caused by chip adhesion or false connected components in traditional methods. Finally, the defect distribution map covering the entire Mini-LED batch is constructed through multiple repair opinion reports, so that the operator can directly locate the defect position of the specific chip without full-surface manual re-inspection, thereby greatly reducing the labor consumption and improving the efficiency of Mini-LED repair. Therefore, the present application can improve the accuracy of Mini-LED product defect detection and reduce the labor consumption of Mini-LED products.

[0188] As Figure 2 shown is a functional module diagram of a Mini-LED repair device based on automatic optical detection according to an embodiment of the present application.

[0189] The Mini-LED repair device based on automatic optical detection 100 can be installed in an electronic device. According to the implemented function, the Mini-LED repair device based on automatic optical detection 100 can include a detection instruction receiving module 101, a chip image shooting module 102, a filtered image enhancement module 103, and a repair report generation module 104. The modules in the present application can also be referred to as units, which refer to a series of computer program segments that can be executed by an electronic device processor and can complete a fixed function, which are stored in the memory of the electronic device.

[0190] The detection instruction receiving module 101 is used to receive an optical detection instruction, determine a set of products to be repaired based on the optical detection instruction, wherein the set of products to be repaired includes a plurality of products to be repaired, and the product to be repaired is a Mini-LED product, and the product to be repaired includes a plurality of Mini-LED chips, a product to be repaired is extracted in the set of products to be repaired, and the product to be repaired is placed in a preset background stage to obtain a product to be detected, wherein the background stage includes: a stage, a high-definition camera and a light source;

[0191] The chip image shooting module 102 is used to shoot the product to be detected by using the high-definition camera and the light source to obtain an optical chip image, perform Gaussian filtering on the optical chip image to obtain a filtered chip image, obtain a set of original chip images, and train a pre-obtained neural network based on the set of original chip images to obtain a defect area detection model, wherein the set of original chip images includes a plurality of original chip images, and the original chip image is a defective chip image or a normal chip image;

[0192] The filtered image enhancement module 103 is used to detect the filtered chip image according to the defect area detection model to obtain a set of defect image regions, wherein the set of defect image regions includes a plurality of defect image regions or the set of defect image regions is an empty set, if the set of defect image regions is not an empty set, the product to be repaired is recorded as a repairable product, the filtered chip image is subjected to local contrast enhancement to obtain a target chip image;

[0193] The repair report generation module 104 is used to identify a single chip in the target chip image to obtain a set of single chip regions, identify a set of chip regions to be repaired in the set of defect image regions according to the set of single chip regions, generate a repair opinion report of the repairable product based on the set of chip regions to be repaired, and summarize the repair opinion report to obtain a plurality of repair opinion reports.

[0194] In detail, the modules in the Mini-LED repair device based on automatic optical detection 100 in the embodiment of the present application are used as described above Figure 1The Mini-LED repair method based on automatic optical detection described in the embodiment of the present application has the same technical means and can produce the same technical effects, and details are not repeated here.

[0195] As shown in Figure 3 is a structural schematic diagram of an electronic device for implementing the Mini-LED repair method based on automatic optical detection provided by an embodiment of the present application.

[0196] The electronic device 1 can include a processor 10, a memory 11 and a bus 12, and can further include a computer program stored in the memory 11 and executable on the processor 10, such as a Mini-LED repair method based on automatic optical detection program.

[0197] The memory 11 includes at least one type of readable storage medium, including flash memory, mobile hard disk, multimedia card, card-type memory (such as SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the electronic device 1, such as a mobile hard disk of the electronic device 1. In other embodiments, the memory 11 can also be an external storage device of the electronic device 1, such as a plug-in mobile hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 1. Further, the memory 11 includes an internal storage unit of the electronic device 1 and also includes an external storage device. The memory 11 can be used not only to store application software and various data installed on the electronic device 1, such as the code of the Mini-LED repair method based on automatic optical detection program, but also to temporarily store data that has been output or will be output.

[0198] The processor 10 can be composed of integrated circuits in some embodiments, such as a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same function or different functions, including one or more central processing units (CPU), microprocessors, digital processing chips, graphics processors and combinations of various control chips, etc. The processor 10 is the control unit of the electronic device, which connects all components of the electronic device through various interfaces and lines, executes or runs programs or modules stored in the memory 11 (such as the Mini-LED repair method based on automatic optical detection program, etc.), and calls data stored in the memory 11 to perform various functions and process data of the electronic device 1.

[0199] The bus 12 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus 12 can be divided into an address bus, a data bus, a control bus, etc. The bus 12 is configured to enable connection and communication between the memory 11, the at least one processor 10, etc.

[0200] Figure 3 Only the electronic device with components is shown, and those skilled in the art can understand that, Figure 3 The structure shown does not constitute a limitation on the electronic device 1, and can include fewer or more components than shown, or combine certain components, or different component arrangements.

[0201] For example, although not shown, the electronic device 1 can also include a power supply (such as a battery) to power each component. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, so that the power management device can implement functions such as charge management, discharge management, and power consumption management. The power supply can also include one or more direct current or alternating current power supplies, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, etc. The electronic device 1 can also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which are not described here.

[0202] Further, the electronic device 1 can also include a network interface, which can optionally include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), and is typically used to establish a communication connection between the electronic device 1 and other electronic devices.

[0203] Optionally, the electronic device 1 can also include a user interface, which can be a display (Display), an input unit (such as a keyboard (Keyboard)), and optionally a standard wired interface, a wireless interface. Optionally, in some embodiments, the display can be an LED display, a liquid crystal display, a touch liquid crystal display, an OLED (Organic Light-Emitting Diode) touch, etc. The display can also be appropriately referred to as a display screen or a display unit, and is used to display information processed in the electronic device 1 and to display a visualized user interface.

[0204] The Mini-LED repair method program based on automatic optical detection stored in the memory 11 in the electronic device 1 is a combination of multiple instructions, which can realize the following technical effects when running in the processor 10:

[0205] receiving an optical detection instruction, determining a set of products to be repaired based on the optical detection instruction, wherein the set of products to be repaired includes multiple products to be repaired, and the product to be repaired is a Mini-LED product, and the product to be repaired includes multiple Mini-LED chips;

[0206] extracting the product to be repaired from the set of products to be repaired in turn, and placing the product to be repaired on a preset background stage to obtain a product to be detected, wherein the background stage includes: a stage, a high-definition camera and a light source;

[0207] using the high-definition camera and the light source to take a picture of the product to be detected to obtain an optical chip image, and performing Gaussian filtering on the optical chip image to obtain a filtered chip image;

[0208] obtaining a set of original chip images, and training a pre-obtained neural network based on the set of original chip images to obtain a defect region detection model, wherein the set of original chip images includes multiple original chip images, and the original chip image is a defective chip image or a normal chip image;

[0209] detecting the filtered chip image according to the defect region detection model to obtain a set of defect image regions, wherein the set of defect image regions includes multiple defect image regions or the set of defect image regions is an empty set;

[0210] if the set of defect image regions is not an empty set, the product to be repaired is recorded as a repairable product, and the filtered chip image is subjected to local contrast enhancement to obtain a target chip image;

[0211] performing single chip recognition on the target chip image to obtain a set of single chip regions, identifying a set of chip regions to be repaired in the set of defect image regions according to the set of single chip regions, and generating a repair opinion report of the repairable product based on the set of chip regions to be repaired;

[0212] summarizing the repair opinion reports to obtain multiple repair opinion reports, and completing the Mini-LED repair based on automatic optical detection based on the multiple repair opinion reports.

[0213] Specifically, the specific implementation method of the processor 10 to the above instructions can refer to Figures 1 to 3 The description of related steps in the corresponding embodiments is omitted here.

[0214] Further, the modules / units integrated in the electronic device 1, if implemented in the form of software function units and sold or used as independent products, can be stored in a computer readable storage medium. The computer readable storage medium can be volatile or non-volatile. For example, the computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM).

[0215] The application further provides a computer readable storage medium, which stores a computer program, and the computer program can implement the following when executed by a processor of an electronic device:

[0216] receiving an optical detection instruction, determining a set of products to be repaired based on the optical detection instruction, wherein the set of products to be repaired includes a plurality of products to be repaired, the product to be repaired is a Mini-LED product, and the product to be repaired includes a plurality of Mini-LED chips;

[0217] extracting the product to be repaired in the set of products to be repaired in turn, and placing the product to be repaired on a preset background stage to obtain a product to be detected, wherein the background stage includes a stage, a high-definition camera and a light source;

[0218] photographing the product to be detected by using the high-definition camera and the light source to obtain an optical chip image, and performing Gaussian filtering on the optical chip image to obtain a filtered chip image;

[0219] obtaining a set of original chip images, and training a pre-obtained neural network based on the set of original chip images to obtain a defect region detection model, wherein the set of original chip images includes a plurality of original chip images, and the original chip image is a defective chip image or a normal chip image;

[0220] detecting the filtered chip image according to the defect region detection model to obtain a set of defect image regions, wherein the set of defect image regions includes a plurality of defect image regions or the set of defect image regions is an empty set;

[0221] if the set of defect image regions is not an empty set, recording the product to be repaired as a repairable product, performing local contrast enhancement on the filtered chip image to obtain a target chip image;

[0222] performing single chip recognition on the target chip image to obtain a set of single chip regions, identifying a set of chip regions to be repaired in the set of defect image regions according to the set of single chip regions, and generating a repair opinion report of the repairable product based on the set of chip regions to be repaired;

[0223] The repair opinion reports are summarized, multiple repair opinion reports are obtained, and the Mini-LED repair based on automatic optical detection is completed based on the multiple repair opinion reports.

[0224] In several embodiments provided by the present application, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative, and actual implementation can have other division manners.

[0225] The modules described as separate components can or can not be physically separated, and the components shown as modules can or can not be physical units, that is, they can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to actual needs.

[0226] In addition, each functional module in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically independently, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of hardware plus software functional module.

[0227] It is obvious for those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, and the present application can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application.

[0228] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solutions of the present application can be modified or replaced equivalently without departing from the spirit and scope of the technical solutions of the present application.

Claims

1. A Mini-LED rework method based on automated optical inspection, characterized in that, The method includes: Receive optical inspection instructions, determine the set of products to be repaired based on the optical inspection instructions, wherein the set of products to be repaired includes multiple products to be repaired, and the products to be repaired are Mini-LED products, and the products to be repaired include multiple Mini-LED chips; Products to be repaired are extracted sequentially from the set of products to be repaired and placed in a preset background stage to obtain the product to be inspected. The background stage includes a stage, a high-definition camera and a light source. The product to be tested is photographed using a high-definition camera and a light source to obtain an image of the optical chip. The optical chip image is then subjected to Gaussian filtering to obtain a filtered chip image. A set of raw chip images is acquired, and a pre-acquired neural network is trained based on the raw chip image set to obtain a defect region detection model. The raw chip image set includes multiple raw chip images, and the raw chip images are defective chip images or normal chip images. The defect region detection model is used to detect the filter chip image to obtain a defect image region group, which may include multiple defect image regions or be an empty set. If the defective image region group is not an empty set, the product to be repaired is recorded as a repairable product, and the local contrast enhancement is performed on the filter chip image to obtain the target chip image. Single chip identification is performed on the target chip image to obtain a single chip region set. Based on the single chip region set, a group of chip regions to be repaired is identified in the defect image region group. A repair opinion report of the reworkable product is generated based on the chip regions to be repaired. The return feedback reports were compiled, resulting in multiple return feedback reports. Based on these multiple return feedback reports, the Mini-LED return was completed using automated optical inspection.

2. The Mini-LED rework method based on automated optical inspection as described in claim 1, characterized in that, The defect region detection model is obtained by training a pre-acquired neural network based on the original chip image set, including: The original chip images are extracted sequentially from the original chip image set, and the image labels of the original chip images are obtained. The image labels include: normal labels or defect labels. The original chip image is divided according to the preset region specifications to obtain multiple original image regions; The original image regions are extracted sequentially from multiple original image regions, and the original image regions are standardized to obtain the target image region. Feature extraction is performed on the target image region to obtain the target region feature vector; The feature vector of the target region is labeled based on the image label, and the labeled region feature vector is obtained; The feature vectors of the labeled regions corresponding to each original image region in multiple original image regions are summarized to obtain a set of labeled region feature vectors. The feature vector sets of the labeled regions corresponding to each original chip image in the original chip image set are then merged to obtain a set of labeled region feature vectors. A loss function is set, and the neural network is trained using the loss function and the feature vector set of the labeled region to obtain a defect region detection model.

3. The Mini-LED rework method based on automated optical inspection as described in claim 2, characterized in that, The setting of the loss function includes: The training dataset of the neural network during the training process is identified, wherein the training dataset includes multiple training data, and each training data includes: defect probability value and true label value; Construct a distribution overlap loss term based on the training dataset, where the distribution overlap loss term is expressed as: Where F1 represents the distribution overlap loss term, n represents the number of training data in the training dataset, and r j r represents the true label value of the j-th training data in the training dataset. j′ Let ω represent the defect probability value of the j-th training data in the training dataset, and let ω represent the preset minimum value. The weighted summation of the distribution overlap loss term and the preset classification loss term yields the loss function, where the classification loss term is the binary cross-entropy loss.

4. The Mini-LED rework method based on automatic optical inspection as described in claim 3, characterized in that, The step of detecting the filter chip image according to the defect region detection model to obtain a group of defect image regions includes: The filter chip image is divided into regions according to the region specifications to obtain multiple divided image regions; Extract segmented image regions sequentially from multiple segmented image regions, and standardize the segmented image regions to obtain standard image regions; Feature extraction is performed on the standard image region to obtain the feature vector of the segmented region; The feature vector of the segmented region is input into the defect region detection model to obtain the region defect probability value; If the probability value of a region defect is greater than the preset standard probability value, then the divided image region is recorded as a defective image region. The defective image regions are summarized to obtain a defective image region group.

5. The Mini-LED rework method based on automatic optical inspection as described in claim 4, characterized in that, The step of extracting features from the standard image region to obtain the region feature vector includes: Identify color feature groups in a standard image region; Generate a gray-level co-occurrence matrix for a standard image region, and calculate a texture feature set for the standard image region based on the gray-level co-occurrence matrix. The texture feature set includes: the contrast of the gray-level co-occurrence matrix, the entropy of the gray-level co-occurrence matrix, and the second angular moment of the gray-level co-occurrence matrix. Morphological detection is performed on a standard image region to obtain a morphological feature set, which includes: edge density, number of connected components, and average area of ​​connected components. The color feature group, texture feature group, and morphology feature group are merged to obtain the region segmentation feature group, and the region segmentation feature vector is constructed based on the region segmentation feature group.

6. The Mini-LED rework method based on automated optical inspection as described in claim 5, characterized in that, The step of performing local contrast enhancement on the filter chip image to obtain the target chip image includes: Set a low-frequency traversal template and use it to traverse the image of the filter chip to obtain multiple traversal regions. Each traversal region includes multiple region pixels, and each region pixel corresponds to a region grayscale value. Extract the traversed regions sequentially from multiple traversed regions, and calculate the average gray value of the pixels in multiple regions within the traversed regions; If the average gray value is not greater than the preset standard gray value, the traversed area is recorded as a low-frequency area. By summarizing the low-frequency regions, multiple low-frequency regions are obtained; If the average gray value is greater than the standard gray value, the traversed area is recorded as a high-frequency area, and the high-frequency area is enhanced by comparison to obtain the enhanced area. By summing up the low-frequency region and the enhancement region respectively, multiple low-frequency regions and multiple enhancement regions are obtained; The target chip image is obtained by combining multiple low-frequency regions and multiple enhancement regions.

7. The Mini-LED rework method based on automated optical inspection as described in claim 6, characterized in that, The enhancement of the high-frequency region to obtain the enhanced region includes: The gray standard deviation of the high-frequency region is calculated based on the average gray value. The high-frequency region includes multiple high-frequency pixels, and each high-frequency pixel corresponds to a high-frequency gray value. Set the background grayscale value of the image of the filter chip; High-frequency pixels are extracted sequentially from multiple high-frequency pixels. Contrast gain is applied to these high-frequency pixels based on a preset gain formula, grayscale standard deviation, and background grayscale value to obtain enhanced pixels. The gain formula is expressed as: Where H′ represents the grayscale value of the enhanced pixel, H back β represents the background grayscale value, σ represents the preset adjustment constant, σ represents the grayscale standard deviation, and H represents the high-frequency grayscale value corresponding to the high-frequency pixel. The enhanced pixels corresponding to each high-frequency pixel in the plurality of high-frequency pixels are summarized to obtain an enhanced pixel set. The high-frequency region is updated based on the enhanced pixel set to obtain the enhanced region.

8. The Mini-LED rework method based on automatic optical inspection as described in claim 7, characterized in that, The step of performing single-chip recognition on the target chip image to obtain a single-chip region set includes: Contour extraction is performed on the target chip image to obtain a set of connected components, which includes multiple connected components. The contour extraction method is to use the Canny operator. Identify the total chip area in the target chip image and set the noise area based on the total chip area; The area of ​​each connected component in the connected component set is identified to obtain the connected component area set. The connected component area set is then filtered based on the total chip area and the noise area to obtain the effective connected component area set. The effective connected component area set includes multiple effective connected component areas, and the effective connected component area is greater than the noise area and less than the total chip area. In the set of connected components, identify the set of valid connected components that corresponds to the set of valid connected component areas; Extract the effective connected components sequentially from the set of effective connected components, and identify the center of each effective connected component, where the center of each connected component is the geometric center of the effective connected component. Identify the image center in the target chip image and calculate the geometric distance between the image center and the center of the connected region, where the image center is the geometric center of the target chip image; By summing the geometric distances, a geometric distance set is obtained; Determine the minimum geometric distance in the geometric distance set, identify the center of the target connected region corresponding to the minimum geometric distance, and use the center of the target connected region as the target seed point; A preset region growing algorithm is executed based on the target seed point to obtain a single-chip region set.

9. The Mini-LED rework method based on automatic optical inspection as described in claim 8, characterized in that, The step of identifying the chip region group to be repaired in the defect image region group based on the single chip region set includes: Perform the following operation on each defect image region in the defect image region group: In a single chip region set, a defect chip region group is determined within the defect image region, wherein the defect chip region group includes multiple defect chip regions, and the defect chip regions are located within the defect image region. Defective chip regions are extracted sequentially from the defective chip region group; Based on a preset binarization method, identify the light-emitting region and electrode region in the defective chip area; Obtain an electrode region template, and use the electrode region template to compare the electrode region to obtain the electrode template matching degree; If the electrode template matching degree is not greater than the preset standard matching degree, the defective chip area is recorded as the chip area to be repaired. If the electrode template matching degree is greater than the standard matching degree, then the elongated connected region of the luminescent area is identified to obtain the connected region of the luminescent area and determine the length of the connected region of the luminescent area. If the length of the connected region is greater than the preset standard scratch length, the defective chip region is recorded as the chip region to be repaired. The areas of chips to be repaired are summarized to obtain a group of chip areas to be repaired.

10. A Mini-LED rework device based on automatic optical inspection, characterized in that, The device includes: The detection instruction receiving module is used to receive optical detection instructions, determine the set of products to be reworked based on the optical detection instructions, wherein the set of products to be reworked includes multiple products to be reworked, and the products to be reworked are Mini-LED products, and each product to be reworked includes multiple Mini-LED chips. The products to be reworked are extracted sequentially from the set of products to be reworked, and the products to be reworked are placed in a preset background stage to obtain the product to be inspected. The background stage includes: a stage, a high-definition camera and a light source. The chip image capturing module is used to capture the product to be inspected using a high-definition camera and a light source to obtain an optical chip image. The optical chip image is then subjected to Gaussian filtering to obtain a filtered chip image. The original chip image set is obtained, and a pre-acquired neural network is trained based on the original chip image set to obtain a defect area detection model. The original chip image set includes multiple original chip images, and the original chip images are either defective chip images or normal chip images. The filter image enhancement module is used to detect the filter chip image according to the defect region detection model to obtain a defect image region group. The defect image region group includes multiple defect image regions or the defect image region group is an empty set. If the defect image region group is not an empty set, the product to be repaired is recorded as a repairable product. The filter chip image is then locally contrast enhanced to obtain the target chip image. The rework report generation module is used to perform single-chip identification on the target chip image to obtain a single-chip region set, identify the chip region group to be repaired in the defect image region group based on the single-chip region set, generate a rework opinion report for the reworkable product based on the chip region group to be repaired, and summarize the rework opinion reports to obtain multiple rework opinion reports.

Images