A small sample Mini LED defect detection method and system based on multi-dimensional measurement

Abstract

The application discloses a kind of based on multi-dimension measurement small sample Mini LED defect detection method and system, method includes the following steps: S1: obtaining Mini LED screen picture and carrying out image gray processing;S2: the image of gray processing is carried out binaryzation processing;S3: the contour coordinates of image after binaryzation processing are determined, and single LED matrix image is cut according to contour coordinates;S4: with ResNet18 as feature extraction module, construct detection model and train, obtain the detection model after training;S5: the single LED matrix image to be detected is input into the detection model of training, and the defect detection result is output.The application can quickly determine LED matrix position by image preprocessing to obtain single LED matrix image, and multi-dimension measurement is used in detection model to overcome the problem that the similarity dimension of classical measurement method is single and cannot express the similarity of image well, so as to improve the accuracy of defect detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of screen defect detection, and more particularly to a small sample Mini LED defect detection method and system based on multi-dimensional measurement. BACKGROUND

[0002] With the rapid development of Mini LED display technology, Mini LED display products have begun to be applied to ultra-large screen high-definition display, such as monitoring command, high-definition broadcasting, high-end cinema, medical diagnosis, advertising display, conference and exhibition, office display, virtual reality, and other commercial fields. However, as the demand for Mini LED increases, various problems in the production process have followed, and various product defect problems have become prominent. Traditional LED screens form a whole, and the photographed pictures are relatively smooth. Defects are relatively clear compared to Mini LED. Mini LED requires that the photographed pictures can clearly show each LED matrix, and requires detecting and identifying defects for each LED matrix, which greatly increases the difficulty of defect detection.

[0003] Traditional visual inspection is based on difference, filtering and some operations. The defect detection effect of ordinary smooth LED screen is significant, and different defects on the screen can be well identified. However, for Mini LED, the independent LED lamp matrix makes it different from ordinary LED, and its detection accuracy must reach single lamp detection. Therefore, traditional machine vision is difficult to accurately detect Mini LED.

[0004] The principle of the template matching method is to use the LED matrix as a template to traverse the entire picture to identify the module coordinates with high similarity to the template picture. This method directly compares the total similarity of the pixel values of the corresponding positions of the image and the template. This single standard cannot well distinguish different defect types. Although increasing the threshold can improve the accuracy and detect the missing lamp defects, it cannot well solve the problem of large rotation angle of the lamp bead, and cannot well detect the defects of the Mini LED screen.

[0005] The target detection-based method also has some problems in detecting Mini LED screens: the structure of a single LED matrix is complex, and pictures of different defects have great similarity. Therefore, directly using target detection to detect them requires a large number of labeled pictures. At present, there are problems in obtaining a large number of Mini LED pictures, and a large amount of manpower and material resources are required to label the obtained pictures, and there are many difficulties.

[0006] The prior art discloses a Mini LED product defect detection method and related equipment, the method comprising: collecting a Mini LED product image and taking the Mini LED product image as a detection image; selecting a template feature database corresponding to the Mini LED product, inputting the detection image into the template feature database corresponding to the Mini LED product, and obtaining a reference image by using a function library; based on the detection image and the reference image, obtaining a residual image by using an ET algorithm; marking pixel points with a gray value greater than a threshold value in the residual image as defect points, and marking pixel points with a gray value not greater than the threshold value as non-defect points. The scheme needs enough samples to construct the template feature database, and cannot realize accurate detection of small samples. SUMMARY

[0007] The present application provides a small sample Mini LED defect detection method and system based on multi-dimensional measurement, which realizes small sample defect detection and improves the accuracy of defect detection.

[0008] The primary object of the present application is to solve the above technical problems, and the technical scheme of the present application is as follows:

[0009] The present application provides a small sample Mini LED defect detection method based on multi-dimensional measurement, comprising the following steps:

[0010] S1: Obtain a Mini LED screen image and perform image gray scale processing;

[0011] S2: Perform binary processing on the gray scale processed image;

[0012] S3: Determine the contour coordinates of the binary processed image, and crop the single LED matrix image according to the contour coordinates;

[0013] S4: Construct a detection model using ResNet18 as a feature extraction module and train it to obtain a trained detection model;

[0014] S5: Input the single LED matrix image to be detected into the trained detection model, and output the defect detection result.

[0015] Further, the mathematical expression of the image gray scale processing in step S1 is:

[0016]

[0017] Wherein, Gray(i,j) represents the gray value of pixel point (i,j), R(i,j) represents the R component of the image pixel point, G(i,j) represents the G component of the image pixel point, and B(i,j) represents the B component of the image pixel point.

[0018] Further, in the binarization operation, if the pixel gray value on the gray image is less than the preset threshold, it is set to 0, and if it is greater than the preset threshold, it is set to the maximum value maxval.

[0019] Further, the contour coordinates of the image after binarization processing are determined, and the specific steps of cutting the single LED matrix image according to the contour coordinates are as follows:

[0020] The contour of the image after binarization processing is determined by the hollow internal point method, the coordinates of the upper left and lower right points of the contour are determined, the circumscribed rectangle of the LED matrix is determined according to the coordinates of the upper left and lower right points, and the single LED matrix image is cut according to the circumscribed rectangle.

[0021] Further, ResNet18 is used as a feature extraction module to construct a detection model, wherein the detection model uses a multi-dimensional measurement method MDM as a measurement method, and a cross-entropy loss function as a loss function for training and prediction.

[0022] Further, the training of the detection model includes three parts: data processing of the network, determination of the training mode of the network, and pre-training of the auxiliary data set.

[0023] Data processing of the network:

[0024] In the single task, N labeled categories are randomly selected from the training data set, K samples are randomly sampled in each category as the training set of the current task, and one or more non-repeated samples are randomly sampled as the test set; the input of the network model is extracted to the feature vector of the training set through a standard feature extraction layer g(θ);

[0025] The test set is used to evaluate the performance of the detection model, and after obtaining the feature extraction result from the images in the training set and the test set, the extracted feature map is input into the measurement function to calculate the MDM similarity;

[0026] Network training mode:

[0027] The training process of the network is to divide the training set into small few-shot subsets, and divide them into training sets and test sets, that is, the training phase of the network is actually the superposition of multiple FSL learning;

[0028] Pre-training of the auxiliary data set:

[0029] The network training is a way of pre-training of the auxiliary data set and fine-tuning of the target data set, wherein the auxiliary training set is the PVEL-AD data set.

[0030] Furthermore, the detection model employs a multi-dimensional measurement approach, which includes: a cosine distance unit and a structural similarity unit. The cosine distance unit is used to calculate the cosine distance, while the structural similarity unit is used to calculate the similarity between two feature vectors from the dimensions of brightness, contrast, and structure.

[0031] Furthermore, the structural similarity units are measured in three aspects: brightness, contrast, and structure.

[0032] Brightness: The brightness value μ is obtained by averaging all pixel values. x ,

[0033]

[0034] Where, x i Let be the value of the i-th pixel in image x, and N be the total number of pixel values;

[0035] Contrast ratio: Measured by the standard deviation (square root of variance) of all pixel values; contrast ratio σ x The expression is:

[0036]

[0037] σ x Represents the contrast of image x;

[0038] Structure: The study focuses on two normalized vectors: (x-μ) x ) / σ x and (y-μ) y ) / σ y The relationship between the structures, and the specific formula for the structure s(x,y):

[0039]

[0040] Define the brightness comparison function, contrast comparison function, and structure comparison function respectively;

[0041] The expression for the brightness comparison function is:

[0042]

[0043] Where, μ x μ represents the brightness of image x. y This represents the brightness of image y, where x and y are the two images being compared;

[0044] The expression for the contrast comparison function is:

[0045]

[0046] σ xContrast of image x, sigma y Contrast of image y; x and y are two images compared;

[0047] The structure comparison function expression is:

[0048]

[0049] Wherein, sigma xy Defined as:

[0050]

[0051] Where C1, C2, C3 are constants, to ensure the stability when the denominator is 0, then:

[0052] C1=(K1L) 2 ,C2=(K2L) 2 ,C3=(K3L) 2

[0053] Define the structure similarity function:

[0054] SSIM(x,y)=[l(x,y)] alpha·[c(x,y)] β ·[s(x,y)] γ

[0055] Where alpha>0, beta>0, gamma>0 represent the relative importance of each metric, simplify the expression, let:

[0056]

[0057] Further, the expression of multi-dimensional metric MDM distance is:

[0058]

[0059] Wherein, alpha is a training variable, which is trained together with the model, and is continuously optimized through gradient descent back propagation, so as to learn the optimal division.

[0060] The second aspect of the application provides a small sample Mini LED defect detection system based on multi-dimensional metric, which comprises a memory and a processor, the memory comprises a small sample Mini LED defect detection method based on multi-dimensional metric, and the small sample Mini LED defect detection method based on multi-dimensional metric is executed by the processor to realize the following steps:

[0061] S1: Obtain Mini LED screen picture and perform image gray processing;

[0062] S2: binaryzation processing is conducted on the image subjected to the gray scale processing;

[0063] S3: contour coordinates of the image subjected to the binaryzation processing are determined, and the single LED matrix image is cropped according to the contour coordinates;

[0064] S4: a detection model is constructed by taking ResNet18 as a feature extraction module and is trained, and the trained detection model is obtained;

[0065] S5: the single LED matrix image to be detected is input into the trained detection model, and a defect detection result is output.

[0066] Compared with the prior art, the beneficial effects of the technical scheme of the present application are:

[0067] The present application can quickly determine the position of the LED matrix through image preprocessing to obtain a single LED matrix image, and the multi-dimensional measurement method is used in the detection model to overcome the problem that the single dimension of the similarity degree calculated by the classical measurement method cannot well express the picture similarity, thereby improving the accuracy of defect detection. BRIEF DESCRIPTION OF DRAWINGS

[0068] Figure 1 A small sample Mini LED defect detection method flow chart based on multi-dimensional measurement for an embodiment of the present application.

[0069] Figure 2 An effect image after gray scale processing for an embodiment of the present application.

[0070] Figure 3 A binaryzation processing effect image for an embodiment of the present application.

[0071] Figure 4 An outline extraction effect image for an embodiment of the present application.

[0072] Figure 5 A structural diagram of a structural similarity unit for an embodiment of the present application.

[0073] Figure 6 A single LED matrix schematic diagram for an embodiment of the present application. DETAILED DESCRIPTION

[0074] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the present application will be further described in detail below in combination with the drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0075] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, it will be apparent to one skilled in the art that the present application can be practiced without the specific details set forth in this description, that the present application can be practiced with other than the described embodiments, and that the present application can be practiced with other than the described embodiments.

[0076] Embodiment 1

[0077] As shown in the figure, the first aspect of the present application provides a small sample Mini LED defect detection method based on multi-dimensional measurement, comprising the following steps: Figure 1

[0078] S1: Obtain the Mini LED screen picture and perform image gray scale processing;

[0079] It should be noted that since the industrial camera collects the Mini LED screen picture with sufficient clarity through line scanning, and then sends it to the PC end for detection, and each LED matrix is independently distributed and has great similarity, the detection standard also requires us to accurately identify the type of each LED matrix, which is relatively strict for detection. If the entire picture is directly detected, it is difficult to achieve a relatively accurate degree, and it will also have a great burden on the hardware device. Therefore, the present application proposes an image preprocessing method to position and segment the input Mini LED image.

[0080] As shown in the figure, it is the effect picture after gray scale processing. Figure 2

[0081] First, the gray scale processing is performed. In image processing, RGB three components (R: Red, G: Green, B: Blue), i.e. red, green, and blue three primary colors, are used to represent true color. The value range of R component, G component, and B component is 0-255. The color of each pixel in a color image is determined by R, G, and B components, and each component has 255 values. Thus, a pixel point can have a color variation range of more than 1.6 million (255*255*255). A gray scale image is a special color image with the same R, G, and B components. The variation range of a pixel point of the gray scale image is 255. Therefore, in digital image processing, various formats of images are generally converted into gray scale images to reduce the subsequent image calculation amount. The description of the gray scale image, like the color image, still reflects the overall and local color and brightness level distribution and characteristics of the entire image. The average brightness of the three components in the color image is obtained to obtain a gray scale value, and the formula is as follows:

[0082]

[0083] ​​Wherein, Gray(i, j) represents the gray value of pixel point (i, j), R(i, j) represents the R component of the image pixel point, G(i, j) represents the G component of the image pixel point, and B(i, j) represents the B component of the image pixel point.

[0084] It should be noted that the average method is used in the present application to perform the gray-scale operation on the input color image, so that the three-channel picture input is converted into a single channel, so as to facilitate some subsequent detection and processing operations.

[0085] S2: performing a binaryzation processing on the gray-scale processed image;

[0086] As Figure 3 The binaryzation processing effect diagram is shown.

[0087] It should be noted that the image binaryzation processing is to obtain a binaryzation image which can still reflect the overall and local features of the image by selecting a proper threshold value from the 256 gray level gray-scale image. In digital image processing, the binary image occupies a very important position. First, the binaryzation of the image is beneficial to the further processing of the image, so that the image becomes simple, the data amount is reduced, and the outline of the target of interest can be highlighted. Second, to process and analyze the binary image, the gray-scale image must be binaryzated first to obtain a binaryzation image.

[0088] In image processing, there are many systems implemented by binary image processing. To process and analyze the binary image, the gray-scale image must be binaryzated first to obtain a binaryzation image, which is beneficial to the further processing of the image. The collective nature of the image is only related to the position of the pixel value of 0 or 255, and no longer involves the multi-level value of the pixel, so that the processing becomes simple, and the data processing and compression amount is small. In order to obtain an ideal binary image, a closed and connected boundary is generally used to define a non-overlapping region. All pixels with a gray value greater than or equal to the threshold value are determined to belong to a specific object, and the gray value of 255 is used to represent it. Otherwise, these pixel points are excluded from the object region, and the gray value of 0 is used to represent the background or the exception irrelevant object region.

[0089] Because of the characteristics of the Mini LED product picture, only the LED matrix is bright, and the other regions are black or have a low gray value. Therefore, the following method is adopted in the present application: in the binaryzation operation, if the gray value of the pixel point on the gray-scale image is less than the preset threshold value, it is set to 0, and if it is greater than the preset threshold value, it is set to the maximum value maxval. By selecting a suitable threshold value, the product region can be highlighted to the greatest extent for subsequent operations.

[0090] S3: determining the outline coordinates of the binaryzation processed image, and cutting the single LED matrix image according to the outline coordinates;

[0091] As Figure 4 The figure shows the contour extraction effect diagram.

[0092] It should be noted that the method used by the present application is to determine the position information of each LED matrix block by finding the contour and fitting its circumscribed rectangle, so as to accurately cut it out for later use.

[0093] The specific steps are as follows:

[0094] The contour of the image after binarization processing is determined by the hollow internal point method, the coordinates of the upper left and lower right points of the contour are determined, the circumscribed rectangle of the LED matrix is determined according to the coordinates of the upper left and lower right points, and the single LED matrix image is cut according to the circumscribed rectangle.

[0095] It should be noted that the position and contour coordinates of the LED matrix are accurately identified by the above method, and the left uppermost and right lowermost point coordinates are marked as the circumscribed rectangle coordinates of the LED matrix. As shown in the figure, the object that fits the object is the accurate contour of the identified object, and the peripheral frame is the circumscribed rectangle determined according to the contour coordinates. The single LED matrix can be cut down for detection based on measurement to determine whether it has defects and the type of defects.

[0096] S4: using ResNet18 as a feature extraction module to construct a detection model and train, to obtain a trained detection model;

[0097] It should be noted that the present application mainly adopts a small sample learning method based on MDM measurement for MiniLED matrix block detection operation, which can be detected after a small amount of picture training, without the need for a large number of labeled pictures for training as in traditional target detection. Based on small sample measurement, a small amount of picture training can meet the detection requirements, greatly saving the training time and labeling cost.

[0098] More specifically, the present application uses ResNet18 as a feature extraction module to construct a detection model, wherein the detection model uses a multi-dimensional measurement method MDM as a measurement method, and a cross-entropy loss function (CrossEntropy Loss) as a loss function for training and prediction.

[0099] Example 2

[0100] This embodiment details the detection model. The training of the detection model in the present application includes three parts: network data processing, determination of network training mode, and pre-training auxiliary data set;

[0101] Network data processing:

[0102] In the single task of the present application, N label categories are randomly selected from the training data set, K samples are randomly sampled in each category as the training set of the current task, and one or more non-repeated samples are randomly sampled as the test set; the input of the network model is extracted into the feature vector of the training set through a standard feature extraction layer g (theta);

[0103] The test set is used to evaluate the performance of the detection model, and after obtaining the feature extraction result from the images of the training set and the test set, the extracted feature map is input into the metric function to calculate the MDM similarity;

[0104] The training mode of the network:

[0105] The training process of the network is to divide the training set into small few-shot subsets, and divide them into training sets and test sets, that is, the training stage of the network is actually the superposition of multiple FSL learning;

[0106] It should be noted that the training process of the matching network of the present application is different from the traditional deep learning method. The present application replicates the data distribution of the test data in the training data. In short, the present application divides the training set into small few-shot subsets, and divides them into training sets and test sets, which actually reproduces the process of few-shot learning in the training process. That is, the training stage of the matching network is actually the superposition of multiple FSL learning.

[0107] Pre-training auxiliary data set:

[0108] The network training is trained in the manner of pre-training of the auxiliary data set and fine-tuning of the target data set, wherein the auxiliary training set is the PVEL-AD data set.

[0109] It should be noted that the PVEL-AD data set, also known as the EL2021 data set, is a data set for benchmark testing of photovoltaic cell abnormal defect detection methods. PVEL-AD contains 36,543 near-infrared images with various internal defects and heterogeneous backgrounds, including 1 class of non-abnormal images and 12 different classes of abnormal defect images, such as cracks (linear and star-shaped), broken grids, black cores, misalignment, thick lines, scratches, fragments, broken corners, and material defects. In addition, 40,000+ real annotation boxes are provided for 12 types of defects for defect detection. Some defect types in the PVEL-AD data set have some similarities with the defects of the MiniLED matrix, and can well learn the differences between different defects in order to effectively detect defects such as non-luminous lamp beads and missing lamp beads. By constructing an N-way K-shot mode on the auxiliary data set for training, the model can learn how to distinguish different features of different defect categories. In addition, some defects in the PVEL-AD data set have great similarities with the defect features in the target data set, so after pre-training, the model can achieve good detection effect after fine-tuning or even without fine-tuning.

[0110] Further, the detection model adopts a multi-dimensional measurement method, which includes a cosine distance unit and a structural similarity unit. The cosine distance unit is used to calculate the cosine distance, and the structural similarity unit is used to calculate the similarity between two feature vectors from the dimensions of brightness, contrast, and structure.

[0111] As shown in Figure 5 , it is a structural diagram of the structural similarity unit.

[0112] Further, the structural similarity unit measures three aspects: brightness, contrast, and structure.

[0113] Brightness: The brightness value μ is obtained by averaging all pixel values. x ,

[0114]

[0115] where x i is the i-th pixel value of image x, and N is the total number of pixel values.

[0116] Contrast: The standard deviation (square root of variance) of all pixel values is taken to measure the contrast σ x , and the expression is:

[0117]

[0118] σ x represents the contrast of image x.

[0119] Structure: The structure comparison is done by using a combined formula, but in essence, it is divided by the standard deviation of the input signal, so the result has units of standard deviation, which gives a more robust comparison. What is actually studied is the relationship between the normalized two vectors: (x-μ x ) / σ x and (y-μ y ) / σ y The specific formula for the structure s(x,y) is:

[0120]

[0121] So far, the mathematical intuition behind these three parameters has been established, and the mathematical operations are not yet complete, there is one more thing. What is missing now is a comparison function that can compare two given images on these parameters, and finally, a combination function that combines the comparison functions together. The present invention defines the comparison function, and finally defines the combination function that produces the similarity index value.

[0122] Define the luminance comparison function, the contrast comparison function, and the structure comparison function, respectively;

[0123] The expression of the luminance comparison function is:

[0124]

[0125] Where μ x represents the luminance of image x, μ y represents the luminance of image y, and x and y are two images to be compared;

[0126] The expression of the contrast comparison function is:

[0127]

[0128] σ x represents the contrast of image x, σ y represents the contrast of image y; x and y are two images to be compared;

[0129] The expression of the structure comparison function is:

[0130]

[0131] Where σ xy is defined as:

[0132]

[0133] Where C1, C2, C3 are constants to ensure stability when the denominator is 0, then:

[0134] C1 = (K1L) 2 C2 = (K2L) 2 C3 = (K3L) 2

[0135] Define the structural similarity function:

[0136] SSIM(x, y) = [l(x, y)]a·[c(x, y)]b·[s(x, y)]g β γ

[0137] where a > 0, b > 0, g > 0 represent the relative importance of each metric,

[0138] For simplicity of expression, let:

[0139]

[0140] Further, the expression of the multi-dimensional metric MDM distance is:

[0141]

[0142] where a is a training variable that is trained together with the model, and is continuously optimized through back propagation of gradient descent, so as to learn the optimal division. It should be noted that the obtained MDM multi-dimensional metric method can well distinguish the differences between different single LED matrices in different classes, and better measure and judge the defect categories.

[0143] S5: input the single LED matrix image to be detected into the trained detection model, and output the defect detection result.

[0144] Embodiment 3

[0145] The second aspect of the application provides a small sample Mini LED defect detection system based on multi-dimensional metric, which comprises a memory and a processor, the memory comprises a small sample Mini LED defect detection method based on multi-dimensional metric, and the small sample Mini LED defect detection method based on multi-dimensional metric is implemented when the processor is executed.

[0146] S1: acquire a Mini LED screen picture and perform image grayscale processing;

[0147] S2: perform binaryzation processing on the grayscale processed image;

[0148] S3: determine the contour coordinates of the binaryzation processed image, and crop the single LED matrix image according to the contour coordinates;

[0149] ​S4: A detection model is constructed and trained with ResNet18 as the feature extraction module, and a trained detection model is obtained;

[0150] S5: The single LED matrix image to be detected is input into the trained detection model, and a defect detection result is output.

[0151] Embodiment 4

[0152] This embodiment is verified and illustrated by specific experiments and data.

[0153] (1) The image preprocessing module segments the picture into independent LED matrixes

[0154] A certain amount of Mini LED picture data is obtained by camera shooting, and is segmented by the preprocessing module. The main methods are as follows:

[0155] First, the captured Mini LED picture is subjected to grayscale processing, and the three-channel picture data is converted into a single channel for subsequent operation.

[0156] Second, the converted single-channel image is subjected to binary threshold processing to convert it into a binary image with white target and black background, so that the target area is more obviously displayed.

[0157] Then, the binary image is subjected to contour extraction to extract the contours of each LED matrix, and the extracted contours are subjected to fitting of the circumscribed rectangle to find the circumscribed rectangle.

[0158] Next, the image is cropped according to the circumscribed rectangle to obtain a single LED matrix, as shown in the following figure.

[0159] Finally, these single LED matrices are classified according to defect types (such as missing lamp beads, excessive rotation of lamp beads, lamp bead cracks, unlit lamp beads, and normal pictures), so they can be roughly divided into five categories of picture sets. After division, they are ready for the next step. Figure 6 The single LED matrix is shown in the following figure.

[0160] (2) Small sample model pre-training based on metrics

[0161] 1) Parameter initialization: initialization of parameters refers to the process of initializing and assigning values to the weights and biases of each node before training the network model.

[0162] Weight initialization: random initialization is a method often used by many people, and the initialized weights in the present application are randomly selected values from a Gaussian or uniform distribution.

[0163] Learning rate: If the learning rate is too small, the convergence is slow. If the learning rate is too large, the loss will fluctuate and even increase. The initial learning rate is generally in the range of 10-6 to 1.0. The general idea is to observe the curve of the loss or accuracy with respect to the learning rate, and then choose the best learning rate as the initial learning rate according to a certain strategy.

[0164] Batchsize: Unlike general deep learning, small sample learning adopts a scenario training mechanism, that is, one batch corresponds to one task, and the data is read according to the set n-way k-shot mode in a single task.

[0165] Training process:

[0166] Pre-training uses the PVEL-AD dataset, and the N-way K-shot data reading mode is constructed in the program for training. This invention uses 5-way 1-shot, that is, five categories of defects are randomly selected each time, and one picture in each category is used as the support set, and 15 pictures in each category are read as the query set.

[0167] In each individual training task, the n-way k-shot mode is used to read data, that is, 80 pictures are read into each task, including 1 support set picture and 15 query set pictures in each category. For each category, 5*15=75 pictures are input into the feature extractor f(θ) to obtain 75 corresponding feature quantities Pi(i=1~75), and the corresponding support set picture is also input into the feature extractor g(θ) for feature extraction to obtain Vj(j=1~5). The obtained query set feature quantities and support set feature vectors are input into the metric module for similarity measurement, and then the similarity scores of the query image for each category are obtained. According to these similarity scores, the loss is calculated to obtain the gradient for backpropagation.

[0168] (3) Defect detection for small sample Mini LED

[0169] Specific to the steps of implementing detection:

[0170] First, read the model weight that has been trained to achieve the ideal accuracy, input it into the model, and input the picture for detection. At the same time, read 1 (1-shot) or 5 (5-shot) pictures for each defect as support set pictures to form a 5-way 1-shot data mode;

[0171] Second, the Mini LED picture taken by the camera in real time is input into the network, and the image preprocessing module is used to locate and segment the LED matrix block. The segmented LED matrix is input into the metric network as the query set for convolution operation for feature extraction;

[0172] Then, a measure is performed on the convolution of the feature matrix representing each LED matrix and the support set, so as to obtain a similarity score of each LED matrix to each class in the support set, and the class with the highest score is the defect.

[0173] Obviously, the above embodiments of the present application are merely exemplary for clearly illustrating the present application, but are not intended to limit the implementation modes of the present application. Based on the above description, other different forms of changes or variations can be made by those skilled in the art. Here, all the implementation modes are not required or can not be exhausted. Any modification, equivalent replacement and improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the claims of the present application.

Claims

1. A method for small sample Mini LED defect detection based on multi-dimension measurement, characterized in that, The method comprises the following steps: S1: Obtain a Mini LED screen picture and perform image grayscale processing; S2: Perform binaryzation processing on the image after grayscale processing; S3: Determine the contour coordinates of the image after binaryzation processing, and crop the single LED matrix image according to the contour coordinates; S4: Construct a detection model by taking ResNet18 as a feature extraction module and train the detection model to obtain a trained detection model; S5: Input the single LED matrix image to be detected into the trained detection model, and output a defect detection result; The specific steps of determining the contour coordinates of the image after binaryzation processing and cropping the single LED matrix image according to the contour coordinates are as follows: Determine the contour of the image after binaryzation processing by using the hollow internal point method, determine the coordinates of the left upper and right lower points of the contour, determine the circumscribed rectangle of the LED matrix according to the coordinates of the left upper and right lower points, and crop the single LED matrix image according to the circumscribed rectangle; The detection model adopts a multi-dimension measurement method (MDM) as a measurement method, and a cross-entropy loss function as a loss function for training and prediction. The training of the detection model comprises three parts: data processing of the network, determination of the training mode of the network, and a pre-training auxiliary data set; Data processing of the network: In a single task, N label categories are randomly selected from the training data set, K samples in each category are randomly sampled as the training set of the current task, and one or more non-repeated samples are randomly sampled as the test set; the input of the network model is extracted into a feature vector of the training set through a standard feature extraction layer g(θ); The test set is used to evaluate the performance of the detection model, and after obtaining the feature extraction result from the images in the training set and the test set, the extracted feature map is input into a measurement function to calculate the MDM similarity; Training mode of the network: The training process of the network is to divide the training set into small few-shot subsets, and divide them into training sets and test sets, that is, the training stage of the network is actually the superposition of multiple FSL learning; Pre-training auxiliary data set: The network training is a way of training the target data set by pre-training the auxiliary data set, wherein the auxiliary training set is a PVEL-AD data set.

2. The small sample Mini LED defect detection method based on multi-dimension measurement according to claim 1, characterized in that, The mathematical expression of the image grayscale processing in step S1 is: Wherein, Gray(i,j) represents the gray value of pixel point (i,j), R(i,j) represents the R component of the image pixel point, G(i,j) represents the G component of the image pixel point, and B(i,j) represents the B component of the image pixel point.

3. The small sample Mini LED defect detection method based on multi-dimension measurement according to claim 1, characterized in that, In the binaryzation operation, if the gray value of the pixel point on the gray image is less than the preset threshold, it is set to 0, and if it is greater than the preset threshold, it is set to the maximum value maxval.

4. The small sample Mini LED defect detection method based on multi-dimension measurement according to claim 1, characterized in that, The detection model adopts a multi-dimension measurement method, which includes a cosine distance unit and a structural similarity unit, wherein the cosine distance unit is used to calculate the cosine distance, and the structural similarity unit is used to calculate the similarity between two feature vectors from the dimensions of brightness, contrast and structure.

5. The small sample Mini LED defect detection method based on multi-dimension measurement according to claim 4, characterized in that, The structural similarity unit measures three aspects: brightness, contrast and structure. Luminance: The average luminance value μ is measured by averaging all pixel values x , where x i is the i-th pixel value of the image x, and N is the total number of pixel values. Contrast: measured as the standard deviation of all pixel values, contrast σ x The expression is: σ x denotes the contrast of the image x; Structure: The study is of the relationship between the normalized two vectors: (x-μ x ) / σ x and (y-μ y ) / σ y The specific formula for structure s(x,y) is: A luminance comparison function, a contrast comparison function, and a structure comparison function are respectively defined. An expression of the luminance comparison function is: where μ x denotes the average luminance value of the image x, μ y denotes the average luminance value of the image y, x and y are the two images being compared; An expression of the contrast comparison function is: σ x σ represents the contrast of image x. y This represents the contrast of image y; x and y are the two images being compared. An expression of the structure comparison function is: where σ xy is defined as: Wherein C1, C2, C3 are constants, and the stability when the denominator is 0 is ensured, so: C1 = (K1L) 2 C2 = (K2L) 2 C3 = (K3L) 2 A structure similarity function is defined as: wherein β>0, γ>0 indicate the relative importance of each metric, simplifying the expression, let:

6. The small sample Mini LED defect detection method based on multi-dimension measurement according to claim 5, characterized in that, An expression of the multi-dimensional metric MDM distance is: Wherein, alpha is a training variable, which is trained together with the model, and is continuously optimized through back propagation of gradient descent, so as to learn the optimal division. 7.A small sample Mini LED defect detection system based on multi-dimension measurement, characterized in that, The application discloses a small sample Mini LED defect detection method based on multi-dimensional metric, and the system comprises a memory and a processor, the memory comprises an application program, and the application program is executed by the processor to realize the following steps: S1: acquiring a Mini LED screen picture and performing image grayscale processing; S2: performing binaryzation processing on the grayscale processed image; S3: determining the contour coordinates of the binaryzation processed image, and cutting the single LED matrix image according to the contour coordinates; S4: taking ResNet18 as a feature extraction module to construct a detection model and train the detection model, and obtaining a trained detection model; S5: inputting a single LED matrix image to be detected into the trained detection model, and outputting a defect detection result.

Images