Mini LED defect detection method, electronic equipment and medium

Abstract

The invention discloses a Mini LED defect detection method, electronic equipment and a medium, and the method comprises the steps: obtaining a defective Mini LED image, marking the key position and category of the image, and taking the image as a training set; constructing and training a defect detection model; the defect detection model comprises the following steps: carrying out feature extraction on a Mini LED image, and carrying out multi-scale feature layer processing to obtain first, second and third scale features; performing feature alignment and global information fusion processing on the first scale feature, the second scale feature and the third scale feature to obtain global fusion information; decomposing the global fusion information to obtain first fusion information and second fusion information; the first fusion information is spliced with the first scale feature after detail enhancement processing, and a first prediction result is output; the second fusion information is spliced with the second scale feature after being subjected to spatial cross processing, and a second prediction result is output; directly outputting the third scale feature as a third prediction result; and inputting a to-be-detected Mini LED image into the originally trained defect detection model to obtain a defect detection result.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and in particular relates to a MiniLED defect detection method, electronic equipment, and medium. Background Art

[0002] MiniLED is a new type of light-emitting device with broad application prospects. It is in great demand in display technology, medical and communication industries. To ensure product life and user experience, high-quality MiniLED must be defect-free. Therefore, defect detection becomes a crucial link. Traditional manual detection methods have the disadvantages of inconsistent judgment standards, low detection efficiency and high human resource costs. These disadvantages limit its reliability in large-scale industrial detection. At the same time, some complex and tiny defects are difficult to be accurately detected by the human eye and cannot meet the requirements of high precision.

[0003] In recent years, researchers have conducted extensive research on deep learning detection algorithms based on neural networks, and gradually developed the target detection model from a two-stage model represented by RCNN and Fast RCNN to a faster one-stage model. Single-stage detection models, especially the YOLO series, are widely welcomed by the industry due to their simple structure and high computational efficiency. However, directly applying the current YOLO method to MiniLED detection still has the following shortcomings:

[0004] (1) The complex production process leads to a wide variety of MiniLED defects, which have different shapes and complex texture information. During the detection process, there are morphological differences within the same type of defects, and the location distribution is random, requiring the algorithm to have flexible data processing capabilities. At the same time, different types of defects may have similar features in appearance, which increases the risk of false detection and missed detection, and aggravates the difficulty of detection.

[0005] (2) The current neck network of the YOLO series usually uses FPN for context feature fusion. However, this network structure has obvious drawbacks in industrial detection. The traditional FPN structure can only transfer features from adjacent layers. When cross-layer information needs to be integrated, it will cause the loss of global information and hinder the full fusion of features. Summary of the invention

[0006] In view of the deficiencies in the prior art, the present invention provides a MiniLED defect detection method, electronic device, and medium.

[0007] In a first aspect, an embodiment of the present invention provides a MiniLED defect detection method, the method comprising:

[0008] Obtain defective MiniLED images, and annotate key locations and categories of defects in the defective MiniLED images as training sets;

[0009] Construct a defect detection model, and set a loss function to train the defect detection model using a training set; the defect detection model includes:

[0010] Feature extraction is performed on the MiniLED image, and after multi-scale feature layer processing, first-scale features, second-scale features, and third-scale features are obtained;

[0011] Perform feature alignment and global information fusion processing on the first scale feature, the second scale feature, and the third scale feature to obtain global fusion information;

[0012] Decomposing the global fusion information to obtain first fusion information and second fusion information;

[0013] The first fusion information is processed by detail enhancement and then spliced ​​with the first scale feature to output a first prediction result; the second fusion information is processed by spatial crossover and then spliced ​​with the second scale feature to output a second prediction result; the third scale feature is directly output as a third prediction result;

[0014] The MiniLED image to be inspected is input into the previously trained defect detection model to obtain the defect detection result.

[0015] In a second aspect, an embodiment of the present invention provides an electronic device, comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned MiniLED defect detection method.

[0016] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the above-mentioned MiniLED defect detection method is implemented.

[0017] In a fourth aspect, an embodiment of the present invention provides a computer program product, including a computer program / instruction, which implements the above-mentioned MiniLED defect detection method when executed by a processor.

[0018] The same as the prior art, the beneficial effects of the present invention are:

[0019] The present invention provides a MiniLED defect detection method, which obtains a MiniLED image data set by processing and annotating MiniLED images in real industry. The defect detection model provided by the present invention achieves better extraction effect by fully fusing global and local context information through an improved distributed fusion neck, and successfully solves the problem of detecting multi-scale targets in MiniLED. At the same time, the SCIoU-Loss function is used as the loss function, which introduces a scale weight based on variance operation, improves the sensitivity to the difference in bounding box size, and accelerates the convergence speed and improves the regression accuracy. The method of the present invention effectively improves the detection accuracy of multi-scale defects. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0021] Figure 1 A schematic diagram of a MiniLED defect detection method provided by an embodiment of the present invention;

[0022] Figure 2 A schematic diagram of a defect detection model provided by an embodiment of the present invention;

[0023] Figure 3 A schematic diagram of the structure of a global fusion module provided in an embodiment of the present invention;

[0024] Figure 4 A schematic diagram of the structure of a spatial cross module provided by an embodiment of the present invention;

[0025] Figure 5 A schematic diagram of the structure of a detail enhancement module provided in an embodiment of the present invention;

[0026] Figure 6 A schematic diagram of a MiniLED defect detection result provided by an embodiment of the present invention;

[0027] Figure 7 The experimental result diagram provided for the embodiment of the present invention;

[0028] Figure 8 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] It should be noted that, in the absence of conflict, the features in the following embodiments and implementations may be combined with each other.

[0031] like Figure 1 As shown, an embodiment of the present invention provides a MiniLED defect detection method, the method comprising the following steps:

[0032] Step S1, obtaining a defective MiniLED image, and marking key positions and categories of defects in the defective MiniLED image.

[0033] Specifically, in this example, a high-precision camera is used to collect images of large-scale MiniLEDs under different lighting conditions. The image size is 360*240, and blurry, jittery, and images with extremely poor lighting conditions are eliminated. Then, the labeling software labelImg is used to annotate the key positions and categories of defects in the collected images to generate an xml file. The xml file contains the boundary box and defect category of the defect. The defect categories include foreign matter, stains, missing solid, solid skew, and solid deviation.

[0034] Step S2, constructing a defect detection model and setting a loss function to train the defect detection model.

[0035] Specifically, Figure 2 As shown, the process of the defect detection model includes:

[0036] Step S201, extracting features from the MiniLED image, and obtaining first scale features, second scale features, and third scale features after multi-scale feature layer processing.

[0037] Furthermore, the MiniLED image is feature extracted and the original feature is recorded as F D×C , after being processed by the multi-scale feature layer, the first scale feature F is obtained 1 , the second scale feature F 2 , the third scale feature F 3 ; Among them, the first scale feature F 1 , the second scale feature F 2 , the third scale feature F 3 The dimensions are {D / 1024, D / 256, D / 64}.

[0038] Step S202: perform feature alignment and global information fusion processing on the first scale feature, the second scale feature, and the third scale feature to obtain global fusion information.

[0039] Furthermore, the feature alignment process is used to retain the feature information of multiple layers as completely as possible, and secondly, to reduce the amount of calculation and network delay. The feature alignment module includes an average pooling layer, a convolution layer and a maximum pooling layer. The average pooling layer divides the input into two parts, one of which is downsampled by a 3*3 convolution with a step size of 2, and the other part is subjected to maximum pooling and a 1*1 convolution with a step size of 1 to achieve efficient feature extraction. Finally, the channel fusion module completes the fusion of the two parts of information. The feature alignment module completes the alignment of multi-scale features through the effective combination of global average pooling and global maximum pooling strategies.

[0040] The expression of global information fusion processing is as follows:

[0041] F Bottleneck =Conv 1×1 {Bottleneck[Conv 1×1 (F in )]}+F in

[0042] F csp =LeakyReLU[BN(F Bottleneck )]

[0043]

[0044] F fuse =ReLU{LN[Conv 1×1 (F in_1 )]}+F in_1

[0045] In the formula, F in Represents the information after feature alignment; F Bottleneck It represents the features after being processed by the residual module, and then processed by the BatchNorm and LeakyReLU layers to obtain F csp . F csp Multiply it with the convolution feature to get the fusion feature F in_1 , and finally after the normalization LayerNorm layer and the activation function ReLU processing, the final global fusion information F is obtained. fuse .

[0046] It should be noted that the global fusion module consists of the BottleneckCSP module and GCNet, which aims to fuse the alignment features into comprehensive features containing rich information. Figure 3As shown. Specifically, the BottleneckCSP module reduces the computational complexity by reducing the number of channels, while retaining a strong feature representation capability to promote information exchange and fusion. This module divides the input alignment features into two parts. The first part is reduced in dimension by 1*1 convolution, then extracted by 3*3 convolution, and finally 1*1 convolution is used for dimensionality increase. The second part of the information is directly transmitted through the residual module and spliced ​​with the processed first part in the channel dimension. This module combines the advantages of CSPNet and Bottleneck, promoting the network's ability to learn detailed features while promoting gradient flow.

[0047] At the same time, GCNet improves global perception through the global context attention mechanism. Specifically, the attention weight is first obtained through 1*1 convolution and Softmax function, and then the global context features are obtained through global average pooling. Secondly, the cascade network structure composed of two layers of 1*1 convolution layers, LayerNorm and ReLU not only completes the efficient transmission of information, but also ensures the simplicity of optimization. LayerNorm, as a regularizer, effectively improves the generalization ability of the network. Finally, the integration algorithm integrates global features into different positions to obtain comprehensive features.

[0048] Step S203, decomposing the global fusion information to obtain first fusion information and second fusion information.

[0049] Step S204, the first fusion information is spliced ​​with the first scale feature after detail enhancement processing, and the first prediction result is output; the second fusion information is spliced ​​with the second scale feature after spatial cross processing, and the second prediction result is output; the third scale feature is directly output as the third prediction result.

[0050] The expression of detail enhancement processing is as follows:

[0051]

[0052] F out_1 =ParaResidual(F avg , F diff )

[0053] In the formula, F fuse_1 represents the first fusion information, F avg Indicates the average value information, F diff Represents the difference value information. ParaResidual represents the parallel convolution residual module. The final feature after detail enhancement is F out_1 .

[0054] It should be noted that the detail enhancement module implements the first scale feature F with the strongest semantic features. 1The fusion of the first fusion information completely retains the details, edges and other tiny information. This module mines the similarities and differences between comprehensive features and local features based on the Laplace structure, and is mainly composed of two branches, such as Figure 5 As shown, the bottom branch extracts feature differences by subtracting the shared information to obtain F diff , the top branch uses the average value of the added statistical pixels to obtain F avg .

[0055] After completing the above feature enhancement processing, the present invention designs a parallel convolution residual module ParaResidual, which mainly includes three branches. The specific operation process is:

[0056]

[0057] F out_1 =Concat(F exp , F res )

[0058] Through the parallel structure, the detail enhancement module fully extracts the differences and similarities of features. exp and F res After channel splicing, we get the detailed output F out_1 .

[0059] The expression of spatial cross processing is as follows:

[0060] F act =bilinear{Sigmoid[Conv 1×1 (F fuse_2 )]}

[0061] F embed =bilinear[Conv 1×1 (F fuse_2 )]

[0062] F out =Attention(F 2 , F act , F embed )

[0063] F out_2 =RepConv(F out )

[0064] In the formula, bilinear represents bilinear interpolation, Sigmoid represents activation function, and Attention represents attention interaction module. out Indicates the features after cross processing. out_2 Represents the final output of the spatial intersection module.

[0065] It should be noted that the spatial cross module uses the attention mechanism to inject rich location information into local information. The detailed structure is as follows: Figure 4 Specifically: the second scale feature F 2 and the second fusion information F fuse_2 As input, the comprehensive features are processed by 1*1 convolution, Sigmoid function and bilinear interpolation to obtain the activation feature F act At the same time, the comprehensive features are passed through another parallel channel consisting of 1*1 convolution and bilinear interpolation to obtain the embedded feature F embed . For local information F 2 , the Attention module uses 1*1 convolution to extract deep information and then act To perform similar fusion, the specific implementation process is as follows:

[0066] F out =Conv 1×1 (F 2 )×F act +F embed

[0067] F out Indicates the features after cross processing. out The RepConv consisting of 3*3 convolution, 1*1 convolution and Identity mapping branches completes the organization and fusion of information, significantly reducing the amount of calculation while improving the expression ability of the model, and obtaining the output F of the final spatial cross module out_2 .

[0068] Furthermore, in this example, the improved SCIoU-Loss is used to calculate the positioning box loss, adding scale constraints to the original IoU-Loss function to improve the regression performance of the network bounding box. SCIoU adds a weight to measure scale differences based on CIoU; the specific expression of the loss function is:

[0069]

[0070] Among them, B gt and B p Represent the pixel points of the predicted target and the true value target respectively, Predict the target center point b by calculation gt and the true target center point b p The ratio of the distance between the two center points to the diagonal c promotes accurate center point positioning. By calculating the angle information, the aspect ratio of the positioning box is limited. gt and h gt Represents the length and width of the truth box, w p and hp Represents the length and width of the predicted box. α represents the weight adjustment parameter, which dynamically adjusts the impact of aspect ratio consistency in SCIoU. v represents the angle ratio adjustment parameter, ρ represents the Euclidean distance, and IoU represents the intersection-over-union ratio of the detection box and the predicted box.

[0071] w calculates the distribution of pixels from a microscopic perspective to improve scale sensitivity, and uses variance to measure the area difference between the predicted box and the true value box, specifically:

[0072]

[0073] Where Var(.) represents variance calculation.

[0074] Step S3, inputting the MiniLED image to be detected into the previously trained defect detection model to obtain the defect detection result.

[0075] In this example, the MiniLED defect detection dataset is used for model training. During the training process, the image size is upsampled to 640*640. The SGD optimizer is used, the initial learning rate is set to 0.01, and the final learning rate is reduced to 0.0001 by linear decay. A total of 150 rounds of training are performed, and the batch_size is set to 8. In the last 10 training rounds, Mosaic enhancement is turned off.

[0076] In actual application, the trained defect detection model is used to detect defects in MiniLED images and output defect borders and categories. Figure 6 The figure shows the detection results of Fusion-YOLO for five types of defects in MiniLED.

[0077] This embodiment uses floating point operations per second (FLOPs), mean average precision (mAP) and class average precision (AP) to evaluate the defect detection results. FLOPs measures the complexity of the algorithm or model. Higher FLOPs means higher computational complexity of the network. mAP and AP represent the detection accuracy of the model for MiniLED defects. The implementation effect of the algorithm of the present invention is as follows: Figure 7As shown. The performance comparison of Fusion-YOLO and conventional YOLOv8l is shown in Table 1. Experimental results show that Fusion-YOLO has obvious advantages over conventional YOLOv8l in terms of efficiency or accuracy. While Fusion-YOLO greatly reduces FLOPs, its mAP is improved by 2.7% compared with YOLOv8l. Among them, for foreign objects of different sizes and random distribution, the Fusion-YOLO in the present invention achieves an AP of 98.4%, which is 5.2% higher than the conventional YOLOv8l, proving that Fusion-YOLO can retain feature information more completely and achieve more accurate positioning. At the same time, for the two similarities of fixed deviation and fixed skew, the recognition accuracy of Fusion-YOLO in the present invention on these two types of defects reached 99.3% and 98.7% respectively, successfully achieving high-quality recognition of confusing samples.

[0078] Table 1: Performance comparison of conventional YOLOv8l and Fusion-YOLO on the MiniLED defect dataset

[0079]

[0080] In summary, the present invention provides a MiniLED defect detection method, which obtains a MiniLED image dataset by processing and annotating MiniLED images in real industry. The defect detection model provided by the present invention achieves better extraction effect by fully fusing global and local context information through an improved distributed fusion neck, and successfully solves the problem of detecting multi-scale targets in MiniLED. At the same time, the SCIoU-Loss function is used as the loss function. The loss function introduces a scale weight based on variance operation, which improves the sensitivity to the difference in bounding box size, accelerates the convergence speed, and improves the regression accuracy. The method of the present invention effectively improves the detection accuracy of multi-scale defects.

[0081] Accordingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the MiniLED defect detection method as described above. Figure 7 As shown, it is a hardware structure diagram of any device with data processing capability where the MiniLED defect detection method provided by the embodiment of the present invention is located, except Figure 7 In addition to the processor, memory and network interface shown, any device with data processing capability in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capability, which will not be described in detail.

[0082] Accordingly, the present application also provides a computer-readable storage medium on which computer instructions are stored, and when the instructions are executed by the processor, the MiniLED defect detection method as described above is implemented. The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the foregoing embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0083] Those skilled in the art will readily appreciate other embodiments of the present application after considering the description and practicing the contents disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application. The description and examples are intended to be exemplary only.

[0084] It should be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. A MiniLED defect detection method, It is characterized in that The method comprises: Obtain defective MiniLED images, and annotate key locations and categories of defects in the defective MiniLED images as training sets; Construct a defect detection model, and set a loss function to train the defect detection model using a training set; the defect detection model includes: Feature extraction is performed on the MiniLED image, and after multi-scale feature layer processing, first-scale features, second-scale features, and third-scale features are obtained; Perform feature alignment and global information fusion processing on the first scale feature, the second scale feature, and the third scale feature to obtain global fusion information; Decomposing the global fusion information to obtain first fusion information and second fusion information; The first fusion information is processed by detail enhancement and then spliced ​​with the first scale feature to output a first prediction result; the second fusion information is processed by spatial crossover and then spliced ​​with the second scale feature to output a second prediction result; the third scale feature is directly output as a third prediction result; The MiniLED image to be inspected is input into the previously trained defect detection model to obtain the defect detection result.

2. A MiniLED defect detection method according to claim 1, It is characterized in that The categories of defects in MiniLED images include: foreign matter, stains, missing components, deflection, and deviation.

3. The MiniLED defect detection method according to claim 1, It is characterized in that The expression of global information fusion processing is as follows: F Bottleneck =Conv 1×1 {Bottleneck[Conv 1×1 (F in )]}+F in F csp =LeakyReLU[BN(F Bottleneck )] F fuse =ReLU{LN[Conv 1×1 (F in_1 )]}+F in_1 In the formula, F in Represents the information after feature alignment; F Bottleneck Represents the features after being processed by the residual module; F csp Indicates F Bottleneck Features obtained after processing by BatchNorm and LeakyReLU layers; F csp Multiply it with the convolution feature to get the fusion feature F in_1 , and finally after the normalization LayerNorm layer and the activation function ReLU processing, the final global fusion information F is obtained. fuse .

4. The MiniLED defect detection method according to claim 1, It is characterized in that The expression of detail enhancement processing is as follows: F out_1 =ParaResidual(F avg ,F diff ) In the formula, F fuse_1 represents the first fusion information, F avg Indicates the average value information, F diff Represents the difference value information. ParaResidual represents the parallel convolution residual module. The final feature after detail enhancement is F iut_1 .

5. The MiniLED defect detection method according to claim 1, It is characterized in that The expression of spatial cross processing is as follows: F act =bilinear{Sigmoid[Conv 1×1 (Ffuse _2 )]} F embed =bilinear[Conv 1×1 (Ffuse _2 )] F out =Attention(F 2 ,F act ,F embed ) F out_2 =RepConv(F out ) In the formula, bilinear represents bilinear interpolation, Sigmoid represents activation function, and Attention represents attention interaction module. out Indicates the features after cross processing, F out_2 Represents the output features after spatial cross processing.

6. The MiniLED defect detection method according to claim 1, It is characterized in that The expression of the loss function is as follows: In the formula, represents the total loss function, w represents the distribution law of pixels, B gt Represents the pixel set of the predicted target, B p The set of pixels representing the true value target, Used to calculate the predicted target center point b gt and the true target center point b p The ratio of the distance between the two center points to the diagonal c, ρ 2 (.) represents the square of the Euclidean distance, It is used to limit the aspect ratio of the positioning box. α represents the weight adjustment parameter, which dynamically adjusts the impact of aspect ratio consistency in SCIoU. v represents the angle ratio adjustment parameter, w gt and h gt Represents the length and width of the truth box, w p and h p It represents the length and width of the prediction box, and IoU represents the intersection over union of the detection box and the prediction box.

7. The MiniLED defect detection method according to claim 6, It is characterized in that The distribution law w of the pixel points uses variance to measure the area difference between the predicted box and the true value box. The expression is as follows: Where Var(.) represents variance calculation.

8. An electronic device comprising a memory and a processor, It is characterized in that The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the MiniLED defect detection method described in any one of claims 1 to 7 above.

9. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the program is executed by a processor, the MiniLED defect detection method as described in any one of claims 1 to 7 is implemented.

10. A computer program product comprising a computer program / instructions, It is characterized in that When the computer program / instruction is executed by a processor, the MiniLED defect detection method described in any one of claims 1-7 is implemented.

Images