MiniLED defect detection method, electronic device, and medium

Abstract

The present invention discloses a MiniLED defect detection method, electronic device, and medium, including: obtaining a defective MiniLED image and marking its key positions and categories as a training set; constructing and training a defect detection model; the defect detection model includes: extracting features from the MiniLED image, and obtaining first, second, and third scale features after multi-scale feature layer processing; aligning the first, second, and third scale features and performing global information fusion processing to obtain global fusion information; decomposing the global fusion information to obtain first and second fusion information; splicing the first fusion information with the first scale feature after detail enhancement processing to output a first prediction result; splicing the second fusion information with the second scale feature after spatial cross processing to output a second prediction result; directly outputting the third scale feature as a third prediction result; inputting the MiniLED image to be detected into the previously trained defect detection model to obtain a defect detection result.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and in particular relates to a MiniLED defect detection method, electronic equipment, and medium. Background Art

[0002] MiniLEDs, a new type of light-emitting device with broad application prospects, are in huge demand in industries such as display technology, healthcare, and communications. To ensure product lifespan and user experience, high-quality MiniLEDs must be defect-free, making defect detection a crucial step. Traditional manual inspection methods suffer from inconsistent judgment standards, low efficiency, and high human resource costs, limiting their reliability in large-scale industrial testing. Furthermore, some complex and subtle defects are difficult for the human eye to accurately detect, failing to meet high-precision requirements.

[0003] In recent years, researchers have conducted extensive research on deep learning detection algorithms based on neural networks, gradually evolving object detection models from two-stage models, such as RCNN and Fast RCNN, to faster single-stage models. Single-stage detection models, particularly the YOLO series, have been widely welcomed by the industry due to their simple structure and high computational efficiency. However, directly applying the current YOLO method to MiniLED detection still has the following shortcomings:

[0004] (1) The complex production process leads to a wide variety of MiniLED defects, which have different shapes and complex texture information. During the detection process, defects of the same type have different shapes and random location distribution, requiring the algorithm to have flexible data processing capabilities. At the same time, different types of defects may have similar appearance features, increasing the risk of false detection and missed detection, and exacerbating the difficulty of detection.

[0005] (2) Current YOLO-series neck networks typically use FPN for contextual feature fusion. However, this network structure has significant drawbacks in industrial detection. The traditional FPN structure can only transfer features from adjacent layers. When cross-layer information needs to be integrated, it will cause global information loss, hindering the full fusion of features. Summary of the Invention

[0006] In response to the shortcomings of the existing technology, the present invention provides a MiniLED defect detection method, electronic equipment, and medium.

[0007] In a first aspect, an embodiment of the present invention provides a MiniLED defect detection method, the method comprising:

[0008] Obtain defective MiniLED images and annotate the key locations and categories of defects in the defective MiniLED images as a training set;

[0009] Construct a defect detection model and set a loss function to train the defect detection model using a training set; the defect detection model includes:

[0010] Feature extraction is performed on the MiniLED image, and after multi-scale feature layer processing, first-scale features, second-scale features, and third-scale features are obtained;

[0011] Perform feature alignment and global information fusion processing on the first scale features, the second scale features, and the third scale features to obtain global fusion information;

[0012] Decomposing the global fusion information to obtain first fusion information and second fusion information;

[0013] The first fused information is processed with detail enhancement and then concatenated with the first scale feature to output a first prediction result; the second fused information is processed with spatial cross-talk and then concatenated with the second scale feature to output a second prediction result; the third scale feature is directly output as a third prediction result;

[0014] The MiniLED image to be inspected is input into the previously trained defect detection model to obtain the defect detection results.

[0015] In a second aspect, an embodiment of the present invention provides an electronic device comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned MiniLED defect detection method.

[0016] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the above-mentioned MiniLED defect detection method when executed by a processor.

[0017] In a fourth aspect, an embodiment of the present invention provides a computer program product, including a computer program / instruction, which implements the above-mentioned MiniLED defect detection method when executed by a processor.

[0018] The same as the prior art, the beneficial effects of the present invention are:

[0019] The present invention provides a MiniLED defect detection method, which processes and annotates MiniLED images from real industries to obtain a MiniLED image dataset. The defect detection model provided by the present invention achieves better extraction effects by fully fusing global and local context information through an improved distributed fusion neck, successfully solving the problem of detecting multi-scale targets in MiniLED. At the same time, the SCIoU-Loss function is used as the loss function. This loss function introduces a scale weight based on variance operation, which increases sensitivity to bounding box size differences, accelerates convergence speed, and improves regression accuracy. The method of the present invention effectively improves the detection accuracy of multi-scale defects. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 A schematic diagram of a MiniLED defect detection method provided by an embodiment of the present invention;

[0022] Figure 2 A schematic diagram of a defect detection model provided by an embodiment of the present invention;

[0023] Figure 3 A schematic diagram of the structure of a global fusion module provided in an embodiment of the present invention;

[0024] Figure 4 A schematic structural diagram of a spatial cross module provided in an embodiment of the present invention;

[0025] Figure 5 A schematic diagram of the structure of a detail enhancement module provided in an embodiment of the present invention;

[0026] Figure 6 A schematic diagram of MiniLED defect detection results provided by an embodiment of the present invention;

[0027] Figure 7 The experimental results provided by the embodiment of the present invention are shown in FIG;

[0028] Figure 8 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0030] It should be noted that, unless there is any conflict, the features in the following embodiments and implementations may be combined with each other.

[0031] like Figure 1 As shown, an embodiment of the present invention provides a MiniLED defect detection method, the method comprising the following steps:

[0032] Step S1: Acquire a defective MiniLED image and mark the key positions and categories of defects in the defective MiniLED image.

[0033] Specifically, in this example, a high-precision camera was used to capture large-scale 360x240 images of MiniLEDs under varying lighting conditions. Blurred, jittery, and images with extremely poor lighting conditions were removed. LabelImg, annotating software, was then used to annotate the key locations and categories of defects in the captured images. XML files were generated containing the defect bounding boxes and defect categories, including foreign matter, stains, missing components, skewed components, and deflected components.

[0034] Step S2: construct a defect detection model and set a loss function to train the defect detection model.

[0035] Specifically, if Figure 2 As shown, the process of the defect detection model includes:

[0036] In step S201 , feature extraction is performed on the MiniLED image, and after multi-scale feature layer processing, first scale features, second scale features, and third scale features are obtained.

[0037] Furthermore, the MiniLED image is subjected to feature extraction, and the original feature is recorded as F D×C After processing by the multi-scale feature layer, the first scale feature F1, the second scale feature F2, and the third scale feature F3 are obtained; among which, the sizes of the first scale feature F1, the second scale feature F2, and the third scale feature F3 are {D / 1024, D / 256, D / 64}.

[0038] In step S202 , feature alignment and global information fusion processing are performed on the first scale feature, the second scale feature, and the third scale feature to obtain global fusion information.

[0039] Furthermore, the feature alignment process is used to preserve the feature information of multiple layers as completely as possible, and secondly, to reduce the amount of computation and network latency. The feature alignment module includes an average pooling layer, a convolution layer, and a maximum pooling layer. The average pooling layer divides the input into two parts, one of which is downsampled through a 3*3 convolution with a stride of 2, and the other part is subjected to maximum pooling and a 1*1 convolution with a stride of 1 to achieve efficient feature extraction. Finally, the channel fusion module completes the fusion of the two parts of information. The feature alignment module completes the alignment of multi-scale features through the effective combination of global average pooling and global maximum pooling strategies.

[0040] The expression of global information fusion processing is as follows:

[0041] F Bottleneck =Conv 1×1 {Bottleneck[Conv 1×1 (F in )]}+F in

[0042] F csp =LeakyReLU[BN(F Bottleneck )]

[0043]

[0044] F fuse =ReLU{LN[Conv 1×1 (F in_1 )]}+F in_1

[0045] Where, F in Represents the information after feature alignment; F Bottleneck It represents the features after being processed by the residual module, and then processed by the BatchNorm and LeakyReLU layers to obtain F csp . F csp Multiply it by the convolution feature to get the fusion feature F in_1 Finally, after the normalization LayerNorm layer and activation function ReLU processing, the final global fusion information F is obtained. fuse .

[0046] It should be noted that the global fusion module consists of the BottleneckCSP module and GCNet, which aims to fuse the aligned features into comprehensive features containing rich information. The structure is as follows Figure 3As shown in the figure. Specifically, the BottleneckCSP module reduces computational complexity by reducing the number of channels while retaining strong feature representation capabilities, promoting information exchange and fusion. This module divides the input alignment features into two parts. The first part is reduced in dimension through 1*1 convolution, then feature extracted by 3*3 convolution, and finally 1*1 convolution is used for dimensionality increase. The second part of the information is directly transmitted through the residual module and spliced ​​with the processed first part in the channel dimension. This module combines the advantages of CSPNet and Bottleneck, promoting gradient flow while enhancing the network's ability to learn detailed features.

[0047] GCNet also enhances global perception through a global contextual attention mechanism. Specifically, attention weights are first obtained through 1x1 convolution and a Softmax function, followed by global contextual features through global average pooling. Secondly, a cascaded network structure consisting of two 1x1 convolutional layers, LayerNorm, and ReLU not only enables efficient information transfer but also ensures ease of optimization. LayerNorm acts as a regularizer, effectively improving the network's generalization capabilities. Finally, an integration algorithm integrates global features at different locations to produce comprehensive features.

[0048] Step S203: decompose the global fusion information to obtain first fusion information and second fusion information.

[0049] In step S204 , the first fused information is processed with detail enhancement and then concatenated with the first scale feature to output a first prediction result; the second fused information is processed with spatial cross-talk and then concatenated with the second scale feature to output a second prediction result; and the third scale feature is directly output as a third prediction result.

[0050] The expression of detail enhancement processing is as follows:

[0051]

[0052] F out_1 =ParaResidual(F avg , F diff )

[0053] Where, F fuse_1 represents the first fusion information, F avg Indicates the average value information, F diff Represents the difference value information. ParaResidual represents the parallel convolution residual module. The final feature after detail enhancement is F out_1 .

[0054] It should be noted that the detail enhancement module achieves the fusion of the first scale feature F1 with the strongest semantic features and the first fusion information, completely retaining the details, edges and other small information. This module is based on the Laplace structure to mine the similarities and differences between comprehensive features and local features, and is mainly composed of two branches, such as Figure 5 As shown, the bottom branch extracts feature differences by subtracting shared information to obtain F diff , the top branch uses the average value of the added statistical pixels to obtain F avg .

[0055] After completing the above feature enhancement processing, the present invention designs a parallel convolution residual module ParaResidual, which mainly includes three branches. The specific operation process is:

[0056]

[0057] F out_1 =Concat(F exp , F res )

[0058] Through the parallel structure, the detail enhancement module fully extracts the differences and similarities of features. exp and F res After channel splicing, we get the detailed output F out_1 .

[0059] The expression of spatial cross processing is as follows:

[0060] F act =bilinear{Sigmoid[Conv 1×1 (F fuse_2 )]}

[0061] F embed =bilinear[Conv 1×1 (F fuse_2 )]

[0062] F out =Attention(F2,F act , F embed )

[0063] F out_2 =RepConv(F out )

[0064] In the formula, bilinear represents bilinear interpolation, Sigmoid represents activation function, and Attention represents attention interaction module. out Indicates the features after cross processing. out_2Represents the final output of the spatial intersection module.

[0065] It should be noted that the spatial cross module uses the attention mechanism to inject rich location information into local information. The detailed structure is as follows: Figure 4 Specifically: the second scale feature F2 and the second fusion information F fuse_2 As input, the comprehensive features are processed by 1*1 convolution, Sigmoid function and bilinear interpolation to obtain the activation feature F act At the same time, the comprehensive features are passed through another parallel channel consisting of 1*1 convolution and bilinear interpolation to obtain the embedded feature F embed For local information F2, the Attention module uses 1*1 convolution to extract deep information and then act The specific implementation process of similar fusion is as follows:

[0066] F out =Conv 1×1 (F2)×F act +F embed

[0067] F out Indicates the features after cross processing. out The RepConv composed of 3*3 convolution, 1*1 convolution and Identity mapping branches completes the information organization and fusion, significantly reducing the amount of calculation while improving the expression ability of the model, and obtaining the output F of the final spatial cross module. out_2 .

[0068] Furthermore, in this example, the improved SCIoU-Loss is used to calculate the positioning box loss, adding scale constraints to the original IoU-Loss function to improve the regression performance of the network bounding box. SCIoU adds a weight to measure scale differences based on CIoU; the specific expression of the loss function is:

[0069]

[0070] Among them, B gt and B p Represent the pixel sets of the predicted target and the true value target respectively, Predict the target center point b by calculating gt and the true target center point b p The ratio of the distance between the two center points to the diagonal line c promotes accurate center point positioning. By calculating the angle information, the aspect ratio of the positioning box is limited. gt and h gt Represents the length and width of the true value box, w p and hp represents the length and width of the predicted box. α is a weight adjustment parameter that dynamically adjusts the impact of aspect ratio consistency in SCIoU. v is an angle scale adjustment parameter, ρ is the Euclidean distance, and IoU is the intersection over union (IoU) between the detection box and the predicted box.

[0071] w calculates the distribution of pixels from a microscopic perspective to improve scale sensitivity, and uses variance to measure the area difference between the predicted box and the true value box, specifically:

[0072]

[0073] Where Var(.) represents variance calculation.

[0074] Step S3: Input the MiniLED image to be detected into the previously trained defect detection model to obtain the defect detection result.

[0075] In this example, the model was trained using a MiniLED defect detection dataset. During training, the images were upsampled to 640*640 pixels. An SGD optimizer was used, with an initial learning rate of 0.01 and a linear decay to 0.0001. Training was performed for 150 epochs with a batch size of 8. Mosaic enhancement was disabled for the final 10 epochs.

[0076] In actual application, the trained defect detection model is used to detect defects in MiniLED images and output defect boundaries and categories. Figure 6 The figure shows the detection results of Fusion-YOLO for five types of defects in MiniLED.

[0077] This embodiment uses floating point operations per second (FLOPs), mean average precision (mAP) and class average precision (AP) to evaluate the defect detection results. FLOPs measures the complexity of the algorithm or model. Higher FLOPs means higher computational complexity of the network. mAP and AP represent the accuracy of the model in detecting MiniLED defects. Figure 7As shown. The performance comparison of Fusion-YOLO and conventional YOLOv8l is shown in Table 1. Experimental results show that Fusion-YOLO has obvious advantages over conventional YOLOv8l in terms of efficiency or accuracy. While Fusion-YOLO greatly reduces FLOPs, its mAP is improved by 2.7% compared with YOLOv8l. Among them, for foreign objects of different sizes and random distribution, the Fusion-YOLO in the present invention achieves an AP of 98.4%, which is 5.2% higher than the conventional YOLOv8l, proving that Fusion-YOLO can retain feature information more completely and achieve more accurate positioning. At the same time, for the two categories of fixed deviation and fixed skew that have similarities, the recognition accuracy of Fusion-YOLO in the present invention on these two types of defects reached 99.3% and 98.7% respectively, successfully achieving high-quality recognition of confusing samples.

[0078] Table 1: Performance comparison of conventional YOLOv8l and Fusion-YOLO on the MiniLED defect dataset

[0079]

[0080] In summary, the present invention provides a MiniLED defect detection method, which obtains a MiniLED image dataset by processing and annotating MiniLED images in real industry. The defect detection model provided by the present invention achieves better extraction effect by fully fusing global and local context information through an improved distributed fusion neck, and successfully solves the problem of detecting multi-scale targets in MiniLED. At the same time, the SCIoU-Loss function is adopted as the loss function. This loss function introduces a scale weight based on variance operation, which improves the sensitivity to the size difference of the bounding box, accelerates the convergence speed, and improves the regression accuracy. The method of the present invention effectively improves the detection accuracy of multi-scale defects.

[0081] Accordingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the MiniLED defect detection method as described above. Figure 7 As shown in the figure, it is a hardware structure diagram of any device with data processing capability where the MiniLED defect detection method provided by the embodiment of the present invention is located. Figure 7 In addition to the processor, memory, and network interface shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0082] Accordingly, the present application also provides a computer-readable storage medium having computer instructions stored thereon, which implement the MiniLED defect detection method as described above when the instructions are executed by the processor. The computer-readable storage medium can be the internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium can also include both the internal storage unit and the external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and can also be used to temporarily store data that has been output or is to be output.

[0083] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the contents disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only.

[0084] It will be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. A MiniLED defect detection method, characterized in that: The method comprises: Obtain defective MiniLED images and annotate the key locations and categories of defects in the defective MiniLED images as a training set; Construct a defect detection model and set a loss function to train the defect detection model using a training set; the defect detection model includes: Feature extraction is performed on the MiniLED image, and after multi-scale feature layer processing, first-scale features, second-scale features, and third-scale features are obtained; Perform feature alignment and global information fusion processing on the first scale features, the second scale features, and the third scale features to obtain global fusion information; Decomposing the global fusion information to obtain first fusion information and second fusion information; The first fused information is processed with detail enhancement and then concatenated with the first scale feature to output a first prediction result; the second fused information is processed with spatial cross-talk and then concatenated with the second scale feature to output a second prediction result; the third scale feature is directly output as a third prediction result; Input the MiniLED image to be inspected into the previously trained defect detection model to obtain the defect detection results; The expression of global information fusion processing is as follows: F Bottleneck =Conv 1×1 {Bottleneck[Conv 1×1 (F in )]}+F in F csp =LeakyReLU[BN(F Bottleneck )] F fuse =ReLU{LN[Conv 1×1 (F in_1 )]}+F in_1 Where, F in Represents the information after feature alignment; F Bottleneck Represents the features after processing by the residual module; F csp Indicates F Bottleneck Features obtained after processing by BatchNorm and LeakyReLU layers; F csp Multiply it by the convolution feature to get the fusion feature F in_1 Finally, after the normalization LayerNorm layer and activation function ReLU processing, the final global fusion information F is obtained. fuse .

2. The MiniLED defect detection method according to claim 1, wherein: The categories of defects in MiniLED images include: foreign matter, stains, missing components, deflection, and deviation.

3. The MiniLED defect detection method according to claim 1, wherein: The expression of detail enhancement processing is as follows: F diff =|F1-F fuse_1 | F out_1 =ParaResidual(F avg ,F diff ) In the formula, F1 represents the first scale feature, F fuse_1 represents the first fusion information, F avg Indicates the average value information, F diff Represents the difference value information; ParaResidual represents the parallel convolution residual module, and the final feature after detail enhancement processing is F out_1 .

4. The MiniLED defect detection method according to claim 1, wherein: The expression of spatial intersection processing is as follows: F act =bilinear{Sigmoid[Conv 1×1 (F fuse_2 )]} F embed =bilinear[Conv 1×1 (F fuse_2 )] F out =Attention(F2,F act ,F embed ) F out_2 =RepConv(F out ) In the formula, bilinear represents bilinear interpolation, Sigmoid represents activation function, Attention represents attention interaction module, F fuse_2 represents the second fusion information, F2 represents the second scale feature, F out Indicates the features after cross processing, F out_2 Represents the output feature after spatial cross processing, F act represents the activation feature, F embed Represents the embedded features.

5. The MiniLED defect detection method according to claim 1, wherein: The expression of the loss function is as follows: Where, represents the total loss function, w represents the distribution law of pixels, B gt Represents the pixel set of the predicted target, B p The set of pixels representing the true value target, Used to calculate the predicted target center point b gt and the true target center point b p The ratio of the distance between the two center points to the diagonal c, ρ 2 (.) represents the square of the Euclidean distance, It is used to limit the aspect ratio of the positioning box. α represents the weight adjustment parameter, which dynamically adjusts the impact of aspect ratio consistency in SCIoU. v represents the angle ratio adjustment parameter, w gt and h gt Represents the length and width of the true value box, w p and h p It represents the length and width of the prediction box, and IoU represents the intersection over union of the detection box and the prediction box.

6. The MiniLED defect detection method according to claim 5, characterized in that: The distribution law w of the pixel points uses variance to measure the area difference between the predicted box and the true value box. The expression is as follows: Where Var(.) represents variance calculation.

7. An electronic device comprising a memory and a processor, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the MiniLED defect detection method according to any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the MiniLED defect detection method as described in any one of claims 1 to 6 is implemented.

9. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the MiniLED defect detection method described in any one of claims 1 to 6 is implemented.

Images