Screen defect detection method based on improved YOLOv11

Abstract

The invention provides an improved YOLOv11-based screen defect detection method, which comprises the following steps of: improving YOLOv11 into a YOLOv11-DEF model through the technical means of replacing an SPPF module in a backbone with an FPC (Flexible Printed Circuit) module and the like, training and evaluating the model, and putting the model into practical application. The method is superior to an existing YOLO series model in the aspects of screen detection accuracy, recall rate, mAP50 and model speed and size. It needs to be emphasized that in the LCD manufacturing industry, detection of optical defects (especially Mura) of a screen is the most challenging link in quality control, the technical barrier and cost of the detection are far higher than those of geometric defect detection, and the detection is a technical difficulty to be solved urgently in the field. Through a series of improvements in the aspects of model design, training and evaluation, not only can the geometric defects of the screen be accurately detected, but also the optical defects of the screen can be accurately detected.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence machine vision detection, and particularly relates to a screen flaw detection method based on an improved YOLOv11. BACKGROUND

[0002] Since 2015, the YOLO series detection model has been continuously optimized from YOLOv1 to YOLOv11, and has achieved significant improvement in precision and speed. YOLOv11 mainly includes a backbone network, a neck network and a detection head, and plays an important role in screen (such as mobile phone display screen, computer display screen, multimedia terminal display screen, etc.) flaw detection. In recent years, the person skilled in the art has made further improvements to YOLOv11. For example, the patent application with the application number 202510917786.1 discloses a high-reflective and high-transmissive material surface flaw detection method based on an improved YOLOv11, which comprises the following steps: S1, collecting mobile phone screen surface defect images in an industrial production line to construct an original surface defect image dataset; pre-processing and labeling the original surface defect image dataset to generate a training dataset containing three types of defects, i.e., scratches, edge collapses and cracks; S2, building an improved YOLOv11 network model and training and optimizing parameters to construct a flaw detection model; S3, dividing the input image into four independent detection regions on average, using a boundary box overlap area weight distribution strategy for cross-region defects, and using the flaw detection model to count the image by region; S4, adjusting the light weight network structure and optimizing the hardware acceleration to construct a surface defect detection network that meets the real-time requirements of the industrial production line, and outputting the detection results and processing frame rate; S5, performing pixel-level mask segmentation on the scratch defects to extract the contour coordinates and geometric features; S6, fitting the actual damage size of the high-reflective and high-transmissive material surface scratch defects according to the segmentation results.

[0003] The above steps constitute the basic technical scheme of the patent application with the application number 202510917786.1, and on this basis, the invention can be additionally attached with various technical means (see the specification 0062-0107 paragraphs) to better solve the technical problems.

[0004] In summary, for the three macroscopic geometric defects (including micro-defects) on the surface of screens (including high-reflectivity and high-transmittance materials used to manufacture screens): scratches, edge chips, and cracks, the technical solution disclosed in Patent Application No. 202510917786.1 (hereinafter referred to as "the Invention") improves the detection accuracy and quantification, and reduces the false positive rate of dense defects. Furthermore, by combining a cross-regional area weighting strategy, the Invention can achieve precise defect tracing.

[0005] However, this invention also has some obvious technical shortcomings, mainly that its detection effect on two types of microscopic optical defects on the screen surface—dot mura and line mura—is not ideal. The specific reasons are as follows:

[0006] The technical means employed in this invention (including the supporting existing technical means, hereinafter the same) are specifically designed for macroscopic geometric defects of screens with clear physical boundaries (including physical sense of unevenness) and geometric features. Scratches, edge chips, and cracks are all macroscopic geometric defects of screens with clear physical boundaries and geometric features (such as length, width, and direction). For example, for scratches / cracks, low-angle lighting or dark-field lighting is used to highlight the unevenness of their surface, making them appear as bright or dark lines in the image. Based on this, the detection model and algorithm of this invention can be used to obtain ideal detection results. As another example, for edge chips, by extracting contours and comparing them with standard contour templates, and combining them with the detection model and algorithm of this invention, the size and location of the defect can be quickly located. However, unlike screen geometric defects with clear physical boundaries and geometric features, dot mura and line mura (hereinafter collectively referred to as mura) on the screen surface do not have clear physical boundaries and geometric features. They arise because, when the screen emits light, issues such as the uniformity of liquid crystal / OLED materials, driving circuits, and thin-film transistor consistency lead to uneven brightness and color, specifically manifesting as blurry, gradient clouds, spots, or lines. This makes it difficult for the technical means employed in this invention to accurately detect mura on the screen surface.

[0007] It should also be noted that existing technologies face the following four challenges and difficulties in detecting mura on the screen surface:

[0008] First, Mura's criteria are inherently subjective. There is no absolutely clear physical boundary between what constitutes "acceptable uniformity" and "unacceptable non-uniformity." This makes setting the threshold extremely difficult.

[0009] Second, low signal-to-noise ratio: The difference in brightness / color between the Mura region and the normal region is very small, often on the same order of magnitude as the camera's own noise and lighting unevenness. How to extract the true Mura signal from the noise is the core challenge.

[0010] Third, the gradient characteristic: Mura's edges are gradient, without sharp boundaries, which renders traditional edge detection algorithms almost ineffective. More complex background modeling, filtering, and image contrast enhancement algorithms are required.

[0011] Fourth, the relevance of the displayed content: Some Muras only appear under specific grayscale or color conditions, requiring multi-mode detection, which increases the complexity and time of detection.

[0012] In summary, in the LCD manufacturing industry, the detection of optical defects (especially mura) in screens is the most challenging aspect of quality control. Its technical barriers and costs are far higher than those for detecting geometric defects. How to accurately and efficiently detect optical defects in screens at a relatively low cost is a technical challenge that urgently needs to be addressed in this field. Summary of the Invention

[0013] The technical problem to be solved by the present invention is how to accurately and efficiently detect screen defects, especially mura on the screen surface, at a relatively low cost, thereby overcoming the technical defects of the prior art.

[0014] To solve the above-mentioned technical problems, the present invention first adopts the following technical solution: One method for improving YOLOv11 includes the following two basic steps: Step A: Model design, improving YOLOv11 to YOLOv11-DEF; unless otherwise specified, the model referred to below refers to YOLOv11-DEF. Step B: Model training and evaluation; Step A includes the following specific steps: AS1: Design the FPC (Feature Pyramid Shared Convolution) module; AS2: Replace the SPPF (attention module) in the backbone network with the FPC module; AS3: Modifying the neck network based on REMAFPN (Improved Multi-Scale Feature Fusion Network); Step B includes the following specific steps: BS1: Dataset preparation involves collecting a large number of screen images and manually labeling the location and type of defects in normal and defective screens using annotation tools. BS2: Training the model involves inputting labeled data into the model, allowing it to learn repeatedly, and finally calculating a set of model parameters for judging screen defects. BS3: Evaluate the model's performance metrics, such as accuracy and speed, using a new set of data that the model has never seen before.

[0015] Based on the above technical solutions, the present invention may be supplemented with the following technical means to better or more specifically solve the technical problems to be solved by the present invention: When AS1 is executed, the FPC module is able to extract features at different scales and reduce redundancy by using Conv3×3 convolutional layers with different dilation rates of 1×1, 3×3, and 5×5, thereby improving model efficiency.

[0016] Furthermore, when executing AS3, the MAFPN module (Multi-Scale Feature Fusion Network Module) is improved by utilizing the idea of ​​the BIFPN module (Bidirectional Feature Pyramid Network Module) to obtain the REMAFPN module (Improved Multi-Scale Feature Fusion Network). Furthermore, the REMAFPN module includes three feature paths: top-down path, bottom-up path, and fusion-split path.

[0017] Furthermore, the top-down path downsamples high-level semantic features to lower levels, thereby enhancing the small target detection capability. The formula is as follows: , In the formula, X td For small target detection capability, This is a downsampling operation, where σ is the activation function, FC is the attention weight generated by the fully connected layer, and X... high This refers to high-level semantics.

[0018] Furthermore, the bottom-up path upsamples low-level details to higher levels, thereby enhancing the ability to detect large targets. Its formula is as follows: , In the formula, X bu To enhance large target detection capabilities, It is an upsampling operation. X low This refers to low-level semantics.

[0019] Furthermore, the lateral connection path fuses multi-scale features and splits the channel dimensions to enhance the detection capability of intermediate targets, as shown in the following formula: , In the formula, X fs For intermediate object detection capabilities, Concat represents feature concatenation. Adjust the number of channels for a 1×1 convolution. Xmid This is a mid-level semantic.

[0020] Furthermore, BS1 includes the following sub-steps: BS1-1, Subset Partitioning: Randomly divide the complete dataset into three mutually exclusive subsets, where the training set accounts for 70%, the validation set accounts for 20%, and the test set accounts for 10%. BS1-2, Hyperparameter Tuning: Evaluate the model performance under different hyperparameters (such as learning rate and number of network layers) to select the optimal hyperparameters; BS1-3, Early Stop: Monitor the model's performance on the validation set and stop training early when the performance no longer improves to prevent overfitting.

[0021] Based on the above technical solution, the present invention further provides a screen defect detection method based on improved YOLOv11, including step C: putting the model obtained by the above improved YOLOv11 method into practical application; Step C includes the following specific steps: CS1: Apply the trained model to the production line or other workplaces, and obtain screen images in real time as input; CS2: The model outputs the results showing whether there are defects on the screen, as well as the location and type of the defects; The defects include screen geometric defects and screen optical defects (especially Mura).

[0022] Compared with the prior art, the main beneficial technical effects of the present invention are as follows:

[0023] This invention, through improvement, training, and evaluation of the YOLOv11 model, yields a YOLOv11-DEF model with accuracy, recall, and mAP50 of 94.5%, 91.3%, and 95.0%, respectively, for screen detection. The model's speed and size are 6.4 milliseconds / frame and 2.12 MB, respectively. This represents a 1.28% improvement in mAP50 accuracy and an 18.5% reduction in model parameters compared to the YOLOv11 benchmark. In summary, compared to existing YOLO series models such as YOLOv11, this invention enables more reliable, faster, and more accurate detection of LCD defects in industrial scenarios, improving the quality and efficiency of inspection work. It is particularly noteworthy that, compared to the high-reflectivity, high-transmittance material surface defect detection method based on improved YOLOv11 disclosed in Patent Application No. 202510917786.1, the YOLOv11-DEF model in this invention can accurately detect not only geometric defects but also optical defects (especially murras). Furthermore, by using techniques such as Conv3×3 convolutional layers with shared parameters, this invention reduces operating costs and improves work efficiency. Attached Figure Description

[0024] Figure 1 This is a flowchart of an embodiment of the present invention. Figure 2 This is a network structure diagram of YOLOv11-DEF in this embodiment; Figure 3 This is a structural diagram of the FPC module in this embodiment; Figure 4 This is a graph showing the dynamic changes of the evaluation metrics during the training period of this embodiment; Figure 5 This is a diagram showing the detection results of this embodiment. Detailed Implementation

[0025] To facilitate a thorough understanding of the technical solution of the present invention by those skilled in the art, an embodiment of the present invention is described below in conjunction with the accompanying drawings.

[0026] like Figure 1 As shown, a screen defect detection method based on improved YOLOv11 includes the following three basic steps: Step A: Model design, improving YOLOv11 to YOLOv11-DEF; unless otherwise specified, the model referred to below refers to YOLOv11-DEF. Step B: Model training and evaluation; Step C: Put the trained model into practical applications.

[0027] In the three basic steps mentioned above, the combination of steps A and B can constitute a complete technical solution with independent protection value, while step C is a specific application based on this technical solution.

[0028] Step A includes the following specific steps: AS1: Design the FPC (Feature Pyramid Shared Convolution) module; the structure of the FPC module is as follows: Figure 3 As shown; AS2: Replace the SPPF (attention module) in the backbone network with the FPC module; AS3: Modify the neck network based on REMAFPN (Improved Multi-Scale Feature Fusion Network).

[0029] Step B includes the following specific steps: BS1: Dataset preparation, collecting a large number of screen images (e.g., collecting 1000 mobile phone screen images of different models, of which 50 mobile phone screens are normal screens and 950 mobile phone screens have defects of different degrees or types), and manually using annotation tools to annotate the normal screens and defective screens with defect location and type. In this embodiment, BS1 includes the following sub-steps: BS1-1, Subset Partitioning: Randomly divide the complete dataset into three mutually exclusive subsets, where the training set accounts for 70%, the validation set accounts for 20%, and the test set accounts for 10%. BS1-2, Hyperparameter Tuning: Evaluate the model performance under different hyperparameters (such as learning rate and number of network layers) to select the optimal hyperparameters; BS1-3, Early Stop: Monitor the model's performance on the validation set and stop training early when the performance no longer improves to prevent overfitting.

[0030] It is important to note that the test set must be completely isolated throughout the entire training and hyperparameter tuning process and should never be used in any part of the training process. The test set should only be used once, after training is fully completed, to evaluate the model's final generalization performance, reflecting its performance on unseen real-world data.

[0031] BS2: Train the model by inputting the labeled data into the model (the aforementioned screen images and their manually labeled information are input into the model), allowing the model to learn repeatedly, and finally calculate a set of model parameters for judging screen defects; BS3: This involves inputting a completely new set of data (also known as a test set) that the model has never seen before. For example, inputting 100 images of mobile phone screens that the model has never encountered before. The model then autonomously determines which screens are normal, which are defective, and the type of defect. During BS3 execution, the model compares its judgments with manually labeled information to evaluate performance metrics such as accuracy and speed.

[0032] Step C includes the following specific steps: CS1: Apply the trained model to the production line or other workplaces, and obtain screen images in real time as input; CS2: The model outputs the results of whether there are defects on the screen and the type of defects.

[0033] like Figure 3As shown, in this embodiment, when executing AS1, a Conv3×3 convolutional layer with shared parameters and different dilation rates (1×1, 3×3, and 5×5) is used. The low dilation rate captures local details, while the high dilation rate captures the global context, enabling the FPC module to extract features at different scales. By using the Conv3×3 convolutional layer with shared parameters, this invention significantly reduces the number of parameters that need to be trained (originally, the required number of parameters = N (number of input channels) × M (number of output channels) × 3 × 3 × 3; by using the Conv3×3 convolutional layer with shared parameters, the actual number of parameters required is reduced to N × M × 3 × 3). Therefore, compared to using independent convolutional layers for each dilation rate, the shared parameter convolutional layer reduces redundancy, improves model efficiency, reduces model storage and computational overhead, and enhances computational efficiency.

[0034] In this embodiment, when executing AS3, the MAFPN module (Multi-Scale Feature Fusion Network Module) is improved using the concept of the BIFPN module (Bidirectional Feature Pyramid Network Module) to obtain the REMAFPN module (Improved Multi-Scale Feature Fusion Network Module). The BIFPN module (Bidirectional Feature Pyramid Network Module) concept refers to the adoption of a bidirectional information flow design. Specifically, in this embodiment, the aforementioned REMAFPN module includes three feature paths: a top-down path (e.g., ... Figure 2 Downward arrow The path it points to), bottom-up path (e.g.) Figure 2 Upward arrow The path to which it points, and the lateral connection path (Fusion-Split Path, for example, Figure 2 Arrow pointing to the right The path it points to, where the combination of top-down and bottom-up paths embodies the idea of ​​the BIFPN module.

[0035] The top-down path downsamples high-level semantic features to lower levels, thereby enhancing the ability to detect small targets. Its formula is as follows: , In the formula, X td For small target detection capability ( X The subscript td is short for top-down. This is a downsampling operation, where σ is the activation function and FC is the attention weight generated by the fully connected layer. X high This refers to high-level semantics.

[0036] The bottom-up path upsamples low-level details to higher levels, thereby enhancing the ability to detect large targets. Its formula is as follows: , In the formula, X bu For large target detection capability ( X subscript bu It's short for bottom-up. It is an upsampling operation. X low This refers to low-level semantics.

[0037] The lateral connection path fuses multi-scale features and splits the channel dimensions to enhance the detection capability of intermediate targets. Its formula is as follows: , In the formula, X fs For intermediate object detection capabilities, Concat represents feature concatenation. Adjust the number of channels for a 1×1 convolution. X mid This is a mid-level semantic.

[0038] The technical solution of one embodiment of the present invention has been described above with reference to the accompanying drawings. The technical effects of the present invention will be further described below.

[0039] like Figure 4 As shown, the model disclosed in this embodiment was tested on 10 different screen models (all of which have defects in point Mura and line Mura) during the training process, and a large number of evaluation metrics were obtained for each screen. Figure 4 The figure illustrates the changing trends of various losses and evaluation metrics during the model's training process. The figure includes curves showing the changes in training loss, validation loss, and accuracy evaluation metrics (precision, recall, mAP). As shown in the figure, the model converged after 400 training iterations. Detailed data for each metric are shown in Table 2. Therefore, it is evident that the technical solution disclosed in this invention can accurately detect screen defects (including point mura and line mura). Figure 5 This further confirms this conclusion.

[0040] It should also be noted that all experiments conducted in this invention were performed on a workstation equipped with an Intel Core i7-9750H processor (6 cores, 12 threads, base frequency 2.6 GHz), 16GB of DDR4 memory, and an NVIDIA GeForce RTX 2070 GPU with 8GB of video memory. The system ran Windows 11 and included CUDA 11.8 and cuDNN 9.0 acceleration libraries. The inventors used Python 3.12 and the PyTorch 2.5.1 framework. SATA III SSD storage was used during training for rapid data loading. Specific parameter settings are shown in Table 1.

[0041] Table 1: Hyperparameter Settings Table Parameter Setting lr0 0.01 lrf 0.01 warmup_epochs 3.0 warmup_momentum 0.8 weight_decay 0.0005 mosaic 1.0 mixup 0.0 batch 8 imgsz 640 optimizer Auto epochs 400

[0042] In Table 1, Parameter refers to hyperparameters (parameters that need to be set in advance before model training). ， Setting point to The hyperparameter settings are as follows: lr0 refers to the initial learning rate, lrf refers to the learning rate decay factor, warmup_epochs refers to the number of epochs in the learning rate warm-up phase, warmup_momentum refers to the momentum value that is gradually increased in the early stages of training, weight_decay refers to weight decay, mosaic refers to mosaic data augmentation, mixup refers to a data augmentation method that combines training samples and their labels through linear interpolation, batch refers to the number of batch training iterations, imgsz is the core parameter controlling the input image size in the YOLO series object detection models, and optimizer refers to the optimizer in the system. epochs This refers to the number of times the YOLOv11-DEF detection model completely traverses the training dataset.

[0043] It should also be noted that, based on comparative experiments, the technical effect of this invention is significantly better than other detection models in the YOLO series, as detailed in Table 2.

[0044] Table 2. Comparison of Algorithm Detection Results Algorithms Precision Recall mAP50 ModelSize (MB) Speed(ms / frame) YOLOv5n 0.847 0.85 0.914 2.2 5.8 YOLOv8n 0.940 0.894 0.937 2.7 5.5 YOLOv11n 0.877 0.895 0.938 2.6 6.4 YOLOv11_DEF 0.945 0.913 0.950 2.12 6.4

[0045] In Table 2, Algorithms refers to the algorithms corresponding to the YOLO series detection models, Precision refers to accuracy, Recall refers to recall, and mAP50 refers to the average accuracy when the intersection-union ratio (IUU) threshold between the predicted and ground truth bounding boxes is fixed at 0.5. Model Size (MB) refers to the storage space occupied by the detection model, and Speed ​​(ms / frame) refers to the processing time per image (unit: milliseconds).

[0046] In summary, through improvements, training, and evaluation of the YOLOv11 model, this invention achieves a YOLOv11-DEF model with accuracy, recall, and mAP50 (mean precision when the intersection-union ratio of predicted and ground truth boxes is 0.5) of 94.5%, 91.3%, and 95.0% respectively in screen detection. The model's speed and size are 6.4 milliseconds / frame and 2.12 MB, respectively. This represents a 1.28% improvement in mAP50 accuracy and a 18.5% reduction in model parameters compared to the YOLOv11 benchmark. In conclusion, compared to existing YOLO models such as YOLOv11, the model proposed in this invention can more reliably and accurately detect LCD dot mura and line mura defects in industrial scenarios, improving the quality and efficiency of screen detection work, while also having a relatively low cost. It should be noted that, compared with the method for detecting surface defects of high-reflectivity and high-transmittance materials based on improved YOLOv11 disclosed in Patent Application No. 202510917786.1, the YOLOv11-DEF model in this invention can not only effectively detect geometric defects of the screen, but also effectively detect optical defects of the screen (including microscopic optical defects that are extremely difficult to detect). Furthermore, by using techniques such as Conv3×3 convolutional layers with shared parameters, this invention reduces operating costs and improves work efficiency.

Claims

1. A method for improving YOLOv11, characterized in that, It includes the following two basic steps: Step A: Model design, improving YOLOv11 to YOLOv11-DEF; the model mentioned below refers to YOLOv11-DEF. Step B: Model training and evaluation; Step A includes the following specific steps: AS1: Design the FPC module; AS2: Replace the SPPF module in backbone with the FPC module; AS3: Modifying the neck network based on REMAFPN; Step B includes the following specific steps: BS1: Dataset preparation involves collecting a large number of screen images and manually labeling the location and type of defects in normal and defective screens using annotation tools. BS2: Training the model involves inputting labeled data into the model, allowing it to learn repeatedly, and finally calculating a set of model parameters for judging screen defects. BS3: Evaluate the model's performance metrics using a completely new set of data that the model has never seen before.

2. The method for improving YOLOv11 as described in claim 1, characterized in that: When AS1 is executed, the FPC module is able to extract features at different scales and reduce redundancy by using Conv3×3 convolutional layers with different dilation rates of 1×1, 3×3, and 5×5, thereby improving model efficiency.

3. The method for improving YOLOv11 as described in claim 1, characterized in that: When executing AS3, the MAFPN module is improved by using the idea of ​​the BIFPN module, resulting in the REMAFPN module.

4. The method for improving YOLOv11 as described in claim 3, characterized in that: The REMAFPN module includes three types of feature paths: top-down path, bottom-up path, and lateral connection path.

5. The method for improving YOLOv11 as described in claim 4, characterized in that: The top-down path downsamples high-level semantic features to lower levels, thereby enhancing the ability to detect small targets. Its formula is as follows: ， In the formula, X td For small target detection capability, This is a downsampling operation, where σ is the activation function, FC is the attention weight generated by the fully connected layer, and X... high This refers to high-level semantics.

6. The method for improving YOLOv11 as described in claim 4, characterized in that: The bottom-up path upsamples low-level details to higher levels, thereby enhancing the ability to detect large targets. The formula is as follows: , In the formula, X bu To enhance large target detection capabilities, It is an upsampling operation. X low This refers to low-level semantics.

7. The method for improving YOLOv11 as described in claim 4, characterized in that: The lateral connection path fuses multi-scale features and splits the channel dimensions to enhance the detection capability of intermediate targets. Its formula is as follows: , In the formula, X fs For intermediate object detection capabilities, Concat represents feature concatenation. Adjust the number of channels for a 1×1 convolution, X mid This is a mid-level semantic.

8. The method for improving YOLOv11 as described in claim 1, characterized in that: The BS1 includes the following sub-steps: BS1-1, Subset Partitioning: Randomly divide the complete dataset into three mutually exclusive subsets, where the training set accounts for 70%, the validation set accounts for 20%, and the test set accounts for 10%. BS1-2, Hyperparameter Tuning: Evaluate the model performance under different hyperparameters to select the optimal hyperparameters; BS1-3, Early Stop: Monitor the model's performance on the validation set and stop training early when the performance no longer improves to prevent overfitting.

9. The method for improving YOLOv11 as described in any one of claims 1 to 8, characterized in that: When executing BS3, the performance metrics include accuracy and speed.

10. A screen defect detection method based on improved YOLOv11, characterized in that: Including step C: putting the model obtained by the improved YOLOv11 method according to any one of claims 1 to 9 into practical application; Step C includes the following specific steps: CS1: Apply the trained model to the production line or other workplaces, and obtain screen images in real time as input; CS2: The model outputs the results showing whether there are defects on the screen, as well as the location and type of the defects; The defects include screen geometric defects and screen optical defects.

Images