Smartphone panel surface defect detection method based on YOLO V5 model

Abstract

The present invention belongs to the field of target detection in computer vision, and specifically designs a method for detecting surface defects of smartphones based on an improved YOLOV5 model. The method comprises: real-time acquisition of smartphone surface defect images; segmentation of the acquired images; image enhancement and annotation, and preparation of a dataset of smartphone panels; construction of an improved YOLOV5 model; training of the improved YOLOV5 model; and detection of the smartphone panel surface using the trained YOLOV5 model. The YOLOV5 model: introduces an attention module into Backbone; adds cross-scale and cross-layer fusion of feature maps into Neck; adds deformable convolution into Head; and simultaneously uses the Varifocalloss cross entropy function; in the anchor frame screening stage, in the anchor frame set, the efficiency and confidence of anchor frame deduplication are improved by suppressing strong neighbors and strengthening weak neighbors; and finally, the structure of the improved YOLOV5 model is lightweight and compressed. The improved YOLOV5 model detects smartphone surface defects with better accuracy and real-time performance than the original YOLOV5 model.

Description

Technical Field

[0001] The present invention belongs to the field of target detection in computer vision, and specifically relates to a method for detecting surface defects of smartphone panels based on the YOLO V5 model. Background Art

[0002] With the advent of the 5G era and the rapid global development of the smartphone industry, mobile phone transactions are increasing year by year, and users' expectations for phone screen quality are also increasing. Today's smartphone screens are covered with a sheet of glass, beneath which the phone's display, touchscreen, camera, and a series of optical sensors are hidden. The processing accuracy and surface quality of the phone's panel directly impact the phone's display quality, touch sensitivity, image quality, and signal reception by the optical sensors. During the production process of mobile phone panels, including CNC cutting, surface polishing, ink printing, and cleaning, defects such as chipped edges, scratches, dead pixels, dust, and dirt are inevitably generated, severely impacting the user experience. Therefore, surface quality inspection of mobile phone panels is essential. However, conventional inspection methods have difficulty capturing surface defects on transparent glass products; the captured surface defects have diverse morphologies; the inspection results are easily affected by the environment; and the inspection speed cannot keep up with production schedules. As a result, the accuracy and real-time performance of mobile phone glass inspections cannot meet the inspection requirements of manufacturers. Therefore, developing a real-time inspection method with strong anti-interference capabilities and high accuracy holds great potential for application.

[0003] In the field of object detection, two-stage detection algorithms such as the R-CNN series offer high accuracy, but struggle to meet real-time detection speed requirements. In contrast, the YOLO series, as a single-stage detection algorithm, balances performance and flexibility, making it more suitable for industrial detection. However, the YOLO series also suffers from issues such as low sensitivity to small-scale objects, insufficient localization accuracy, and inefficient anchor box selection. Therefore, optimizing the YOLO algorithm to achieve even better performance is a research topic.

[0004] YOLO V5-based object detection methods for smartphone panel surface defects face the following challenges. First, smartphone surface defects have diverse morphologies and a wide range of sizes. Some defects are densely distributed and have a high overlap rate. Second, smartphone panels are thin and transparent, making optical image acquisition difficult. Third, laser scattering scanning is susceptible to environmental dust and surface contamination. Therefore, while improving detection accuracy, high requirements are placed on algorithm generalization and interference resistance. Summary of the Invention

[0005] In order to solve the problems of difficulty in collecting defects on the surface of transparent glass products; detection results are easily affected by the environment; defects are densely distributed and have different shapes, etc., the present invention proposes a surface defect detection method for smartphone panels based on the YOLO V5 model.

[0006] A method for detecting surface defects of smartphone panels based on the YOLO V5 model. The technical solution includes:

[0007] S1, capture defect images. To obtain clearer defect images, the acquisition method is based on the principle of laser scattering. The acquisition device mainly receives the scattered light generated by the laser beam at the defect. The image acquisition card converts the optical signal into a digital signal to synthesize the image data.

[0008] S2, defect image cutting. The optimal size of the detected image in the YOLO model is 640×640. Therefore, to avoid the loss of original image information, the collected image needs to be cut and then the images with defect information need to be filtered out.

[0009] S3: Perform image enhancement and annotation on the filtered image. The annotation information of the YOLO model is stored in a text file. Therefore, when annotating the defect location in the image, the defect category and location information are saved as a text file, and the image and annotation file correspond one to one. The text format is preferably TXT.

[0010] Furthermore, we augment the labeled images by adding noise, changing brightness and chroma, and rotating them to improve the generalization of the model. At the same time, the corresponding annotation files are generated synchronously with the augmented images.

[0011] Furthermore, the labeled images and corresponding annotation files are used to create a dataset of mobile phone panels, and the training set, validation set and test set are divided into a ratio of 8:1:1, which are used for model training and testing respectively.

[0012] S4, builds an improved YOLO V5 network, including:

[0013] S4-1: Backbone Improvements. The Cross-Stage Local Network (CSP) contains a BottleNeck structure, which consists of a 1×1 CBL module (convolutional layer module), a 3×3 CBL module, and a residual fusion structure. Replacing the 3×3 CBL module with the PSA module improves the backbone network's feature extraction capabilities without changing the model size.

[0014] Furthermore, the stacking ratio of CSP in the original YOLO network is 1:2:3:1. In order to further improve the feature extraction capability of the backbone network, the stacking ratio of the PSA-CSP structure after adding the PSA module is adjusted to 1:3:3:1.

[0015] S4-2, Improved Neck. For the feature maps output by the backbone network, based on the original YOLO Neck network PANet, we add information fusion between multiple feature maps of adjacent sizes at different levels and feature maps of the same size across levels.

[0016] Furthermore, before feature maps are fused, if the sizes are inconsistent, average pooling or bilinear interpolation is required to scale the feature maps.

[0017] Furthermore, when fusing multiple feature maps of different dimensions, a trainable weighting parameter is introduced to perform weighted stacking in the dimension. Specifically, it can be expressed as:

[0018]

[0019] Among them, num is the number of fusion feature maps, w i ,w j are the weighted parameters corresponding to the i-th and j-th feature maps, F j is the jth feature map of the input. Set the learning rate ε0 to a minimum value to avoid numerical instability, F in is the output feature map, and e is a natural constant.

[0020] S4-3, Improved Head. Based on the original YOLO's ordinary convolution, a deformable convolution is added to improve the detection head's perception of the target's spatial position and shape.

[0021] Furthermore, we improve the feature maps of 80×80, 40×40 and 20×20 sizes output by YOLO V5, and unify the number of channels of the three feature maps of different dimensions through a common 1×1 convolution. The unified number of channels is n c =(cls+5)*n a , where cls is the number of defect categories of mobile phone panels, n a is the number of anchor boxes generated for each grid of the feature map.

[0022] After unifying the dimensions, the three scaled feature maps are split into a eigenvalue modulation tensor mask and an offset tensor offer along the channel dimension, which are then fed into the Deformable V2 convolutional neural network. Training the offset tensor offer enhances the detection head's ability to identify regions of interest and extract target features.

[0023] S4-4, regarding anchor box selection. The improved YOLO V5 model first calculates the intersection-over-union (IoU) ratio between each anchor box and the annotated box. Anchor boxes with an IoU ratio exceeding a set threshold are grouped together. Therefore, all anchor boxes in the same group identify the same object.

[0024] Furthermore, in the same set, a virtual line is formed between two nodes, and the positive and negative effects of confidence are continuously transmitted along the line.

[0025] Furthermore, among the adjacent anchor frames that are connected to the target anchor frame, if the confidence is higher than that of the target anchor frame, it is a strong neighbor anchor frame; otherwise, if the confidence is weaker than that of the target anchor frame, it is a weak neighbor anchor frame.

[0026] Furthermore, the confidence of the target anchor box is strengthened by the weak neighboring anchor boxes and weakened by the strong neighboring anchor boxes. The confidence of the final predicted box will be strengthened, while the confidence of the redundant boxes will be continuously and flexibly weakened.

[0027] The positive influence of the weak neighbor set is expressed as:

[0028]

[0029] The negative impact of a strong neighbor set is expressed as:

[0030]

[0031] Among them, b i is the target anchor box, b j is a weak neighbor anchor box, b jj is a strong neighbor anchor frame, M p (i) As the positive influence of the target anchor box, p(b i ) is the confidence of the target anchor box. is the weak neighbor set of the target anchor box, Q is the anchor box b j The number of p(b j ) is the anchor box b j The confidence level, p(b i ) is the target anchor box b i Confidence level. n (i) As the negative influence of the target anchor box, p(b i ) is the confidence of the target anchor box. IOU represents the intersection-over-union ratio of two anchor boxes. is the strong neighbor set of the target anchor box, p(b jj ) is the anchor box b jj Confidence level. jj,i≤ξ is the minimum supremum of the strong neighbor matrix to prevent the target anchor box from being repeatedly suppressed by a strong neighbor anchor box. ζ is a hyperparameter and is determined according to the experimental situation. IOU represents the intersection-and-union ratio of two anchor boxes. In the strong neighbor set, θ is the threshold of the intersection-and-union ratio. Only when IOU(b i ,b jj )>θ to produce an inhibitory effect. α0 is a negative impact factor, and the value of α0 determines the selection of different strong neighbor anchor boxes b jj To punish the target anchor box, during training, the box with the largest confidence value (α0=1.0) and the box with the highest overlap (α0=0.0) are selected alternately.

[0032] The final confidence of the target anchor box P:

[0033] p=p(b i )+M P (i)-M n (i)

[0034] S4-5 uses the cross-entropy function. To improve the network's detection accuracy for dense, small-sized objects, the cross-entropy function Varifocal loss is used to calculate the loss value of the filtered anchor boxes. This function uses the asymmetric weighting principle for positive and negative samples and introduces the intersection-over-union (IoU) parameter. The specific function is as follows:

[0035]

[0036] Among them, q iou is the intersection-over-union ratio between the anchor box and the annotation box, VEL is the loss value of the cross entropy function, which represents the difference between the predicted anchor box and the annotation box. Through the learning of the neural network, the loss value gradually decreases, indicating that the algorithm's prediction is gradually fitting to the true value. The loss value is composed of the differences between all generated anchor boxes and annotation boxes. In the detection process, when q iou When q > 0, it means that the anchor box recognizes the defect, which is a positive sample. iou The higher the positive sample, the greater the contribution to the loss value. iou =0, it means that only the background is detected in the anchor frame, that is, the negative sample. α and β are adjustable coefficients, which reduce the contribution of negative samples to the loss value.

[0037] S4-6, the improved YOLO V5 model has significantly improved detection performance compared to the original YOLO model, but the model size has also increased to a certain extent. Therefore, the detection speed can be improved by compressing the network structure.

[0038] Furthermore, the batch normalization (BN) and convolutional layers are compressed. The CSP architecture in YOLO has a 1×1 CBL module at the end, which increases the number of feature map channels. Therefore, compressing the convolutional and batch normalization layers in this architecture can reduce the number of trainable parameters without compromising accuracy.

[0039] Furthermore, for the fused convolutional layer and BN layer, the output tensor y is calculated as follows:

[0040]

[0041] Where x is the input tensor, y is the output tensor, μ, σ are the calculated mean and variance, δ is the offset parameter, γ is the scaling factor, ε′ is a very small number to prevent the denominator from being zero, and m i and C i As convolution weights and bias parameters.

[0042] In step S5, the improved YOLO V5 model is trained using the mobile phone panel dataset to obtain the optimal target parameters and target weights for mobile phone panel detection. The target parameters and target weights are used as the parameters and weights of the trained YOLO V5 model.

[0043] S6 uses the trained YOLO V5 model to detect surface defects on mobile phone panels and outputs the defect category and location information.

[0044] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0045] Based on the optimal weight file generated during the training phase, the trained improved YOLO V5 model was tested on images from the test set. Using the same hardware, software, training parameters, and iteration count, and setting an Intersection-over-Union threshold of 0.5, the improved YOLO V5 model achieved a 3.5% improvement in average accuracy for smartphone panel surface defect detection compared to the original YOLO V5, while maintaining minimal speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a flow chart of the method of the present invention;

[0047] Figure 2 This is the schematic diagram of image acquisition;

[0048] Figure 3 To improve the neural network diagram of YOLO V5;

[0049] Figure 4 To improve the Backbone network structure diagram of YOLO V5;

[0050] Figure 5 To improve the Neck network structure diagram of YOLO V5;

[0051] Figure 6 Schematic diagram of feature fusion to improve Neck of YOLO V5. DETAILED DESCRIPTION

[0052] The technical solution of the present invention is further described below in conjunction with the drawings and specific implementation methods of this embodiment. Obviously, the described embodiments are only part of the embodiments of the present invention, but are not intended to limit the present invention.

[0053] Example

[0054] A method for detecting surface defects of smartphone panels based on the YOLO V5 model. Figure 1 , including the following steps:

[0055] S1, acquisition of defect images.

[0056] like Figure 2 During the image acquisition process, the phone panel to be inspected is transported on a roller-type conveyor. A laser beam is collimated into a parallel beam by a collimating system. The rotating scanning mirror reflects the laser beam onto the phone panel surface, scanning it line by line. When scanning a clean, defect-free glass panel, the light is primarily refracted and reflected. However, when scanning a defect, scattering occurs. A low-light detector receives the scattered laser light signal, and the image acquisition card converts the optical signal into a digital signal, synthesizing the defect image and sending it to the host computer for display.

[0057] S2, cutting of defect image.

[0058] The image size of the laser scattering scan is 4500×6600. If directly fed into the YOLO detection algorithm, it would be scaled to 640×640 after image adaptation. Such a large image scaling would result in the loss of features of small defects. Therefore, before creating the dataset, the images were cropped to 640×640. From these cropped images, 5470 images containing defect information were selected.

[0059] S3, image enhancement and annotation of the filtered images, including:

[0060] To minimize the impact of the inspection environment on the results, smartphone panel surface defects were categorized into five categories: bad pixels, scratches, chipped edges, dust, and dirt. LabelImg software was then used to annotate the defect categories and locations in the images. This annotation information was saved as a TXT file, generating a corresponding TXT annotation file for each defect image.

[0061] The annotated images require data augmentation. First, the images are rotated 90, 180, and 270 degrees. Then, the brightness and chromaticity of each image are randomly adjusted, Gaussian noise is added, and multiple images are randomly superimposed. This ultimately expands the image data by a factor of four. The corresponding annotation files are generated simultaneously with the image augmentation. The resulting mobile phone panel dataset has a training, validation, and test set ratio of 8:1:1, making it suitable for model training and testing.

[0062] S4, build the improved YOLO V5 model, according to Figure 3 Shown, including:

[0063] S4-1, according to Figure 4 As shown in Figure 1, the PSA module is introduced into Backbone. To improve Backbone's feature map extraction capabilities, the attention module PSA replaces the 3×3 CBL module in the CSP residual structure to form a new module PSA-CSP. The new Backbone network is then constructed using a stacking ratio of 1:3:3:1 for PSA-CSP.

[0064] S4-2, according to Figure 5 As shown in Figure 1, the improved Neck. For the feature map output by the backbone network, through average pooling and bilinear interpolation operations, on the basis of PANet, the feature fusion of p2-p3′, p3-p′4, p4-p5″, p4-p3′ and p′4-p5″ is added; there is also a cross-level fusion of p4-p′4′. The fusion process is referenced in Figure 6 , trainable weight parameters are introduced to perform weighted stacking on the channels.

[0065] In S4-3, the head layer adds deformable convolution to the standard convolution layer of the original YOLO, improving the detection head's perception of the spatial position and shape of the target. After passing through a standard convolution layer, the number of channels of the feature maps at the three scales is unified. They are then split into a eigenvalue modulation tensor (mask) and an offset tensor (offer) along the channel dimension, and then fed into the deformable convolution layer (Deformable V2).

[0066] S4-4, in order to balance positive and negative samples and improve the detection accuracy of dense small-sized targets, the cross entropy function Varifocal loss is used.

[0067] In S4-5, regarding anchor box selection, when anchor boxes with an IoU ratio greater than 0.6 identify the same defect, they are divided into multiple sets to detect different defects in the feature map. Each anchor box is used as the target anchor box in turn. Anchor boxes in this set with a higher confidence than the target anchor box are called strong neighbor sets, while those with a lower confidence than the target anchor box are called weak neighbor sets. Strong neighbor anchor boxes have a negative impact on the target anchor box, while weak neighbor anchor boxes have a positive impact.

[0068] S4-6, model compression. Compressing the convolutional layer and batch normalization layer in the last 1×1 CBL layer in the CSP structure can reduce the model size without affecting accuracy.

[0069] S5, uses the prepared mobile phone panel data set to train the improved YOLO V5 model, obtains the best weight file, and outputs the best weight parameters to detect the performance test indicators of the network.

[0070] S6, uses the optimal weight file and test set to perform detection and output the category and location information of the defect.

[0071] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting surface defects of smartphone panels based on the YOLO V5 model, characterized in that: The method comprises the following steps: S1, collect defect images; S2, cutting defective images and filtering out images with defect information; S3, performs image enhancement and annotation on the screened images and generates a mobile phone panel dataset, including a training set, a validation set, and a test set; S4, builds an improved YOLO V5 network, including: S4-1, Backbone introduces the PSA module, specifically replacing the 3×3CBL module in the CSP residual structure with the attention module PSA to form a new module PSA-CSP; S4-2, Neck adds multiple cross-scale and cross-level feature fusion; S4-3, in the Head part, add deformable convolution to improve the detection head's perception of the target's spatial position and shape; S4-4, when screening anchor frames, the anchor frames are divided into multiple sets according to the identified defects. Among them, the anchor frames in the same set are divided into strong neighbor sets and weak neighbor sets according to the confidence. The strong neighbor set suppresses the confidence of the target anchor frame, and the weak neighbor set strengthens the confidence of the anchor frame; S4-5, use the cross entropy function Varifocal loss to calculate the loss value of the anchor box: Among them, VEL is the cross entropy loss value of the anchor box, q iou is the intersection-over-union ratio between the target anchor box and the annotation box, when q iou = 0, it means that the anchor box is a negative sample. iou When >0, it means that the anchor box is a positive sample, P is the predicted probability of the anchor box for the defect, and α and β are adjustable coefficients; S4-6, compress the convolutional layer and BN layer in the last 1×1 CBL layer in the CSP structure to reduce the model size. The compression method is as follows: Where x is the input tensor, y is the output tensor, μ, σ are the calculated mean and variance, δ is the offset parameter, γ is the scaling factor, ε′ is a very small number to prevent the denominator from being zero, and m i and C i As convolution weights and bias parameters; S5, uses the mobile phone panel dataset to train the improved YOLO V5 model and obtain the optimal weight file; S6, uses the optimal weight file and test set to perform detection and output the category and location information of the defect.

2. A method for detecting surface defects of smartphone panels based on the YOLO V5 model according to claim 1, characterized in that: The ratio of the training set, validation set, and test set in step 3 is 8:1:

1.

3. A method for detecting surface defects of a smart phone panel based on the YOLO V5 model according to claim 1, characterized in that S3 include: S3-1, Data Annotation The surface defects of smartphone panels are classified into five categories: bad pixels, scratches, chipped edges, dust, and dirt. The annotation information includes the defect category, the center coordinates of the annotation box, and the length and width of the annotation box. The annotation information is saved in text format, and a text-formatted annotation file is generated for each defect image. S3-2, Image Enhancement First, the images are rotated at different angles, and then the brightness and chromaticity of each image are randomly adjusted, Gaussian noise is added, and multiple images are randomly superimposed. Finally, the image data is enhanced, and the corresponding annotation files are generated synchronously with the image enhancement.

4. A method for detecting surface defects of smartphone panels based on the YOLO V5 model as claimed in claim 1, characterized in that: S4-2 also includes: When fusion of features is performed, trainable weight parameters are introduced to perform weighted stacking on the channel. The feature fusion function is: Among them, num is the number of fusion feature maps, w i ,w j are the weighted parameters corresponding to the i-th and j-th feature maps, F j is the jth feature map of the input, ε0 is the learning rate, F in is the output feature map, and e is a natural constant.

5. A method for detecting surface defects of smartphone panels based on the YOLO V5 model as claimed in claim 1, characterized in that: S4-3 also includes: In the channel dimension of the feature map, it is split into two sets of tensors: offer and mask, where the number of channels of the feature map is n c =(cls+5)*n a , where cls is the number of defect categories of mobile phone panels, n a is the number of anchor boxes generated for each grid of the feature map.

6. A method for detecting surface defects of smartphone panels based on the YOLO V5 model as claimed in claim 1, characterized in that: S4-4 also includes: The positive impact of the weak neighbor set is expressed as: The negative impact of a strong neighbor set is expressed as: Among them, b i is the target anchor box, b j is a weak neighbor anchor box, b jj is a strong neighbor anchor frame, M p (i) As the target anchor box b i The positive impact of is the target anchor box b i The weak neighbor set of p(b j ) is a weak neighbor anchor box b j The confidence level, p(b i ) is the target anchor box b i Confidence, Q is the weak neighbor anchor box b j The number of n (i) Negative impact as a target anchor box, is a strong neighbor anchor box b jj The set of p(b jj ) is the confidence of the strong neighbor anchor box, α0 is the negative impact factor, SUP jj,i ≤ξ is the minimum supremum of the strong neighbor matrix, ζ is a hyperparameter, which is determined according to the experimental situation, IOU represents the intersection-union ratio of two anchor frames, θ is the threshold of the intersection-union ratio, and in the strong neighbor set, only when IOU(b i ,b jj )>θ to produce the suppression effect, and the confidence of the final target anchor box is: p(b i )=p(b i )+M P (i)-M n (i)。

Images