Defect detection method based on smart phone display screen

Abstract

The invention belongs to the technical field of mobile phone display screen detection, and particularly discloses a defect detection method based on a smart mobile phone display screen. Graying and denoising the collected surface defect image of the original display screen; performing feature labeling and expansion on the preprocessed image; on the basis of the U-Net network, a residual block is inserted, common convolution is replaced by depth separable convolution, and an attention module is added; performing batch training on the cross entropy loss function by using a stochastic gradient descent method, continuously updating parameters of the network model, and optimizing the parameters of the network model by using an Adam method; when the output value of the loss function is reduced and does not change any more, iteration is stopped; and inputting a to-be-detected mobile phone display screen image into the trained network model for detection, if the mobile phone display screen is detected to have a defect, outputting a defect binary image of the mobile phone display screen, and calculating the real size of the defect. According to the invention, surface defects such as scratches and black spots of the mobile phone display screen can be rapidly and accurately detected.

Description

Technical Field

[0001] The present invention relates to the technical field of mobile phone display screen detection, and in particular to a defect detection method based on a smart phone display screen. Background Art

[0002] With the rapid development of information technology, information products have penetrated into every aspect of people's lives. In recent years, artificial intelligence technology, especially deep learning, has demonstrated powerful capabilities in many fields. Applying artificial intelligence technology to the inspection and testing of information products is one of the important research directions.

[0003] The U²-Net convolutional neural network was formally proposed by the University of Alberta at the CVPR conference in 2020. It is mainly a network model for salient object detection tasks. The main body of the U²-Net network is a structure similar to that of U-Net, which also has encoding and decoding modules. Each stage is composed of residual U-shaped blocks (RSU) of multi-scale features. This model has the traditional advantages of the U-Net network, and because it mixes receptive fields of different sizes, it can capture more image features from different scales, and this structure increases the depth of the entire network architecture without significantly increasing the computational cost. The overall design framework of the U²-Net network model is a two-layer nested U-shaped network structure, and each level follows an encoder-decoder structure similar to that in the U-Net model. In the U²-Net model, the outermost layer consists of 11 independent large U-shaped structures, each of which consists of encoder and decoder blocks and is filled with small residual U-shaped blocks.

[0004] Mobile phone display defect detection is a crucial link in the mobile phone manufacturing process. It involves the identification and evaluation of various potential defects in the display. Traditional defect detection algorithms have disadvantages such as complexity and low accuracy, and can no longer meet the needs of high-precision detection of mobile phone displays. Summary of the invention

[0005] In view of the shortcomings of the prior art described above, the purpose of the present invention is to provide a defect detection method based on a smartphone display, which is used to solve the problems in the prior art that the defect detection algorithm loses some features of the defects during the network training process, the gradient disappears when the network is deepened, and the training difficulty is high.

[0006] To achieve the above-mentioned purpose and other related purposes, the present invention provides a defect detection method based on a smart phone display screen, comprising: Preprocessing: grayscale the collected original display screen surface defect image to obtain a single-channel grayscale image, and perform Gaussian bilateral filtering to filter out related noise; normalize the filtered image; Create defect dataset: perform feature annotation on preprocessed images, then perform sample expansion on the annotated images, and divide the annotated images into training sets and test sets according to the proportion; Build an improved U²-Net network: Based on the U²-Net network, insert a residual block, replace the ordinary convolution in the RSU block in the upsampling stage with a depth-wise separable convolution, and add an attention module; The training set is used to improve the U²-Net network model and train it: the cross loss function is batch trained using the stochastic gradient descent method, the parameters of the network model are continuously updated, and the parameters of the network model are optimized using the Adam method; when the output value of the loss function decreases and no longer changes, the iteration stops and the optimal network model is formed; the test set data is input into the optimal network model to obtain the prediction results; Detection of mobile phone display screens: Collect images of mobile phone display screens to be detected and input them into the trained improved U²-Net network model for detection. If defects are detected in the mobile phone display screen, output the defect binary image of the mobile phone display screen. Defect measurement: The defect size calculation method is used to obtain the two-dimensional size of defects on the mobile phone display.

[0007] Optionally, the step of Gaussian bilateral filtering processing specifically includes: Load the original defect image; Perform Gaussian bilateral filtering on the original defect image to retain the edge linearity of the image and suppress noise. The calculation of Gaussian bilateral filtering is as follows: In the formula, and are the input image and the output image respectively, is the pixel position in the output image, is the pixel position in the input image, For the space window, and are two Gaussian functions, express and The Euclidean distance between is the normalized weight; Output the filtered defect image.

[0008] Optionally, the annotated image is expanded by using rotation, flipping, and translation, including: Define the image to rotate around the center point at 45°, 90°, and 135° respectively; transform the image along the axis line; translate the image horizontally or vertically on the plane according to the set distance.

[0009] Optionally, on the basis of the U²-Net network, a residual block is inserted, specifically comprising: in the U²-Net, the residual block is inserted between the convolutional layers in the encoder, the residual block comprises three convolutional layers, a batch normalization layer, a ReLu activation function and a residual connection, the output of the convolutional layer is added to the residual connection to obtain the output of the residual block, and its function expression is: In the formula, is the input tensor, is the residual mapping, is the output tensor.

[0010] Optionally, the depthwise separable convolution first convolves the input data through a depthwise convolution layer, and then performs a pointwise convolution operation on the output of the depthwise convolution layer. In the depthwise separable convolution, the input is , the output feature map dimension is , is the spatial size of the convolution kernel, M is the number of convolution kernels, that is, the number of channels of the input feature map, It is the number of convolution kernels, that is, the number of channels of the output feature map; In the depth convolution operation, the number of channels of the convolution kernel of the depth convolution is 1, and the convolution kernel is , then the amount of depth convolution calculation is: In the point-by-point convolution operation, use The 1×1 convolution of the channel operates on the structure of the depth convolution and obtains 1, the computational cost of point-by-point convolution is; Then the computational cost of depth-wise separable convolution is: .

[0011] Optionally, the attention module is a CBAM module, the CBAM module is embedded in the skip connection, and the CBAM module includes a channel attention module and a spatial attention module; The channel attention module performs global average pooling and global maximum pooling operations on the original feature map to generate two channel descriptors; the two channel descriptors are respectively input into the shared fully connected layer and transformed by the ReLu activation function; the transformed channel descriptors are added and the channel attention weight is obtained by the Sigmoid function; the channel attention weight is multiplied by the original feature map to achieve recalibration of the channel dimension feature; The spatial attention module performs average pooling and maximum pooling operations in the channel dimension on the feature map processed by the channel attention module to generate two spatial descriptors; concatenates the two spatial descriptors and fuses them through a convolutional layer; obtains the spatial attention weight through the Sigmoid function; multiplies the spatial attention weight with the original feature map to achieve feature recalibration of the spatial position.

[0012] Optionally, the semantic segmentation of the mobile phone display defect image belongs to a binary classification problem, and a binary classification cross entropy loss function is used, which is expressed as follows: In the formula, is the pixel coordinate, are the image height and width, and Represent the pixel values ​​of the background and the predicted image respectively.

[0013] Optionally, the update strategy of the Adam method optimization function for the network model parameters is: In the formula, , are first-order matrix estimation and second-order matrix estimation respectively, , are the decay rates of the first-order matrix and the second-order matrix respectively; is the initialization learning rate; are the model parameters at the current time step, are the updated model parameters.

[0014] Optionally, the method of obtaining the two-dimensional size of the defect on the mobile phone display screen by using the defect size calculation method specifically includes: Performing thinning processing on the defect binary image; Count the number of skeleton pixels to get the pixel size of the defect; Determine the image resolution, and use the defect size calculation formula to calculate the actual size of the defect, that is, the two-dimensional size of the mobile phone display defect. The defect size calculation formula is: In the formula, is the pixel size, including width, height or diameter, is the image resolution, is the actual size of the defect.

[0015] As described above, the defect detection method based on the smartphone display screen proposed by the present invention has the following beneficial effects: A Gaussian bilateral filtering algorithm is used to remove noise interference in the image, and then the image is annotated. In order to solve the problem of insufficient sample data, the data set is expanded by using image flipping, rotation and translation. The production of microscopic dark field image data set of surface defects of optical mobile phone display screens is completed, providing data support for deep learning defect extraction model.

[0016] The improved U²-Net network introduces an attention module and a residual block based on the U²-Net network, which effectively solves the problem that some features of surface defects of mobile phone displays are lost during network training. At the same time, it solves the gradient vanishing problem that occurs as the network deepens, reduces the training difficulty of the network model, improves the effect of defect extraction on mobile phone displays, and the output defect binary image can calculate the actual size of the defect. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Shown is a flow chart of a defect detection method based on a smart phone display screen according to an embodiment of the present invention; Figure 2 Shown is a schematic diagram of the structure of a residual block according to an embodiment of the present invention; Figure 3 Shown is a schematic diagram of comparison of an original image, a label image and a binary image on a mobile phone display screen according to an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The following describes the embodiments of the present invention through specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention.

[0019] See also Figure 1 , Figure 1 The present application provides a method for defect detection based on a smartphone display screen. Figure 1 As shown, this embodiment discloses a defect detection method based on a smartphone display screen, comprising: Preprocessing: Grayscale the collected original display screen surface defect image to obtain a single-channel grayscale image, and perform Gaussian bilateral filtering to filter out related noise; normalize the filtered image to normalize all collected defect images to an image size of 512×512, which is conducive to network training; Among them, microscopic dark field imaging detection technology is used to collect defect images of mobile phone displays. Microscopic dark field imaging detection technology is a technology that presents bright and clear images of objects in a dark background field of view. It is mainly used to image the surface defects of mobile phone displays and obtain their defect images. Microscopic dark field imaging technology can provide high-resolution defect images, which provides a guarantee for the collection of surface defect image data sets.

[0020] The grayscale value of the defective pixel on the surface of the mobile phone display is higher than the grayscale value of the surrounding pixels. In the process of filtering and denoising, the non-defective area should be suppressed as much as possible, while the contrast of the defective area should be improved. The steps of Gaussian bilateral filtering processing include: Load the original defect image; Perform Gaussian bilateral filtering on the original defect image to retain the edge linearity of the image and suppress noise. The calculation of Gaussian bilateral filtering is as follows: In the formula, and are the input image and the output image respectively, is the pixel position in the output image, is the pixel position in the input image, For the space window, and are two Gaussian functions, express and The Euclidean distance between is the normalized weight; Output the filtered defect image.

[0021] Create defect data set: feature annotate the preprocessed images, then expand the samples of the annotated images, and expand the annotated images by using rotation, flipping, translation and other means to increase the diversity of the training data set. Specifically, define the image to rotate around the center point at 45°, 90°, and 135° respectively; perform axisymmetric transformation on the image along the axis; translate the image in the horizontal or vertical direction on the plane according to the set distance; divide the annotated images into training set and test set in a ratio of 4:1; The labelme image annotation tool is used to annotate the surface defect images. Specifically, the regions of interest in the images are manually annotated and converted into .json format files. The .json format files are parsed to obtain the label images corresponding to the defect images. Each label image contains only two categories: white defect images and black background images, which are used to train and evaluate the network model.

[0022] Build an improved U²-Net network: Based on the U²-Net network, insert a residual block, replace the ordinary convolution in the RSU block in the upsampling stage with a depth-wise separable convolution, and add an attention module; use the He initialization method to initialize the model parameters, set the number of iterations to 200, the training batch size to 3, and the initial value of the learning rate to 0.001.

[0023] Among them, on the basis of the U²-Net network, the residual block is inserted, specifically including: in the U²-Net, the residual block is inserted between the convolutional layers in the encoder, such as Figure 2 As shown in the figure, the residual block consists of three convolutional layers, a batch normalization layer, a ReLu activation function and a residual connection. The output of the convolutional layer is added to the residual connection to obtain the output of the residual block. Its function expression is: In the formula, is the input tensor, is the residual mapping, is the output tensor.

[0024] The introduction of residual blocks can make the U²-Net network model deeper, thereby improving the network's representation and approximation capabilities. At the same time, by introducing residual connections, the impact of gradient vanishing and gradient exploding can be significantly reduced, the reliability of the model can be enhanced, and the accuracy and generalization ability of the network can be further improved.

[0025] Depthwise separable convolution first convolves the input data through the depthwise convolution layer, and then performs pointwise convolution on the output of the depthwise convolution layer. In depthwise separable convolution, the input is , the output feature map dimension is , is the spatial size of the convolution kernel, M is the number of convolution kernels, that is, the number of channels of the input feature map, It is the number of convolution kernels, that is, the number of channels of the output feature map; In the depth convolution operation, the number of channels of the convolution kernel of the depth convolution is 1, and the convolution kernel is , then the amount of depth convolution calculation is: In the point-by-point convolution operation, use The 1×1 convolution of the channel operates on the structure of the depth convolution and obtains 1, the computational cost of point-by-point convolution is; Then the computational cost of depth-wise separable convolution is: .

[0026] The computational complexity of ordinary convolution is: Compare and analyze the computational complexity of depth-wise separable convolution and ordinary convolution: If a 3×3 convolution kernel is used (i.e. =3), the number of parameters of its depthwise separable convolution is reduced by 8~9 times compared with the standard convolution. And the accuracy of the model will not be affected, which greatly improves the training speed of the model.

[0027] The attention module is a CBAM module, which is embedded in a skip connection structure and includes a channel attention module and a spatial attention module; The channel attention module performs global average pooling and global maximum pooling operations on the original feature map to generate two channel descriptors; the two channel descriptors are respectively input into the shared fully connected layer and transformed by the ReLu activation function; the transformed channel descriptors are added and the channel attention weight is obtained by the Sigmoid function; the channel attention weight is multiplied by the original feature map to achieve recalibration of the channel dimension feature; The spatial attention module performs average pooling and maximum pooling operations in the channel dimension on the feature map processed by the channel attention module to generate two spatial descriptors; concatenates the two spatial descriptors and fuses them through a convolutional layer; obtains the spatial attention weight through the Sigmoid function; multiplies the spatial attention weight with the original feature map to achieve feature recalibration of the spatial position.

[0028] By introducing the CBAM module, the feature map in the upsampling process can be combined with the feature map of the same size in the downsampling stage, which effectively solves the problem of feature loss of small mobile phone display surface defect targets during the downsampling process.

[0029] The training set is used to improve the U²-Net network model and train it: the cross entropy loss function is batch trained using the stochastic gradient descent method, the parameters of the network model are continuously updated, and the parameters of the network model are optimized using the Adam method; when the output value of the loss function decreases and no longer changes, the iteration stops and the optimal network model is formed; the test set data is input into the optimal network model to obtain the prediction results; Among them, the semantic segmentation of the mobile phone display defect image belongs to a binary classification problem, and the binary classification cross entropy loss function is used, and the expression is as follows: In the formula, is the pixel coordinate, are the image height and width, and Represent the pixel values ​​of the background and the predicted image respectively.

[0030] The update strategy of the Adam optimization function for the network model parameters is: In the formula, , are first-order matrix estimation and second-order matrix estimation respectively, , are the decay rates of the first-order matrix and the second-order matrix respectively; is the initialization learning rate; are the model parameters at the current time step, are the updated model parameters.

[0031] When testing the optimal network model on the test set, the effect and performance of the model are evaluated by similarity coefficient, intersection-over-union ratio, accuracy and recall evaluation indicators. The value range of these indicators is between 0 and 1. When the extraction result is optimal, the indicator value is 1; when it is the worst, the indicator value is 0. Therefore, the higher the indicator value, the better the extraction effect of the improved U²-Net model.

[0032] If the similarity coefficient, intersection-over-union ratio, accuracy and recall rate perform well and are higher than the set threshold, they can be used to detect mobile phone displays.

[0033] Detection of mobile phone display screen: collect the image of the mobile phone display screen to be detected, and input it into the trained improved U²-Net network model for detection. If defects are detected in the mobile phone display screen, output the defect binary image of the mobile phone display screen; Figure 3 The following is a schematic diagram showing the comparison of the original image (a), the labeled image (b) and the binary image (c) of the display screen.

[0034] Defect measurement: The defect size calculation method is used to obtain the two-dimensional size of defects on the mobile phone display.

[0035] Specifically, the defect size calculation method is used to obtain the two-dimensional size of the mobile phone display defect, including: Performing thinning processing on the defect binary image, simplifying the defect object of the defect binary image into a skeleton with a width of a single pixel; Count the number of skeleton pixels to get the pixel size of the defect; Determine the image resolution, and use the defect size calculation formula to calculate the actual size of the defect, that is, the two-dimensional size of the mobile phone display defect. The defect size calculation formula is: In the formula, is the pixel size, including width, height or diameter, is the image resolution, is the actual size of the defect.

[0036] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A defect detection method based on a smart phone display screen, characterized in that: include: Preprocessing: Grayscale the collected original display screen surface defect image to obtain a single-channel grayscale image, and perform Gaussian bilateral filtering to filter out related noise; Normalize the filtered image; Create defect datasets: perform feature annotation on preprocessed images, then perform sample expansion on the annotated images, and divide the annotated images into training sets and test sets according to the ratio; Build an improved U²-Net network: Based on the U²-Net network, insert a residual block, replace the ordinary convolution in the RSU block in the upsampling stage with a depth-wise separable convolution, and add an attention module; The training set is used to improve the U²-Net network model and train it: the cross entropy loss function is batch trained using the stochastic gradient descent method, the parameters of the network model are continuously updated, and the parameters of the network model are optimized using the Adam method; when the output value of the loss function decreases and no longer changes, the iteration stops and the optimal network model is formed; the test set data is input into the optimal network model to obtain the prediction results; Detection of mobile phone display screens: Collect images of mobile phone display screens to be detected and input them into the trained improved U²-Net network model for detection. If defects are detected in the mobile phone display screen, output the defect binary image of the mobile phone display screen. Defect measurement: Use the defect size calculation method to obtain the two-dimensional size of defects on the mobile phone display.

2. A defect detection method based on a smart phone display screen according to claim 1, characterized in that: The steps of the Gaussian bilateral filtering process specifically include: Load the original defect image; Perform Gaussian bilateral filtering on the original defect image to retain the edge linearity of the image and suppress noise. The calculation of Gaussian bilateral filtering is as follows; In the formula, and are the input image and the output image respectively, is the pixel position in the output image, is the pixel position in the input image, For the space window, and are two Gaussian functions, express and The Euclidean distance between is the normalized weight; Output the filtered defect image.

3. The defect detection method based on a smart phone display screen according to claim 1, characterized in that: The annotated image is expanded by using rotation, flipping, and translation, including: Define the image to rotate around the center point at 45°, 90°, and 135° respectively; transform the image along the axis line; translate the image horizontally or vertically on the plane according to the set distance.

4. The defect detection method based on a smart phone display screen according to claim 1, characterized in that: On the basis of the U²-Net network, a residual block is inserted, specifically including: in the U²-Net, the residual block is inserted between the convolutional layers in the encoder, the residual block includes three convolutional layers, a batch normalization layer, a ReLu activation function and a residual connection, and the output of the convolutional layer is added to the residual connection to obtain the output of the residual block, and its function expression is: In the formula, is the input tensor, is the residual mapping, is the output tensor.

5. The defect detection method based on a smart phone display screen according to claim 4 is characterized in that: The depth-wise separable convolution first convolves the input data through the depth convolution layer, and then performs point-by-point convolution on the output of the depth convolution layer. In the depth-wise separable convolution, the input is , the output feature map dimension is , is the spatial size of the convolution kernel, M is the number of convolution kernels, that is, the number of channels of the input feature map, It is the number of convolution kernels, that is, the number of channels of the output feature map; In the depth convolution operation, the number of channels of the convolution kernel of the depth convolution is 1, and the convolution kernel is , then the amount of depth convolution calculation is: In the point-by-point convolution operation, use The 1×1 convolution of the channel operates on the structure of the depth convolution and obtains 1, the computational cost of point-by-point convolution is; Then the computational cost of depth-wise separable convolution is: 。 6. A defect detection method based on a smart phone display screen according to claim 5, characterized in that: The attention module is a CBAM module, which is embedded in a skip connection structure and includes a channel attention module and a spatial attention module; The channel attention module performs global average pooling and global maximum pooling operations on the original feature map to generate two channel descriptors; the two channel descriptors are respectively input into the shared fully connected layer and transformed by the ReLu activation function; the transformed channel descriptors are added and the channel attention weight is obtained by the Sigmoid function; the channel attention weight is multiplied by the original feature map to achieve recalibration of the channel dimension feature; The spatial attention module performs average pooling and maximum pooling operations in the channel dimension on the feature map processed by the channel attention module to generate two spatial descriptors; concatenates the two spatial descriptors and fuses them through a convolutional layer; obtains the spatial attention weight through the Sigmoid function; multiplies the spatial attention weight with the original feature map to achieve feature recalibration of the spatial position.

7. The defect detection method based on a smart phone display screen according to claim 1, characterized in that: The semantic segmentation of mobile phone display defect images belongs to a binary classification problem, and the binary classification cross entropy loss function is used. The expression is as follows: In the formula, is the pixel coordinate, are the image height and width, and Represent the pixel values ​​of the background and the predicted image respectively.

8. The defect detection method based on a smart phone display screen according to claim 7, characterized in that: The update strategy of the Adam method optimization function for the network model parameters is: In the formula, , are first-order matrix estimation and second-order matrix estimation respectively, , are the decay rates of the first-order matrix and the second-order matrix respectively; is the initialization learning rate; are the model parameters at the current time step, are the updated model parameters.

9. The defect detection method based on a smart phone display screen according to claim 1, characterized in that: The method for calculating the defect size is used to obtain the two-dimensional size of the defect on the mobile phone display screen, specifically comprising: Performing thinning processing on the defect binary image; Count the number of skeleton pixels to get the pixel size of the defect; Determine the image resolution, and use the defect size calculation formula to calculate the actual size of the defect, that is, the two-dimensional size of the mobile phone display defect. The defect size calculation formula is: In the formula, is the pixel size, including width, height or diameter, is the image resolution, is the actual size of the defect.

Images