A LCD screen defect detection method based on multi-scale feature fusion

Abstract

The application discloses a LCD screen defect detection method based on multi-scale feature fusion, and the construction steps comprise the following steps: image acquisition, LCD screen images are collected by using an industrial camera; a data set is constructed, the collected images are preprocessed, defect areas in the images are labeled, and a training set and a test set are prepared; a backbone feature extraction network is constructed, a MobileNetV3 algorithm model is built and trained, preprocessed images are taken as inputs of the MobileNetV3 network, and four feature maps with different sizes are obtained; and defect detection is carried out by inputting the extracted feature maps into an improved PSPNet network. The application adopts a lightweight backbone network and carries out multi-scale fusion on feature information, so that the accuracy of defect detection is effectively improved, and the requirement of industrial application is met.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of LCD screen defect detection, and particularly relates to a method for detecting defects of an LCD screen based on a semantic segmentation model of a PSPNet (Pyramid Scene Parseing Network). BACKGROUND

[0002] With the rapid popularization of the Internet, more and more people use smart phones as the main communication tool. According to the data of the Ministry of Industry and Information Technology, as of the end of November 2023, the number of mobile phone users reached 1.726 billion, and the number of mobile phone Internet users reached 1.509 billion. The massive use of mobile phones has led to a rapid increase in demand for various components of mobile phones. The quality of the mobile phone screen, as one of the main components of the mobile phone, directly affects the user experience. The material of the mobile phone screen is mainly the liquid crystal display (LCD) panel. Compared with traditional screens, LCD screens have many advantages, such as small size, moderate thickness, low energy consumption, and high safety.

[0003] However, due to the large number of liquid crystal molecules that need to be injected into the LCD screen and the high requirements for the electric field control system, the production of LCD screens requires extremely complex and standardized processes. With the current level of technology, it is still impossible to avoid the occurrence of defects. Any mistake in the production process will indirectly affect the display function of the liquid crystal body, causing the LCD screen to be unable to truly and effectively display the received image, affecting the user's experience. Common defects of mobile phone screens include bright spots, dark spots, bright lines, dark lines, and yellow spots, etc. Therefore, we need to strictly, efficiently, and quickly detect the manufactured LCD screens.

[0004] Computer liquid crystal display screens have relatively obvious defects due to their large display area. The detection of defects and control circuit faults of specific types of computer screens is currently a mature technology, and there are national detection standards to follow. However, there is currently no perfect identification and detection method for the defects of small LCD screens such as mobile phone screens.

[0005] The machine vision detection method can greatly improve the accuracy of detection. Peng Ganquan uses the machine vision method to cut, dust, filter and denoise the image, then uses local threshold segmentation to extract the defect area, and finally screens out the defective products by combining local area contrast analysis; Peng Yu et al. propose a method for detecting the external contour size and surface defect condition of slender products by using machine vision technology, which uses the detection method of cann algorithm and simple blob detector feature points to extract the part contour and stain contour. Xiong Kangdi et al. propose a mobile phone graphite heat dissipation sheet defect detection method based on machine vision, which uses an adaptive local noise reduction homomorphic filter to remove image noise, and uses an affine transformation method to correct the image and other preprocessing operations. The defective image is extracted and displayed by using difference operation and binaryzation. In addition, there are many methods for defect detection using machine vision, but these methods have problems such as low automation degree, low detection precision, low efficiency, high environmental requirements and the like.

[0006] On the other hand, deep learning has become an important tool for defect detection and identification classification. The defect detection method based on deep learning convolutional neural network improves the detection precision and accuracy, and improves the work efficiency. Chen Renzong et al. propose a metal mechanical part surface defect detection method based on deep learning, which mainly uses deep learning algorithm to extract full connection layer features and train multiple cascade classifiers, and obtains the posterior probability of the target image by fitting the output results of the classifier. This method improves the detection efficiency, but the precision of subtle defect detection needs to be improved, and the detection range needs to be expanded. Ren Bingyin et al. propose a small target segmentation and micro defect detection method based on classification network + Attentio U-net for automatic detection of slight scratches on mobile phone screens. The detection precision of micro scratches is improved, but the learning model parameters are too many, the image annotation task is heavy, the global information acquisition is insufficient, and the detection target is single.

[0007] In general, the defect detection method based on deep learning can more effectively detect the results, and the detection efficiency is better and the accuracy is higher than that of the machine vision detection method, but there is no efficient and comprehensive defect detection method for mobile phone LCD screens at present. Based on the above background, an effective method for effectively detecting defects of mobile phone screens is urgently needed at present, so the present application arises at the historic moment. SUMMARY

[0008] In order to overcome the defects of insufficient precision, more model parameters and insufficient global information acquisition in the prior art, the application provides a mobile phone LCD screen defect detection method based on MobileNetV3 and PSPNet network, which can effectively reduce model parameters and shorten training time. At the same time, through the fusion of multi-scale feature information, the context information of different regions is aggregated, the defect detection accuracy is improved, and the automation level of LCD screen production is improved.

[0009] A LCD screen defect detection method based on multi-scale feature fusion, comprising the following steps:

[0010] S1: using an industrial camera to collect LCD screen images and classifying common defects.

[0011] S2: pre-processing the collected images, enhancing effective semantic information, and obtaining a relatively sufficient sample size. Labeling the defect area in the image, making a training set and a test set corresponding to the defect type.

[0012] S3: building and training a MobileNetV3 algorithm model, taking the pre-processed image as the input of the MobileNetV3 network, extracting defect features, and obtaining four feature maps of different sizes.

[0013] S4: inputting the feature map extracted in step S3 into the improved PSPNet network, fusing and splicing the semantic information of different scales, and obtaining the final prediction result through convolution operation to complete defect detection.

[0014] Further, in the step S1, an 8K line scanning lens of LS12056A model is used, and the focal length is 120mm. In order to meet the actual demand, it is necessary to expand the image field of view range to 70mm, so a length of 149mm extension ring is added based on the original lens. In this way, the collected image can be clearer and more accurate, and the requirement of focal length of the system can also be met. In order to better highlight the defect features and reduce the detection error, a white LED ring light source is used, and a low-angle lighting method is used to reduce the problem of screen surface reflection.

[0015] Then, according to the collected samples, the defects are divided into SQ, Mura, TP and Line four types of defects.

[0016] Further, in the step S2, since the industrial camera used in the manufacturing industry has high resolution, the direct input of the deep learning model requires a hardware device with excellent performance to process, and the high-resolution image will also reduce the accuracy of defect detection to a certain extent. Therefore, in the above scheme, the original image is cropped into an image with a resolution of 480*480 from left to right and from top to bottom, and the insufficient part is filled with a full black background.

[0017] In the above scheme, the image preprocessing includes a pixel transformation method and a geometric transformation method, wherein the pixel transformation method includes adjusting brightness, saturation, histogram equalization and Gaussian blur, and the geometric transformation method includes flipping and rotating. Through the above method, the data is enhanced to meet the data set size required for training the deep learning framework.

[0018] Further, in the step S3, the MobileNetV3 network is used for feature extraction. The MobileNetV3 network adopts an inverted residual structure and is composed of 15 bottleneck units. In each bottleneck unit, the dimension is first increased by 1×1 convolution, then the feature is extracted by depthwise separable convolution (DW), and finally the dimension is reduced by 1×1 convolution. Its structure is exactly the opposite of the residual structure, but in some bottleneck units with stride=1 and the same number of input image and output image channels, the "skip layer" connection of the residual is still retained. Among them, after the dimension increasing and convolution operation, the h-swish activation function is introduced, and its expression is:

[0019]

[0020] Wherein, ReLU6(x) = min(max(x, 0), 6).

[0021] In the 15 bottleneck units of the above MobileNetV3 network, the feature maps obtained after extracting three bottleneck units are retained, and the sizes are 120 2 ×24, 60 2 ×40, 30 2 ×112, and the final obtained 30 2 ×160 feature Figure One is input into the improved PSPNet network of step S4.

[0022] Further, in the step S4, the four feature maps obtained in the step S3 are input into the improved PSPNet network. In the improved PSPNet network, the 30 2 ×160, 30 2 ×112, 60 2X 40, 120 2 X 24 Four size feature maps are divided into 1x1, 2x2, 3x3, 6x6 sub-regions respectively, and average pooling operation is performed on each sub-region. Then through a 1x1 convolution, the channel number of the four feature maps is unified to 40. Then bilinear interpolation upsampling is used to 30x30 size, and is concatenated with the input 30 2 X 160 image to obtain a 30 2 X 320 feature map. The feature map is further filtered through the CBMA attention module to obtain the key semantic information, and finally a convolution layer is used to obtain the defect detection result of the input image.

[0023] The present application has the beneficial effects compared with the prior art:

[0024] The present application is a LCD screen defect detection method based on multi-scale feature fusion to solve the problems of insufficient precision, more model parameters and insufficient global information acquisition in the prior art. The improved PSPNet network is used for defect detection, and the traditional backbone feature extraction network ResNet is replaced by the lightweight network MobileNetV3, which greatly reduces the parameter amount of the model, makes the model more lightweight, and is more in line with the needs of industrial application. At the same time, the improved pyramid pooling module further fuses features of different scales, effectively aggregates context information in different regions. Through the addition of the attention mechanism module, the accuracy of defect detection is also effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 The method flowchart of the present application is shown in the figure;

[0026] Figure 2 The structure schematic diagram of image acquisition of the present application is shown in the figure;

[0027] Figure 3 The schematic diagram of four kinds of defects classified by the present application is shown in the figure;

[0028] Figure 4 The complete structure diagram of the MobileNetV3 network used by the present application is shown in the figure;

[0029] Figure 5 The bottleneck structure schematic diagram of the MobileNetV3 network in the present application is shown in the figure;

[0030] Figure 6 The overall network structure diagram of the present application is shown in the figure;

[0031] Figure 7 The CBMA module schematic diagram used by the present application is shown in the figure; DETAILED DESCRIPTION

[0032] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. In order to better illustrate the embodiments, some components in the drawings can be omitted, enlarged or reduced, and do not represent the actual size of the product; it can be understood by those skilled in the art that some well-known structures and their descriptions can be omitted in the drawings.

[0033] The present application provides a LCD screen defect detection method based on multi-scale feature fusion, and the method comprises the steps of Figure 1 The method flowchart of the present application case, the method is realized by the following steps:

[0034] S1: using an industrial camera to collect LCD screen images, and classifying common defects.

[0035] S2: pre-processing the collected images, enhancing effective semantic information, and obtaining a relatively sufficient sample size. Labeling the defect area in the image, and making a training set and a test set corresponding to the defect type.

[0036] S3: building and training a MobileNetV3 algorithm model, taking the pre-processed images as the input of the MobileNetV3 network, extracting defect features, and obtaining four feature maps of different sizes.

[0037] S4: inputting the feature maps extracted in step S3 into an improved PSPNet network, fusing and splicing the semantic information of different scales, and obtaining the final prediction result through convolution operation to complete defect detection.

[0038] In the specific implementation process, the present example adopts an 8K line scanning lens of LS12056A model, and the focal length is 120mm. In order to meet the actual demand, it is necessary to expand the image field of view range to 70mm, so a length of 149mm extension ring is added based on the original lens. In this way, the collected images can be clearer and more accurate, and the requirements of the system on focal length can also be met. In order to better highlight the defect features and reduce the detection error, a white LED ring light source is adopted, and a low-angle lighting method is adopted to reduce the reflection problem of the screen surface. The image acquisition structure is as Figure 2 shown.

[0039] After that, according to the collected samples, the defects are divided into SQ, Mura, TP and Line four types of defects. SQ defect refers to circular, elliptical or shape close to circular defects; Mura defect refers to irregular spot defects; TP defect refers to square defects, the color is white or black; Line defect refers to linear defects, generally through the entire screen, there are horizontal and vertical, and a small number of discontinuous line defects. The specific defect is shown in Figure 3 .

[0040] More specifically, in step S2, since the resolution of the industrial camera used in the manufacturing industry is high, direct input into the deep learning model requires a hardware device with excellent performance to process, and high-resolution images will also reduce the accuracy of defect detection to some extent. Therefore, in the specific implementation process, the original image will be cut into a 480*480 resolution image in the order of left to right and top to bottom, and the insufficient part will be filled with a full black background.

[0041] In the image preprocessing stage, this example comprehensively uses the preprocessing methods of pixel transformation and geometric transformation. In terms of pixel transformation, the image quality and recognizability are effectively improved by adjusting the brightness and saturation of the image, implementing histogram equalization and Gaussian blur. In terms of geometric transformation, flip and rotation operations are used to increase the diversity and generalization ability of the data. The preprocessed data set is divided into training data set and test data set in the ratio of 7:3; the training set will be used to train the defect detection network model, and the test set will be used to verify the effectiveness of the model.

[0042] More specifically, in step S3, MobileNetV3 network is a lightweight network model for image classification, which is used as backbone in this application for feature extraction of images, so in the specific implementation process, the network structure of MobileNetV3 model is adjusted, and the specific network structure is shown in Figure 4 . The Operator column refers to the specific operation performed on the input image, and Bneck corresponds to the Bottleneck unit shown in Figure 5 . The Exp size column refers to the number of channels of the input image after the dimension increasing operation in the bottleneck unit, and the #out column refers to the number of channels of the image output by the unit after the dimension reduction operation. The SE column refers to whether to use the SE channel attention mechanism, and "√" indicates that the SE attention mechanism is added in the corresponding bottleneck unit, and "-" indicates that the attention mechanism is not used. The NL column refers to the activation function, and HS represents the h-swish activation function, whose expression is:

[0043]

[0044] where ReLU6(x) = min(max(x, 0), 6). RE represents the RuLU activation function, and its expression is ReLU(x) = max(x, 0). The s column refers to the convolution stride, and the d column corresponds to the dilated convolution coefficient. When the coefficient is 1, it is a normal convolution.

[0045] More specifically, the above MobileNetV3 network adopts an inverted residual structure and is composed of 15 bottleneck units. The structure diagram of each bottleneck unit is as shown in Figure 5 In each bottleneck unit, dimensionality is first increased by 1x1 convolution, then feature extraction is performed by depthwise separable convolution (DW), and finally dimensionality is reduced by 1x1 convolution. The structure is just the opposite of the residual structure, but in some bottlenecks with stride = 1 and the same number of input image and output image channels, the "skip layer" connection of the residual is still retained. And an activation function is added after the dimensionality increasing and convolution operation, and the specific function used is as shown in Figure 4 .

[0046] Among the 15 bottleneck units of the above MobileNetV3 network, the feature maps output by three bottleneck units are extracted for reservation, and the sizes are 120 2 ×24, 60 2 ×40, 30 2 ×112, and the finally obtained 30 2 ×160 feature Figure One is input into the improved PSPNet network.

[0047] More specifically, in the step S4, the four feature maps obtained in the step S3 are input into the improved PSPNet network. In the improved PSPNet network, the four feature maps with sizes of 30 2 ×160, 30 2 ×112, 60 2 ×40, and 120 2 ×24 are respectively divided into 1x1, 2x2, 3x3, and 6x6 sub-regions, and average pooling operation is performed on each sub-region. Then, through a 1x1 convolution, the channel numbers of the four feature maps are unified to 40. Then, bilinear interpolation upsampling is used to the size of 30x30, and concat is performed with the input 30 2 ×160 image to obtain a 30 2The feature map is input into a CBMA attention module to further filter out key semantic information, and finally a convolutional layer is used to obtain the defect detection result of the input image. Figure 6 The complete network structure is shown in

[0048] More specifically, the CBMA attention module is a lightweight attention module. Figure 7 The CBMA attention module is a lightweight attention module.

[0049] The embodiments of the present application are preferred embodiments, but are not limited thereto, and those skilled in the art can easily understand the spirit of the present application according to the above embodiments, and make different inferences and changes, as long as they do not deviate from the spirit of the present application, and are within the protection scope of the present application.

Claims

1.A method for LCD screen defect detection based on multi-scale feature fusion, characterized in that Comprising the following steps: S1: using an industrial camera to collect LCD screen images and classify common defects; S2: pre-processing the collected images, enhancing effective semantic information, and obtaining a relatively sufficient sample size; labeling the defect area in the image, and making a training set and a test set corresponding to the defect type; S3: build and train the MobileNetV3 algorithm model, and take the preprocessed image as the input of the MobileNetV3 network to perform feature extraction; in the 15 bottleneck units of the MobileNetV3 network, the feature maps obtained after extracting three different bottleneck units are reserved, and the sizes are 120 2 × 24, 60 2 × 40, 30 2 × 112, and the feature maps of 30 2 × 160 output at last are input into the improved PSPNet network of step S4; S4: input the feature map obtained in step S3 into the improved PSPNet network; in the improved PSPNet network, first divide the feature maps of four sizes, 302*160, 302*112, 602*40, and 1202*24, into 1*1, 2*2, 3*3, and 6*6 sub-regions, respectively, and perform average pooling operation on each sub-region; then, through a 1*1 convolution, the channel numbers of the four feature maps are unified to 40; then, using bilinear interpolation up-sampling to 30*30, and performing concat with the input 302*160 image, a 302*320 feature map is obtained; the feature map is further filtered through the CBMA attention module to further filter out key semantic information, and finally a convolution layer is used to obtain the defect detection result of the input image. 2.The LCD screen defect detection method based on multi-scale feature fusion according to claim 1, characterized in that The industrial camera used in step S1 is an 8K line scan lens of LS12056A model with a focal length of 120mm; in order to meet the actual needs, the image field of view needs to be expanded to 70mm, so a length of 149mm is added to the original lens; white LED ring light source and low-angle lighting method are used for image acquisition; then according to the collected samples, the defects are divided into SQ, Mura, TP, and Line four types of defects. 3.The LCD screen defect detection method based on multi-scale feature fusion according to claim 1, characterized in that In step S2, the specific steps of the image pre-processing are: First, the original image is evenly cropped into a 480*480 resolution image from left to right and top to bottom, and the insufficient part is filled with a black background; then pixel transformation method and geometric transformation method are used for further processing; The pixel transformation method includes adjusting brightness, saturation, histogram equalization and Gaussian blur, and the geometric transformation method includes flipping and rotating; through the above methods, the data is enhanced, and the pre-processed data set is divided into training data set and test data set in the ratio of 7:3; wherein the training set is used to train the defect detection network model, and the test set is used to verify the effectiveness of the model. 4.The LCD screen defect detection method based on multi-scale feature fusion according to claim 1, characterized in that In step S3, the specific steps of the model built are: The MobileNetV3 network adopts an inverted residual structure and is composed of 15 bottleneck units; in each bottleneck unit, First, dimensionality is increased by 1x1 convolution, then feature extraction is performed by depth separable convolution, and finally dimensionality is reduced by 1x1 convolution, which is exactly opposite to the residual structure, but in the bottleneck where part stride=1 and the channel number of the input image and the output image is the same, the "skip layer" connection of the residual is still retained; wherein, after the dimensionality increasing and convolution operation, the h-swish activation function is introduced, and the expression is: ； wherein, ReLU6(x) = min(max(x, 0), 6).

Images