Liquid crystal display screen defect detection method, system, device and medium
Through the improved YOLOv8 model, combined with the lightweight backbone network and the adaptive selection space attention module, the problem of difficulty in both accuracy and real-time detection of LCD display defects is solved, and efficient and accurate LCD screen defect detection is achieved. In mid-to-low-end hardware.
Patent Information
- Application Number
- CN202411050976.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-01
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-08-01
AI Technical Summary
The existing LCD screen defect detection methods cannot meet the requirements of accuracy and real-time at the same time. Larger models have slow inference speed, and the defect targets have fine-grained feature information are lost, and it is difficult to accurately identify small targets in complex backgrounds.
The improved YOLOv8 model is adopted, including a lightweight backbone network that enhances feature extraction, a fine-grained channel fusion block and an adaptive selection spatial attention module. The key features are extracted through the C2f_FELB and FCFB modules, combined with a small object detection head, and dynamically adjust the receptive field. The bounding box regresses the loss function using the minimum point distance.
It realizes accurate detection of LCD screen defects under low latency and high real-time performance, improves the recognition accuracy of small-size defect targets, reduces system costs, and is suitable for mid- and low-end hardware to meet practical application needs.
Smart Images

Figure CN118570207B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of display screen defect detection, and in particular to a liquid crystal display screen defect detection method, system, equipment and medium. Background Art
[0002] With the continuous advancement of science and technology and the rapid development of display technology, liquid crystal displays (LCDs) have been widely used in various electronic devices, including smartphones, tablets, computer monitors, and TVs. With the continuous advancement of LCD manufacturing technology, its quality requirements are getting higher and higher. However, due to various reasons in the production process, LCD screens will produce different types of defects, such as bright spots, dark spots, line defects, color spots, etc. These defects not only affect the appearance and user experience of the product, but may also lead to more serious quality problems. Therefore, LCD screen defect detection technology has become an important means to ensure the quality of LCD screen products.
[0003] Traditional LCD screen defect detection mainly relies on manual visual inspection. The inspector observes the surface of the LCD screen with the naked eye to determine whether there are defects such as scratches, bubbles, foreign matter, bright spots, dark spots, etc. Although this method is intuitive, it has disadvantages such as low efficiency, unstable test results, and easy to be affected by subjective factors. At the same time, with the continuous improvement of the production scale and speed of LCD screens, manual inspection has been difficult to meet the needs of efficient and high-precision inspection. In order to solve the above problems, automated inspection technology has gradually been introduced into the field of LCD screen defect detection. Automated inspection systems are usually composed of high-speed cameras, light sources, image processing software, etc., which capture images of the LCD screen surface and use image processing algorithms to identify and classify defects. Automated inspection technology improves inspection efficiency and accuracy, but still faces some challenges, especially the balance between inspection accuracy and inspection speed.
[0004] At present, the mainstream methods of LCD screen defect detection mainly include methods based on traditional image processing algorithms and methods based on deep learning algorithms. Among them, methods based on traditional image processing algorithms: Traditional image processing algorithms usually rely on edge detection, template matching, morphological processing and other technologies. These methods work well when processing regular shapes and obvious defects, but the detection effect is poor for complex backgrounds, tiny defects and irregular defects. In addition, traditional image processing algorithms are sensitive to ambient lighting and shooting angles, and it is difficult to maintain stable detection performance in complex production environments. Methods based on deep learning algorithms: With the development of deep learning technology, defect detection methods based on convolutional neural networks (CNNs) have gradually become a research hotspot. Deep learning algorithms can automatically learn image features and have strong generalization capabilities. In the development history of defect target detection, one-stage and two-stage defect target detection algorithms represent two different ideas. The one-stage detection algorithm directly regresses the position and category of the defect target through a convolutional neural network, such as SSD, RetinaNet and YOLO series, which has good real-time performance and is suitable for scenes with high real-time requirements such as LCD screen defect detection, but the detection accuracy is relatively low and needs further improvement and optimization. The two-stage detection algorithm divides the defect target detection task into two stages: first extracting candidate areas, and then classifying and locating these areas. For example, R-CNN, Faster R-CNN, etc. have high detection accuracy, but the reasoning speed is relatively slow, which is not suitable for the real-time detection of LCD screen defects.
[0005] Although the existing defect target detection algorithms have improved the accuracy of LCD defect detection to a certain extent, there are still some key problems:
[0006] (1) Although a larger model improves accuracy, the model has too many parameters, computational complexity, and memory access requirements, which results in a slow inference speed of the LCD defect detection algorithm and makes it difficult to meet the application requirements of real-time detection.
[0007] (2) The defects on the surface of LCD screens are usually small in size. The existing network uses strided convolutional layers or pooling layers for downsampling multiple times during the forward propagation process, which can easily cause the loss of fine-grained feature information of the defect target. This is particularly serious when processing low-resolution images or when the defect target is small in size. The detection accuracy needs to be improved.
[0008] (3) The small defect target itself lacks feature information and is difficult to distinguish from the complex background. The receptive field scale of the existing defect detection network is limited and fixed, and it fails to fully utilize the contextual information around the defect target, which easily leads to missed detection and false detection of LCD screen defects.
[0009] In summary, existing LCD defect detection methods cannot meet the requirements of accuracy and real-time performance at the same time. Summary of the invention
[0010] Therefore, the technical problem to be solved by the present invention is to overcome the problem that the accuracy and real-time performance of the liquid crystal display screen defect detection method in the prior art cannot be satisfied at the same time.
[0011] In order to solve the above technical problems, the present invention provides a method for detecting defects in a liquid crystal display screen, comprising:
[0012] Step S1: Acquire an image of the LCD screen;
[0013] Step S2: performing defect detection on the image of the LCD screen through a preset neural network, wherein:
[0014] The preset neural network is improved based on the YOLOv8 model, and includes a backbone network, a neck network and a head network. The backbone network is used to extract image features, the neck network is used to perform feature fusion on the image features extracted by the backbone network, and the head network is used to perform defect prediction on the features after fusion of the neck network;
[0015] The backbone network and the neck network of the preset neural network in step S2 are both provided with a C2f_FELB module for extracting key feature information, wherein:
[0016] The C2f_FELB module first adjusts the number of channels of the feature map through the first 1×1 convolution. The output of the first 1×1 convolution is divided into the first two parts, one part is used for residual connection, and the other part is divided into the second two parts through the spilt operation. One part is used for residual connection and the other part enters the main branch; the main branch extracts features through n series-connected feature extraction bottlenecks FEBs and maintains residual connection; then, the output of the part used for residual connection obtained by the first 1×1 convolution, the output of the part used for residual connection obtained by the spilt operation, and the output of the n series-connected feature extraction bottlenecks FEBs maintaining residual connection are all spliced through the concat operation to obtain the spliced feature map; finally, the spliced feature map is processed through the second 1×1 convolution to obtain the final output of the C2f_FELB module.
[0017] In one embodiment of the present invention, the feature extraction bottleneck FEB includes a first partial convolution PConv, a first point-by-point convolution PWConv, a second partial convolution PConv and a second point-by-point convolution PWConv connected in sequence, and the input of the first partial convolution PConv and the output of the second point-by-point convolution PWConv are element-wise added as the output of the feature extraction bottleneck FEB, wherein a normalization layer BN and a SiLU activation function are further arranged between the second partial convolution PConv and the second point-by-point convolution PWConv.
[0018] In one embodiment of the present invention, the backbone network of the neural network preset in step S2 is further provided with an FCFB module for downsampling, wherein:
[0019] The FCFB module includes: for a given size Feature map , H is the feature map height, W is the feature map width, C is the number of feature map channels, and the feature map is scaled by a factor of 2 Perform interleaved downsampling to obtain four sub-feature maps , the size of each sub-feature map is , expressed as:
[0020] ;
[0021] in, Represents the feature map The sub-feature map obtained after downsampling by a scale factor of 2;
[0022] The four sub-feature maps obtained Connect along the channel dimension to obtain the intermediate feature map , The size is , whose channel dimension is expanded to feature map 4 times, the spatial dimension is reduced to the feature map of ;
[0023] Then use a non-strided convolution with a convolution kernel size of 3×3 to reduce the number of channels of the feature map and convert the intermediate feature map Convert to size The resulting feature map , the resulting feature map It is the final output of the FCFB module.
[0024] In one embodiment of the present invention, the neck network of the preset neural network in step S2 is provided with an ASSA module, and the ASSA module comprises:
[0025] The 23×23 convolution is decomposed into two convolutions with different scales of receptive fields in series, and then two feature maps with different receptive fields are generated. The convolution kernel sizes of the two convolutions after decomposition are , Expansion Rate And the theoretical receptive field is defined as:
[0026] ;
[0027] Use two decomposed convolutions with different scale receptive fields to extract feature information, expressed as:
[0028] ;
[0029] in, is the input feature map of the ASSA module, represents the first and second convolutions, is the feature map obtained by the corresponding convolution; then the spatial adaptive selection mechanism is used to realize the dynamic selection of the kernel, including: first Splice in the channel dimension to form a feature set :
[0030] ;
[0031] Again Extracted using the maximum pooling operation based on the channel dimension The spatial feature relationship of :
[0032] ;
[0033] in, Represents the maximum pooling operation based on the channel dimension; then the convolution layer is used for nonlinear transformation and feature integration to convert the maximum pooling spatial feature map Converted to two spatial attention maps :
[0034] ;
[0035] For the two obtained spatial attention maps , using the Sigmoid activation function to generate a spatial selection mask :
[0036] ;
[0037] in, is the Sigmoid activation function;
[0038] Will The corresponding Perform weighted operation and then pass through the convolution layer The weighted features are fused to obtain the attention feature map :
[0039] ;
[0040] Last passed and The output of the ASSA module is calculated by element-by-element product between :
[0041] .
[0042] In one embodiment of the present invention, the head network of the neural network preset in step S2 includes four Dectect layers, namely, a first Dectect layer, a second Dectect layer, a third Dectect layer, and a fourth Dectect layer, wherein the first Dectect layer, the second Dectect layer, the third Dectect layer, and the fourth Dectect layer are respectively connected to different ASSA modules in the neck network.
[0043] In one embodiment of the present invention, assuming that the input image size of the preset neural network is 640×640×3, the output size of the first Dectect layer is 160×160×64, the output size of the second Dectect layer is 80×80×128, the output size of the third Dectect layer is 40×40×256, and the output size of the fourth Dectect layer is 20×20×512.
[0044] In one embodiment of the present invention, before obtaining the preset neural network in step S2, the preset neural network is also trained, and during the training, a bounding box regression loss function based on the minimum point distance is used to determine whether the preset neural network converges, wherein:
[0045] The method for calculating the bounding box regression loss function based on the minimum point distance includes:
[0046] ;
[0047] in, Indicates the diagonal length of the minimum enclosing rectangle of the real box and the predicted box, , Respectively represent the center point coordinates of the predicted box and the real box, is the width and height of the prediction box, is the width and height of the real frame;
[0048] Assume that the height and width of the input image are , the real box and the predicted box are represented by the coordinates of the upper left corner and the lower right corner respectively. , , then the bounding box regression loss function based on the minimum point distance is expressed as:
[0049] ;
[0050] in, is the Euclidean distance between the upper left point of the predicted box and the true box, is the Euclidean distance between the lower right point of the predicted box and the true box, is a measure of the similarity between the predicted bounding box and the true bounding box. is the bounding box regression loss function based on the minimum point distance, To take the union, To take the intersection.
[0051] In order to solve the above technical problems, the present invention provides a liquid crystal display screen defect detection system, comprising:
[0052] Acquisition module: used to acquire images about the LCD screen;
[0053] Detection module: used to perform defect detection on the image of the LCD screen through a preset neural network, wherein:
[0054] The preset neural network is improved based on the YOLOv8 model, and includes a backbone network, a neck network and a head network. The backbone network is used to extract image features, the neck network is used to perform feature fusion on the image features extracted by the backbone network, and the head network is used to perform defect prediction on the features after fusion of the neck network;
[0055] The backbone network and the neck network of the preset neural network in the detection module are both provided with a C2f_FELB module for extracting key feature information, wherein:
[0056] The C2f_FELB module first adjusts the number of channels of the feature map through the first 1×1 convolution. The output of the first 1×1 convolution is divided into the first two parts, one part is used for residual connection, and the other part is divided into the second two parts through the spilt operation. One part is used for residual connection and the other part enters the main branch; the main branch extracts features through n series-connected feature extraction bottlenecks FEBs and maintains residual connection; then, the output of the part used for residual connection obtained by the first 1×1 convolution, the output of the part used for residual connection obtained by the spilt operation, and the output of the n series-connected feature extraction bottlenecks FEBs maintaining residual connection are all spliced through the concat operation to obtain the spliced feature map; finally, the spliced feature map is processed through the second 1×1 convolution to obtain the final output of the C2f_FELB module.
[0057] To solve the above technical problems, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned liquid crystal display defect detection method when executing the computer program.
[0058] In order to solve the above technical problems, the present invention provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned liquid crystal display defect detection method are implemented.
[0059] The above technical solution of the present invention has the following advantages compared with the prior art:
[0060] The present invention constructs a lightweight backbone network for enhanced feature extraction, uses the feature extraction lightweight block C2f_FELB to efficiently extract the key feature information of the feature map, and designs the fine-grained channel fusion block FCFB as a downsampling module for the improved backbone network, thereby avoiding the loss of fine-grained feature information of the defect target; the backbone network fully considers the limitations of the hardware device on the model scale, significantly reduces the number of model parameters, calculations and memory access, and improves the network's feature extraction capabilities, thereby achieving real-time and accurate detection of LCD screen defects, meeting the low-latency and high-real-time defect detection requirements in practical applications. The improved backbone network can still maintain high detection accuracy when processing low-resolution images or small defect targets, and effectively identify smaller LCD defect targets;
[0061] The present invention constructs a small target detection head SODH, integrates the shallow feature map of the main network into the feature fusion process of the neck network, fully extracts the position information and detail features of the defective small target, effectively compensates for the loss of feature information, improves the detection accuracy of small-size defect targets on the LCD screen, and can more accurately identify and locate tiny defects;
[0062] The computational efficiency of the EA-YOLO model constructed by the present invention is greatly improved, so that high-precision defect detection can be achieved on mid- and low-end hardware. This design reduces the overall cost of the system, is easy to promote and apply on a large scale, promotes the practical application of LCD screen defect detection technology, and has broad practical application value. BRIEF DESCRIPTION OF THE DRAWINGS
[0063] In order to make the contents of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments of the present invention in conjunction with the accompanying drawings.
[0064] Figure 1 is a flow chart of the method of the present invention;
[0065] Figure 2 1 is a schematic diagram of the overall structure of the EA-YOLO model in an embodiment of the present invention;
[0066] Figure 3 Schematic diagram of a partial convolution structure in an embodiment of the present invention;
[0067] Figure 4is a schematic diagram of the feature extraction bottleneck FEB structure in an embodiment of the present invention;
[0068] Figure 5 2 is a schematic diagram of the structure of a feature extraction lightweight block C2f_FELB in an embodiment of the present invention;
[0069] Figure 6 is a schematic diagram of the structure of a fine-grained channel fusion block FCFB in an embodiment of the present invention;
[0070] Figure 7 is a schematic diagram of the structure of the adaptive selection spatial attention module ASSA in an embodiment of the present invention;
[0071] Figure 8 is a schematic diagram of the structure of the Dectect layer in an embodiment of the present invention;
[0072] Fig. 9 Schematic diagram of the upper left point and lower right point of the predicted box or the real box in an embodiment of the present invention. DETAILED DESCRIPTION
[0073] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention. Embodiment 1
[0074] Reference Figure 1 The present invention relates to a method for detecting defects in a liquid crystal display screen, comprising:
[0075] Step S1: Acquire an image of the LCD screen;
[0076] Step S2: performing defect detection on the image of the LCD screen through a preset neural network, wherein:
[0077] The preset neural network is improved based on the YOLOv8 model, and includes a backbone network, a neck network and a head network. The backbone network is used to extract image features, the neck network is used to perform feature fusion on the image features extracted by the backbone network, and the head network is used to perform defect prediction on the features fused by the neck network.
[0078] The following is a detailed introduction to this embodiment:
[0079] The LCD defect detection method proposed in the present invention adopts a preset neural network, which is referred to as the EA-YOLO network. The EA-YOLO network is improved on the YOLOv8 benchmark network. The YOLOv8 network includes three components: the backbone network uses the CSPDarkNet-53 structure, the neck network uses the PANet structure, and the head network uses the Decoupled Head decoupled detection head structure. YOLOv8 can output three different scales of predicted feature maps. This multi-scale design achieves better defect detection performance. The overall structure of the EA-YOLO network is shown in the figure below. Figure 2 As shown in the figure. Firstly, a lightweight backbone network with enhanced feature extraction is constructed, and the feature extraction lightweight block C2f_FELB is used to efficiently extract key feature information. A fine-grained channel fusion block FCFB is designed as the downsampling module of the backbone network to effectively avoid the loss of fine-grained feature information of defective targets. The improved backbone network can significantly improve the feature extraction ability and reasoning efficiency of the model. Secondly, a small target detection head SODH is constructed, and the shallow feature map P2 in the improved backbone network is integrated into the feature fusion process in the neck network to fully extract the location information and detail features of small defective targets and make up for the loss of feature information. Then, the proposed adaptive selection spatial attention module ASSA is used to dynamically adjust the different scales of the spatial receptive field to more effectively capture and process the extensive contextual information around different types of defective targets. Finally, the bounding box regression loss function CIoU used in the original network is replaced by MPDIoU, which fully considers the actual deviations of the overlapping or non-overlapping area, center point distance, width and height between the predicted box and the true box, and minimizes the distance between the upper left point and the lower right point between the predicted box and the true box to improve the efficiency and accuracy of bounding box regression in complex background LCD screen defect detection. The following will be divided into 4 parts to describe in detail the implementation details and functional advantages of each module.
[0080] 1. Lightweight backbone network for enhanced feature extraction
[0081] In order to reduce the redundant calculation amount, parameter amount and memory access number of the model, and at the same time improve the feature extraction capability of the network to build a high-performance and lightweight LCD screen defect detection model, the present invention constructs a lightweight backbone network with enhanced feature extraction, uses the feature extraction lightweight block C2f_FELB to quickly extract feature information, and constructs a fine-grained channel fusion block FCFB to effectively avoid the loss of fine-grained feature information, thereby being efficiently suitable for detecting low-resolution images and LCD screen defect detection tasks.
[0082] Specifically, the lightweight operator partial convolution PConv is introduced to make full use of the high similarity of features between different channels of the feature map. Only on the first or last continuous quarter of the input channels of the feature map, conventional convolution Conv is used to extract features, and the remaining input channels of the feature map remain unchanged. The structure of partial convolution PConv is as follows: Figure 3 shown.
[0083] Assume that the size of the input feature map is , the convolution kernel size is , the input and output feature maps have the same number of channels, then the parameters Params, calculation FLOPs and memory access number MAC of the partial convolution PConv are:
[0084] ;
[0085] ;
[0086] ;
[0087] For regular convolution, use The size is The convolution kernel is used for feature extraction. The size of the output feature map obtained by default with padding operation is , then the parameters Params, calculation FLOPs and memory access MAC of the conventional convolution Conv are: ;
[0088] Therefore, the number of parameters, computational complexity, and memory access of the partial convolution PConv are respectively , , , which effectively utilizes the local correlation between channels, reduces the amount of calculation while extracting sufficient feature information, achieves higher computational efficiency, and is conducive to building a lightweight, high-throughput LCD screen defect target detection network. Point-by-point convolution PWConv is introduced after the partial convolution PConv to fully mine the information of all input channels of the feature map, promote information interaction between channels, and enhance the feature expression ability of the network.
[0089] Based on partial convolution PConv and point-by-point convolution PWConv, a feature extraction bottleneck FEB is constructed, and its structure is as follows: Figure 4As shown. The feature extraction bottleneck FEB includes the first partial convolution PConv, the first point-by-point convolution PWConv, the second partial convolution PConv and the second point-by-point convolution PWConv connected in sequence. The two partial convolution PConv and the two point-by-point convolution PWConv are combined into a reverse residual block. The middle layer of this module has an extended number of channels, which is conducive to the effective flow of gradients. At the same time, the input features are reused through shortcut connections. The normalization layer BN and SiLU activation function are inserted after the second partial convolution PConv in the feature extraction bottleneck FEB to fully retain the diversity of features and achieve faster inference speed. The feature extraction bottleneck FEB makes full use of the combination of partial convolution PConv and point-by-point convolution PWConv, so that the network can efficiently extract the key feature information of the feature map and improve the efficiency of inference calculation.
[0090] The feature extraction bottleneck FEB is used to replace the Bottleneck structure of the C2f module in the overall network to construct a feature extraction lightweight block C2f_FELB (referred to as C2f_FELB module). The specific structure of this module is as follows: Figure 5 First, the number of channels of the feature map is adjusted through the first 1×1 convolution. The output of the first 1×1 convolution is divided into two parts: one part (the output size is H×W×0.5C OUT , H is the feature map height, W is the feature map width, C OUT is the number of feature map channels) is used for residual connection, and the other part (output size is H×W×C OUT ) After the spilt operation, the feature map is divided into the second two parts, one part (the output size is H×W×0.5C OUT ) is used for residual connection, and the other part (output size is H×W×0.5C OUT ) enters the main branch. The main branch extracts features through n series-connected feature extraction bottlenecks FEB (i.e., FEBottleneck) and maintains residual connections, thereby reducing the amount of parameters and calculations while obtaining a richer gradient flow. Subsequently, a part of the first 1×1 convolution is used for the output of the residual connection, a part of the output obtained by the spilt operation is used for the output of the residual connection, and the output of the n series-connected feature extraction bottlenecks FEB with residual connections are all spliced through the concat operation to integrate richer gradient information and further enhance the feature expression capability. Finally, the spliced feature map is processed through the second 1×1 convolution to obtain the final output of the feature extraction lightweight block C2f_FELB (size H×W×C OUT ). The feature extraction lightweight block C2f_FELB significantly reduces the amount of parameters, calculations, and memory access while ensuring detection accuracy, which is conducive to building a lightweight, high-performance LCD defect detection network.
[0091] The size of the defect target of the LCD screen is small, and the original feature extraction network uses the strided convolution layer for downsampling operations, which easily causes the loss of fine-grained feature information of the defect target, thereby reducing the performance of the LCD screen defect detection algorithm, which is particularly obvious when the input image resolution is low and the defect target size is small. To address this problem, this embodiment improves a fine-grained channel fusion block FCFB (FCFB module for short) as each downsampling layer in the improved backbone network, effectively improving the accuracy of defect target detection without increasing the model parameters and calculation amount too much. The structure of this module is as follows: Figure 6 shown.
[0092] The fine-grained channel fusion block FCFB first performs the Perform spatial cutting and deep connection to retain and fuse the fine-grained feature information of the defect target and avoid the loss of detail feature information. Specifically, for a given size Input feature map , H is the feature map height, W is the feature map width, C is the number of feature map channels, and the scale factor (Stride) is 2 pairs Perform interleaved downsampling to obtain four sub-feature maps , the size of each sub-feature map is , specifically expressed as follows:
[0093] ;
[0094] in, Represents the feature map The sub-feature maps obtained by downsampling by a scale factor of 2 are connected along the channel dimension to obtain the intermediate feature map. , The size is , and its channel dimension is expanded to 4 times, the spatial dimension is reduced to of Then, a non-strided convolution with a kernel size of 3×3 is used to reduce the number of channels of the feature map. Convert to size The resulting feature map The purpose of using non-strided convolution is to further achieve the effective retention of all key feature information in the feature map. The fine-grained channel fusion block FCFB, as a downsampling module in the improved backbone network, effectively solves the problem of loss of fine-grained feature information of defective targets. It is highly suitable for low-resolution images and small defective target detection scenarios, and significantly improves the accuracy of LCD defect detection.
[0095] The lightweight backbone network for enhanced feature extraction consists of four stages. The fine-grained channel fusion block FCFB in each stage is used as a downsampling layer for spatial downsampling and expanding the number of channels of the feature map. The function of the feature extraction lightweight block C2f_FELB in each stage is to efficiently extract feature information. The backbone network can simultaneously reduce the amount of parameters, calculations and memory access, and enhance the feature extraction capability of the network, thus achieving a better balance between detection accuracy and inference speed.
[0096] 2. Build a small target detection head
[0097] LCD screen defect targets have a low pixel area and limited feature representation. The original network is difficult to fully learn their location information and detail features, and the feature information of small defect targets is easily lost. To address this problem, based on the three detection heads of the original network, a detection head SODH specifically for small defect targets is constructed, that is, the shallow feature map P2 in the lightweight backbone network with enhanced feature extraction is used, and P2 is integrated into the feature fusion process in the neck network. Through shallow feature fusion, a predicted feature map with a higher spatial resolution of 160×160 is finally generated, which improves the network's perception of small LCD screen defect targets.
[0098] By constructing a small target detection head, more location information about defective small targets can be fully retained, allowing the network to better extract detail information and shape features of small targets, thereby effectively reducing the probability of missed detection and false detection of defective small targets, significantly improving the detection performance of the model, and improving the network's robustness and accuracy for LCD screen defect detection.
[0099] 3. Adaptive Selection of Spatial Attention Module
[0100] Defect targets are small in size, and it is difficult to achieve accurate detection based only on appearance feature information. The contextual information around the defect targets can provide other important features, so making full use of the extensive contextual information around the defect targets is the key to improving detection performance. At the same time, different types of defect targets require contextual information of different ranges, so it is necessary to use a receptive field of reasonable scale to achieve accurate detection. The receptive field of the original network is insufficient and single, and only considers the limited contextual information around the defect targets, which can easily lead to incorrect defect classification and detection positioning.
[0101] In order to make full use of the extensive contextual information around the defect target to improve the performance of LCD defect detection, an adaptive selection spatial attention module ASSA (ASSA module for short) is improved. By using large kernel convolution and spatial adaptive selection mechanism, the different scales of the spatial receptive field are dynamically adjusted to more effectively capture and process the contextual information around the defect target, thereby improving the performance of the defect detection algorithm. The adaptive selection spatial attention module ASSA consists of two decomposed large kernel convolutions with different scales of receptive fields and a spatial adaptive selection mechanism. The specific structure is as follows Figure 7 shown.
[0102] First, using the explicit decomposition strategy, the constructed large kernel convolution (23×23 convolution) is decomposed into two series of regular convolutions (Large Conv1 and Large Conv2) with different scales of receptive fields, thereby generating two feature maps with reasonable receptive fields. Specifically, the convolution kernel size of the two regular convolutions after decomposition is , (in practice is 5×5, =7×7), expansion rate And the theoretical receptive field The definition of is:
[0103] ;
[0104] formula The theoretical receptive field is ensured to be expanded rapidly by increasing the convolution kernel size and the dilation rate. At the same time, in order to avoid the dilated convolution introducing gaps between feature maps, the upper bound of the dilation rate is set as the theoretical receptive field of the first convolution after decomposition. Then, two conventional convolutions with different scales of receptive fields after decomposition are used to fully extract feature information, which can be expressed as:
[0105] ;
[0106] in, is the input feature map of the ASSA module, Represents the first convolution (LargeConv1), the second convolution (Large Conv2), are the feature maps obtained by the first and second convolutions. Then, the spatial adaptive selection mechanism is used to realize the dynamic selection of kernels to better adapt to the scales of different targets and the required context information, enhance the network's perception of different types of small targets, and improve the accuracy of small target detection. Specifically, Splice in the channel dimension to form a feature set :
[0107] ;
[0108] Then Use the channel-based maximum pooling operation to fully extract The spatial feature relationship of :
[0109] ;
[0110] in, Represents the channel-based maximum pooling operation. Then the convolution layer is used for nonlinear transformation and feature integration to convert the pooled features into two spatial attention maps :
[0111] ;
[0112] For the two obtained spatial attention maps , using the Sigmoid activation function to generate a spatial selection mask :
[0113] ;
[0114] in, is the Sigmoid activation function. The corresponding Perform weighted operations and then pass through the convolution layer The weighted features are fully integrated to obtain the attention feature map :
[0115] ;
[0116] Finally, through and The output of ASSA is calculated by element-by-element product between :
[0117] .
[0118] ASSA can adaptively adjust the reasonable receptive fields required for different targets to improve the network's attention to the most relevant spatial context areas when detecting defect targets of different scales, thereby further improving the performance of LCD screen defect target detection.
[0119] The following combination Figure 2 The EA-YOLO model of this embodiment (based on the improvement of the YOLOv8 network) is summarized as follows:
[0120] The EA-YOLO model includes a backbone network (Backbone), a neck network (Neck) and a head network (Head). The backbone network is used to extract image features, the neck network is used to fuse the image features extracted by the backbone network, and the head network is used to predict LCD screen defects based on the features fused by the neck network.
[0121] The backbone network (Backbone) includes a first convolutional layer (Conv), a first FCFB module, a first C2f_FELB module, a second FCFB module, a second C2f_FELB module, a third FCFB module, a third C2f_FELB module, a fourth FCFB module, a fourth C2f_FELB module and an SPPF module, which are connected in sequence. Among them, the function of the SPPF module is to expand the receptive field and enrich the feature expression through multi-scale pooling operations, thereby improving the accuracy and robustness of detection. Since the SPPF module is an existing technology of the original YOLOv8 network, it will not be repeated in this embodiment.
[0122] The neck network (Neck) includes the first upsampling layer (Upsample), the first Concat layer, the fifth C2f_FELB module, the second upsampling layer (Upsample), the second Concat layer, the sixth C2f_FELB module, the third upsampling layer (Upsample), the third Concat layer, the seventh C2f_FELB module, the first ASSA module, the second convolutional layer (Conv), the fourth Concat layer, the eighth C2f_FELB module, the second ASSA module, the third convolutional layer (Conv), the fifth Concat layer, the ninth C2f_FELB module, the third ASSA module, and the fourth convolutional layer. Convolution layer (Conv), sixth Concat layer, tenth C2f_FELB module, fourth ASSA module; wherein, the SPPF module is respectively connected to the first upsampling layer (Upsample) and the sixth Concat layer; the third C2f_FELB module is connected to the first Concat layer; the second C2f_FELB module is connected to the second Concat layer; the first C2f_FELB module is connected to the third Concat layer, and the output of the first C2f_FELB module is the above-mentioned shallow feature map P2; the fifth C2f_FELB module is also connected to the fifth Concat layer; the sixth C2f_FELB module is also connected to the fourth Concat layer.
[0123] It should be noted that each convolutional layer (Conv) in the backbone network (Backbone) and the neck network (Neck) is connected to the first BatchNorm layer (batch normalization layer) and the first SiLU function.
[0124] The head network (Head) consists of four Dectect layers. The structure of the Dectect layer is detailed in Figure 8 , each Dectect layer includes two branches, the first branch is used to obtain the bounding box of the LCD defect, and the second branch is used to obtain the confidence of the LCD defect (that is, what type of defect it is). The first branch includes the first CBS unit, the fifth convolutional layer (Conv), and the Bbox Loss layer (used to calculate the regression loss) connected in sequence, and the second branch includes the second CBS unit, the sixth convolutional layer (Conv), and the Cls Loss layer (used to calculate the classification loss) connected in sequence. The first CBS unit and the second CBS unit both include the seventh convolutional layer (Conv), the second BatchNorm layer (batch normalization layer), and the second SiLU function connected in sequence. The first Dectect layer is connected to the first ASSA module, the second Dectect layer is connected to the second ASSA module, the third Dectect layer is connected to the third ASSA module, and the fourth Dectect layer is connected to the fourth ASSA module. It is worth mentioning that the first Dectect layer is the small target detection head SODH mentioned above. In this embodiment, the shallow feature map P2 is integrated into the feature fusion process in the neck network, so that the first Dectect layer can fully extract the location information and detail features of the defective small target.
[0125] Assuming that the input image size of the preset neural network is 640×640×3, the output size of the first Dectect layer is 160×160×64, the output size of the second Dectect layer is 80×80×128, the output size of the third Dectect layer is 40×40×256, and the output size of the fourth Dectect layer is 20×20×512.
[0126] 4. Optimize the loss function
[0127] The baseline network YOLOv8 uses CIoU as the bounding box regression loss function. CIoU additionally considers the aspect ratio of the predicted bounding box and the true bounding box on the basis of DIoU. However, this penalty term describes the relative values of the width and height of the predicted box and the true box, and cannot reflect the actual error between the two. In addition, CIoU will fail when the predicted box and the true box have the same aspect ratio but completely different width and height values, resulting in a decrease in convergence speed and detection accuracy. In order to solve this problem, a bounding box regression loss function based on the minimum point distance is introduced, which fully considers the actual deviation of the overlapping or non-overlapping area, center point distance, width and height between the predicted box and the true box, and minimizes the distance between the upper left point and the lower right point between the predicted bounding box and the true bounding box to improve the convergence speed and accuracy of the LCD screen defect detection model.
[0128] The coordinates of the upper left corner and lower right corner of the predicted box and the real box can fully represent all the factors considered by the CIoU loss function. The conversion formula is as follows:
[0129] ;
[0130] in, Represents the diagonal length of the minimum enclosing rectangle of the real bounding box and the predicted bounding box, , Respectively represent the center point coordinates of the predicted box and the real box, is the width and height of the prediction box, is the width and height of the real frame.
[0131] This shows This not only takes into account the considerations, but also simplifies the calculation process, which can reach convergence faster in the bounding box regression process and achieve higher detection accuracy. Assume that the height and width of the input image are , the real box and the predicted box are represented by the coordinates of the upper left corner and the lower right corner respectively. , , The specific expressions are as follows:
[0132] ;
[0133] in, is the Euclidean distance between the upper left point of the predicted box and the true box, is the Euclidean distance between the lower right point of the predicted box and the true box, is a measure of the similarity between the predicted bounding box and the true bounding box. is the bounding box regression loss function based on the minimum point distance, To take the union, To obtain the intersection. In this embodiment, the upper left point and lower right point of the predicted box or the real box can be found in Fig. 9 .
[0134] The specific use process of the technical solution is described in detail below through specific embodiments.
[0135] 1. Construction and application of LCD screen defect detection system
[0136] The construction of the LCD screen defect detection system mainly involves deploying the EA-YOLO model proposed in this embodiment on a mid-to-low-end hardware platform such as NVIDIA Jetson TX2-NX. First, the hardware environment includes the hardware platform processor, display and high-resolution industrial camera, and the software environment includes the Ubuntu operating system, PyTorch deep learning framework and other necessary dependent libraries. Then, the model training process is carried out: (1) Dataset preparation: collect and annotate image datasets containing LCD screen defects (such as black spots, bright spots, scratches, etc.). (2) Data preprocessing: normalize the dataset and perform data enhancement (such as random cropping, rotation, etc.) to improve the generalization ability of the model. (3) Model training: use the improved EA-YOLO model for training, set appropriate hyperparameters such as learning rate and batch size, and use the Adam optimizer for optimization. (4) Model evaluation: evaluate the performance of the model on the validation set, and use indicators such as Precision, Recall, mAP0.5, mAP0.5:0.95, and FPS to measure the detection performance of the model. After the deployment is completed, the system goes through model conversion, inference environment configuration and real-time detection process, that is, the LCD screen image is captured in real time by the camera, and the EA-YOLO model proposed in this embodiment is used to detect defect targets. The detection results are displayed on the screen in real time, and finally accurate real-time detection of LCD screen defects is achieved to meet the application requirements of low latency and high real-time performance.
[0137] 2. Practical application process
[0138] An LCD defect detection system is deployed in the inspection link of the LCD production line. A high-resolution industrial camera is installed above the production line to ensure that the camera can cover the entire LCD screen. When the LCD screen passes through the inspection area from the production line conveyor belt, the camera captures the image in real time and transmits it to the hardware platform device for defect detection using the EA-YOLO network. After a defect is detected, the system marks the defect location through the display and alarm system, and automatically records the defect information. The inspection results are stored in the database for subsequent analysis and quality control.
[0139] It is not difficult to find that the present invention can effectively improve the accuracy and real-time performance of LCD defect detection, enhance the adaptability and robustness of the system, and has wide practical application value. Embodiment 2
[0140] This embodiment provides a liquid crystal display screen defect detection system, including:
[0141] Acquisition module: used to acquire images about the LCD screen;
[0142] Detection module: used to perform defect detection on the image of the LCD screen through a preset neural network, wherein:
[0143] The preset neural network is improved based on the YOLOv8 model, and includes a backbone network, a neck network and a head network. The backbone network is used to extract image features, the neck network is used to perform feature fusion on the image features extracted by the backbone network, and the head network is used to perform defect prediction on the features after fusion of the neck network;
[0144] The backbone network and the neck network of the preset neural network in the detection module are both provided with a C2f_FELB module for extracting key feature information, wherein:
[0145] The C2f_FELB module first adjusts the number of channels of the feature map through the first 1×1 convolution. The output of the first 1×1 convolution is divided into the first two parts, one part is used for residual connection, and the other part is divided into the second two parts through the spilt operation. One part is used for residual connection and the other part enters the main branch; the main branch extracts features through n series-connected feature extraction bottlenecks FEBs and maintains residual connection; then, the output of the part used for residual connection obtained by the first 1×1 convolution, the output of the part used for residual connection obtained by the spilt operation, and the output of the n series-connected feature extraction bottlenecks FEBs maintaining residual connection are all spliced through the concat operation to obtain the spliced feature map; finally, the spliced feature map is processed through the second 1×1 convolution to obtain the final output of the C2f_FELB module. Embodiment 3
[0146] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the liquid crystal display defect detection method described in the first embodiment are implemented. Embodiment 4
[0147] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the liquid crystal display screen defect detection method described in the first embodiment are implemented.
[0148] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The schemes in the embodiments of the present application may be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.
[0149] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0150] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0151] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0152] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.
[0153] Obviously, the above embodiments are merely examples for clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from these are still within the protection scope of the invention.
Claims
1. A method for detecting defects in a liquid crystal display screen, Features: include: Step S1: Acquire an image of the LCD screen; Step S2: performing defect detection on the image of the LCD screen through a preset neural network, wherein: The preset neural network is improved based on the YOLOv8 model, and includes a backbone network, a neck network and a head network. The backbone network is used to extract image features, the neck network is used to perform feature fusion on the image features extracted by the backbone network, and the head network is used to perform defect prediction on the features after fusion of the neck network; The backbone network and the neck network of the preset neural network in step S2 are both provided with a C2f_FELB module for extracting key feature information, wherein: The C2f_FELB module first adjusts the number of channels of the feature map through the first 1×1 convolution. The output of the first 1×1 convolution is divided into the first two parts, one part is used for residual connection, and the other part is divided into the second two parts through the spilt operation. One part is used for residual connection and the other part enters the main branch; the main branch extracts features through n series-connected feature extraction bottlenecks FEBs and maintains residual connection; then, the output of the part used for residual connection obtained by the first 1×1 convolution, the output of the part used for residual connection obtained by the spilt operation, and the output of the n series-connected feature extraction bottlenecks FEBs maintaining residual connection are all spliced through the concat operation to obtain the spliced feature map; finally, the spliced feature map is processed through the second 1×1 convolution to obtain the final output of the C2f_FELB module; The backbone network of the neural network preset in step S2 is also provided with an FCFB module for downsampling, wherein: The FCFB module includes: for a given size Feature map , H is the feature map height, W is the feature map width, C is the number of feature map channels, and the feature map is scaled by a factor of 2 Perform interleaved downsampling to obtain four sub-feature maps , the size of each sub-feature map is , expressed as: ; in, Represents the feature map The sub-feature map obtained after downsampling by a scale factor of 2; The four sub-feature maps obtained Connect along the channel dimension to obtain the intermediate feature map , The size is , whose channel dimension is expanded to feature map 4 times, the spatial dimension is reduced to the feature map of ; Then use a non-strided convolution with a convolution kernel size of 3×3 to reduce the number of channels of the feature map and convert the intermediate feature map Convert to size The resulting feature map , the resulting feature map It is the final output of the FCFB module.
2. The method for detecting defects in a liquid crystal display according to claim 1, Features: The feature extraction bottleneck FEB includes a first partial convolution PConv, a first point-by-point convolution PWConv, a second partial convolution PConv and a second point-by-point convolution PWConv connected in sequence, and the input of the first partial convolution PConv and the output of the second point-by-point convolution PWConv are element-wise added as the output of the feature extraction bottleneck FEB, wherein a normalization layer BN and a SiLU activation function are also arranged between the second partial convolution PConv and the second point-by-point convolution PWConv.
3. The method for detecting defects in a liquid crystal display according to claim 1, Features: The neck network of the preset neural network in step S2 is provided with an ASSA module, and the ASSA module includes: The 23×23 convolution is decomposed into two convolutions with different scales of receptive fields in series, and then two feature maps with different receptive fields are generated. The convolution kernel sizes of the two convolutions after decomposition are , Expansion Rate And the theoretical receptive field is defined as: ; Use two decomposed convolutions with different scale receptive fields to extract feature information, expressed as: ; in, is the input feature map of the ASSA module, represents the first and second convolutions, is the feature map obtained by the corresponding convolution; then the spatial adaptive selection mechanism is used to realize the dynamic selection of the kernel, including: first Splice in the channel dimension to form a feature set : ; Again Extracted using the maximum pooling operation based on the channel dimension The spatial feature relationship of : ; in, Represents the maximum pooling operation based on the channel dimension; then the convolution layer is used for nonlinear transformation and feature integration to convert the maximum pooling spatial feature map Converted to two spatial attention maps : ; For the two obtained spatial attention maps , using the Sigmoid activation function to generate a spatial selection mask : ; in, is the Sigmoid activation function; Will The corresponding Perform weighted operation and then pass through the convolution layer The weighted features are fused to obtain the attention feature map : ; Last passed and The output of the ASSA module is calculated by element-by-element product between : 。 4. The method for detecting defects in a liquid crystal display screen according to claim 3, Features: The head network of the preset neural network in step S2 includes four Dectect layers, namely the first Dectect layer, the second Dectect layer, the third Dectect layer, and the fourth Dectect layer, wherein the first Dectect layer, the second Dectect layer, the third Dectect layer, and the fourth Dectect layer are respectively connected to different ASSA modules in the neck network.
5. The method for detecting defects in a liquid crystal display screen according to claim 4, Features: Assuming that the input image size of the preset neural network is 640×640×3, the output size of the first Dectect layer is 160×160×64, the output size of the second Dectect layer is 80×80×128, the output size of the third Dectect layer is 40×40×256, and the output size of the fourth Dectect layer is 20×20×512.
6. The method for detecting defects in a liquid crystal display screen according to claim 1, Features: Before obtaining the preset neural network in step S2, the preset neural network is also trained, and during the training, a bounding box regression loss function based on the minimum point distance is used to determine whether the preset neural network converges, wherein: The method for calculating the bounding box regression loss function based on the minimum point distance includes: ; in, Indicates the diagonal length of the minimum enclosing rectangle of the real box and the predicted box, , Respectively represent the center point coordinates of the predicted box and the real box, is the width and height of the prediction box, is the width and height of the real frame; Assume that the height and width of the input image are , the real box and the predicted box are represented by the coordinates of the upper left corner and the lower right corner respectively. , , then the bounding box regression loss function based on the minimum point distance is expressed as: ; in, is the Euclidean distance between the upper left point of the predicted box and the true box, is the Euclidean distance between the lower right point of the predicted box and the true box, is a measure of the similarity between the predicted bounding box and the true bounding box. is the bounding box regression loss function based on the minimum point distance, To take the union, To take the intersection.
7. A liquid crystal display defect detection system, Features: include: Acquisition module: used to acquire images about the LCD screen; Detection module: used to perform defect detection on the image of the LCD screen through a preset neural network, wherein: The preset neural network is improved based on the YOLOv8 model, and includes a backbone network, a neck network and a head network. The backbone network is used to extract image features, the neck network is used to perform feature fusion on the image features extracted by the backbone network, and the head network is used to perform defect prediction on the features after fusion of the neck network; The backbone network and the neck network of the preset neural network in the detection module are both provided with a C2f_FELB module for extracting key feature information, wherein: The C2f_FELB module first adjusts the number of channels of the feature map through the first 1×1 convolution. The output of the first 1×1 convolution is divided into the first two parts, one part is used for residual connection, and the other part is divided into the second two parts through the spilt operation. One part is used for residual connection and the other part enters the main branch; the main branch extracts features through n series-connected feature extraction bottlenecks FEBs and maintains residual connection; then, the output of the part used for residual connection obtained by the first 1×1 convolution, the output of the part used for residual connection obtained by the spilt operation, and the output of the n series-connected feature extraction bottlenecks FEBs maintaining residual connection are all spliced through the concat operation to obtain the spliced feature map; finally, the spliced feature map is processed through the second 1×1 convolution to obtain the final output of the C2f_FELB module; The backbone network of the preset neural network in the detection module is also provided with an FCFB module for downsampling, wherein: The FCFB module includes: for a given size Feature map , H is the feature map height, W is the feature map width, C is the number of feature map channels, and the feature map is scaled by a factor of 2 Perform interleaved downsampling to obtain four sub-feature maps , the size of each sub-feature map is , expressed as: ; in, Represents the feature map The sub-feature map obtained after downsampling by a scale factor of 2; The four sub-feature maps obtained Connect along the channel dimension to obtain the intermediate feature map , The size is , whose channel dimension is expanded to feature map 4 times, the spatial dimension is reduced to the feature map of ; Then use a non-strided convolution with a convolution kernel size of 3×3 to reduce the number of channels of the feature map and convert the intermediate feature map Convert to size The resulting feature map , the resulting feature map It is the final output of the FCFB module.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, Features: When the processor executes the computer program, the steps of the liquid crystal display defect detection method according to any one of claims 1 to 6 are implemented.
9. A computer-readable storage medium having a computer program stored thereon, Features: When the computer program is executed by a processor, the steps of the liquid crystal display defect detection method according to any one of claims 1 to 6 are implemented.
