A method for detecting low contrast and small defects in display screens

Abstract

The present invention discloses a method for detecting low-contrast and small defects on display screens, which relates to the technical field of automated detection of display screen defects in the field of machine vision. The method comprises the following steps: preprocessing a captured display screen defect image; extracting features from the preprocessed display screen defect image to obtain a multi-level feature map from shallow to deep layers; fusing the extracted feature maps using a feature pyramid network, a shallow semantic depth fusion module, and a path enhancement network, and inserting a dual-focus contrast enhancement attention at the feature fusion site to obtain three fused feature maps of different scales; and applying a convolution operation to the three fused feature maps for the detection of large, medium, and small defect targets, respectively. The present invention can effectively improve the detection performance of small defects such as dots and stains, and oil stains, lines, and mura defects under low contrast. It can be applied to the unified detection of lighting defects and appearance defects on display screens, reducing the cost of industrial defect detection.

Description

Technical Field

[0001] The present invention relates to the technical field of automated detection of display screen defects in the field of machine vision, and in particular to a method for detecting low-contrast and small defects in display screens. Background Art

[0002] In recent years, display screens have been widely used in televisions, personal computers, mobile phones, and other fields. However, due to the complexity of their manufacturing process, display screens are prone to failure during production. These failures can be primarily categorized as lighting defects and cosmetic defects. Lighting defects primarily include spots, lines, and mura, while cosmetic defects primarily include oil stains, scratches, and stains. These serious defects can significantly affect the user's visual experience. Therefore, to ensure product quality and user satisfaction, newly manufactured products must undergo a rigorous defect inspection process before leaving the factory.

[0003] Currently, LCD defect detection is primarily categorized into traditional image processing methods and deep learning approaches. Traditional image processing methods rely heavily on manually designed algorithms for feature extraction, resulting in poor robustness and generalization. Furthermore, traditional methods have high computational complexity and lack the ability to utilize GPU parallel computing, making them difficult to apply for real-time, rapid detection on industrial production lines. Existing deep learning methods can effectively address the shortcomings of traditional processing methods, but they suffer from low defect detection accuracy in low-contrast scenes and a significant risk of missing even small defects. Furthermore, all current LCD defect detection methods are unable to uniformly detect both lighting defects and appearance defects. Summary of the Invention

[0004] The present invention provides a method for detecting low contrast and small defects of a display screen, which can solve the above problems.

[0005] In order to solve the above problems, the technical solutions adopted by the present invention are as follows:

[0006] The present invention provides a method for detecting low contrast and small defects of a display screen, comprising the following steps:

[0007] S1. Perform data enhancement processing, scale unification processing and anchor frame automatic generation processing on the collected display screen defect image;

[0008] S2. Use the backbone network to extract features from the preprocessed display defect image to obtain a multi-level feature map from shallow to deep layers;

[0009] S3, use the feature pyramid network, shallow semantic depth fusion module and path enhancement network to perform feature fusion on the feature map extracted in step S2, and insert dual focus contrast enhancement attention at the feature fusion site to obtain three fused feature maps of different scales;

[0010] S4. The fused feature maps of three different scales are subjected to a convolution operation and applied to the detection of large, medium and small defect targets respectively.

[0011] Specifically, in step S1, data enhancement includes flipping, rotation, cropping, scaling, translation and dithering; scale unification processing is to uniformly scale all images to h×w×c, where h, w, and c are the length, width and number of channels of the image respectively; the automatic generation of anchor frames is to cluster all label frames using the Kmeans++ algorithm to generate preliminary anchor frames, and then use the heuristic optimization algorithm to further optimize the preliminary anchor frames.

[0012] Specifically, step S2 includes: using a large kernel convolution to extract features from the preprocessed display defect image, outputting a preliminary feature map of size h / 2×w / 2×c, and then using a four-step cross-stage local (CSP) network and a convolutional combination (CBS) module to extract shallow and deep features. The proportion of the residual module of the CSP network gradually decreases from the shallow layer to the deep layer. Finally, a fast spatial pyramid pooling module (SPPF) is used to fuse the local and global features of the last layer of deep feature map to obtain a deep feature map of size h / 32×w / 32×16c.

[0013] Specifically, the CBS module consists of a convolution, a batch normalization, and an activation function in series. Input convolution combination module, the calculation process is expressed as:

[0014]

[0015] Among them, F input is the input feature map, CBS() is the operation after the CBS module, Conv is the convolution operation, BatchNorm2d is the feature map batch normalization, and SiLU is the activation function.

[0016] Specifically, the CSP network divides the input feature map into two branches, one branch passes through the CBS module, and the other part is connected to multiple residual modules after passing through a CBS module. The two branches are then combined on the channel and output through another CBS module. In the four-fold CSP network, the proportion of residual modules gradually decreases from the shallow layer to the deep layer, which means that the proportion of residual structures in the shallow network is large and the residual structure is small in the deep network.

[0017] Specifically, in step S3, feature fusion is to first perform top-down feature fusion on the feature map extracted in step S2 using a feature pyramid, then use a shallow semantic depth fusion module to enhance the small target feature fusion, and then perform bottom-up feature fusion based on a path enhancement network.

[0018] Specifically, the feature map is first fused from top to bottom using a feature pyramid, which is completed in series by four operation modules: a CBS module, an upsampling module, a channel fusion module, and a CSP network. After completing a cycle of modules, the alternating operation is continued. Among them, the CBS module reduces the number of channels, and the upsampling module uses a bilinear interpolation method to expand the feature map. When the feature map Input feature pyramid network, the computation process of two cycles of feature pyramid network is expressed as:

[0019]

[0020]

[0021] Among them, Up 2× Indicates 2x upsampling, F Bkmap Represents the feature map of the same size and channel, F output is the output graph after the feature pyramid network.

[0022] Specifically, the shallow semantic depth fusion module introduces shallower semantic features into the loss calculation without adding additional detection heads, and upsamples the output F in the feature pyramid network. output After that, it is further upsampled and fused with the original shallow network feature map, which can be expressed as follows:

[0023]

[0024] in, It is the output feature map of the feature pyramid after the shallow semantic depth fusion module.

[0025] Specifically, the feature map The bottom-up feature fusion is performed based on the path enhancement network, which is completed in series by three operation modules: a CBS module, a channel fusion, and a CSP network. After completing a cycle of modules, the alternating operation is continued until a feature map of the same size as the output map of the backbone network is generated. When the feature map Enter the path enhancement network. The calculation process of the three cycles of the path enhancement network is expressed as:

[0026]

[0027]

[0028]

[0029] Among them, f output is the output feature map after the PANet module, f LSDFRepresented as feature maps of the same size and channel in the LSDF module.

[0030] Specifically, the dual-focus contrast enhancement attention is composed of spatial attention and channel attention in parallel, and the residual method is used to combine and generate three-dimensional attention to enrich the semantic information of defects in low-contrast scenes, which is expressed as:

[0031]

[0032]

[0033] Among them, F out For dual focus contrast enhanced attention output map, F in is the input feature map, is a three-dimensional attention vector composed of spatial attention and channel attention, is the channel attention output vector, is the spatial attention output vector, is point-by-point multiplication;

[0034] Channel weight vector F c The calculation formula is:

[0035] F c =σ(MLP(maxpool(F in ))+MLP(avgpool(F in )))

[0036] Where σ is the sigmiod activation function, and MLP is a parameter-sharing multi-layer perceptron;

[0037] The weight map F for each position in each space s , is expressed as:

[0038] F s =σ(C 7×7 (concat(maxpool(F in ),avgpool(F in ))))

[0039] Among them, C 7×7 Represents a convolution operation with a kernel size of 7×7.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] The backbone network can enhance the model's ability to perceive details, the pyramid semantic enhancement network can effectively alleviate the problem of shallow information loss after deep convolution, and the dual-focus contrast enhancement network can guide the model to more effectively focus on the spatial information and key features of defect targets. The overall solution combining these three can effectively improve the detection performance of small defects such as dots and stains, as well as oil stains, lines, and mura defects in low contrast. At the same time, this solution can be applied to the unified detection of lighting defects and appearance defects, thereby significantly reducing the cost of industrial defect detection.

[0042] In order to make the above-mentioned objects, features and advantages of the present invention more clearly understood, embodiments of the present invention are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0044] Figure 1 This is a diagram of a defect detection model based on deep learning according to an embodiment of the present invention;

[0045] Figure 2 This is a defect sample diagram of an embodiment of the present invention;

[0046] Figure 3 The backbone network LACSPDarknet53 of the embodiment of the present invention;

[0047] Figure 4 The dual-focus contrast enhancement attention mechanism of an embodiment of the present invention;

[0048] Figure 5 This is the shallow semantic depth fusion module of an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0050] The embodiment of the present invention provides a method for detecting defects in low-contrast scenes and small-sized display screens, and the defect detection model used is as follows: Figure 1 As shown, the method of the present invention specifically comprises the following steps:

[0051] S1. Preprocessing the collected display screen defect image to obtain more scale-consistent image data and anchor frame data, that is, the preprocessed display screen defect image.

[0052] In this embodiment, the collected display screen defect images include lighting and appearance defect data. Lighting defects include dots, lines, and mura, and appearance defects include oil stains, scratches, and stains. The background colors in the lighting defect data include seven types: black, white, red, green, gray, light gray, and black and white. The defect colors mainly include six types: red, white, black, yellow, blue, and green. The appearance defect data includes various styles of black and white mobile phone shells as the background. The experimental samples are as follows. Figure 2 shown.

[0053] The preprocessing process includes three processing operations: data enhancement, scale unification, and automatic generation of anchor boxes.

[0054] Data enhancement includes geometric transformation methods such as flipping, rotation, cropping, scaling, translation, and jittering.

[0055] Scaling is to uniformly scale all images to h×w×c. In this example, h, w, and c are 640, 640, and 3 respectively.

[0056] The automatic generation of anchor frames uses the Kmeans++ algorithm to cluster all label frames to generate preliminary anchor frames, and then uses a heuristic optimization algorithm to further optimize the preliminary anchor frames. The heuristic algorithm used in this example is a genetic algorithm.

[0057] S2. Use the backbone network LACSPDakrnet53 to extract features from the preprocessed display defect image and obtain a multi-level feature map from shallow to deep layers.

[0058] Specifically, a large kernel convolution is first used to extract features from the h×w×3 feature map (the preprocessed display defect map), outputting a preliminary feature map of size h / 2×w / 2×c. A quad-step local (CSP) network and a convolutional binning (CBS) module are then used to extract shallow and deep features. To enhance the extraction of small object features, the proportion of residual modules in the CSP module gradually decreases from shallow to deep layers. Finally, fast spatial pyramid pooling (SPPF) is used to fuse the local and global features of the last deep feature map, resulting in a deep feature map of size h / 32×w / 32×16c.

[0059] In this example, the backbone network LACSPDarknet53 model framework used is as follows Figure 3 As shown, the preprocessed display screen defect image is its input image, and the scale is 640×640×3, that is, the input image length h is 640, the width w is 640, and the number of channels c is 3.

[0060] The output image scale after backbone network feature extraction is 20×20×1024.

[0061] Large kernel convolution uses a convolution kernel of size 6×6 with a stride of 2.

[0062] In this example, the CBS module consists of a convolution, a batch normalization, and an activation function in series. Enter the CBS module, and its calculation process is expressed as:

[0063]

[0064] Among them, F input The input feature map is CBS(), which is the CBS module operation, Conv is the convolution operation, BatchNorm2d is the batch normalization of the feature map, and SiLU is the activation function. In this example, the convolution operation uses a 3×3 kernel with a stride of 2.

[0065] In this example, the Cross-Stage Local (CSP) network divides the input feature map into two branches, one branch passes through the CBS module, and the other branch is connected to multiple residual modules after passing through a CBS module. The two branches are then combined on the channel and output through another CBS module.

[0066] In the four-pass CSP network, the ratio of residual modules gradually decreases from shallow to deep layers, indicating that the shallow network has a higher proportion of residual structure and the deep network has a lower proportion of residual structure. In this example, the ratio of residual structure in the four CSP networks from shallow to deep is set to 3:2:1:1 respectively.

[0067] S3. Use the FPN (feature pyramid) network, shallow semantic depth fusion module and PANet (path enhancement network) to fuse the feature map extracted in step S2, and insert dual focus contrast enhancement attention at the feature fusion site.

[0068] Specifically, the feature maps output by the backbone network are first fused from top to bottom using the feature pyramid (FPN), and then the shallow semantic depth fusion module is used to enhance the fusion of small target features, and then the features are fused from bottom to top based on the path enhancement network (PANet).

[0069] In this example, the input image size of the FPN network (that is, the output image of the backbone network) is 20×20×1024, and the output image size is 80×80×256. After passing through the shallow semantic deep fusion module, the output image size is 160×160×128. In other words, the input image size of the PANet network is also 160×160×128, and the output image size after passing through PANet is 20×20×1024.

[0070] Specifically, the feature fusion process is as follows:

[0071] 1) Take the backbone network output map (i.e., the feature map extracted in step S2) as the input of feature fusion, and first perform feature fusion based on FPN top-down, which is completed in series by four operation modules: a CBS module, an upsampling module, a channel fusion module, and a CSP network module. After completing a cycle of modules, continue to alternate operations. The CBS module reduces the number of channels, and the upsampling module uses a bilinear interpolation method to expand the feature map. When the backbone network outputs the feature map Input FPN, the calculation process of two cycles of FPN is expressed as:

[0072]

[0073]

[0074] Among them, Up 2× Indicates 2x upsampling, F Bkmap Represented as the feature map of the same size and channel in the backbone network, F output This is the output image after the FPN module. In this example, h2, w2, and c2 are 20, 20, and 1024 respectively.

[0075] 2) The shallow semantic depth fusion module introduces shallower semantic features into the loss calculation without adding additional detection heads. The shallow semantic depth fusion module is as follows: Figure 5 As shown. After the FPN network is upsampled to 4h2×4w2×c2 / 4, the output F output Finally, in order to strengthen the attention of small objects in the shallow semantics, the 4h2×4w2×c2 / 4 is further upsampled to 8h2×8w2×c2 / 8 and fused with the original shallow network feature map. It can be expressed as follows:

[0076]

[0077] in, It is the output feature map of FPN after the shallow semantic depth fusion module.

[0078] 3) Based on PANet, the bottom-up feature fusion is performed. The fusion process is completed in series by three operation modules: a CBS module, a channel fusion module, and a CSP module. After completing a cycle of modules, the alternating operation is continued until a feature map of the same size h2×w2×c2 as the output map of the backbone network is generated. The CBS module is used to reduce the size of the feature map and increase the number of channels. When the feature map Enter PANet. The calculation process of PANet's three cycles can be expressed as:

[0079]

[0080]

[0081]

[0082] Among them, f output is the output feature map after the PANet module, f LSDF Represented as feature maps of the same size and channel in the LSDF module.

[0083] In this example, dual focus contrast enhancement attention is inserted into the feature fusion of FPN (feature pyramid) network, shallow semantic deep fusion module and PANet (path enhancement network). Figure 4 As shown, attention does not change the size and number of channels of the feature map.

[0084] The dual-focus contrast-enhanced attention is composed of spatial attention and channel attention in parallel, and the residual method is used to combine and generate three-dimensional attention to enrich the semantic information of defects in low-contrast scenes, which is expressed as:

[0085]

[0086]

[0087] Among them, F out For dual focus contrast enhanced attention output map, F in is the input feature map, is a three-dimensional attention vector composed of spatial attention and channel attention, is the channel attention output vector, is the spatial attention output vector, is point-by-point multiplication.

[0088] In this example, the channel attention mechanism uses global maximum pooling and global average pooling to compress each feature map into two 1×1×C vectors to obtain global information between channels. The two vectors are then input into the multi-layer perceptron (MLP) for stimulation to learn the weight or importance between each channel. On this basis, the two vectors with weight information are added together and the sigmoid function is used for activation to obtain the weight vector F for each channel. c .

[0089] F c =σ(MLP(maxpool(F in))+MLP(avgpool(F in )))

[0090] Among them, σ is the sigmiod activation function, and MLP is a parameter-sharing multi-layer perceptron.

[0091] In this example, spatial attention first uses global maximum pooling and global average pooling to compress all feature maps into two h×w×1 feature maps in space to obtain global information in space, and then splices the two feature maps into h×w×2. On this basis, a 7×7 convolution kernel is used to reduce the dimension of the spliced feature map to h×w×1, and then the sigmoid function is used for activation to obtain the weight map F for each position in each space. s , can be expressed as:

[0092] F s =σ(C 7×7 (concat(maxpool(F in ),avgpool(F in ))))

[0093] Among them, C 7×7 Represents a convolution operation with a kernel size of 7×7.

[0094] S4. Utilize fused features for defect target detection. That is, the feature maps of three different sizes (h / 8×w / 8, h / 16×w / 16, and h / 32×w / 32) output from the PANet network are subjected to a convolution operation and applied to the detection of large, medium, and small defect targets respectively.

[0095] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A method for detecting low contrast and small defects on a display screen, characterized in that: The following steps are involved: S1. Perform data enhancement processing, scale unification processing and anchor frame automatic generation processing on the collected display screen defect image; S2. Use the backbone network to extract features from the preprocessed display defect image to obtain a multi-level feature map from shallow to deep layers. Specifically, it includes: using a large kernel convolution to extract features from the preprocessed display defect image, outputting a preliminary feature map of size h / 2×w / 2×c, and then using four cross-stage local networks and convolution combination modules to extract shallow and deep features. Among them, the proportion of residual modules in the cross-stage local network gradually decreases from shallow to deep layers. Finally, a fast spatial pyramid pooling module is used to fuse local and global features of the last layer of deep feature map to obtain a deep feature map of size h / 32×w / 32×16c; where h, w, and c are the length, width, and number of channels of the image, respectively; S3, use the feature pyramid network, shallow semantic depth fusion module and path enhancement network to perform feature fusion on the feature map extracted in step S2, and insert dual focus contrast enhancement attention at the feature fusion site to obtain three fused feature maps of different scales; S4. The fused feature maps of three different scales are subjected to a convolution operation and applied to the detection of large, medium and small defect targets respectively.

2. The method for detecting low contrast and small defects of a display screen according to claim 1, wherein: In step S1, data enhancement includes flipping, rotation, cropping, scaling, translation, and dithering; scale unification processing is to uniformly scale all images to h×w×c; the anchor frame automatic generation processing is to cluster all label frames using the Kmeans++ algorithm to generate preliminary anchor frames, and then use the heuristic optimization algorithm to further optimize the preliminary anchor frames.

3. The method for detecting low contrast and small defects of a display screen according to claim 2, wherein: The convolution combination module consists of a convolution, a batch normalization and an activation function in series. Input convolution combination module, the calculation process is expressed as: Among them, F input is the input feature map, CBS(·) is the convolution combination module operation, Conv is the convolution operation, BatchNorm2d is the feature map batch normalization, and SiLU is the activation function.

4. The method for detecting low contrast and small defects of a display screen according to claim 2, wherein: The cross-stage local network divides the input feature map into two branches, one branch passes through the convolution combination module, and the other part is connected to multiple residual modules after passing through a convolution combination module. The two branches are then combined on the channel and output through another convolution combination module. In the four-time cross-stage local network, the proportion of residual modules gradually decreases from the shallow layer to the deep layer, which means that the proportion of residual structures in the shallow network is large and the proportion of residual structures in the deep network is small.

5. The method for detecting low contrast and small defects of a display screen according to claim 1, wherein: In step S3, feature fusion is to first use the feature pyramid to perform top-down feature fusion on the feature map extracted in step S2, then use the shallow semantic depth fusion module to enhance the small target feature fusion, and then perform bottom-up feature fusion based on the path enhancement network.

6. The method for detecting low contrast and small defects of a display screen according to claim 5, characterized in that: The feature map is first fused from top to bottom using a feature pyramid, which is completed in series by four operation modules: a convolution combination module, an upsampling module, a channel fusion module, and a cross-stage local network. After completing a cycle of modules, the alternating operation is continued. Among them, the convolution combination module reduces the number of channels, and the upsampling uses a bilinear interpolation method to expand the feature map. When the feature map Input feature pyramid network, the computation process of two cycles of feature pyramid network is expressed as: Among them, Up 2× Indicates 2x upsampling, F Bkmap Represents the feature map of the same size and channel, F output is the output graph after the feature pyramid network.

7. The method for detecting low contrast and small defects of a display screen according to claim 6, wherein: The shallow semantic depth fusion module introduces shallower semantic features into the loss calculation without adding additional detection heads, and upsamples the output F in the feature pyramid network. output After that, it is further upsampled and fused with the original shallow network feature map, which can be expressed as follows: in, It is the output feature map of the feature pyramid after the shallow semantic depth fusion module.

8. The method for detecting low contrast and small defects of a display screen according to claim 7, wherein: Feature Map The bottom-up feature fusion is performed based on the path enhancement network, which is completed in series by three operation modules: a convolution combination module, a channel fusion module, and a cross-stage local network. After completing a cycle of modules, the alternating operation is continued until a feature map of the same size as the output map of the backbone network is generated. When the feature map Enter the path enhancement network. The calculation process of the three cycles of the path enhancement network is expressed as: Among them, f output is the output feature map after the PANet module, f LSDF Represented as feature maps of the same size and channel in the LSDF module.

9. The method for detecting low contrast and small defects of a display screen according to claim 1, wherein: The dual-focus contrast enhancement attention is composed of spatial attention and channel attention in parallel, and the residual method is used to combine and generate three-dimensional attention to enrich the semantic information of defects in low-contrast scenes, which is expressed as: Among them, F out For dual focus contrast enhanced attention output map, F in is the input feature map, is a three-dimensional attention vector composed of spatial attention and channel attention, is the channel attention output vector, is the spatial attention output vector, is point-by-point multiplication; Channel weight vector F c The calculation formula is: F c =σ(MLP(maxpool(F in ))+MLP(avgpool(F in ))) Where σ is the sigmiod activation function, and MLP is a parameter-sharing multi-layer perceptron; The weight map F for each position in each space s , is expressed as: F s =σ(C 7×7 (Concat(maxpool(F in ),avgpool(F in )))) Among them, C 7×7 Represents a convolution operation with a kernel size of 7×7.

Images