Liquid crystal module defect detection method based on multi-difference fusion and multi-branch pyramid network

Abstract

The invention discloses a multi-difference fusion and multi-branch pyramid network liquid crystal module defect detection method, and belongs to the technical field of screen detection. According to the utility model, the MDFM module is designed; the module uses a plurality of differential operators to enhance defect details in a liquid crystal screen, highlight edges and reduce background noise, and is beneficial to retaining and detecting small defects in a deep network. According to the invention, an MFEFPN structure is provided; by adopting an efficient up-sampling and fusion method, the synergistic effect among different level features is improved, the detection capability on large defects and small defects is enhanced, and the problem of information loss in the traditional FPN is solved. According to the utility model, the lightweight ASLD is redesigned; by sharing the parameters among the feature maps, information flow is improved, the parameter quantity is reduced, and the detection performance is improved. According to the method, the Inner-GIoU loss function is introduced to improve the positioning of the bounding box, the positioning error is reduced, and the robustness of the model is enhanced.

Description

Technical Field

[0001] This invention belongs to the field of screen inspection technology, and specifically relates to a method for detecting defects in liquid crystal modules using multi-differential fusion and multi-branch pyramid networks. Background Technology

[0002] LCD screen defect detection is a crucial step in ensuring product quality and directly impacts a company's quality control level. With the advancement of new industrialization, intelligent manufacturing has become a core driving force for modern industrial development. Against this backdrop, the industrial sector is increasingly demanding higher standards for production automation, efficiency, safety, and product quality, especially in the production of electronic products. The rapid development of deep learning technology has enabled many companies to combine object detection algorithms with robotic arms, further improving detection efficiency. Therefore, promoting the development of efficient LCD screen defect detection technology not only enhances the quality control level of production lines but also has significant practical implications for improving overall production efficiency and reducing costs.

[0003] NT Le et al. proposed a deep learning method combining CNN and FPN for multi-scale optical film defect detection. By extracting features of defects of different sizes through a multi-branch network, the perceptual ability for small and low-contrast defects was improved. The FPN structure facilitates the fusion of features at different levels and performs well in detecting thin, elongated line defects in LCD screens. Image enhancement techniques expanded the training dataset and improved the model's generalization ability. However, the robustness of this method in high-noise backgrounds still needs improvement, especially for defects with irregular edges or similar backgrounds. W Ming et al. proposed a CNN method combining an adaptive weighting mechanism for detecting small defects in TFT-LCD panels. The adaptive weighting mechanism enhanced the network's ability to perceive small defects, and the detection accuracy was improved through multi-scale feature fusion. However, false positives and false negatives still exist in environments with large changes in illumination, especially in bubble detection, where the similarity between transparent bubbles and the background poses a challenge to the model. YWang et al. (2024) proposed a lightweight YOLOv4 network architecture for real-time defect detection in TFT-LCD displays. By reducing network parameters through pruning and quantization, the real-time detection capability was improved. This method performs well in detecting large-area leakage defects, but in detecting low-contrast point defects, changes in illumination and background lead to missed detections, resulting in lower-than-expected accuracy. The YOLOv7-SiamFF framework proposed by Yi et al. improves defect feature representation and suppresses background interference through parallel Siamese networks, introduces a deep information feature fusion module to improve the detection accuracy of small targets, and integrates an attention mechanism to enhance the ability to recognize defect features. However, this model may still miss detections in high-noise environments and has high computational resource requirements, affecting real-time detection efficiency. Furthermore, over-reliance on the attention mechanism may lead to overfitting to detailed features, reducing generalization ability. The cross-scale weighted feature fusion network proposed by Zhang et al. improves defect recognition and localization accuracy through Laplacian sharpening and k-means clustering modules, but still has some shortcomings. First, the model may be affected by interference in complex backgrounds, leading to false detections. Second, although it has good robustness in multi-scale feature representation, accuracy may decrease when processing small target defects. Finally, the computational cost of the clustering module is high, which may affect real-time performance.

[0004] LCD screen defects are diverse and complex, such as Figure 1As shown, these problems mean that existing detection technologies still face many challenges. Small object detection has always been a difficult problem in the field of target detection, especially in low-contrast situations, where defects are often confused with the surrounding area, leading to missed detections. Point defects, as the most common type of tiny defect, usually appear as bright or dark spots. These defects are extremely small on the screen and have low contrast, making it difficult for general detection methods to accurately distinguish them from complex backgrounds. The detection of line defects and leakage defects also faces challenges. Line defects, due to their elongated shape and low contrast, are easily confused with the background, while leakage defects, due to their irregular shape and blurred edges, make it difficult to define a clear detection area. In addition, due to their transparent or semi-transparent properties, bubble defects cannot be effectively handled by traditional detection methods due to their subtle optical differences. These problems indicate that LCD screen defect detection not only requires efficient feature extraction and fusion capabilities but also needs to solve the problem of detecting small objects and irregular defects in multi-scale, complex backgrounds. In recent years, emerging technologies such as the YOLO series, Feature Pyramid Network (FPN), and self-attention mechanism have made significant progress in detection accuracy and speed, providing new solutions for LCD screen defect detection. By introducing multi-scale feature fusion and optimizing the loss function, the new detection model can effectively deal with these complex defect types while improving accuracy. Summary of the Invention

[0005] Based on the existing problems in LCD screen defect detection, this patent focuses on feature enhancement and multi-scale feature fusion. To address the issues of detail loss in point defects and low detection accuracy caused by the simultaneous occurrence of multiple defects, this patent proposes a novel detection algorithm. A new feature extraction module is designed, which enhances the perception of small-scale defects, especially point defects, through differential convolution. Secondly, an efficient feature pyramid structure is proposed, which solves the information loss problem in the fusion of shallow and deep features in traditional FPN by efficiently interacting and fusing shallow and deep features. Simultaneously, a lightweight shared parameter detection head is designed to improve the detection capability for defects of different scales. Finally, by optimizing the loss function and introducing geometric constraints, the positioning accuracy is improved. The technical solution is as follows: This invention provides a method for detecting defects in liquid crystal modules using multi-difference fusion and multi-branch pyramid networks, the method comprising: S1: Obtain samples and classify them according to defects.

[0006] S2: Construct the MDMB-YOLO model based on the YOLO11 model, with the following adjustments: (1) Replace the C3k2 modules in layers 6 and 8 of the backbone network with MDFM modules. The C3k2 modules in layers 2, 4, 6, and 10, and the C2PSA modules in layers 10 are all output to the neck network.

[0007] (2) The neck network adopts a multi-branch fusion efficient feature pyramid network, which includes 6 Fusion feature fusion modules, 2 EUCB upsampling modules, 6 CSP_MSCB feature extraction modules, 3 1×1 Conv and 7 3×3 Conv.

[0008] (3) The detection head extracts features from three levels through three 3×3 convolutions, then shares the feature map parameters of the three levels through a shared convolution block, then adjusts the number of channels and performs group normalization using a 1×1 convolution, and finally inputs the feature maps into the three detection heads for classification and localization.

[0009] (4) The model is optimized using Inner-GIoU loss.

[0010] S3: Use samples to train and test the MDMB-YOLO model to obtain the optimal model.

[0011] S4: Acquire images of the LCD screen and perform detection using the optimal model.

[0012] The beneficial effects of the technical solution provided by this invention are as follows: This patent effectively improves the detection accuracy of the model for various defect targets and reduces missed detections and false detections. First, the feature extraction part was redesigned, and an MDFM module was constructed using the differential convolution concept. This module combines multiple differential strategies and fuses differential information, significantly enhancing the perception ability of small-scale defects, especially point defects in LCD screens. Second, an MFEFPN structure was designed, introducing a multi-branch design and efficient interaction and fusion of shallow and deep features, enhancing the model's ability to utilize global information and solving the problem of information loss or interference when fusing shallow and deep features in conventional FPN. Then, a lightweight shared parameter detection head was designed for defect targets of different scales, reducing the number of model parameters and computational load, and effectively improving the detection capability for defects of different scales. Finally, the loss function was optimized, and fine geometric constraints were added to help the network better focus on the target boundary and reduce localization errors. Experimental results show that, compared with the baseline model, this patent improves accuracy and recall by 4.4 and 3.3 percentage points, respectively, and improves mAP50 and mAP50-95 by 2.8 and 0.9 percentage points, respectively, while reducing the number of parameters and computational complexity by 23.3% and 8%, respectively. Attached Figure Description

[0013] Figure 1 These are various defect style diagrams; Figure 2 This is a block diagram of MDMB-YOLO; Figure 3 This is a block diagram of the MDFM module and the MDCM_bottleneck unit; Figure 4 This is a diagram illustrating the weight reset process in the MDCM module; Figure 5 These are comparison images of sensory fields; Figure 6 This is a schematic diagram of the EUCB module processing procedure; Figure 7 This is a block diagram of the CSP_MSCB structure; Figure 8 It is a comparison diagram of the detection head structure before and after the improvement; Figure 9 It is the number of labels for each sample in the dataset; Figure 10 This is a comparison chart of the accuracy and number of parameters of different models; Figure 11 This is a comparison chart of the feature extraction performance of YOLO11 and MDMB-YOLO at different levels; Figure 12 These are images showing the detection results of MDMB-YOLO on defect data under different backgrounds.

[0014] Figure 5 In the figure, Figure (a) shows the receptive field size of different layers of the backbone in the baseline model, and Figure (b) shows the receptive field size of different layers of the backbone after adding the MDFM structure. Figure 8 In the figure, Figure (a) shows the YOLO11 decoupling head structure, and Figure (b) shows the ASLD adaptive shared lightweight detection head used in MDMB-YOLO. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings.

[0016] Example 1 See Figure 1-8 Example 1 provides a method for detecting defects in liquid crystal modules using multi-difference fusion and multi-branch pyramid networks. The method includes: S1: Obtain samples and classify them according to defects; The samples were actual samples obtained by photographing defective LCD screens with a high-precision industrial camera. The defects were categorized into five types: bubble defects, point defects, line defects, leakage defects, and multi-pixel fault defects.

[0017] S2: The MDMB-YOLO model is built based on the YOLO11 model. The MDMB-YOLO model has the following adjustments: (1) See Figure 2 The C3k2 modules in layers 6 and 8 of the backbone network are replaced with MDFM modules. The C3k2 modules in layers 2, 4, 6, and 10, and the C2PSA module in layers 10 are all output to the neck network.

[0018] Specifically, the MDFM module is based on the C3k2 module, with the MDCM_bottleneck unit replacing the C3k unit. The MDCM_bottleneck unit is also based on the C3k unit, with two cascaded MDCMs replacing the bottleneck. See [link to documentation]. Figure 3 and 4 MDCM updates the weights by performing center difference, angular difference, horizontal difference, vertical difference, and additional convolution, and then sums the updated weights and biases to obtain a weight matrix containing richer information. The calculation process of MDCM is as follows:

[0019]

[0020]

[0021]

[0022]

[0023] .

[0024] in, express The weighting coefficient of location, Indicates after reset and update The weighting coefficient of location, , , as well as These represent the weight coefficients after the central difference, angular difference, horizontal difference, and vertical difference operations, respectively. This represents the weight coefficients after one convolution following initialization; It is an adjustable parameter used to control the size of the convolution kernel weights; This represents the original weight coefficients output from the previous layer; This indicates the difference direction of the angular difference convolution. This parameter is also adjustable. Here, the difference is performed in a counterclockwise direction.

[0025] (2) See Figure 2The neck network employs a multi-branch fusion efficient feature pyramid network, which includes 6 Fusion feature fusion modules, 2 EUCB upsampling modules, 6 CSP_MSCB feature extraction modules, 3 1×1 Convs and 7 3×3 Convs.The structure of the neck network is as follows: the C3k2 module of layer 2 outputs to the first 3×3 Conv; the C3k2 module of layer 4 outputs to the first 1×1 Conv; the MDFM module of layer 6 outputs to the second 1×1 Conv; the C2PSA module of layer 10 outputs to the third 1×1 Conv; the first 1×1 Conv outputs in two ways: one output goes to the first Fusion feature fusion module and is fused with the outputs of the first 3×3 Conv and the first EUCB upsampling module; the other output goes to the second 3×3 Conv; the second 1×1 Conv outputs in two ways: one output goes to the second Fusion feature fusion module and is fused with the outputs of the second 3×3 Conv and the second EUCB upsampling module. The outputs of the EUCB upsampling module are fused, and another output is sent to the third 3×3 Conv; the first Fusion feature fusion module outputs to the first CSP_MSCB feature extraction module; the second Fusion feature fusion module outputs to the second CSP_MSCB feature extraction module; the third Fusion feature fusion module fuses the outputs of the third 1×1 Conv and the third 3×3 Conv and outputs them to the third CSP_MSCB feature extraction module; the first CSP_MSCB feature extraction module outputs in two ways, respectively to the fourth Fusion feature fusion module and the fourth 3×3 Conv; the second CSP_MSCB feature extraction module outputs in three ways, respectively to the fourth Fusion feature fusion module and the fourth 3×3 Conv. The outputs are sent to the first EUCB upsampling module, the fifth Fusion feature fusion module, and the fifth 3×3 Conv module; the third CSP_MSCB feature extraction module outputs in two paths, one to the second EUCB upsampling module and the other to the sixth Fusion feature fusion module; the fourth Fusion feature fusion module fuses the outputs of the first CSP_MSCB feature extraction module and the first EUCB upsampling module and outputs them to the fourth CSP_MSCB feature extraction module; the fifth Fusion feature fusion module fuses the outputs of the second CSP_MSCB feature extraction module, the second EUCB upsampling module, the fourth 3×3 Conv module, and the sixth 3×3 Conv module and outputs them in two paths. The outputs are distributed as follows: one output goes to the fifth CSP_MSCB feature extraction module, and the other output goes to the fifth 3×3 Conv; the sixth Fusion feature fusion module fuses the outputs of the third CSP_MSCB feature extraction module, the fifth 3×3 Conv, and the seventh 3×3 Conv and outputs them to the sixth CSP_MSCB feature extraction module; the fourth CSP_MSCB feature extraction module outputs in two ways, one to the small detection head and the other to the sixth 3×3 Conv; the fifth CSP_MSCB feature extraction module outputs in two ways, one to the medium detection head and the other to the seventh 3×3 Conv; the sixth CSP_MSCB feature extraction module outputs to the large detection head.

[0026] The Fusion feature fusion module, two EUCB upsampling modules, three 1×1 Conv modules, and seven 3×3 Conv modules are all standard modules. See [link / reference] Figure 7 The CSP_MSCB feature extraction module is a non-standard module. Its processing involves dividing the original feature map into two parts (using a Split unit). One part is used for feature extraction (using CBR, MSDC, and Conv+BN units), and the other part is directly merged with the extracted features (using a Concatenation unit). Convolutional kernels of 1*1, 3*3, and 5*5 sizes are used to capture features during the feature extraction process. The CSP_MSCB feature extraction module is based on the CSP concept but with modifications, replacing the C3k2 unit with an MSDC unit.

[0027] Specifically, see Figure 7 The CSP_MSCB feature extraction module includes a Split unit, a CBR unit, an MSDC unit, a Conv+BN unit, and a Concatenation unit. The Split unit outputs two paths: one to the CBR unit and the other to the Concatenation unit. The CBR, MSDC, and Conv+BN units process the features sequentially, while the Concatenation unit merges the features output from the Split and Conv+BN units. The MSDC unit includes DWC1*1, DWC3*3, DWC5*5, summation, channel shuffling, three BN operations, and three ReLU6 operations. The CBR unit outputs four paths, which are respectively output to DWC1*1, DWC3*3, DWC5*5, and the summation operation. DWC1*1, the first BN, and the first ReLU6 process the features sequentially and output them to the summation. DWC3*3, the second BN, and the second ReLU6 process the features sequentially and output them to the summation. DWC5*5, the third BN, and the third ReLU6 process the features sequentially and output them to the summation. The summation is then output to the ChannelShuffe, and the ChannelShuffe outputs to the Conv+BN unit. The algorithm for the MSDC unit is as follows: .

[0028] in, Represents the kernel size. Represents input, Represents depthwise convolution. This represents the activation function, which undergoes nonlinear computation using ReLU6.

[0029] (3) The detection head extracts features from three levels through three 3×3 convolutions (specifically, the first 3×3DBB, the second 3×3DBB, and the third 3×3DBB). Then, it shares the feature map parameters of the three levels through a shared convolutional block (including the fourth 3×3DBB and 1×1Conv_GN). Then, it adjusts the number of channels and performs group normalization using a 1×1 convolution. Finally, the feature maps are input into the three detection heads for classification and localization.

[0030] Specifically, see Figure 8 The detection head consists of four 3×3DBB blocks, one 1×1Conv_GN block, a large detection head, a medium detection head, and a small detection head. The fourth, fifth, and sixth CSP_MSCB feature extraction modules output to the first, second, and third 3×3DBB blocks, respectively. The first, second, and third 3×3DBB blocks output to the fourth 3×3DBB block, which outputs to the 1×1Conv_GN block. The 1×1Conv_GN block outputs to the large, medium, and small detection heads. The bounding box regression loss (Bbox.Loss) of the large, medium, and small detection heads is adjusted using a scaling factor. The detection head loss includes Bbox.Loss (bounding box regression loss) and Cls.Loss.

[0031] (4) The model is optimized using Inner-GIoU loss; the Inner-GIoU definition formula is as follows:

[0032]

[0033]

[0034]

[0035] .

[0036] Where gt represents the true bounding box, and r, l, b, and t represent the four sides of the left, right, bottom, and top rectangles, respectively; the width and height of the true bounding box are represented by... and This indicates that the width and height of the prediction box are represented by... and The value is indicated by "ratio"; "ratio" represents an auxiliary factor, which can be adjusted according to different tasks. This represents the area of ​​the minimum bounding rectangle shared by the ground truth bounding box and the predicted bounding box. This represents the union of the ground truth boxes and the predicted boxes.

[0037] S3: Use samples to train and test the MDMB-YOLO model to obtain the optimal model.

[0038] S4: Acquire images of the LCD screen and perform detection using the optimal model.

[0039] Steps S3 and S4 are consistent with the prior art, and detailed descriptions are omitted in this embodiment.

[0040] Example 2 Example 2 explains the principles and advantages of each improvement as follows: See Figure 2 Based on MDMB-YOLO in Example 1, this patent combines MDFM (Multiple Differential Fusion Model) differential convolutions for feature extraction in the MDMB-YOLO backbone network, while retaining the traditional CBS convolution module. Unlike the original YOLO11 architecture, the MDFM module designed in this patent captures features from multiple directions more effectively than standard convolutions, thereby enhancing the detection capability for defects of different shapes. In the neck structure, this patent introduces the MFEFPN (Multi-Branch Fusion Efficient Feature Pyramid Network) design. The integration of these components improves the network's local perception capability and multi-scale feature fusion, making the detection of different types of defects more accurate. Feature extraction capability is enhanced by using MDFM multiple differential convolutions in the backbone. Then, the neck structure is redesigned to introduce shallower information to enrich feature representation capabilities. Finally, ASLD-X is used to represent three detection heads of different sizes to enhance the overall detection capability of the network.

[0041] Specifically, for MDFM (Multi-Differential Fusion Model), point defect features are weak after the LCD screen is lit, leading to missed detections during the detection process. As network depth increases, feature information suffers some loss during layer-by-layer transmission, resulting in a gradual reduction in the semantic information carried by point defect features in high-level feature maps. Furthermore, the limitation of the receptive field makes it difficult for the network to effectively distinguish small objects from the background. Therefore, to address this issue, an MDFM module is constructed, such as... Figure 3As shown, the MDFM module aims to enhance the feature extraction capability of small objects from two aspects. First, from the perspective of increasing feature diversity, it adopts a multi-branch convolutional structure to extract various semantic information, thereby enhancing the expressive power of the feature map. Through the multi-branch convolutional operation of the residual structure, these branch structures can capture more diverse features, including detailed information and background differences, thus effectively improving the recognition accuracy of small objects. Second, from the perspective of enhancing the receptive field, the internal MDCM module effectively expands the receptive field by introducing various types of convolutional operations (such as central difference convolution, horizontal difference convolution, vertical difference convolution, and angular difference convolution). These difference convolutional operations not only enhance the sensitivity to local features, but also further improve the expressive power of subtle changes through the design of different convolutional kernels, making the detection of small objects more accurate and effectively suppressing background interference. The overall structure diagram of the MDCM module is shown below. Figure 4 As shown.

[0042] like Figure 4 As shown, MDCM comprises five processes, from top to bottom: central differencing, angular differencing, horizontal differencing, vertical differencing, and additional convolution. The original weights are updated using these five methods, and the updated weights are summed with the biases to form a weight matrix containing richer information. In the diagram, red letters represent the updated weights, and the final red boxes represent the areas where the updated weights are more focused.

[0043] It inherits some features from the original C3k2 module, while using differential convolution to alleviate the problem that ordinary convolution cannot capture edge contour information well. The specific weight update method is as follows:

[0044]

[0045]

[0046]

[0047]

[0048] .

[0049] It is worth noting that the size of the receptive field has a significant impact on object detection performance. A larger receptive field helps the model capture more contextual information, thereby improving detection accuracy. Compared to the feature extraction module in the baseline model, MDFM enhances the model's receptive field, enabling it to obtain more comprehensive target feature information when handling complex scenes, thus improving detection performance. Figure 5This demonstrates the advantages of MDFM over the feature extraction module of the baseline model in terms of receptive field.

[0050] Specifically, for MFEFPN (Multi-Branch Fusion Efficient Feature Pyramid Network), defects in LCD screen detection tasks exhibit high irregularity. First, defects are randomly distributed in the image, potentially appearing in different areas of the screen with unpredictable locations. This dispersion increases the difficulty of detection, especially in noisy backgrounds. Second, defects vary in size and shape, ranging from tiny point defects to large, irregular regional defects. Due to the large differences in defect scale, traditional detection frameworks struggle to handle them uniformly, leading to the omission of small-scale defects or false detection of large-scale defects. Furthermore, the lack of consistency in visual features such as morphology, texture, and color of defects increases the difficulty of feature extraction and classification. To address this issue, a multi-branch efficient feature pyramid structure (MFEFPN) is proposed. This structure uses a multi-branch design to specifically extract features of defects at different scales, ensuring effective handling of both small and large-scale defects. Information is shared among the branches, enhancing the model's adaptability and robustness in multi-scale scenarios. Meanwhile, the improved feature pyramid structure utilizes hierarchical feature fusion to enhance the connection between low-level details and high-level semantic information, thereby improving the detection capability for small-scale defects. By introducing a dynamic weighting mechanism, the model can adaptively adjust the importance of features at different levels, thus improving the recognition accuracy for diverse defects. Overall, the MFEFPN structure effectively overcomes the limitations of traditional methods in handling irregularly distributed, multi-scale, and diverse defects, achieving high-precision detection in complex scenes.

[0051] MFEFPN mainly consists of four parts: ten ordinary convolutions (including three 1×1 Convs and seven 3×3 Convs), a Fusion feature fusion module, an EUCB upsampling module, and a CSP_MSCB feature extraction module. First, convolutional operations allow the model to process features independently at different scales. To address the limitations of small-scale points, features are extracted from near the input layer, avoiding information loss in deeper network layers. For feature fusion, a Fusion feature fusion module is used, employing a Bidirectional Feature Pyramid Network (BiFPN) approach. This module weights the feature maps of different inputs with learnable weights, allowing the contribution of each feature map to adaptively adjust. This weighted fusion method improves the feature fusion effect, avoids information loss, and enhances the model's adaptability and robustness. Since feature map alignment is required before fusion, upsampling deep feature maps and restoring their spatial dimensions is a challenge. Traditional convolutional upsampling methods typically use interpolation techniques to directly enlarge the feature maps, which can lead to increased computation. To reduce computational burden while restoring the spatial dimensions of the feature maps, the Fusion feature fusion module combines upsampling and depthwise separable convolution. First, EUCB restores the spatial dimensions of the feature maps through a 2x scaling upsampling operation, then applies a 3×3 depthwise separable convolution for further processing, and finally uses a 1×1 convolution to adjust the number of channels to match the feature maps in the next stage. In this way, the EUCB module not only effectively reduces computation but also improves the upsampling efficiency of feature maps, making it particularly suitable for real-time LCD screen defect detection. The structure of the EUCB module is as follows: Figure 6 As shown, this structure comprises four processes: double upsampling, depthwise convolution, channel shuffling, and pointwise convolution.

[0052] Assume the input feature map is , in Indicates the number of channels. and Let represent the height and width of the feature map, respectively. The specific calculation process is shown in formulas (1)-(4). First, the EUCB module performs an Upsample operation on the input feature map. Upsampling is performed with a scale factor of 2, which means the spatial resolution of the feature map is increased from... Increase to This operation expands the pixels in the feature map through interpolation, thereby providing more spatial information for subsequent convolution operations.

[0053] (1).

[0054] Next, the feature map is further processed by depthwise convolution. Depthwise convolution performs convolution operation on each channel independently. Compared with standard convolution, the number of convolution kernels in depthwise convolution is greatly reduced, thus reducing the amount of computation and the number of parameters.

[0055] (2).

[0056] Next, the EUCB module performs a channel rearrangement operation. The purpose of this operation is to shuffle the channel information after the convolution operation to enhance the information flow between different channels and improve the expressive power of the features. Assume the feature map has channels. Divide it into Groups, each group contains Each channel. The channel rearrangement operation first reshapes the feature map into... Then, the shape is reshaped by rearranging the channel dimensions. Finally, restore it to To match the shape of the feature map.

[0057] (3).

[0058] The feature map after channel rearrangement is processed by 1x1 convolution to adjust the number of channels, and the final feature map Y is output.

[0059] (4).

[0060] Inspired by the BIFPN structure, MFEFPN fuses multi-scale features through adaptive weights. Given the effectiveness of multi-branch convolutions in object detection networks such as ConvNeXt, Detectors, and EfficientDet, it introduces multi-scale deep convolutions with different kernel sizes to process feature maps in parallel, thereby extracting important information from multiple receptive fields and capturing contextual details. Figure 7 As shown, a CSP_MSCB module was designed, inspired by CSP (Cross Stage Partial Network) and multi-branch convolution. By splitting the input feature map, one part is directly retained, while the other part is fused with the original feature map through multi-branch convolution, reducing redundant computation while maintaining feature diversity and improving computational efficiency.

[0061] like Figure 7 As shown, this structure combines CSP and the multi-branch multi-kernel idea, dividing the original feature map into two parts. One part is used for feature extraction, and the other part is directly merged with the extracted features. This enhances the feature extraction capability while reducing the computational load. In the feature extraction part, three convolution kernels of different sizes, 1*1, 3*3, and 5*5, are used to capture features.

[0062] CSP_MSCB internally includes a Multi-Scale Deep Convolution (MSDC) module. In addition to conventional residual connections, convolutions, and normalization operations, this module includes multiple parallel deep convolutions, each capable of capturing contextual information at different spatial scales. To address LCD screen defects, this module enhances the ability to perceive defects at various scales through parallel processing of convolutional kernels of different sizes, thereby improving the model's robustness in various environments. Through a multi-branch convolution design, the output features retain important information and reduce redundancy by fusing the results of different convolutions. The fused features undergo channel shuffling to further enhance the interaction between features. The formula for Multi-Branch Convolution MSDC is as follows: .

[0063] In the equation, This represents the kernel size, calculated using three different kernels: 1, 3, and 5. Represents input, Represents depthwise convolution. This represents the activation function; in the experiment, ReLU6 was used to introduce nonlinear computation.

[0064] For ASLD (Adaptive Shared Lightweight Detection Head), the YOLO model typically uses a separate detection head when detecting feature maps of multiple different scales. This leads to redundant model parameters, and the decoupled head design also introduces a significant computational load. Statistics show that in previous generations of YOLO models, the detection head accounted for approximately 25-30% of the total computation. Therefore, redesigning the detection head to balance computational load and accuracy is particularly important.

[0065] like Figure 8 As shown in (a), YOLO11 comprises three detection heads, used for identifying small, medium, and large targets, respectively. Each detection head employs a widely adopted decoupled head design, separating the classification and localization tasks. In this design, one head handles target recognition, typically evaluated using bounding box regression loss, which includes CIOU loss and distributed focus loss (DFL). The other head handles the classification task, optimized using a classification loss function (CLSLoss). The classification head utilizes a depthwise separable convolution design, enabling the network to process each channel individually, thus effectively reducing computational cost.

[0066] To reduce the computational burden on the detection head, an adaptive lightweight shared convolutional head (such as...) was designed. Figure 8 (b) shows that while feature maps at different scales contain different semantic information, they are similar in scale. Therefore, features are first extracted using 3×3 convolutions, then the feature map parameters of the three levels are shared using shared convolutional blocks, followed by adjusting the number of channels using 1×1 convolutions and performing group normalization to improve training stability and avoid the limitations of batch normalization. Finally, the feature maps are input into three detection heads for classification and localization. This design reduces computation and improves efficiency by sharing parameters, while also enhancing model robustness. With the help of parameter sharing, the model can still efficiently perform object detection tasks even with limited computational resources. In addition, channel alignment optimizes cross-scale information transmission, avoids information asymmetry, and enhances the adaptability of the detection heads.

[0067] The DBB module optimizes computation through reparameterization. During training, the module computes features through different paths (such as convolution and average pooling) and fuses them during forward propagation. During inference, reparameterization merges the convolutional kernels and biases from different paths into an equivalent convolutional layer, reducing computation and improving inference speed. In this way, the DBB module utilizes more feature branches during training and improves inference efficiency by merging convolutional operations during inference, thus optimizing model deployment performance. Furthermore, due to the use of shared convolutions, a scaling factor is introduced to address the issue of the detection head detecting different target scales, allowing features after the localization layer to be scaled individually according to each feature level.

[0068] like Figure 8 As shown in the figure, the DBB diversified branch block is a reparameterization operation. The part with shared parameters is in the red box. Conv_GN represents the convolution and group normalization operations. The light gray Scale represents the scale factor, which can be adjusted to address the problem of inconsistent target scales detected by each detector head.

[0069] For loss functions, bounding box regression methods based on the IoU (Intersection over Union) metric accelerate convergence during training by introducing an additional loss term, but often overlook some inherent limitations of IoU loss itself. While IoU, as a standard for measuring the overlap between predicted and ground truth bounding boxes, is widely used in many object detection tasks, in bounding box regression, especially for small or highly overlapping targets, IoU loss often fails to adequately express the fine differences in bounding box regression, thus affecting training stability and final performance. In practical regression processes, distinguishing different types of regression samples and combining multi-scale auxiliary bounding boxes to calculate loss has been proven to effectively accelerate the bounding box regression process. This is because, in the early stages of training, large-scale bounding boxes often correspond to high-IoU samples, while low-IoU samples typically appear in small-scale targets. By selecting auxiliary bounding boxes of appropriate scales to calculate loss, the regression accuracy of high-IoU samples can be improved, while avoiding training instability caused by low-IoU samples. Traditional IoU loss may lead to difficulty in model convergence due to excessive overlap between small targets. To address this issue, this paper introduces the concept of Inner-IoU loss and, based on this, adopts the Inner-GIoU method.

[0070] In the context of LCD screen defect detection, defects on LCD screens typically manifest as small, densely packed abnormal areas, which traditional target detection methods may struggle to effectively distinguish. Introducing Inner-GIoU loss and optimizing the calculation of auxiliary bounding boxes at different scales makes the regression of small defects on LCD screens more accurate, thereby improving the accuracy of defect detection. Especially when detecting minute flaws on LCD screens, Inner-GIoU can effectively accelerate convergence while reducing missed and false detections, thus improving the overall performance of the detection system. The Inner-GIoU definition formula is as follows:

[0071]

[0072]

[0073]

[0074] .

[0075] Model Validation I. Dataset and Experimental Environment The experimental data was provided by a smart LCD screen manufacturing company in Jiangxi Province. To ensure data authenticity, all data was collected on-site at the factory by our personnel, comprising 2498 LCD screen sample images. Since the purpose of this patent is to include as many LCD screen defect patterns as possible, the data consists entirely of defect samples. Based on different patterns, the defects are divided into five categories: bubble defects, dot defects, line defects, leakage defects, and multi-pixel fault defects. This data was captured using a high-precision industrial camera, covering most defect situations that may occur in actual production, and is helpful for the development and research of automatic LCD screen inspection equipment systems.

[0076] To meet experimental requirements, the dataset was divided into three parts: 1748 images were used as the training set, 249 images as the validation set, and 501 images as the test set. Details of the dataset are shown in Table 1. Figure 9 As shown.

[0077] Table 1 Dataset Structure

[0078] To ensure the consistency of the experiments, all experiments were conducted on the same server. The experiments were carried out in an environment of Ubuntu 22.04.4, Python 3.10.14 and PyTorch 2.2.2. Other relevant hardware configurations and model parameters are shown in Table 2.

[0079] Table 2 Experimental Environment Configuration

[0080] II. Comparative Experiments Between Different Models To evaluate the model's performance, a series of comparative experiments were designed to compare the proposed model with current mainstream target detection models. These experiments aimed to demonstrate the effectiveness of the MDMB-YOLO model in LCD screen defect detection. The model was compared with other mainstream detection models on six key metrics (precision P, recall R, mAP@50, mAP@50-95, params, and FLOPs). The results are shown in Table 3. Figure 10 As shown. All experiments were performed on the same dataset, and the hardware environment and training protocol were kept consistent to ensure fairness.

[0081] Table 3 Comparative Experiments

[0082] In terms of precision and recall, the model in this patent outperforms most mainstream models, demonstrating excellent performance in both precision and recall, with scores of 0.859 and 0.832 respectively. Compared to other models, especially the baseline models YOLO11 (P=0.815, R=0.799) and YOLOv3-tiny (P=0.839, R=0.752), it shows significant improvements. This means that MDMB-YOLO can not only detect targets more accurately but also capture all possible targets better, achieving a good balance between target completeness and accuracy. In terms of Map50 and Map50-95, MDMB-YOLO's mAP50 is 0.852, second only to Dino (0.896) and Detr (0.841) based on the Transformer architecture. However, its mAP50-95 (0.49) is leading among most models, surpassing most, especially models like Faster-Rcnn (0.408) and YOLOv5-n (0.46). Finally, in terms of parameter count and computational complexity, MDMB-YOLO's Params are 1.98M, the smallest among all models, meaning it is more lightweight and suitable for deployment in resource-constrained environments. In contrast, models like Cascade (69.164M) and Dino (47.548M) have significantly increased parameter counts, not only increasing storage requirements but also potentially causing a decrease in inference speed. Meanwhile, MDMB-YOLO's computational requirements are very low at 5.8G FLOPS. Compared to DiNO and other high-precision models, MDMB-YOLO strikes a balance between accuracy and computational efficiency, significantly reducing the model's hardware requirements without sacrificing too much accuracy. Therefore, MDMB-YOLO exhibits excellent performance and efficiency, making it more suitable for deployment in practical applications such as LCD screen inspection.

[0083] III. Ablation Experiments Between Modules To investigate the effectiveness of individual structures, ablation experiments were designed to evaluate how improvements to individual structures enhance model performance and how their interactions produce better results. YOLO11 was used as the baseline model in this experiment, and the hyperparameters and hardware / software environment were identical throughout. '√' indicates that this module was added to this group of experiments. See Table 4 for details.

[0084] Table 4 Ablation Experiment

[0085] By integrating prior information into the model using the MDFM structure, map50 and map50-95 can be effectively improved without adding extra parameters or computational burden. MFEFPN, through its multi-branch design, utilizes feature fusion at different levels to enhance the connection between low-level details and high-level semantic information. This allows the model to maintain efficiency while avoiding the loss of low-level features, thereby improving the detection capability of small-scale defects. The dynamic weighting mechanism can adaptively adjust the importance of features at different levels, further improving the recognition accuracy for diverse defect styles and sizes. Simultaneously, MFEFPN reduces the number of model parameters due to its efficient upsampling module and the presence of deep convolution in the feature extraction part. The ALSCD detection head significantly improves the model's accuracy through shared detection head parameters and reparameterization operations, ensuring the model can more accurately detect target defects. Feature maps at multiple scales can share parameters at the same level, thereby reducing the number of parameters and improving computational efficiency, enhancing the model's robustness while maintaining detection accuracy. Combining the three modules reveals improvements in precision, recall, map50, and map50-95 metrics. Due to the lightweight design, the number of parameters and computational cost are significantly reduced compared to the baseline model. Finally, a more precise inner-Giou loss function is introduced for localization, enabling the model to identify objects more accurately without increasing computational cost, thus effectively improving the model's detection performance.

[0086] IV. Loss Function Comparison Experiment To further explore the impact of the loss function on the final detection results, a series of comparative experiments related to the loss function were designed, as shown in the table below. The loss function in each group of experiments introduced the concept of inner IOU, and the optimal loss function was verified through experimental results.

[0087] Table 5. Comparison of Loss Functions

[0088] In the experiments, various IoU loss functions incorporating the inner mechanism were compared, including Inner-GIOU, Inner-DIOU, Inner-EIOU, Inner-SIOU, and Inner-PIOU, aiming to identify the most comprehensive loss function in object detection. First, the Inner-GIOU loss function performed exceptionally well on several key metrics, including P, R, mAP50, and Bubble. Compared to other IoU variants, Inner-GIOU achieved higher scores on these metrics, particularly excelling in complex scenes, dense targets, and small target detection tasks. By introducing constraints on the inner region, Inner-GIOU can more accurately optimize the regression of the bounding box, effectively reducing the blank area in IoU calculation and improving the localization accuracy of small targets, thus achieving optimal performance on multiple evaluation metrics. Therefore, Inner-GIOU was confirmed as the most comprehensive and effective loss function in this experiment. Second, although Inner-DIOU and Inner-EIOU performed well on some metrics, their main advantages are reflected in specific scenarios. Inner-DIOU significantly improves target localization accuracy when dealing with large distances between targets, making it particularly suitable for detecting large-scale targets or targets with significant positional deviations. However, Inner-DIOU performs slightly worse when dealing with small or densely packed targets, and is susceptible to target overlap, leading to decreased bounding box regression accuracy. Inner-EIOU shows some advantages in complex scenes, especially when there is significant overlap between targets, where Inner-EIOU performs exceptionally well. However, it fails to provide sufficient improvement in simpler scenes. While Inner-PIOU and Inner-PIOU2 perform well in large target detection, their strengths are primarily in detecting regularly shaped objects. For small targets or irregularly shaped objects, the Inner-PIOU series falls short. This makes Inner-PIOU less of the optimal choice for detecting diverse targets. Finally, while Inner-SIOU demonstrates stable overall performance, it falls slightly short of Inner-GIOU in several complex scenarios, particularly in evaluation targets such as Bubble and Multi-pixel, where it fails to achieve comparable results. Inner-SIOU is more effective for certain specific targets, but overall, its performance improvement is far less significant than that of Inner-GIOU.

[0089] In summary, while other Inner-IoU variants perform well in certain specific tasks, such as large object detection or scene-specific detection, Inner-GIOU offers more balanced and superior performance overall, particularly in detecting small objects, densely packed objects, and complex scenes, demonstrating stronger robustness and accuracy. Therefore, Inner-GIOU is undoubtedly the best-performing loss function in this experiment, exhibiting the best overall performance. V. Results and Discussion After training and testing on an LCD screen defect dataset, MDMB-YOLO achieved a map50 score of 85.2, demonstrating high accuracy compared to other models of similar size. Furthermore, compared to benchmark models, the model exhibits a 23.3% reduction in parameters and an 8% reduction in computational complexity, enabling deployment on edge devices and effectively improving real-time performance. In practical testing, MDMB-YOLO achieved an FPS of 150, sufficient for automated defect detection processes in real-world applications. Figure 11 This paper presents a single-channel visualization of the output feature maps of MDMB-YOLO and YOLO11 at different network layers and with three detection heads. The results show that, compared to the original YOLO11 backbone structure, the network using the MDFM module performs significantly better in extracting defect feature information. Although some feature information is lost with increasing network depth, the MDFM module maximizes the preservation of the target's contour and detail information, ensuring the effective transmission of key features in the deep network. Further analysis of the final output feature map reveals that, with the MFEFPN feature pyramid structure and ALSD, the output feature map of the detection head is clearer than that of YOLO11, and the boundaries of the target object are more distinct, thus significantly improving the localization accuracy. MFEFPN further enhances the fusion of information, and the introduction of ALSD enhances the model's sensitivity to targets at different scales, enabling the network to more accurately locate objects and edge regions, effectively improving the overall detection performance of the model. In conclusion, compared to YOLO11, MDMB-YOLO effectively improves the accuracy of defect detection, especially in target contour preservation and detail capture. Figure 11 In the middle, from left to right, the first two images compare the network layers 7 and 9 with and without the MDFM module; the last three images compare the output of the original three detector heads with the effect of the ALSD detector head. Figure 12As can be seen, compared with YOLO11, MDMB-YOLO exhibits stronger adaptability and higher accuracy under various backgrounds and defect types. It can accurately identify defects of different scales, and its performance in detecting small defects is particularly outstanding in complex backgrounds. Furthermore, MDMB-YOLO can effectively distinguish overlapping defects, avoiding missed or false detections, highlighting its effectiveness and robustness in complex scenarios.

Claims

1. A method for detecting defects in liquid crystal modules using multi-difference fusion and multi-branch pyramid networks, characterized in that, The method includes: S1: Obtain samples and classify them according to defects; S2: Construct the MDMB-YOLO model based on the YOLO11 model, with the following adjustments: (1) Replace the C3k2 modules in layers 6 and 8 of the backbone network with MDFM modules. The C3k2 modules in layers 2, 4, 6, and 10, and the C2PSA modules in layers 10 are all output to the neck network. (2) The neck network adopts a multi-branch fusion efficient feature pyramid network, which includes 6 Fusion feature fusion modules, 2 EUCB upsampling modules, 6 CSP_MSCB feature extraction modules, 3 1×1 Conv and 7 3×3 Conv; (3) The detection head extracts features from three levels through three 3×3 convolutions, then shares the feature map parameters of the three levels through a shared convolution block, then adjusts the number of channels and performs group normalization using a 1×1 convolution, and finally inputs the feature maps into the three detection heads for classification and localization. (4) The model is optimized using Inner-GIoU loss; S3: Use samples to train and test the MDMB-YOLO model to obtain the optimal model; S4: Acquire images of the LCD screen and perform detection using the optimal model.

2. The method according to claim 1, characterized in that, The MDFM module is an adjustment based on the C3k2 module, replacing the C3k unit with the MDCM_bottleneck unit. The MDCM_bottleneck unit is an adjustment based on the C3k unit, replacing the bottleneck with two cascaded MDCMs. The MDCM resets and updates the weights through center difference, angular difference, horizontal difference, vertical difference, and additional convolution, and then adds the updated weights and biases to obtain a weight matrix containing richer information.

3. The method according to claim 2, characterized in that, The calculation process of the MDCM is as follows: in, express The weighting coefficient of location, Indicates after reset and update The weighting coefficient of location, , , as well as These represent the weight coefficients after the central difference, angular difference, horizontal difference, and vertical difference operations, respectively. This represents the weight coefficients after one convolution following initialization. It is an adjustable parameter used to control the size of the convolution kernel weights. This represents the original weight coefficients output from the previous layer. This indicates the difference direction of the angular difference convolution.

4. The method according to claim 1, characterized in that, The structure of the neck network is as follows: The output of the C3k2 module in the second layer is sent to the first 3×3 Conv; The output of the C3k2 module in layer 4 is sent to the first 1×1 Conv; The MDFM module of layer 6 outputs to the second 1×1 Conv; The output of the C2PSA module at layer 10 is sent to the third 1×1 Conv; The first 1×1 Conv is split into two outputs. One output is sent to the first Fusion feature fusion module and fused with the outputs of the first 3×3 Conv and the first EUCB upsampling module. The other output is sent to the second 3×3 Conv. The second 1×1 Conv is split into two outputs: one output is sent to the second Fusion feature fusion module and fused with the outputs of the second 3×3 Conv and the second EUCB upsampling module; the other output is sent to the third 3×3 Conv. The first Fusion feature fusion module outputs to the first CSP_MSCB feature extraction module; The output of the second Fusion feature fusion module is sent to the second CSP_MSCB feature extraction module; The third Fusion feature fusion module fuses the outputs of the third 1×1 Conv and the third 3×3 Conv and outputs them to the third CSP_MSCB feature extraction module; The first CSP_MSCB feature extraction module outputs two paths, which are respectively output to the fourth Fusion feature fusion module and the fourth 3×3 Conv; The second CSP_MSCB feature extraction module outputs three paths, which are respectively output to the first EUCB upsampling module, the fifth Fusion feature fusion module, and the fifth 3×3Conv; The third CSP_MSCB feature extraction module outputs two paths, which are respectively output to the second EUCB upsampling module and the sixth Fusion feature fusion module; The fourth Fusion feature fusion module fuses the outputs of the first CSP_MSCB feature extraction module and the first EUCB upsampling module and outputs them to the fourth CSP_MSCB feature extraction module. The fifth Fusion feature fusion module fuses the outputs of the second CSP_MSCB feature extraction module, the second EUCB upsampling module, the fourth 3×3Conv and the sixth 3×3Conv and outputs them in two ways: one output is sent to the fifth CSP_MSCB feature extraction module and the other output is sent to the fifth 3×3Conv. The sixth Fusion feature fusion module fuses the outputs of the third CSP_MSCB feature extraction module, the fifth 3×3Conv and the seventh 3×3Conv and outputs them to the sixth CSP_MSCB feature extraction module; The fourth CSP_MSCB feature extraction module has two outputs: one outputs to the small detection head, and the other outputs to the sixth 3×3 Conv. The fifth CSP_MSCB feature extraction module has two outputs: one outputs to the middle detection head, and the other outputs to the seventh 3×3 Conv. The sixth CSP_MSCB feature extraction module outputs to the large detection head.

5. The method according to claim 1, characterized in that, The CSP_MSCB feature extraction module divides the original feature map into two parts: one part is used for feature extraction, and the other part is directly merged with the extracted features. In the feature extraction part, three convolution kernels of different sizes, 1*1, 3*3, and 5*5, are used to capture features.

6. The method according to claim 5, characterized in that, The CSP_MSCB feature extraction module is an adjustment based on the CSP concept, replacing C3k2 with MSDC units; The CSP_MSCB feature extraction module includes a Split unit, a CBR unit, an MSDC unit, a Conv+BN unit, and a Concatenation unit. The Split unit outputs two paths: one to the CBR unit and the other to the Concatenation unit. The CBR, MSDC, and Conv+BN units process the features sequentially, while the Concatenation unit merges the features output from the Split and Conv+BN units. The MSDC unit includes DWC1*1, DWC3*3, DWC5*5, summation, and a Channel. The CBR unit consists of a Shuffe, three BNs, and three ReLU6s. It outputs four channels to DWC1*1, DWC3*3, DWC5*5, and a summation unit. DWC1*1, the first BN, and the first ReLU6 process the features sequentially and output to the summation unit. Similarly, DWC3*3, the second BN, and the second ReLU6 process the features sequentially and output to the summation unit. DWC5*5, the third BN, and the third ReLU6 process the features sequentially and output to the summation unit. The summation is then output to the Channel Shuffe, which in turn outputs to the Conv+BN unit.

7. The method according to claim 6, characterized in that, The algorithm for the MSDC unit is as follows: in, Represents the kernel size. Represents input, Represents depthwise convolution. This represents the activation function, which undergoes nonlinear computation using ReLU6.

8. The method according to claim 4, characterized in that, The detection head includes a 1×1 Conv_GN, a large detection head, a medium detection head, a small detection head, and four 3×3 DBBs. The fourth CSP_MSCB feature extraction module, the fifth CSP_MSCB feature extraction module, and the sixth CSP_MSCB feature extraction module output to the first 3×3 DBB, the second 3×3 DBB, and the third 3×3 DBB, respectively. The first 3×3 DBB, the second 3×3 DBB, and the third 3×3 DBB output to the fourth 3×3 DBB. The fourth 3×3 DBB outputs to the 1×1 Conv_GN. The 1×1 Conv_GN outputs to the large, medium, and small detection heads. The bounding box regression loss Bbox.Loss of the large, medium, and small detection heads is adjusted by a scaling factor. The detection head loss includes Bbox.Loss and Cls.Loss.

9. The method according to claim 1, characterized in that, The Inner-GIoU definition formula is as follows: Where gt represents the true bounding box, and r, l, b, and t represent the four sides of the left, right, bottom, and top rectangles, respectively; the width and height of the true bounding box are represented by... and This indicates that the width and height of the prediction box are represented by... and The ratio represents the auxiliary factor. This represents the area of ​​the minimum bounding rectangle shared by the ground truth bounding box and the predicted bounding box. This represents the union of the ground truth boxes and the predicted boxes.

10. The method according to claim 1, characterized in that, In step S1, the sample is an actual sample obtained by photographing the defective LCD screen with a high-precision industrial camera; the defects are divided into five categories, namely bubble defects, point defects, line defects, leakage defects and multi-pixel fault defects.

Images