Method and system for defect detection of electronic display panels

Abstract

The application discloses a kind of defect detection method and system of electronic display panel, applied to computer vision technical field, method includes: the image data of display panel to be measured is obtained;Image data is input to the neural network model trained in advance, the position and type of the defect area of display panel are obtained;Wherein, neural network model includes main network, neck network and detection head, main network is used to carry out feature extraction to image data, obtains multiple initial feature maps;Neck network is used to carry out feature interaction to multiple initial feature maps, obtains the target feature map corresponding to multiple initial feature maps respectively;Detection head is used to carry out target detection based on the target feature map corresponding to multiple initial feature maps respectively, obtains the position and type of the defect area of display panel.The application can improve the defect detection precision of display panel.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a defect detection method and system for electronic display panels. Background Technology

[0002] Display panels are core components in electronic devices used to display images, text, or video, visualizing information by controlling optical properties (such as brightness and color) through electrical signals. With the continuous advancement of display technology and the emergence of complex, high-end display panels, the probability of surface defects in display panels has increased. Therefore, detecting surface defects in display panels is crucial. In related technologies, existing image sample sets are used to train neural network models, resulting in trained detection models used to detect surface defects in display panels. However, surface defects in display panels typically include various types such as point defects, line defects, Mura cluster defects, and bubble defects, with complex and diverse feature information. Current detection models are often single-stage detection models such as YOLO (You Only Look Once) and SSD (Single Shot MultiBox Detector) or two-stage detection models such as Faster R-CNN, which have limited feature extraction capabilities, leading to low accuracy in display panel defect detection. Summary of the Invention

[0003] This application provides a defect detection method and system for electronic display panels, which improves the defect detection accuracy of display panels.

[0004] On one hand, embodiments of this application provide a defect detection method for an electronic display panel, including the following steps:

[0005] Acquire image data of the display panel under test;

[0006] The image data is input into a pre-trained neural network model to obtain the location and type of the defect area of ​​the display panel;

[0007] The neural network model includes:

[0008] The backbone network is used to extract features from the image data to obtain multiple initial feature maps;

[0009] A neck network is used to perform feature interaction on multiple initial feature maps to obtain target feature maps corresponding to the multiple initial feature maps respectively;

[0010] The detection head is used to perform target detection based on the target feature maps corresponding to the multiple initial feature maps, and to obtain the location and type of the defect area of ​​the display panel.

[0011] On the other hand, embodiments of this application provide a defect detection system for an electronic display panel, comprising:

[0012] The acquisition module is used to acquire image data of the display panel under test;

[0013] The detection module is equipped with a pre-trained neural network model. The detection module is used to input the image data into the pre-trained neural network model to obtain the location and type of the defect area of ​​the display panel.

[0014] The neural network model includes:

[0015] The backbone network is used to extract features from the image data to obtain multiple initial feature maps;

[0016] A neck network is used to perform feature interaction on multiple initial feature maps to obtain target feature maps corresponding to the multiple initial feature maps respectively;

[0017] The detection head is used to perform target detection based on the target feature maps corresponding to the multiple initial feature maps, and to obtain the location and type of the defect area of ​​the display panel.

[0018] According to the defect detection method and system for electronic display panels provided in this application, firstly, image data of the display panel to be tested is acquired; then, the image data is input into a pre-trained neural network model to obtain the location and type of defective regions of the display panel. The neural network model includes a backbone network, a neck network, and a detection head. The backbone network is used to extract features from the image data to obtain multiple initial feature maps; the neck network is used to perform feature interaction between the multiple initial feature maps to obtain target feature maps corresponding to each initial feature map; the detection head is used to perform target detection based on the target feature maps corresponding to the multiple initial feature maps to obtain the location and type of defective regions of the display panel. According to the technical solution of this application, the complex defect features of the display panel to be tested are fully explored through both shallow and deep dimensions, improving the neural network model's ability to extract complex defect features, thereby effectively improving the defect detection accuracy of the display panel. Attached Figure Description

[0019] Figure 1 This is a flowchart of a defect detection method for an electronic display panel provided in this application;

[0020] Figure 2 This is a structural diagram of the neural network model provided in this application;

[0021] Figure 3 This is a structural diagram of the feature extraction module provided in this application;

[0022] Figure 4This is a structural diagram of the attention backbone module provided in this application;

[0023] Figure 5 This is a structural diagram of the multi-scale pooling structure provided in this application;

[0024] Figure 6 This is an example image of the sample image provided in this application;

[0025] Figure 7 This is an example diagram illustrating the effect of the frequency domain processing provided in this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0027] The present application will be further described below with reference to the accompanying drawings and specific embodiments. The described embodiments should not be considered as limitations on the present application, and all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of the present application.

[0028] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0030] In view of the shortcomings of related technologies, this application provides a method and system for detecting defects in electronic display panels, aiming to effectively improve the accuracy of defect detection in display panels.

[0031] First, the defect detection method for an electronic display panel provided in this application will be described in detail below with reference to the accompanying drawings.

[0032] This application provides a defect detection method for electronic display panels, which can be applied to terminals, servers, or software running on either terminal or server. Terminals can be tablets, laptops, desktop computers, etc., but are not limited to these. Servers can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Furthermore, a server can be a node server in a blockchain network, but is not limited to these. Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms.

[0033] Reference Figure 1 and Figure 2 The defect detection method for the electronic display panel may include the following steps S101-S102:

[0034] S101, acquire image data of the display panel under test;

[0035] S102, the image data is input into a pre-trained neural network model to obtain the location and type of the defect area of ​​the display panel; wherein, the neural network model includes a backbone network, a neck network and a detection head, the backbone network is used to extract features from the image data to obtain multiple initial feature maps; the neck network is used to perform feature interaction between the multiple initial feature maps to obtain target feature maps corresponding to the multiple initial feature maps respectively; the detection head is used to perform target detection based on the target feature maps corresponding to the multiple initial feature maps respectively to obtain the location and type of the defect area of ​​the display panel.

[0036] In this embodiment, firstly, an image of the display panel under test is acquired to obtain image data of the display panel under test, which is the image under test. The type of display panel under test is an electronic device display panel. Furthermore, the application direction of the display panel under test can be flexibly set according to actual conditions; for example, it can be an automotive display panel, a mobile device display panel, etc., but it is not limited to these.

[0037] Then, the image data of the display panel under test is input into a pre-trained neural network model for defect detection. This neural network model is trained using multiple preset display panel sample images and the corresponding label information for each image. The label information for each sample image can include the location and type of defective regions. The defect type can be any one of point defects, line defects, Mura cluster defects, or bubble defects. Mura cluster defects are localized, blocky visual imperfections in the display panel caused by uneven brightness or color, manifesting as dark spots, bright spots, or colored spots. By using this neural network model for defect detection, the location and type of defective regions in the display panel can be determined.

[0038] Specifically, such as Figure 2 As shown, the neural network model includes a backbone network, a neck network, and a head. Image data of the display panel under test is input into the neural network model. First, the backbone network extracts features from the image data, obtaining multiple initial feature maps. This aims to initially uncover shallow defect features of the display panel at a shallow level. Next, the neck network interacts with these initial feature maps to obtain target feature maps corresponding to each initial feature map. This promotes complementarity and fusion among various shallow defect features, thereby capturing more complex and detailed deep defect features in the display panel at a deeper level. Finally, the head performs target detection based on the target feature maps corresponding to the initial feature maps, obtaining the location and type of defect areas in the display panel, thus achieving defect detection. Therefore, this embodiment fully explores the complex defect features of the display panel under test from both shallow and deep dimensions, improving the neural network model's ability to extract complex defect features and effectively enhancing the defect detection accuracy of the display panel.

[0039] In some implementations, refer to Figure 2 The aforementioned backbone network may include:

[0040] The first convolutional layer is used to extract features from the image data to obtain the first convolutional feature map;

[0041] The feature extraction structure includes four sequentially connected feature extraction modules. The input of the first feature extraction module is the first convolutional feature map, the output of the second feature extraction module is the first initial feature map, and the output of the third feature extraction module is the second initial feature map. Each feature extraction module is used to perform convolution operations and feature extraction on the input of the feature extraction module to obtain the output of the feature extraction module.

[0042] A multi-scale pooling structure is used to perform multi-scale pooling processing on the output of the fourth feature extraction module to obtain the third initial feature map.

[0043] In this embodiment, such as Figure 2 As shown, the backbone network may include a first convolutional layer, a feature extraction structure (CA-C2fs), and a multi-scale pooling structure (SA-SPPF). Image data of the display panel under test is input into the backbone network, where:

[0044] First, the image data is convolved using a first convolutional layer to obtain a first convolutional feature map, which is intended for preliminary feature extraction. The kernel and stride of the first convolutional layer can be flexibly set according to the actual situation. For example, the kernel of the first convolutional layer can be 3×3 and the stride can be 2, but it is not limited to these.

[0045] Then, the first convolutional feature map is used as input to the feature extraction structure, which can include four sequentially connected feature extraction modules (CA-C2f). The input of the first feature extraction module is the first convolutional feature map, and the inputs of the other feature extraction modules are the outputs of the previous feature extraction module. Each feature extraction module performs convolution operations and feature extraction on its input to obtain the output of the feature extraction module, which is intended for further feature extraction. Specifically, the output of the second feature extraction module is the first initial feature map, and the output of the third feature extraction module is the second initial feature map. Specifically, the first feature extraction module can initially extract defect features from the feature map; the second feature extraction module can further extract defect features at different scales; the third feature extraction module can extract features with a greater focus on and further abstraction of defect features; and the last feature extraction module can extract more complex defect features. This progressive feature extraction structure effectively captures defect features from low-level to high-level in the display panel under test, such as defect edges, textures, and shapes.

[0046] Finally, the output of the fourth feature extraction module is used as the input to a multi-scale pooling structure. This multi-scale pooling structure then performs multi-scale pooling on the input, resulting in a third initial feature map. Here, the output of the fourth feature extraction module represents high-level defect features. The multi-scale pooling operation of the multi-scale pooling structure refines these high-level defect features, effectively capturing multi-scale contextual defect features (such as irregular bubbles, sloping line defects, and other spatial features). This yields three types of initial feature maps.

[0047] Therefore, it can be seen that through the above backbone network, the first shallow convolutional layer is used to achieve preliminary feature extraction. Then, the progressive feature extraction structure can effectively capture the defect features of the display panel under test from low level to high level, such as the edge, texture and shape of the defect. Finally, the multi-scale pooling operation of the multi-scale pooling structure is used to refine the features of the output of the fourth feature extraction module, which can effectively capture multi-scale contextual defect features (such as spatial features such as irregular bubbles and tilted line defects). In this way, the shallow defect features of the display panel under test can be accurately mined at the shallow level, which helps to improve the feature extraction capability of the neural network model for complex defect features.

[0048] In some implementations, refer to Figure 3 The aforementioned feature extraction module may include:

[0049] The first convolutional structure is used to perform feature extraction and dimensionality reduction operations on the input of the feature extraction module to obtain the output of the first convolutional structure;

[0050] The feature grouping layer is used to divide the output of the first convolutional structure into a first feature map and a second feature map.

[0051] The attention backbone structure is used to perform multiple channel attention processes on the second feature map to obtain multiple backbone feature maps.

[0052] The first fusion layer is used to fuse the first feature map and multiple backbone feature maps to obtain the first fused feature map;

[0053] The second convolutional layer is used to perform dimensionality reduction on the first fused feature map to obtain the output of the feature extraction module.

[0054] In this embodiment, in the feature extraction module, firstly, feature extraction and dimensionality reduction operations are performed on the input of the feature extraction module through a first convolutional structure to obtain the output of the first convolutional structure. This can initially extract defective features from the input of the feature extraction module and reduce the number of input channels, thereby reducing computational load and parameter count. The first convolutional structure may include a first sub-convolutional layer and a second sub-convolutional layer connected sequentially. The first sub-convolutional layer is used to extract features from the input of the feature extraction module, obtaining its output. The second sub-convolutional layer is used to perform dimensionality reduction operations on the output of the first sub-convolutional layer, obtaining its output as the output of the first convolutional structure. The convolution kernels and strides of the first and second sub-convolutional layers can be flexibly set according to actual conditions. For example, the convolution kernel of the first sub-convolutional layer can be 3×3 with a stride of 2, and the convolution kernel of the second sub-convolutional layer can be 1×1 with a stride of 1, but it is not limited to these.

[0055] Secondly, the feature splitting layer divides the output of the first convolutional structure into two parts along the channel dimension through parallel processing and multi-branch feature extraction. One part is the first feature map, which serves as a shallow detail branch and is directly input into the first fusion layer. The other part is the second feature map, which serves as a deep semantic branch and is input into the attention backbone structure (CA-Bottlenecks) for processing. This splits the output of the first convolutional structure into parallel branches, preserving the combination of original information and deep features. By directly passing the original information, the model can effectively avoid excessive loss of detail information, while by performing complex processing on only part of the information, the computational cost can be effectively reduced.

[0056] Then, in the attention backbone structure, the second feature map is subjected to multiple attention operations through the Coordinate Attention (CA) mechanism to obtain multiple backbone feature maps. Different backbone feature maps can represent defect features at different levels, which can capture the defect features of deep semantics and enhance the expression of defect features, thus helping to improve the feature extraction capability of the neural network model.

[0057] Subsequently, the outputs of the first feature map and the attention backbone structure are passed to the first fusion layer, which fuses the first feature map with multiple backbone feature maps to obtain the first fused feature map. The fusion method of the first fusion layer can be flexibly set according to the actual situation; for example, it can be concatenated, but it is not limited to this. The first feature map represents the raw information of shallow details, while each backbone feature map represents the defect features of different deep semantics. Fusing shallow details and various deep semantics in the first fusion layer can effectively increase the diversity of defect features and enhance the neural network model's ability to express complex defect features.

[0058] For example, in some embodiments, the feature grouping layer divides the output of the first convolutional structure into feature map F1 and feature map F2. Feature map F1 is directly input into the first fusion layer. Feature map F2 is processed by the attention backbone structure to generate feature map F3 and feature map F4. Feature map F3 and feature map F4 are input into the first fusion layer, and the first fusion layer fuses feature maps F1 to F4 to obtain the first fused feature map.

[0059] Finally, the first fused feature map is input into the second convolutional layer, which reduces the dimensionality of the first fused feature map to obtain the output of the feature extraction module. This effectively reduces the number of channels in the output of the feature extraction module, thus reducing computation and parameter requirements. Optionally, the kernel and stride of the second convolutional layer can be flexibly set according to actual conditions; for example, the kernel of the second convolutional layer can be 1×1 with a stride of 1, but it is not limited to this.

[0060] Therefore, through the aforementioned feature extraction module, firstly, the first convolutional structure performs feature extraction and dimensionality reduction on its input, obtaining the output of the first convolutional structure to achieve preliminary feature extraction and reduce computational cost. Next, the feature grouping layer divides the feature-extracted and dimensionality-reduced input into a first feature map and a second feature map. The first feature map is passed to the first fusion layer, and the second feature map is passed to the attention backbone structure to preserve the combination of original information and deep features. Directly passing the original information effectively avoids excessive loss of detailed information by the model, while performing complex processing only on a portion of the information effectively reduces computational cost. The second feature map undergoes feature extraction and enhancement along the channel dimension by the attention backbone structure, resulting in multiple backbone feature maps to capture and enhance the expression of deep semantic defect features. Then, the first fusion layer fuses the first feature map (original information of shallow details) and multiple backbone feature maps (defect features of different deep semantics) to obtain a first fused feature map, increasing the diversity of defect features. Finally, the second convolutional layer reduces the dimensionality of the first fused feature map to obtain the output of the feature extraction module, further reducing computational cost. Thus, the aforementioned feature extraction module can enhance the interaction of defect feature information between channels, fully capture complex defects of multiple scales such as fine point defects and large-area Mura defects in the display panel under test, thereby effectively improving the effect of shallow feature extraction and assisting the neural network model in capturing more complex defect features of shallow dimensions.

[0061] In some implementations, refer to Figure 3 and Figure 4 The aforementioned attention backbone structure may include multiple sequentially connected attention backbone modules. The input of the first attention backbone module is the second sub-feature map, and the input of other attention backbone modules is the output of the previous attention backbone module. Each attention backbone module may include a composite convolutional structure, channel attention modules, and a feature fusion layer. The composite convolutional structure is used to perform dimensionality reduction and feature extraction on the input of the attention backbone module to obtain the output of the composite convolutional structure; the channel attention module is used to perform channel attention operations on the output of the composite convolutional structure to obtain a channel attention feature map; the feature fusion layer is used to fuse the channel attention feature map and the input of the attention backbone module to obtain the backbone feature map corresponding to the attention backbone module.

[0062] In this embodiment, the aforementioned attention backbone structure may include multiple sequentially connected attention backbone modules (CA-Bot tleneck). The input of the first attention backbone module is the second sub-feature map, and the input of other attention backbone modules is the output of the previous attention backbone module. The number of attention backbone modules can be flexibly set according to actual conditions. For example, along the network propagation direction, the attention backbone structure of the first feature extraction module has one attention backbone module, the attention backbone structures of the second and third feature extraction modules have two attention backbone modules, and the attention backbone structure of the last feature extraction module has one attention backbone module, but it is not limited to this.

[0063] Each attention backbone module can include a composite convolutional structure, a channel attention module, and a feature fusion layer. For example... Figure 4 As shown, in each attention backbone module, a composite convolutional structure is used to perform dimensionality reduction and feature extraction on the input of the attention backbone module, resulting in the output of the composite convolutional structure. This allows for the initial extraction of defective features from the input of the attention backbone module and reduces the number of input channels, thus lowering computational and parameter requirements. The composite convolutional structure can include a third and fourth sub-convolutional layer connected sequentially. The third sub-convolutional layer performs dimensionality reduction on the input of the attention backbone module, yielding its output. The fourth sub-convolutional layer extracts features from the output of the third sub-convolutional layer, and its output serves as the output of the composite convolutional structure, thereby extracting features and increasing the receptive field. The kernels and strides of the third and fourth sub-convolutional layers can be flexibly set according to actual conditions; for example, the kernel of the third sub-convolutional layer can be 1×1, and the kernel of the fourth sub-convolutional layer can be 3×3, but this is not a limitation.

[0064] The output of the composite convolutional structure is input into the channel attention module. In this module, channel attention operations are performed on the output to obtain the channel attention feature map. Here, a lightweight channel attention mechanism is introduced. This mechanism obtains the channel weights of each defect feature through global average pooling and adaptively adjusts the importance of defect features in each channel. This strengthens the inter-channel dependencies of defect features, especially improving the feature representation ability for low-contrast defects (such as light-colored patches like Mur a), and capturing more detailed defect features.

[0065] The channel attention feature map is input to the feature fusion layer, and simultaneously, the input of the attention backbone module is passed to the feature fusion layer. The feature fusion layer fuses the attention feature map and the input of the attention backbone module to obtain the backbone feature map corresponding to the attention backbone module, which is the output of the attention backbone module. The fusion method of the feature fusion layer can be flexibly set according to the actual situation; for example, the fusion method can be element-wise addition, but it is not limited to this. Here, the input of the attention backbone module represents the original defect features, while the backbone feature map represents the channel-dimensional defect features. Fusing the original defect features and the channel-dimensional defect features in the feature fusion layer can further increase the diversity of defect features and enhance the neural network model's ability to express complex defect features such as low contrast.

[0066] Through the aforementioned attention backbone module, firstly, the input is subjected to dimensionality reduction and feature extraction using a composite convolutional structure, yielding the output of the composite convolutional structure to reduce computational cost and achieve preliminary feature extraction. Next, a channel attention module with a channel attention mechanism is used to further mine features from the dimensionality-reduced and feature-extracted input, strengthening the inter-channel dependencies of defect features and capturing more detailed defect features. Finally, a feature fusion layer fuses the original defect features and the channel-dimensional defect features to further increase the diversity of defect features. In this way, the aforementioned attention backbone module can further enhance the interaction of defect features across the channel dimensions, capture more multi-scale defect features, thereby further improving the effect of shallow feature extraction and enhancing the neural network model's ability to extract complex defect features.

[0067] In some implementations, refer to Figure 5 The above multi-scale pooling structure can include:

[0068] The third convolutional layer is used to perform dimensionality reduction on the output of the fourth feature extraction module to obtain the second convolutional feature map;

[0069] The pooling module is used to perform multiple parallel pooling operations on the second convolutional feature map to obtain multiple pooled feature maps.

[0070] The second fusion layer is used to fuse the second convolutional feature map and multiple pooling feature maps to obtain the second fused feature map;

[0071] The spatial attention module is used to perform spatial attention processing on the second fused feature map to obtain a spatial attention feature map.

[0072] The fourth convolutional layer is used to perform dimensionality reduction on the spatial attention feature map to obtain the third initial feature map.

[0073] In this embodiment, the output of the fourth feature extraction module in the feature extraction structure is input into a multi-scale pooling structure. In the multi-scale pooling structure, firstly, the third convolutional layer performs dimensionality reduction on the output of the fourth feature extraction module to obtain a second convolutional feature map. This effectively reduces the number of channels in the output of the fourth feature extraction module, thus reducing computational complexity and parameter count. The kernel and stride of the third convolutional layer can be flexibly set according to actual conditions; for example, the kernel of the third convolutional layer can be 1×1, but it is not limited to this.

[0074] Secondly, the pooling module performs multiple parallel pooling operations on the second convolutional feature map, resulting in multiple pooled feature maps, each representing defect features within a different receptive field. This multi-stage pooling operation provides receptive fields of varying scales, facilitating the comprehensive capture of contextual defect features at different scales (e.g., details of defects in small targets and the overall outline of defects in large targets). Furthermore, designing the pooling operation as a parallel structure allows for independent gradient propagation, avoiding gradient path dependence in serial structures. This ensures that multi-scale defect features are comprehensively captured and enhanced, while also effectively reducing computational cost.

[0075] Then, multiple pooling feature maps and the second convolutional feature map are passed to the second fusion layer, which fuses the second convolutional feature map and the multiple pooling feature maps to obtain the second fused feature map. This enables the extraction of defect features from different receptive fields, enhancing the representation ability of multi-scale contextual defect features in the display panel under test. The fusion method of the second fusion layer can be flexibly set according to the actual situation; for example, the fusion method of the second fusion layer can be splicing along the channel dimension, but it is not limited to this.

[0076] Subsequently, the second fused feature map is input into the spatial attention module, where spatial attention is calculated to obtain a spatial attention feature map. This generates a spatial weight map, effectively suppressing background interference and focusing on the spatial features of defects such as irregular bubbles and tilted line defects.

[0077] Specifically, such as Figure 5As shown, the aforementioned spatial attention module may include a spatial pooling module, a first spatial fusion layer, a fifth sub-convolutional layer, and a second spatial fusion layer connected in sequence. The spatial pooling module includes a max pooling layer and an average pooling layer connected in parallel. An activation function is set between the fifth sub-convolutional layer and the second spatial fusion layer. In the spatial attention module, firstly, the spatial pooling module performs global max pooling and global average pooling on the second fused feature map, respectively, to obtain two spatial statistical maps. Then, the first spatial fusion layer fuses the two spatial statistical maps, and the fifth sub-convolutional layer generates a single-channel weight map. The fusion method of the first spatial fusion layer can be flexibly set according to the actual situation; for example, the fusion method of the first spatial fusion layer can be splicing, but it is not limited to this. In addition, the kernel size and stride of the fifth sub-convolutional layer can be flexibly set according to the actual situation; for example, the kernel size of the fifth sub-convolutional layer is 3×3, but it is not limited to this. Afterwards, the single-channel weight map is activated by the activation function, and its weight value ranges from [0, 1], representing the importance of each spatial location. The larger the value, the more attention is paid to the defect region. The type of activation function can be flexibly set according to the actual situation; for example, the activation function can be Sigmoid, ReLU, etc., but it is not limited to this. Finally, the second spatial fusion layer fuses the second fused feature map and the activated single-channel weight map to obtain the spatial attention feature map. The fusion method of the second spatial fusion layer can be flexibly set according to the actual situation; for example, the fusion method of the second spatial fusion layer can be element-wise multiplication, but it is not limited to this. In this way, by generating a spatial weight map, background interference is effectively suppressed, and spatial features of defect shapes such as irregular bubbles and tilted line defects are focused.

[0078] Finally, the fourth convolutional layer performs dimensionality reduction on the spatial attention feature map to obtain the third initial feature map. This effectively reduces the number of channels in the third initial feature map, thus reducing computational cost and parameter count. The kernel and stride of the fourth convolutional layer can be flexibly set according to actual needs; for example, the kernel can be 1×1, but it is not limited to this.

[0079] Therefore, the multi-scale pooling structure described above adopts a multi-scale hybrid pooling structure that combines max pooling and average pooling. First, the third convolutional layer reduces the dimensionality of the input to obtain a second convolutional feature map, thereby reducing computational cost. Next, a pooling module configured with parallel pooling operations processes the second convolutional feature map to obtain multiple pooled feature maps, providing receptive fields at different scales. This helps to comprehensively capture contextual defect features at different scales while reducing computational cost. Then, a second fusion layer fuses the second convolutional feature map and multiple pooled feature maps to obtain a second fused feature map, which extracts defect features from different receptive fields and enhances the representation ability of multi-scale contextual defect features in the display panel under test. Afterward, a spatial attention module configured with a spatial attention mechanism extracts and enhances features from the second fused feature map to obtain a spatial attention feature map, which effectively suppresses background interference and focuses on the spatial features of defect shapes such as irregular bubbles and tilted line defects. Finally, a fourth convolutional layer reduces the dimensionality of the spatial attention feature map to obtain a third initial feature map, thereby reducing computational cost. Thus, the aforementioned multi-scale pooling structure can not only effectively capture the spatial features of defect shapes such as irregular bubbles and tilted line defects, but also comprehensively capture the features of targets of different sizes, such as the details of small targets and the overall outline of large targets. It also has the ability to suppress background noise, thereby effectively improving the adaptability and performance of neural network models in complex defect detection scenarios, especially for small target defect detection.

[0080] In some implementations, refer to Figure 5 The aforementioned pooling module may include multiple parallel pooling branches, each of which may include parallel max-pooling layers and average-pooling layers. Specifically, the max-pooling layer performs max-pooling on the second convolutional feature map to obtain the pooled feature map corresponding to the max-pooling layer; the average-pooling layer performs average-pooling on the second convolutional feature map to obtain the pooled feature map corresponding to the average-pooling layer.

[0081] In this embodiment, the pooling module may include multiple parallel pooling branches. Each pooling branch may include parallel max pooling layers and average pooling layers. The input of each pooling branch is the second convolutional feature map. The max pooling layer can extract the maximum value of the local region through max pooling operation, and retain the extreme value features of the defect such as edges and corners (such as the sharp edges of linear features). The average pooling layer can calculate the average value of the local region through average pooling operation, and retain the overall brightness and grayscale distribution features of the defect (such as the low contrast area of ​​the Mura defect), thereby enhancing the robustness of the neural network model in the case of uneven brightness.

[0082] The number of pooling branches can be flexibly set according to the actual situation. For example, the number of pooling branches can be 3, that is, a total of 6 pooling feature maps are output. If the number of channels of the output of the fourth feature extraction module is 512, and the number of channels of each pooling feature map is consistent with it, then the number of channels of the second fused feature map is expanded from 512 to 512 (original) + 3 × 512 (max pooling) + 3 × 512 (average pooling) = 512 × 7 = 3584, thereby achieving the effect of extracting defect features of different receptive fields and enhancing the representation ability of multi-scale contextual defect features in the display panel under test, but it is not limited to this.

[0083] Furthermore, the kernel size for each pooling branch is different and can be flexibly set according to the actual situation. For example, with three pooling branches, the kernel sizes for the three branches (the average pooling layer and the global pooling layer share the same kernel size) are 5, 9, and 13 respectively. Compared to using other kernel sizes, this combination of 5, 9, and 13 has relatively controllable computational complexity. The larger the kernel size, the greater the computational cost. Using these three kernel sizes ensures the extraction of multi-scale defect features without making the computation too large, maintaining a certain level of computational efficiency while ensuring the performance of the neural network model. Moreover, the stride (i.e., the step size by which the pooling window slides across the input data) for each pooling branch is the same and can be flexibly set according to the actual situation. For example, the stride for each pooling branch is 1 to ensure that the number of channels in the pooled feature map output by each pooling branch is consistent with the number of channels in the output of the fourth feature extraction module.

[0084] Therefore, the pooling module has multiple parallel, independently running pooling branches, each using a different pooling kernel size. The features extracted by these different kernel sizes are complementary, providing receptive fields at different scales. These three values ​​(kernel, kernel, and scale) are continuously increasing, and the differences between them are relatively uniform. This incremental design ensures a continuous and smooth change in the receptive field, avoiding excessive jumps that could lead to the loss of certain scale information. Furthermore, the pooling modules rely on parallel max pooling and average pooling. Max pooling highlights texture details in defect areas, while average pooling smooths global information in defect areas, thus enabling a more comprehensive capture of features from targets of different sizes, such as the details of small targets and the overall outline of large targets.

[0085] In some implementations, refer to Figure 2 The aforementioned neck network may include:

[0086] The second convolutional structure is used to adjust the channels of multiple initial feature maps respectively, so as to obtain adjusted feature maps corresponding to the multiple initial feature maps;

[0087] The first interaction path is used to perform feature interaction between the adjusted feature maps corresponding to the multiple initial feature maps to obtain the first interaction feature maps corresponding to the multiple initial feature maps.

[0088] The second interaction path is used to perform feature interaction between the adjusted feature map corresponding to the first initial feature map and the first interaction feature maps corresponding to the multiple initial feature maps respectively, to obtain the second interaction feature maps corresponding to the multiple initial feature maps respectively.

[0089] The third fusion layer is used to fuse the first and second interactive feature maps of each initial feature map to obtain the target feature map corresponding to the initial feature map.

[0090] In this embodiment, the basic structure of the aforementioned neck network is a Feature Pyramid Network (FPN), whose main function is to interact and fuse multi-scale features. In the neck network, firstly, three initial feature maps are input into a second convolutional structure. The second convolutional structure adjusts the channels of each initial feature map, resulting in adjusted feature maps corresponding to each initial feature map. This rationalizes the number of channels in each initial feature map, balances the feature representation of initial feature maps at different scales, helps reduce the computational cost of high-level defect features, and improves the expressive power of low-level defect features. The second convolutional structure may include a sixth, seventh, and eighth sub-convolutional layer, which are used to adjust the channels of the first, second, and third initial feature maps, respectively.

[0091] Then, the adjusted feature maps corresponding to the multiple initial feature maps are input into the first interaction path. In the first interaction path, the adjusted feature maps corresponding to the multiple initial feature maps are interacted to obtain the first interactive feature maps corresponding to the multiple initial feature maps. Here, the first interaction path is a top-down path, which fuses the high-level, mid-level, and low-level defect features in the order from the third initial feature map to the first initial feature map. This allows the strong semantic information of the high-level defect features (the third initial feature map) to be passed to the low-level defect features (the first initial feature map). The high-level defect features have lower resolution but provide strong semantic information such as the category and shape of the defect, while the low-level defect features have higher resolution but retain details such as edges and textures. By combining low-resolution, semantically strong defect features with high-resolution, semantically weak defect features, it not only enhances the expression of high, mid, and low-level defect features but also achieves semantic alignment of multi-level defect features, which helps to construct high-level semantic defect feature maps at all scales.

[0092] Subsequently, the first interactive feature maps corresponding to the multiple initial feature maps are laterally passed to the second interactive path, while the adjusted feature map corresponding to the first initial feature map is also passed to the second interactive path. In the second interactive path, the adjusted feature map corresponding to the first initial feature map and the first interactive feature maps corresponding to the multiple initial feature maps are interacted to obtain the second interactive feature maps corresponding to the multiple initial feature maps. Here, the second interactive path is a bottom-up path, fusing low-level, mid-level, and high-level defect features sequentially from the first to the third initial feature map. The feature map processed by the first interactive path is then fused with the bottom-down feature map of the same size. By progressively passing shallow defect features upwards and combining them with deep defect features, and by interacting and fusing defect features at all levels, strong semantic information and fine-resolution features can be extracted, thereby improving the probability of the neural network model capturing different defect targets and the detection accuracy of defect targets at different scales.

[0093] Finally, for each initial feature map, its corresponding first and second interactive feature maps can be obtained. The third fusion layer then fuses these first and second interactive feature maps to obtain the target feature map corresponding to the initial feature map. The fusion method of the third fusion layer can be flexibly set according to the actual situation; for example, it can be element-wise addition, but it is not limited to this. This allows for the combination of the outputs from the top-down and bottom-up paths to generate the final multi-scale feature map. For example, the third fusion layer includes a first sub-fusion layer, a second sub-fusion layer, and a third sub-fusion layer. The first sub-fusion layer adds the first and second interactive feature maps of the first initial feature map element-wise to obtain the target feature map of the first initial feature map. The second sub-fusion layer adds the first and second interactive feature maps of the second initial feature map element-wise to obtain the target feature map of the second initial feature map. The third sub-fusion layer adds the first and second interactive feature maps of the third initial feature map element-wise to obtain the target feature map of the third initial feature map. Bidirectional feature fusion can enhance the depth of defect details and help improve the feature extraction accuracy of minute defect features (such as scratch features). In this process, after element-wise addition, the channels of each target feature map are adjusted through convolutional layers. For example, the target feature map of the first initial feature map is adjusted to 256, the target feature map of the second initial feature map is adjusted to 512, and the target feature map of the third initial feature map is adjusted to 1024.

[0094] Therefore, through the aforementioned neck network, firstly, the second convolutional structure adjusts the channels of multiple initial feature maps to obtain adjusted feature maps corresponding to each initial feature map, thus balancing the feature representation of initial feature maps at different scales. Next, the first interaction path interacts the adjusted feature maps corresponding to the multiple initial feature maps to obtain first interaction feature maps corresponding to each initial feature map, thereby enhancing the defect feature representation at high, medium, and low levels and achieving semantic alignment of defect features across multiple levels. Then, the second interaction path interacts the first initial feature map and the first interaction feature maps corresponding to the multiple initial feature maps to obtain second interaction feature maps corresponding to each initial feature map, thus achieving the extraction of strong semantic information and fine-resolution features. Finally, the third fusion layer fuses the first and second interaction feature maps corresponding to each initial feature map to obtain the target feature map corresponding to each initial feature map, thereby enhancing the depth of defect details. In this way, the deep, high-semantic information can be further refined into the semantic expression of shallow features through a top-down path, while the high-resolution features of the shallow layer can be passed back to the deep layer through a bottom-up path to supplement the detailed information of the deep features. This bidirectional circulating data flow can deeply mine the deep defect features of the display panel under test at a deep dimension, which helps to improve the feature extraction capability of the neural network model for complex defect features, and improve the detection capability of the neural network model for small and large defect targets.

[0095] In some implementations, when the second convolutional structure is used to perform channel adjustment on multiple initial feature maps to obtain adjusted feature maps corresponding to the multiple initial feature maps, it is specifically used to perform the following operations:

[0096] Perform upscan adjustment on the first initial feature map;

[0097] Make appropriate channel adjustments to the second initial feature map;

[0098] The third initial feature map is adjusted by reducing the number of channels.

[0099] In this implementation, traditional FPNs typically use the same channels for feature maps of different scales. This results in low-resolution feature maps containing a large number of redundant defect features, which can easily lead to a waste of computational resources, especially in industrial production, given limited computing resources. Therefore, to address this waste, the second convolutional structure employs a channel optimization mechanism. It sets different channel dimensions for feature maps of different scales and dynamically adjusts the number of channels based on the feature map's resolution, reducing the computational load on high-level defect features and improving the expressive power of low-level defect features. From the three initial feature maps output by the backbone network, the resolution decreases from the first to the third initial feature map, and the number of channels gradually increases. Channel adjustments are performed on each of these three initial feature maps, adjusting their channel dimensions. For example, the dimension of the first initial feature map is adjusted from 80×80×128 to 80×80×1024; the dimension of the second initial feature map is adjusted from 40×40×256 to 40×40×512; and the dimension of the third initial feature map is adjusted from 20×20×512 to 20×20×256, but this is not a limitation. Thus, the high-level feature map (i.e., the third initial feature map) is processed by reducing the number of channels to retain its core semantic defect features; the low-level feature map (i.e., the first initial feature map) is processed by increasing the number of channels to enhance its detailed defect features; and the middle-level feature map (i.e., the second initial feature map) is processed by adjusting the number of channels to balance its semantic defect features and detailed defect features.

[0100] In some implementations, refer to Figure 2 The aforementioned first interaction path may include:

[0101] The first parameterization module is used to reparameterize the adjusted feature map corresponding to the third initial feature map to obtain the first interactive feature map corresponding to the third initial feature map.

[0102] The first interaction structure is used to perform element-wise addition and reparameterization on the adjusted feature map corresponding to the second initial feature map and the first interaction feature map corresponding to the third initial feature map to obtain the first interaction feature map corresponding to the second initial feature map.

[0103] The second interaction structure is used to perform element-wise addition and reparameterization of the adjusted feature map corresponding to the first initial feature map and the first interaction feature map corresponding to the second initial feature map to obtain the first interaction feature map corresponding to the first initial feature map.

[0104] In this embodiment, the existing FPN uses only concatenation to achieve feature fusion in its top-down path, which not only leads to an increase in the number of channels and a surge in computation, but also results in insufficient feature extraction. To address this, this embodiment provides a new top-down path, which may include a first parameterization module, a first interaction structure, and a second interaction structure.

[0105] The first parameterization module is CSPStage. It performs reparameterization on the adjusted feature map corresponding to the third initial feature map to obtain the first interactive feature map corresponding to the third initial feature map. This enhances the expressive power of high-level defect features and fully explores the detailed information of small and large defect targets hidden in the high-level defect features.

[0106] The first interaction feature map corresponding to the third initial feature map is input into the first interaction structure. In the first interaction structure:

[0107] First, the first upsampling layer upsamples the first interactive feature map corresponding to the third initial feature map to adjust the scale of the first interactive feature map corresponding to the third initial feature map to match the scale of the adjusted feature map corresponding to the second initial feature map. Figure 1 The upsampling method can be flexibly set according to the actual situation, such as nearest neighbor interpolation upsampling, but it is not limited to this. For example, if the scale of the first interactive feature map corresponding to the third initial feature map is 20×20, and the scale of the adjusted feature map corresponding to the second initial feature map is 40×40, then the scale of the two can be unified by upsampling.

[0108] Then, the first interactive fusion layer adds the adjusted feature map corresponding to the second initial feature map and the first interactive feature map corresponding to the upsampled third initial feature map element-wise to obtain the first interactive fused feature map, enabling feature interaction and transfer between the two. Element-wise addition only requires element-wise operations on feature maps of the same size, which not only does not increase the number of channels and greatly reduces computational overhead, but also emphasizes the direct correspondence of spatial positions of feature maps, thereby improving the detection accuracy of small targets or defects with rich details.

[0109] Subsequently, the first interactive fusion feature map is reparameterized by the first parameterization unit based on CSPStage to obtain the first interactive feature map corresponding to the second initial feature map, so as to enhance the expressive power of the defect features and fully explore the detailed information of small and large defect targets hidden in the defect features.

[0110] The second interaction structure may include a second upsampling layer, a second interaction fusion layer (which outputs a second interaction fusion feature map), and a second parameterization unit. Its data flow is the same as that of the first interaction structure, and will not be described in detail here.

[0111] Therefore, in the aforementioned top-up path, firstly, the defect features of high-level, mid-level, and low-level features are fused sequentially from the third initial feature map to the first initial feature map. This combines low-resolution, semantically strong defect features with high-resolution, semantically weak defect features, which not only enhances the expression of defect features at high, mid, and low levels but also achieves semantic alignment of defect features across multiple levels, facilitating the construction of high-level semantic defect feature maps at all scales. Secondly, the reparameterization operation of CSPStage is introduced. CSPStage is a network module design based on a cross-stage partial connection mechanism, typically used to optimize the computational efficiency and feature fusion capability of convolutional neural networks. This effectively enhances the expressive power of defect features at each scale, fully mining the detailed information of small and large defect targets hidden in the defect features at each scale. This helps improve the feature extraction capability of the neural network model for complex defect features and enhances the detection capability of the neural network model for small and large defect targets. Moreover, it can reduce the computational load of the neural network model and solve the problem of computational redundancy. Finally, an element-wise addition operation is introduced. The existing splicing operation in FPN only stacks channel information and lacks targeted enhancement of spatial position consistency, making it difficult to accurately locate defect details in complex scenes. Element-wise addition emphasizes the direct correspondence of feature map spatial positions to improve the detection accuracy of small targets or defects with rich details. Moreover, it can reduce the amount of computation and solve the problem of computational redundancy.

[0112] In some implementations, refer to Figure 2 The second interaction path mentioned above may include:

[0113] The second parameterization module is used to reparameterize the adjusted feature map corresponding to the first initial feature map to obtain the second interactive feature map corresponding to the first initial feature map.

[0114] The third interaction structure is used to perform element-wise addition and reparameterization on the first interaction feature map corresponding to the second initial feature map and the second interaction feature map corresponding to the first initial feature map to obtain the second interaction feature map corresponding to the second initial feature map.

[0115] The fourth interaction structure is used to perform element-wise addition and reparameterization of the first interaction feature map corresponding to the third initial feature map and the second interaction feature map corresponding to the second initial feature map to obtain the second interaction feature map corresponding to the third initial feature map.

[0116] In this embodiment, a new bottom-up path is provided, which may include a second parameterization module, a third interaction structure, and a fourth interaction structure. The second parameterization module is CSPStage, which reparameterizes the adjusted feature map corresponding to the first initial feature map to obtain the second interactive feature map corresponding to the first initial feature map. This enhances the expressive power of low-level defect features and fully explores the detailed information of small and large defect targets hidden in the low-level defect features.

[0117] The second interactive feature map corresponding to the first initial feature map is input into the third interactive structure. In the third interactive structure:

[0118] First, the first downsampling layer downsamples the second interactive feature map corresponding to the first initial feature map to adjust its scale to match the scale of the first interactive feature map corresponding to the second initial feature map. The first downsampling layer can be flexibly configured according to the actual situation; for example, it can be a convolutional layer with a 3×3 kernel, but it is not limited to this. For instance, if the scale of the second interactive feature map corresponding to the first initial feature map is 80×80, while the scale of the first interactive feature map corresponding to the second initial feature map is 40×40, the downsampling operation can unify their scales.

[0119] Then, the third interactive fusion layer adds the first interactive feature map corresponding to the second initial feature map and the second interactive feature map corresponding to the downsampled first initial feature map element-wise to obtain the third interactive fusion feature map, enabling feature interaction and transfer between the two. Element-wise addition only requires operations on corresponding elements of feature maps of the same size, which not only does not increase the number of channels and greatly reduces computational overhead, but also emphasizes the direct correspondence of spatial positions of feature maps, thereby improving the detection accuracy of small targets or defects with rich details.

[0120] Subsequently, the third parameterization unit of CSPStage reparameterizes the third interactive fusion feature map to obtain the second interactive feature map corresponding to the second initial feature map, thereby enhancing the expressive power of the defect features and fully mining the detailed information of small and large defect targets hidden in the defect features.

[0121] The fourth interaction structure may include a second downsampling layer, a fourth interaction fusion layer (which outputs a fourth interaction fusion feature map), and a fourth parameterization unit. Its data flow is similar to that of the third interaction structure, and will not be described in detail here.

[0122] Therefore, in the aforementioned bottom-down path, firstly, the defect features of the low-level, middle-level, and high-level layers are fused sequentially according to the order of the first initial feature map to the third initial feature map. The feature map processed by the first interaction path is then fused with the bottom-down feature maps of the same size. By progressively passing shallow defect features upwards and combining them with deeper defect features, and by interacting and fusing defect features at all levels, strong semantic information and fine-resolution features can be extracted, thereby improving the probability of the neural network model capturing different defect targets and the detection accuracy of defect targets at different scales. Secondly, the reparameterization operation of CSPStage is introduced. CSPStage is a network module design based on a cross-stage partial connection mechanism, typically used to optimize the computational efficiency and feature fusion capability of convolutional neural networks. This effectively enhances the expressive power of defect features at each scale, fully mining the detailed information of small and large defect targets hidden within the defect features at each scale. This helps improve the feature extraction capability of the neural network model for complex defect features and enhances the detection capability of the neural network model for small and large defect targets. Furthermore, it can reduce the computational load of the neural network model and solve the problem of computational redundancy. Finally, an element-wise addition operation is introduced. The existing splicing operation in FPN only stacks channel information and lacks targeted enhancement of spatial position consistency, making it difficult to accurately locate defect details in complex scenes. Element-wise addition emphasizes the direct correspondence of feature map spatial positions to improve the detection accuracy of small targets or defects with rich details. Moreover, it can reduce the amount of computation and solve the problem of computational redundancy.

[0123] In some implementations, the detection head uses the YOLOHEAD algorithm from YOLO, meaning the prediction module is a YOLO prediction head. This detection head has three detection branches; the target feature maps of the three initial feature maps are input into their respective detection branches to obtain the location and type of the defective region in the display panel under test.

[0124] In some embodiments, the above method may further include:

[0125] During neural network model training, the following loss function is used to update the neural network model:

[0126] The localization loss function is obtained by taking the predicted bounding box, the orientation angle of the predicted bounding box, the ground truth bounding box, the orientation angle of the ground truth bounding box, the orientation penalty coefficient, and the area of ​​the matrix of the minimum bounding box of the predicted bounding box and the ground truth bounding box.

[0127] The classification loss function is obtained by using the precision, focus loss adjustment factor, and label smoothing coefficient of the neural network model.

[0128] The contour distance loss function is obtained by measuring the distance between the contour edges of the predicted bounding box and the contour edges of the ground truth bounding box.

[0129] In this embodiment, to address the geometric diversity of display panel defects (e.g., the elongated shape of linear defects and the irregular contour of bubble defects) and class imbalance (e.g., the limited number of Mura defect samples), a multi-task joint loss function (MTDLoss) is designed, which integrates localization, classification, and shape constraint losses to improve the performance of the neural network model in complex defect detection scenarios.

[0130] First, traditional GIoU only considers the minimum bounding rectangle when dealing with elongated defects, failing to utilize the defect's orientation information, leading to positioning errors. To address this issue, the orientation angle θ of the defect bounding box is introduced, and the orientation-aware positioning loss function (OA-GIoU) ​​is defined as shown in the following formula (1):

[0131]

[0132] In equation (1), L OA-GIoU Let A represent the localization loss function; B represent the ground truth bounding box; M represent the area of ​​the minimum bounding box matrix between the predicted and ground truth bounding boxes; α represent the preset orientation penalty coefficient, for example, α = 0.5; θ represent the orientation angle of the predicted bounding box; θ * Indicates the orientation angle of the actual bounding box.

[0133] The aforementioned localization loss function forces the neural network model to learn the directional information of defects, improving the localization accuracy of linear and tilted defects, and reducing localization errors, especially in complexly arranged display panels.

[0134] Secondly, in the display panel defect dataset, there are relatively few difficult samples such as Mura cluster defects, and the traditional cross-entropy loss function is prone to causing the model to underfit the minority class. To solve this problem, the Focal-LS Loss classification loss function is introduced, as shown in the following formula (2):

[0135] L Classification =-(1-P) γ [(1-∈)logP+∈log(1-P)] (2);

[0136] In equation (2), L Classificationdenoted by , P represents the precision of the neural network model; ∈ represents the preset label smoothing coefficient, which can balance the label smoothing strength and the learning efficiency of defect features to alleviate overfitting, for example, ∈ = 0.1; γ represents the preset focus loss adjustment factor, which can effectively reduce the loss weight of easy samples (such as background, non-defect areas) in class imbalance scenarios (such as when there are few Mura defect samples), forcing the neural network model to pay more attention to difficult samples (such as low-contrast Mura defects), for example, γ = 2.

[0137] The aforementioned classification loss function can reduce the weight of easy samples and enhance the classification ability of difficult samples (low-contrast Mura defects), thus improving the accuracy of the neural network model in imbalanced class scenarios.

[0138] Furthermore, display panel defects often have specific shape priors (such as near-circular or irregular block shapes). Traditional loss functions do not explicitly utilize this information. Therefore, shape prior constraints are introduced, and the distance between the predicted box contour and the true box contour is calculated as the contour distance loss function, as shown in the following formula (3):

[0139]

[0140] In equation (3), L Shape This represents the contour distance loss function; dH represents the distance between the contour edges of the predicted bounding box and the contour edges of the ground truth bounding box. Its type can be flexibly set according to the actual situation, for example, the distance can be the Hausdorff distance. Represents the outline edge of the predicted bounding box for the i-th sample; represents the outline edge of the ground truth bounding box of the i-th sample; N represents the total number of samples.

[0141] The aforementioned contour distance loss function can force the neural network model to learn the shape features of defects, reduce misjudgments of non-defect areas (circuit textures), and improve detection confidence in complex backgrounds.

[0142] In summary, the above three loss functions are combined to form a multi-task joint loss function, as shown in formula (4) below:

[0143] L MTDLoss =λ1L OA-GIoU +λ2L Classification +λ3L Shape (4);

[0144] In equation (4), L MTDLoss λ represents the joint loss function for multiple tasks. i i = 1, 2, 3 represents the preset weight coefficients, for example, λ1 = 1, λ2 = 2, λ3 = 3.

[0145] In some embodiments, the above method may further include:

[0146] Acquire sample images of multiple display panels;

[0147] Multiple display panel sample images are preprocessed to obtain an image dataset; the preprocessing includes frequency domain processing, region of interest extraction, data augmentation, and labeling.

[0148] The neural network model is trained and tested using an image dataset to obtain a pre-trained neural network model.

[0149] In this embodiment, multiple display panel sample images are acquired. The number of display panel sample images can be set according to actual needs; for example, there may be 500 display panel sample images, containing four types of defects: 350 point defects, 33 line defects, 66 Mura cluster defects, and 51 bubble defects. Figure 6 As shown, (a) is a point defect, (b) is a line defect, (c) is a cluster of Mura defects, and (d) is a bubble defect.

[0150] Display panel sample images are crucial for building neural network models. To maximize their effectiveness, they require a sufficient sample size and rich scene representation, enabling the neural network model to achieve high recognition accuracy. Therefore, preprocessing of the display panel sample images is necessary, as follows:

[0151] First, the sample image of the display panel is processed in the frequency domain. The core idea of ​​frequency domain image processing is to transform the image from the spatial domain to the frequency domain for analysis and manipulation. The specific operation process is as follows:

[0152] A Gaussian filter is generated in the frequency domain to control the retention or suppression of different frequency components in the frequency domain of the display panel sample image. A frequency domain Gaussian high-pass filter (GHPF) is used to retain high-frequency components to enhance defect edges or details in the display panel sample image, as shown in the following formula (5):

[0153]

[0154] In equation (5), H(u,v) represents the Gaussian filtered sample image of the display panel; D(u,v) represents the distance from the frequency domain point (u,v) of the sample image of the display panel to the center point (M / 2,N / 2); D0 represents the cutoff frequency, which is used to control the smoothing intensity of the filter. The larger D0 is, the more frequencies are retained.

[0155] The purpose of performing a Fourier transform (DFT) on the sample image of the display panel is to transform the image from the spatial domain to the frequency domain and obtain its frequency distribution, as shown in the following formula (6):

[0156]

[0157] In equation (6), F(u,v) represents the Fourier transform of the display panel sample image, which is the complex value of the point with frequency (u,v) in the frequency domain; f(x,y) represents the complex value of the point with coordinates (x,y) in the spatial domain; M and N represent the width and height of the signal in the spatial domain, i.e., the signal is composed of M×N points; (u,v) represents the frequency coordinates in the frequency domain, with values ​​ranging from [0,M-1] to [0,N-1]; (x,y) represents the coordinates in the spatial domain, with values ​​ranging from [0,M-1] to [0,N-1]; j represents the imaginary unit, j 2 =-1.

[0158] Using a Gaussian filter as the convolution kernel, the Fourier-transformed display panel sample image is convolved. The purpose is to achieve spatial domain convolution through frequency domain multiplication, suppressing or enhancing specific frequency components. The operation involves multiplying the Fourier-transformed display panel sample image and the Gaussian-filtered display panel sample image point by point.

[0159] The convolved display panel sample image is restored to a spatial domain image, resulting in a frequency domain processed display panel sample image. This increases the defect area of ​​the display panel sample image. Specifically, an inverse Fourier transform is used, as shown in the following formula (7):

[0160]

[0161] like Figure 7 As shown, (a) is before processing and (b) is after processing. Frequency domain processing can enhance the representation of defective regions in the display panel sample image, improve the quality of the display panel sample image, and thus help ensure the training effect of the neural network model.

[0162] Next, the Rembg algorithm is used to extract the Region of Interest (ROI) from the surface of the frequency-domain processed display panel sample image. The Rembg algorithm framework, as an open-source tool, is suitable for automated background removal. It detects foreground objects in the image and separates them from the background, thus creating objects with transparent backgrounds and achieving ROI extraction. The Rembg framework achieves significant segmentation performance in the field of image segmentation through deep learning models, especially convolutional neural networks based on the U-Net structure. The U-Net model combines upsampling and downsampling segmentation networks, capturing global image information without neglecting local details, and possesses powerful foreground and segmentation capabilities.

[0163] Subsequently, data augmentation was performed on the extracted display panel sample images using random brightness variations, random angle rotations, and mirror transformations. This increased the diversity of the training data and adapted to real-world production conditions, such as positional shifts in the display panel due to production line movement and uneven brightness in the production line lighting / illumination devices due to voltage fluctuations. This improved the model's robustness. The specific process is as follows:

[0164] Random rotation. By rotating the image around its center point by a certain angle, either clockwise or counterclockwise. This transformation aims to simulate the change in image position angle caused by deviations in the position of the object being measured during actual acquisition, which affects the recognition accuracy of the algorithm model. As shown in the following formula (8), given an image with center point coordinates (a, b), the coordinates of any vertex of the image's bounding box are (x, b). i ,y i When an image is rotated by a random angle θ, the coordinates of the image center point remain unchanged, while the coordinates of the vertices of the bounding box will be adjusted accordingly to (X...). i ,Y i ):

[0165]

[0166] Random brightness variation. To obtain clear imaging results, it is necessary to set up different light sources to achieve the best imaging effect and reduce the false negative rate of surface defects on the test object. Image surface brightness refers to the brightness of the light shining on the surface of an object. When the image brightness increases, the image will appear more dazzling or glaring; when the brightness decreases, the image will appear dark. However, in actual production, unstable lighting voltage is very likely to occur, which will affect the actual detection effect of the algorithm model. By randomly adjusting the brightness, images under different lighting conditions can be simulated, increasing the diversity of training samples and enabling the model to better generalize to different real-world scenarios.

[0167] Mirror transformation is a geometric transformation that alters the spatial distribution of an image by flipping it along a specific axis. Mirror transformation serves both as a low-cost data augmentation technique to improve the performance of neural network models and as a means to optimize the visual representation of an image through symmetry adjustments.

[0168] Finally, the Labelimg data annotation tool was used to annotate defects in the enhanced display panel sample images. For example, corresponding labels were assigned to the images, including four types: point defects, line defects, Mura defects, and bubble defects. To avoid visual interference for the operator during actual inspection, point defects were labeled as 0, line defects as 1, Mura clump defects as 2, and bubble defects as 3. Rectangular boxes were used for annotation, and the results were exported as XML files. These XML files needed to be converted to TXT format, which the neural network model could recognize.

[0169] After preprocessing, an image dataset is obtained, and the neural network model is trained and tested using the image dataset to obtain a pre-trained neural network model.

[0170] The effectiveness of the defect detection method for an electronic display panel provided in the embodiments of this application will be verified below.

[0171] Verification Example 1: The experimental environment for verification was as follows: Operating System: Windows 11; Memory: 32GB; CPU: 13th Gen Intel(R) Core(TM) i5-13400 2.5GHz; GPU: RTX 4060 (8GB)*1; Hard Disk: 1TB SSD; Programming Language: Python; Deep Learning Framework: PyTorch 1.10.0; Development Environment: Python 3.8. The evaluation metrics selected for verification were precision (P), recall (R), and mean average precision (mAP). YOLOv8n, YOLOv5, and YOLOv6 were selected as comparison networks for verification. The dataset selected for verification was one of the existing publicly available datasets.

[0172] Ablation experiments were conducted by adding each module in sequence to verify the effectiveness of each improved module. The results are shown in Table 1 below. GFPN refers to the neck network. It can be seen that compared with the comparison network YOLOv8n (i.e., the first row), the neural network model of this application (i.e., the last row) improved by 5.7 percentage points in mAP@0.5 / %, and the mAP@0.5:0.95 / % improved by 1.6 percentage points.

[0173] Table 1: Ablation Experiment Results

[0174] CA-C2f+SA-SPPF GFPN MTDLoss mAP@0.5 / % mAP@0.5:0.95 / % 78.7 52.8 √ 78.8 50.4 √ 82.3 54.3 √ 81.5 53.3 √ √ 84.1 53.6 √ √ 83.6 53.4 √ √ 83.6 56.7 √ √ √ 84.4 54.4

[0175] The neural network model of this application was compared with the comparative networks, and the results are shown in Table 2 below. It can be seen that the neural network model of this application achieved the best precision and mAP@0.5 in the display panel defect detection task. Specifically, the AP was second best in point defect detection, while the AP for online defects, Mura defects, and bubble defects all achieved the best values. Therefore, the performance of the neural network model of this application is superior to all the comparative networks.

[0176] Table 2: Results of the comparative experiment

[0177]

[0178]

[0179] Verification Example 2: The experimental environment for verification was as follows: Operating System: Windows 11; Memory: 32GB; CPU: 13th Gen Intel(R) Core(TM) i5-13400 2.5GHz; GPU: RTX 4060 (8GB)*1; Hard Disk: 1TB SSD; Development Language: Python; Deep Learning Framework: PyTorch 1.10.0; Development Environment: Python 3.8. The evaluation metrics selected for verification were precision (P), recall (R), and mean average precision (mAP). YOLOv8n was selected as the comparison network for verification. The dataset selected for verification was another existing public dataset containing three typical mobile phone screen surface defects: surface oil stains, scratches, and dirt. Each defect category had 400 samples. The dataset was subsequently expanded to 3600 samples through data augmentation. The 3600 samples were divided into training, testing, and validation sets in an 8:1:1 ratio using a custom script. The validation results are shown in Table 3 below.

[0180]

[0181] As shown in Table 3, the neural network model of this application achieved the best precision and mAP@0.5 in the display panel defect detection task, and achieved the best AP in oil stain defect, scratch detection, and stain detection. Therefore, the neural network model of this application outperforms the comparative networks.

[0182] The above verification demonstrates the superiority, advancement, and effectiveness of the neural network model proposed in this application.

[0183] Furthermore, this application embodiment also provides a defect detection system for an electronic display panel. The system may include: an acquisition module and a detection module. The acquisition module acquires image data of the display panel under test. The detection module is equipped with a pre-trained neural network model. The detection module inputs the image data into the pre-trained neural network model to obtain the location and type of the defective area of ​​the display panel. The neural network model includes: a backbone network, a neck network, and a detection head. The backbone network extracts features from the image data to obtain multiple initial feature maps. The neck network performs feature interaction between the multiple initial feature maps to obtain target feature maps corresponding to each initial feature map. The detection head performs target detection based on the target feature maps corresponding to the multiple initial feature maps to obtain the location and type of the defective area of ​​the display panel.

[0184] The content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0185] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

[0186] The above is a detailed description of the preferred embodiments of this application, but this application is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A method for detecting defects in an electronic display panel, characterized in that, Includes the following steps: Acquire image data of the display panel under test; The image data is input into a pre-trained neural network model to obtain the location and type of the defect area of ​​the display panel; The neural network model includes: The backbone network is used to extract features from the image data to obtain multiple initial feature maps; A neck network is used to perform feature interaction on multiple initial feature maps to obtain target feature maps corresponding to the multiple initial feature maps respectively; The neck network includes: The second convolutional structure is used to perform channel adjustment on multiple initial feature maps to obtain adjusted feature maps corresponding to each initial feature map. Specifically, the second convolutional structure adopts a channel optimization mechanism to set different channel dimensions for feature maps of different scales and dynamically adjusts the channel dimensions according to the resolution of the feature maps. From the three initial feature maps output by the backbone network, the resolution of the first initial feature map to the third initial feature map decreases from high to low, and the channel dimensions gradually increase. Channel adjustment is performed on these three initial feature maps respectively. The dimension of the first initial feature map is adjusted from 80×80×128 to 80×80×1024, the dimension of the second initial feature map is adjusted from 40×40×256 to 40×40×512, and the dimension of the third initial feature map is adjusted from 20×20×512 to 20×20×256. The first interaction path is used to perform feature interaction on the adjusted feature maps corresponding to the multiple initial feature maps respectively, so as to obtain the first interaction feature maps corresponding to the multiple initial feature maps respectively. The second interaction path is used to perform feature interaction between the adjusted feature map corresponding to the first initial feature map and the first interaction feature maps corresponding to the multiple initial feature maps respectively, to obtain the second interaction feature maps corresponding to the multiple initial feature maps respectively. The third fusion layer is used to fuse the first interactive feature map and the second interactive feature map of each initial feature map to obtain the target feature map corresponding to the initial feature map. The detection head is used to perform target detection based on the target feature maps corresponding to the multiple initial feature maps, and to obtain the location and type of the defect area of ​​the display panel; The backbone network includes: The first convolutional layer is used to extract features from the image data to obtain a first convolutional feature map; The feature extraction structure includes four sequentially connected feature extraction modules. The input of the first feature extraction module is the first convolutional feature map, the output of the second feature extraction module is the first initial feature map, and the output of the third feature extraction module is the second initial feature map. Each feature extraction module is used to perform convolution operations and feature extraction on the input of the feature extraction module to obtain the output of the feature extraction module. A multi-scale pooling structure is used to perform multi-scale pooling processing on the output of the fourth feature extraction module to obtain the third initial feature map. The multi-scale pooling structure includes: The third convolutional layer is used to perform dimensionality reduction on the output of the fourth feature extraction module to obtain the second convolutional feature map; The pooling module is used to perform multiple parallel pooling operations on the second convolutional feature map to obtain multiple pooled feature maps. The second fusion layer is used to fuse the second convolutional feature map and multiple pooling feature maps to obtain a second fused feature map; A spatial attention module is used to perform spatial attention processing on the second fused feature map to obtain a spatial attention feature map; The fourth convolutional layer is used to perform dimensionality reduction on the spatial attention feature map to obtain the third initial feature map; The pooling module includes multiple parallel pooling branches, each including a max-pooling layer and an average-pooling layer. The max-pooling layer performs max-pooling on the second convolutional feature map to obtain a pooled feature map corresponding to the max-pooling layer. The average-pooling layer performs average-pooling on the second convolutional feature map to obtain a pooled feature map corresponding to the average-pooling layer. Specifically, there are 3 pooling branches, and the pooling kernel sizes of the 3 pooling branches are 5, 9, and 13 respectively. The average pooling layer and the global pooling layer share the same pooling kernel size. The step size of each pooling branch is 1 to ensure that the number of channels of the pooling feature map output by each pooling branch is consistent with the number of channels of the output of the fourth feature extraction module. The step size is the step size by which the pooling window slides on the input data.

2. The method according to claim 1, characterized in that, The feature extraction module includes: The first convolutional structure is used to perform feature extraction and dimensionality reduction operations on the input of the feature extraction module to obtain the output of the first convolutional structure; A feature grouping layer is used to divide the output of the first convolutional structure into a first feature map and a second feature map. An attention backbone structure is used to perform multiple channel attention processes on the second feature map to obtain multiple backbone feature maps. The first fusion layer is used to fuse the first feature map and multiple backbone feature maps to obtain a first fused feature map; The second convolutional layer is used to perform dimensionality reduction on the first fused feature map to obtain the output of the feature extraction module.

3. The method according to claim 1, characterized in that, The first interaction path includes: The first parameterization module is used to perform reparameterization processing on the adjusted feature map corresponding to the third initial feature map to obtain the first interactive feature map corresponding to the third initial feature map. The first interaction structure is used to perform element-wise addition and reparameterization processing on the adjusted feature map corresponding to the second initial feature map and the first interaction feature map corresponding to the third initial feature map to obtain the first interaction feature map corresponding to the second initial feature map. The second interaction structure is used to perform element-wise addition and reparameterization processing on the adjusted feature map corresponding to the first initial feature map and the first interaction feature map corresponding to the second initial feature map to obtain the first interaction feature map corresponding to the first initial feature map.

4. The method according to claim 1, characterized in that, The second interaction path includes: The second parameterization module is used to reparameterize the adjusted feature map corresponding to the first initial feature map to obtain the second interactive feature map corresponding to the first initial feature map. The third interaction structure is used to perform element-wise addition and reparameterization processing on the first interaction feature map corresponding to the second initial feature map and the second interaction feature map corresponding to the first initial feature map to obtain the second interaction feature map corresponding to the second initial feature map. The fourth interaction structure is used to perform element-wise addition and reparameterization processing on the first interaction feature map corresponding to the third initial feature map and the second interaction feature map corresponding to the second initial feature map to obtain the second interaction feature map corresponding to the third initial feature map.

5. The method according to claim 1, characterized in that, The method further includes: During the training of the neural network model, the following loss function is used to update the neural network model: The localization loss function is obtained by using the predicted bounding box, the orientation angle of the predicted bounding box, the ground truth bounding box, the orientation angle of the ground truth bounding box, the orientation penalty coefficient, and the matrix area of ​​the minimum bounding box of the predicted bounding box and the ground truth bounding box. The classification loss function is obtained by using the precision, focus loss adjustment factor, and label smoothing coefficient of the neural network model. The contour distance loss function is obtained by the distance between the contour edges of the predicted bounding box and the contour edges of the ground truth bounding box.

6. The method according to claim 1, characterized in that, The method further includes: Acquire sample images of multiple display panels; Multiple display panel sample images are preprocessed to obtain an image dataset; wherein, the preprocessing includes frequency domain processing, region of interest extraction processing, data augmentation processing, and labeling processing; The neural network model is trained and tested using the image dataset to obtain the pre-trained neural network model.

7. A defect detection system for an electronic display panel, characterized in that, include: The acquisition module is used to acquire image data of the display panel under test; The detection module is equipped with a pre-trained neural network model. The detection module is used to input the image data into the pre-trained neural network model to obtain the location and type of the defect area of ​​the display panel. The neural network model includes: The backbone network is used to extract features from the image data to obtain multiple initial feature maps; A neck network is used to perform feature interaction on multiple initial feature maps to obtain target feature maps corresponding to the multiple initial feature maps respectively; The neck network includes: The second convolutional structure is used to perform channel adjustment on multiple initial feature maps to obtain adjusted feature maps corresponding to each initial feature map. Specifically, the second convolutional structure adopts a channel optimization mechanism to set different channel dimensions for feature maps of different scales and dynamically adjusts the channel dimensions according to the resolution of the feature maps. From the three initial feature maps output by the backbone network, the resolution of the first initial feature map to the third initial feature map decreases from high to low, and the channel dimensions gradually increase. Channel adjustment is performed on these three initial feature maps respectively. The dimension of the first initial feature map is adjusted from 80×80×128 to 80×80×1024, the dimension of the second initial feature map is adjusted from 40×40×256 to 40×40×512, and the dimension of the third initial feature map is adjusted from 20×20×512 to 20×20×256. The first interaction path is used to perform feature interaction on the adjusted feature maps corresponding to the multiple initial feature maps respectively, so as to obtain the first interaction feature maps corresponding to the multiple initial feature maps respectively. The second interaction path is used to perform feature interaction between the adjusted feature map corresponding to the first initial feature map and the first interaction feature maps corresponding to the multiple initial feature maps respectively, to obtain the second interaction feature maps corresponding to the multiple initial feature maps respectively. The third fusion layer is used to fuse the first interactive feature map and the second interactive feature map of each initial feature map to obtain the target feature map corresponding to the initial feature map. The detection head is used to perform target detection based on the target feature maps corresponding to the multiple initial feature maps, and to obtain the location and type of the defect area of ​​the display panel; The backbone network includes: The first convolutional layer is used to extract features from the image data to obtain a first convolutional feature map; The feature extraction structure includes four sequentially connected feature extraction modules. The input of the first feature extraction module is the first convolutional feature map, the output of the second feature extraction module is the first initial feature map, and the output of the third feature extraction module is the second initial feature map. Each feature extraction module is used to perform convolution operations and feature extraction on the input of the feature extraction module to obtain the output of the feature extraction module. A multi-scale pooling structure is used to perform multi-scale pooling processing on the output of the fourth feature extraction module to obtain the third initial feature map. The multi-scale pooling structure includes: The third convolutional layer is used to perform dimensionality reduction on the output of the fourth feature extraction module to obtain the second convolutional feature map; The pooling module is used to perform multiple parallel pooling operations on the second convolutional feature map to obtain multiple pooled feature maps. The second fusion layer is used to fuse the second convolutional feature map and multiple pooling feature maps to obtain a second fused feature map; A spatial attention module is used to perform spatial attention processing on the second fused feature map to obtain a spatial attention feature map; The fourth convolutional layer is used to perform dimensionality reduction on the spatial attention feature map to obtain the third initial feature map; The pooling module includes multiple parallel pooling branches, each of which includes a parallel max pooling layer and an average pooling layer. The max pooling layer is used to perform max pooling on the second convolutional feature map to obtain a pooled feature map corresponding to the max pooling layer. The average pooling layer is used to perform average pooling on the second convolutional feature map to obtain a pooled feature map corresponding to the average pooling layer. Specifically, there are 3 pooling branches, and the pooling kernel sizes of the 3 pooling branches are 5, 9, and 13 respectively. The average pooling layer and the global pooling layer share the same pooling kernel size. The step size of each pooling branch is 1 to ensure that the number of channels of the pooling feature map output by each pooling branch is consistent with the number of channels of the output of the fourth feature extraction module. The step size is the step size by which the pooling window slides on the input data.

Images