Display panel defect detection method, device, electronic device and storage medium

Abstract

The present application discloses a defect detection method, defect detection device, electronic device, and storage medium for a display panel. The defect detection method includes: obtaining an original image of a display panel bonded with a U-shaped film, preprocessing the original image to obtain a preprocessed image, performing multi-scale feature extraction on the preprocessed image using a defect detection model to obtain a set of multi-scale first feature maps, the defect detection model is constructed and trained by a deep learning multi-layer convolutional network based on a feature map pyramid network, and panel defects are classified according to the first feature maps. As a result, it is possible to monitor the defective phenomena of the display panel after the U-lami process in a timely, rapid, and accurate manner, thereby improving the timeliness of the factory's defect monitoring. This makes it easier for subsequent relevant personnel to analyze the defects and make further improvements, thereby improving the overall product quality and enhancing the competitiveness of the enterprise.

Description

Technical Field

[0001] The present application relates to the field of big data, and in particular to a display panel defect detection method, a defect detection device, an electronic device, and a storage medium. Background Art

[0002] Typically, after a flexible AMOLED completes the TFE process, the bottom film attached to the lower surface of the cut panel is removed and a U-lami is attached to replace the back film. The U-lami ensures that the U-lami and the panel do not separate when the flexible AMOLED panel is bent. At the same time, the U-lami also serves as an intermediate adhesive layer for the subsequent film attached to the lower layer. Currently, foreign matter, bubbles, punctures and other undesirable phenomena are prone to occur during the U-lami process. Automated Optical Inspection (AOI) equipment is required to inspect the U-lami. However, existing AOI equipment can only determine whether the U-lami is qualified, but it is difficult to effectively monitor specific undesirable phenomena to make further improvements. Summary of the Invention

[0003] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application provides a display panel defect detection method, a display panel defect detection device, an electronic device, and a storage medium.

[0004] The display panel defect detection method of the present application includes:

[0005] Acquire an original image of the display panel laminated to the U-shaped film;

[0006] Preprocessing the original image to obtain a preprocessed image;

[0007] Performing multi-scale feature extraction on the preprocessed image using a defect detection model to obtain a set of multi-scale first feature maps, wherein the defect detection model is constructed and trained by a deep learning multi-layer convolutional network based on a feature map pyramid network;

[0008] The panel defects are classified according to the first feature map.

[0009] In some embodiments, the preprocessed image includes a display area preprocessed image and a wiring area preprocessed image, the display panel is provided with a partition mark, and the preprocessing of the original image to obtain the preprocessed image includes:

[0010] determining panel information according to the original image;

[0011] identifying the partition mark of the display panel in the original image;

[0012] Dividing the original image into a display area and a wiring area according to the panel information and the partition mark;

[0013] The display area and the wiring area are pre-processed respectively to obtain the display area pre-processed image and the wiring area pre-processed image.

[0014] In some embodiments, preprocessing the display area and the wiring area to obtain a display area preprocessed image and a wiring area preprocessed image respectively includes:

[0015] performing bilateral filtering on the display area to obtain edge complex information of the display area;

[0016] cropping the display area to remove complex edge information of the display area;

[0017] Performing image segmentation on the display area after removing the complex edge information to obtain a mask coverage area;

[0018] The mask covered area is divided into blocks according to a preset size to obtain the display area pre-processed image.

[0019] In some embodiments, preprocessing the display area and the wiring area to obtain a display area preprocessed image and a wiring area preprocessed image respectively includes:

[0020] Differentiating the wiring into foreground and background to determine a detection range;

[0021] trimming the wiring area to remove complex edge information of the wiring area;

[0022] performing morphological processing and bilateral filtering on the clipped wiring area;

[0023] The processed wiring area is subjected to image segmentation to obtain a pre-processed image of the wiring area.

[0024] In some embodiments, performing multi-scale feature extraction on the pre-processed image using a defect detection model to obtain a set of multi-scale first feature maps includes:

[0025] Performing feature extraction on the preprocessed image through a residual convolutional network to obtain a set of second feature maps with different resolutions;

[0026] constructing a first feature map pyramid based on the set of second feature maps;

[0027] Upsampling is performed from top to bottom according to the first feature map pyramid to generate the set of multi-scale first feature maps with different resolutions.

[0028] In some embodiments, the set of multi-scale first feature maps forms a second feature map pyramid corresponding to the first feature map pyramid, and the upsampling from top to bottom according to the first feature map pyramid to generate the set of multi-scale first feature maps with different resolutions includes:

[0029] Determine a first feature map at the top of the second feature map pyramid according to the second feature map at the top of the first feature map pyramid;

[0030] The first feature map of the current layer is obtained by fusing a result of upsampling the first feature map of a layer above the second feature map pyramid and the second feature map of the current layer of the first feature map pyramid.

[0031] In some embodiments, fusing the upsampling result of the first feature map of a previous layer of the second feature map pyramid with the second feature map of the current layer of the first feature map pyramid to obtain the first feature map of the current layer includes:

[0032] Performing convolution processing on the second feature map of the current layer of the first feature map pyramid to reduce the number of convolution kernels of the second feature map;

[0033] Upsampling the first feature map of a layer above the second feature map pyramid to obtain a current upsampled feature map, so that the current upsampled feature map and the second feature map of the current layer have the same resolution;

[0034] The second feature map after convolution processing and the current up-sampled feature map are pixel-superimposed to obtain the first feature map of the current layer.

[0035] In certain embodiments, the defect detection method includes:

[0036] Establish an algorithm model for the defect type to be detected;

[0037] Training the algorithm model using the training image;

[0038] Utilizing the trained algorithm model to detect the verification image to obtain a verification detection result to optimize the algorithm model;

[0039] Repeat the above training steps. When the accuracy of the verification test result reaches a preset value, it is determined that the algorithm model training is completed to serve as the defect detection model.

[0040] In certain embodiments, the defect detection method includes:

[0041] The classification results are output and labeled.

[0042] The display panel defect detection device of the present application is used for panel defect detection of U-shaped film lamination, including:

[0043] An acquisition module, configured to acquire an original image of the display panel attached to the U-shaped film;

[0044] A processing module, configured to preprocess the original image to obtain a preprocessed image;

[0045] an extraction module, configured to perform multi-scale feature extraction on the preprocessed image using a defect detection model to obtain a set of multi-scale first feature maps, wherein the defect detection model is constructed and trained by a deep learning multi-layer convolutional network based on a feature map pyramid network; and

[0046] A classification module is used to classify the panel defects according to the first feature map.

[0047] The electronic device of the present application includes: a processor, a memory, and a program, wherein the program is stored in the memory and executed by the processor, and the program includes instructions for executing any one of the above-mentioned defect detection methods.

[0048] The non-volatile computer-readable storage medium of the computer program of the present application, when the computer program is executed by one or more processors, enables the processors to execute any one of the above-mentioned defect detection methods.

[0049] In the display panel defect detection method, defect detection device, electronic device, and computer storage medium of the embodiments of the present application, an original image of a panel attached to a U-shaped film is obtained, and the original image is preprocessed using a visual processing algorithm to obtain a preprocessed image. A defect detection model constructed and trained using a deep learning multi-layer convolutional network based on a feature map pyramid network then performs multi-scale feature extraction on the preprocessed image to obtain a set of multi-scale first feature maps that can characterize specific defects in the U-shaped film. The panels attached to the U-shaped film can then be classified based on the first feature maps. This allows analysis based on the classified panels to facilitate subsequent improvements, effectively improving overall product quality and enhancing corporate competitiveness.

[0050] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become obvious from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0052] Figure 1is a schematic flow chart of a defect detection method according to certain embodiments of the present application;

[0053] Figure 2 is a schematic diagram of a module of a defect detection device according to certain embodiments of the present application;

[0054] Figure 3 is a schematic diagram of a module of an electronic device according to some embodiments of the present application;

[0055] Figure 4 is a schematic diagram of a feature graph pyramid network of certain embodiments of the present application;

[0056] Figure 5 is a schematic flow chart of a defect detection method according to certain embodiments of the present application;

[0057] Figure 6 Schematic diagram of an original image of a panel laminating a U-shaped film according to certain embodiments of the present application;

[0058] Figure 7-11 is a schematic flow chart of a defect detection method according to certain embodiments of the present application;

[0059] Figure 12 It is a module schematic diagram of a defect detection device in certain embodiments of the present application.

[0060] Description of main component symbols:

[0061] Defect detection device 10, acquisition module 11, processing module 12, extraction module 13, classification module 14, creation module 15, training module 16, optimization module 17, determination module 18, electronic device 100, processor 20, memory 30, program 32. DETAILED DESCRIPTION

[0062] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0063] See also Figure 1 The present application provides a method for detecting defects in a display panel, which is used for detecting defects in a panel with a U-shaped film laminate. The method comprises the following steps:

[0064] 01. Obtain the original image of the display panel attached to the U-shaped film;

[0065] 02, preprocessing the original image to obtain a preprocessed image;

[0066] 03. Perform multi-scale feature extraction on the pre-processed image using a defect detection model to obtain a set of multi-scale first feature maps. The defect detection model is constructed and trained by a deep learning multi-layer convolutional network based on a feature map pyramid network; and

[0067] 04. Classify panel defects according to the first feature map.

[0068] See also Figure 2 In one embodiment of the present application, a display panel defect detection device 10 is provided. The defect detection device 10 includes an acquisition module 11, a processing module 12, an extraction module 13, and a classification module 14. Step 01 can be implemented by the acquisition module 11, step 02 can be implemented by the processing module 12, step 03 can be implemented by the extraction module 13, and step 04 can be implemented by the classification module 14.

[0069] In other words, the acquisition module 11 can be used to acquire the original image of the display panel attached to the U-shaped film;

[0070] The processing module 12 can be used to preprocess the original image to obtain a preprocessed image.

[0071] The extraction module 13 can be used to perform multi-scale feature extraction on the preprocessed image through a defect detection model to obtain a set of multi-scale first feature maps. The defect detection model is constructed and trained by a deep learning multi-layer convolutional network based on a feature map pyramid network.

[0072] The classification module 14 may be configured to classify panel defects according to the first feature map.

[0073] See also Figure 3 The electronic device 100 of the present application also includes one or more processors 20, a memory 30; and one or more programs 32, wherein the one or more programs 32 are stored in the memory 30 and executed by the one or more processors 20, and the program 32 is executed by the processor 20 to execute the instructions of the above-mentioned defect detection method.

[0074] See also Figure 4 The embodiment of the present application also provides a non-volatile computer-readable storage medium, which stores a computer program. When the computer program is executed by one or more processors 20, the processor 20 executes the above-mentioned defect detection method.

[0075] In the defect detection method, detection device 10, electronic device 100, and storage medium of the embodiments of the present application, an original image of the U-shaped film is acquired and preprocessed using a visual processing algorithm to obtain a preprocessed image. A defect detection model constructed and trained using a deep learning multi-layer convolutional network based on a feature map pyramid network then performs multi-scale feature extraction on the preprocessed image to obtain a set of multi-scale first feature maps that can characterize specific defects in the U-shaped film. This first feature map can then be used to classify panels bonded to the U-shaped film. This allows analysis based on the classified panels to facilitate subsequent improvements, effectively improving overall product quality and enhancing corporate competitiveness.

[0076] In some embodiments, the electronic device 100 may be a server that can communicate with automated optical inspection (AOI) equipment used to inspect the display panel's U-shaped film. This allows the AOI equipment to obtain an original image of the display panel's U-shaped film. The server may include a big data platform that integrates data access, data processing, data storage, query retrieval, analysis and mining, and application interfaces. Thus, the electronic device 100 can implement the defect detection method of the embodiments of this application.

[0077] It should be noted that AOI equipment is a device that uses optical principles to detect common defects encountered in welding production. During automatic inspection, the AOI equipment automatically scans through the camera and captures the original image of the display panel attached to the U-shaped film.

[0078] In some embodiments, the detection device 10 may be a part of the electronic device 100 . In other words, the electronic device 100 includes the detection device 10 .

[0079] In some embodiments, the detection device 10 may be discrete components assembled in a certain manner to have the aforementioned functions, or a chip in the form of an integrated circuit having the aforementioned functions, or a computer software code segment that enables the computer to have the aforementioned functions when running on the computer.

[0080] Please further combine Figure 3 In some embodiments, the electronic device 100 may further include a communication module, through which the electronic device 100 outputs the data after the detection process is completed, and / or obtains the original image of the U-shaped film to be processed by the electronic device 10 from an external device (such as an AOI device). For example, the communication module is connected to the database of a factory that produces display panels, thereby obtaining the original image of the U-shaped film in the database.

[0081] The processor 20 can pre-process the original image of the U-shaped film collected by a visual algorithm such as the Open CV algorithm to obtain a pre-processed image. OPEN CV (Open Source Computer Vision Library) is an open source computer vision library that provides many algorithm functions that efficiently implement computer vision algorithms. The Open CV algorithm can pre-process the original image, including positioning and segmentation. That is, after acquiring the original image, the Open CV algorithm can be used to position the original image, and then the positioned original image can be segmented to obtain a pre-processed image.

[0082] A defect detection model is a mathematical model used to detect and determine defects in images of U-shaped films attached to display panels. The detection model can be established based on preset logic and mathematical algorithms. The preset logic is business logic, which refers to the rules and processes that one entity must follow to provide services to another. The mathematical algorithm can be a deep learning network algorithm based on Feature Pyramid Networks (FPN).

[0083] It should be noted that Feature Pyramid Networks (FPN) is a network proposed in 2017. FPN mainly solves the multi-scale problem in object detection. By simply changing the network connection, it greatly improves the performance of small object detection without basically increasing the computational complexity of the original model.

[0084] Please combine Figure 4 , the feature map pyramid network includes bottom-up lines, top-down lines and lateral connections. The bottom-up process is the ordinary forward propagation process of the neural network. The feature map is calculated by the convolution kernel and usually becomes smaller and smaller. Top-down: The top-down process is to upsample the more abstract and semantically stronger high-level feature maps (upsampling), while the lateral connection is to merge the upsampling results with the feature map of the same size generated from the bottom up. The two layers of features of the horizontal connection have the same spatial size, which can utilize the bottom-level positioning detail information. The low-resolution feature map is upsampled by 2 times (for simplicity, nearest neighbor upsampling is used). The upsampled map is then merged with the corresponding bottom-up map by element-wise addition. This process is iterative until the final first feature map is generated.

[0085] In this application, the defect detection model may include multiple sub-models such as a puncture detection model, a foreign body detection model, and a bubble detection model. The puncture detection model is used to process the pre-processed image to detect whether the corresponding U-shaped film has puncture wounds, the foreign body detection model is used to process the pre-processed image to detect whether the corresponding U-shaped film has foreign bodies, and the bubble detection model is used to process the pre-processed image to detect whether the corresponding U-shaped film has bubbles.

[0086] Furthermore, the first feature map can be processed by the Open CV algorithm to perform judgment and screening based on the first feature map to determine whether the panel corresponding to the first feature map has defects, thereby classifying the panel according to the panel defects to obtain a classification result. The classification result may include but is not limited to good panels, defective panels with punctures, defective panels with foreign objects, defective panels with bubbles, panels with punctures and foreign objects, panels with punctures and bubbles, panels with foreign objects and bubbles, and panels with punctures, foreign objects and bubbles, etc.

[0087] Furthermore, in some implementations, to facilitate intuitive viewing of the classification results of the inspected panels, the classification results can be labeled and then displayed. This allows personnel to monitor specific defects in the panels, quickly identifying and addressing them, minimizing the impact of defects on quality and production capacity.

[0088] In some other embodiments, the processor 20 may also generate and send a notification of adverse fluctuations based on the classification results to notify relevant personnel. For example, if a large number of panels with poor puncture marks appear, this notification can facilitate quick review and analysis by relevant personnel. The notification method is not limited to SMS notification, phone notification, email notification, etc. For example, after monitoring an adverse fluctuation, the processor 20 can notify the process manager via email in the form of a real-time alert, allowing the process manager to quickly view the analysis details via email, so as to promptly identify problems and make improvements.

[0089] In addition, as can be seen from the above example, the defect detection method in this application is described using data obtained during the production process of a display panel. It is understandable that the above is only an example of a defect detection method and does not strictly limit the objects to which the defect detection method in the embodiments of the present invention can be applied.

[0090] See also Figure 5 In some embodiments, the pre-processed image includes a display area pre-processed image and a wiring area pre-processed image, and the display panel is provided with a partition mark. Step 02 further includes the steps of:

[0091] 021, determine panel information according to the original image;

[0092] 022, identifying the partition mark of the display panel in the original image;

[0093] 023, dividing the original image into a display area and a wiring area according to the panel information and the partition mark;

[0094] 024, pre-processing the display area and the wiring area to obtain a display area pre-processed image and a wiring area pre-processed image respectively.

[0095] Please further combine Figure 2 In some embodiments, steps 021-024 can be implemented by processing module 12. Specifically, processing module 12 can be configured to determine panel information based on the original image and identify partition markings of the display panel in the original image. Processing module 12 can also be configured to divide the original image into a display area and a wiring area based on the panel information and partition markings, and to preprocess the display area and wiring area to obtain a display area preprocessed image and a wiring area preprocessed image, respectively.

[0096] Please further combine Figure 3 In some embodiments, the processor 20 may be configured to determine panel information based on the original image and identify partition markings of the display panel in the original image. The processor 20 may also be configured to divide the original image into a display area and a wiring area based on the panel information and the partition markings, and to preprocess the display area and the wiring area to obtain a preprocessed display area image and a preprocessed wiring area image, respectively.

[0097] Panel information may include product size, model, panel partition markings, etc.

[0098] Please combine Figure 6 Specifically, the processor 20 can identify the partition mark of the display panel in the original image, divide the original image into two areas, the display area and the wiring area, according to the panel information and the partition mark, and then preprocess the display area and the wiring area respectively through the Open CV algorithm to obtain the display area preprocessed image and the wiring area preprocessed image.

[0099] In this way, by dividing the original image into the display area and the wiring area according to the partition mark, and then preprocessing them separately to obtain the display area preprocessed image and the wiring area preprocessed image, it is beneficial to subsequently process the display area preprocessed image and the wiring area preprocessed image separately through the defect detection model, thereby improving the accuracy of the defect detection model detection. At the same time, it can also further determine the specific location of the defect.

[0100] See also Figure 7 In some embodiments, step 024 includes the sub-steps of:

[0101] 0241, performing bilateral filtering on the display area to obtain complex edge information of the display area;

[0102] 0242, cropping the display area to remove complex information at the edge of the display area;

[0103] 0243, performing image segmentation on the display area after removing complex edge information to obtain a mask coverage area;

[0104] 0244, the mask covered area is divided into blocks according to a preset size to obtain a pre-processed image of the display area.

[0105] In some embodiments, sub-steps 0241-0244 can be implemented by the processing module 12, or in other words, the processing module 12 is also used to perform bilateral filtering on the display area to obtain complex edge information of the display area, and crop the display area to remove the complex edge information of the display area, and then perform image segmentation on the display area after removing the complex edge information to obtain a mask coverage area, and divide the mask coverage area into blocks according to a preset size to obtain a pre-processed image of the display area.

[0106] In some embodiments, the processor 20 can be used to perform bilateral filtering on the display area to obtain complex edge information of the display area, and crop the display area to remove the complex edge information of the display area, and then perform image segmentation on the display area after removing the complex edge information to obtain a mask coverage area, and divide the mask coverage area into blocks according to a preset size to obtain a pre-processed image of the display area.

[0107] Specifically, after obtaining the display area, the display area can be subjected to bilateral filtering processing using a bilateral filtering algorithm to enhance the edge features of the display area, so that the central detection area of ​​the display area is clearly contrasted with the edge area. The display area is then cropped according to the partition mark in the display area to remove the complex information in the edge area of ​​the display area. In this way, interference of the complex edge information with the positioning of the partition mark is avoided.

[0108] Furthermore, the cropped display area is segmented to obtain a mask coverage area with an image resolution of 15000×15000. The segmentation process can be implemented using a region growing segmentation method. Those skilled in the relevant technical field can understand that region growing is the process of aggregating pixels or sub-regions into larger regions based on pre-defined criteria. Its basic idea is to start from a set of growth points (a growth point can be a single pixel or a small area), merge adjacent pixels or areas with similar properties to the growth point with the growth point to form a new growth point, and repeat this process until growth is no longer possible. The similarity between the growth point and the similar area can be determined based on image information such as grayscale value, texture, and color. Bilateral filtering is a nonlinear filter that can achieve the effect of maintaining edges, reducing noise and smoothing. Like other filtering principles, bilateral filtering also uses a weighted average method, using the weighted average of the brightness values ​​of surrounding pixels to represent the intensity of a pixel. The weighted average used is based on a Gaussian distribution.

[0109] Furthermore, the mask coverage area is divided into blocks according to a preset size (for example, 512×512 pixels), and the pixels of the latter part that is less than the preset size are overlapped with the pixels of the former block part (the blocks can also be adjusted as needed), and finally N×M display area pre-processed images of the preset size are generated.

[0110] In this way, the original interference area in the display area is removed by bilateral filtering, cropping, segmentation and blocking of the display area to obtain a preprocessed image of the display area, which is beneficial to the subsequent defect detection model processing of the preprocessed image of the display area.

[0111] See also Figure 8 In some embodiments, step 024 includes the sub-steps of:

[0112] 0245, distinguish the wiring into foreground and background to determine the detection range;

[0113] 0246, trimming the wiring area to remove complex edge information of the wiring area;

[0114] 0247, perform morphological processing and bilateral filtering on the cropped wiring area;

[0115] 0248, perform image segmentation on the processed wiring area to obtain a wiring area preprocessing image.

[0116] In some embodiments, sub-steps 02455-0248 can be implemented by the processing module 12, or in other words, the processing module 12 is also used to divide the wiring area into foreground and background to determine the detection range, and to crop the wiring area to remove the complex edge information of the wiring area, and then perform morphological processing and bilateral filtering on the cropped wiring area, and perform image segmentation on the processed wiring area to obtain a pre-processed image of the wiring area.

[0117] In some embodiments, the processor 20 can be used to divide the wiring area into foreground and background to determine the detection range, and to crop the wiring area to remove complex edge information of the wiring area, and then perform morphological processing and bilateral filtering on the cropped wiring area, and perform image segmentation on the processed wiring area to obtain a pre-processed image of the wiring area.

[0118] After obtaining the wiring area, image recognition can be performed on the wiring area image, thereby dividing the wiring area into two areas: foreground and background. The foreground is the primary detection area (such as the center area of ​​the wiring area), and the background is the secondary detection area (such as the edge area). The image recognition algorithm is not limited. For example, the Otsu method (OTSU) algorithm can be used for division. The Otsu method (OTSU) is an algorithm for determining the threshold for image binarization segmentation. It divides the image into background and foreground parts according to the grayscale characteristics of the image. It is considered to be the best algorithm for threshold selection in image segmentation. It is simple to calculate and is not affected by image brightness and contrast. Furthermore, based on the division of the foreground and background of the wiring area, the wiring area is cropped. It is understandable that since the background area is mainly the edge area and the edge area has black edges, the subsequent image algorithm processing process is prone to over-detection of the black edges. Therefore, the black edge boundary can be determined and marked based on the black edges in the image, and then the black edges of the wiring area are cropped based on the marks.

[0119] Furthermore, the cropped wiring area is subjected to morphological processing and bilateral filtering processing, wherein the morphological processing is used to remove small black dots in the wiring area, and the bilateral filtering processing is used to enhance edge features of the wiring area.

[0120] It's important to note that morphology, or mathematical morphology, is one of the most widely used techniques in image processing. It primarily extracts image components that are meaningful for expressing and depicting regional shapes, enabling subsequent recognition to capture the most essential (and most discriminative) shape features of the target object, such as boundaries and connected regions. Morphological operations primarily include erosion, dilation, and opening and closing operations.

[0121] Furthermore, the wiring area after morphological processing and bilateral filtering is subjected to image segmentation to obtain a wiring area preprocessing image. The image segmentation can be implemented using the Otsu method (OTSU).

[0122] See also Figure 9 In some embodiments, step 03 includes the sub-steps of:

[0123] 031, extracting features from the preprocessed image through a residual convolutional network to obtain a set of second feature maps with different resolutions;

[0124] 032, constructing a first feature map pyramid based on a set of second feature maps;

[0125] 033, upsampling is performed from top to bottom according to the first feature map pyramid to generate a set of multi-scale first feature maps with different resolutions.

[0126] Please further combine Figure 2 In some embodiments, sub-steps 031-033 can be implemented by the extraction module 13. In other words, the extraction module 13 can be used to extract features from the pre-processed image using a residual convolutional network to obtain a set of second feature maps with different resolutions. The extraction module 13 can also be used to construct a first feature map pyramid based on the second feature maps, and to perform top-down upsampling on the first feature map pyramid to generate a set of multi-scale first feature maps with different resolutions.

[0127] In certain embodiments, the processor 20 may be configured to perform feature extraction on the preprocessed image using a residual convolutional network to obtain a set of second feature maps having different resolutions. The processor 20 may also be configured to construct a first feature map pyramid based on the second feature maps, and to perform top-down upsampling on the first feature map pyramid to generate a set of multi-scale first feature maps having different resolutions.

[0128] It should be noted that residual networks are a type of convolutional neural network. They are easy to optimize and can improve accuracy by increasing their depth. The residual blocks within them use skip connections, which alleviates the vanishing gradient problem associated with increasing the depth of deep neural networks.

[0129] Since the preprocessed image includes a display area preprocessed image and a wiring area preprocessed image, the residual network may include multiple networks. Some of the residual networks may perform feature extraction on the display area preprocessed image to obtain a set of second feature maps with different resolutions related to the display area preprocessed image, and some of the residual networks may perform feature extraction on the wiring area preprocessed image to obtain a set of second feature maps with different resolutions related to the wiring area preprocessed image.

[0130] Furthermore, by connecting the four vertices of a set of second feature images of different resolutions, a top-down feature map pyramid similar to a real pyramid can be constructed. Specifically, each second feature map in a set of second feature maps is upsampled to obtain multiple sampling layers, each sampling layer corresponds to the corresponding second feature map, and then the top sampling layer is first reduced in dimension using a 1×1 convolution, and then the two are added (corresponding elements are added), and finally a 3×3 convolution operation is performed. Then, a 3×3 convolution is performed on each sampling layer, followed by two paths, each connected to a 1×1 convolution for classification and regression operations to obtain candidate ROIs, and the candidate ROIs are input to the sampling layer for ROIPOOL operations. Finally, based on the previous step, two 1024-layer fully connected network layers are connected, and then two branches are divided to connect the corresponding classification layer and regression layer to generate a set of multi-scale first feature maps with different resolutions.

[0131] See also Figure 10In some embodiments, a set of multi-scale first feature maps is formed in a second feature map pyramid corresponding to the first feature map pyramid. Step 033 includes the following sub-steps:

[0132] 0331, determining the first feature map at the top of the second feature map pyramid according to the second feature map at the top of the first feature map pyramid;

[0133] 0332, the result of upsampling the first feature map of the previous layer of the second feature map pyramid is fused with the second feature map of the current layer of the first feature map pyramid to obtain the first feature map of the current layer.

[0134] Please further combine Figure 2 In some embodiments, sub-step 0331 can be implemented by the extraction module 13. In other words, the extraction module 13 can be used to determine the first feature map of the topmost layer of the second feature map pyramid based on the second feature map of the topmost layer of the first feature map pyramid, and to fuse the upsampling result of the first feature map of the upper layer of the second feature map pyramid with the second feature map of the current layer of the first feature map pyramid to obtain the first feature map of the current layer.

[0135] In some embodiments, the processor 20 can be used to determine the first feature map of the top layer of the second feature map pyramid based on the second feature map of the top layer of the first feature map pyramid, and fuse the result of upsampling the first feature map of the upper layer of the second feature map pyramid with the second feature map of the current layer of the first feature map pyramid to obtain the first feature map of the current layer.

[0136] Specifically, the first feature map of the topmost layer of the second feature map pyramid can be determined according to the second feature map of the topmost layer of the first feature map pyramid, convolution processing is performed on the second feature map of the current layer of the first feature map pyramid to reduce the number of convolution kernels of the second feature map, and upsampling processing is performed on the first feature map of the previous layer of the second feature map pyramid to obtain the current upsampled feature map, so that the current upsampled feature map and the second feature map of the current layer have the same resolution, and then the convolved second feature map and the current upsampled feature map are pixel-superimposed to obtain the first feature map of the current layer.

[0137] In this way, a set of multi-scale first feature maps can be obtained, so that the corresponding panels can be classified according to the first feature maps.

[0138] See also Figure 11 In some embodiments, the defect detection method further comprises:

[0139] 001, for the defect type to be detected, an algorithm model is established through a deep learning multi-layer convolutional neural network based on a feature graph pyramid network;

[0140] 002, use training images to train the algorithm model;

[0141] 003. Use the trained algorithm model to detect the verification image to obtain the verification detection results to optimize the algorithm model.

[0142] 004. Repeat the above training steps. When the accuracy of the verification test results reaches the preset value, determine that the algorithm model training is completed and use it as the defect detection model.

[0143] Please further combine Figure 12 In some embodiments, the defect detection device may further include a creation module 15, a training module 16, an optimization module 17, and a determination module 18.

[0144] Step 001 can be implemented by the creation module 15, step 002 can be implemented by the training module 16, the optimization module 17 can be implemented by 003, and step 004 can be implemented by the determination module.

[0145] In other words, the creation module 15 is used to establish an algorithm model for the defect type to be detected through a deep learning multi-layer convolutional neural network based on a feature graph pyramid network.

[0146] The training module 16 is used to train the algorithm model using training images.

[0147] The optimization module 17 is used to use the trained algorithm model to detect the verification image to obtain the verification detection result so as to optimize the algorithm model.

[0148] The determination module 18 is used to repeat the above training steps, and when the accuracy of the verification detection result reaches a preset value, determine that the algorithm model training is completed to serve as the defect detection model.

[0149] In certain embodiments, the processor 20 may be configured to establish an algorithm model for the defect type to be detected using a deep learning multi-layer convolutional neural network based on a feature graph pyramid network, train the algorithm model using training images, and then use the trained algorithm model to detect verification images to obtain verification detection results to optimize the algorithm model. The aforementioned training steps are repeated, and when the accuracy of the verification detection results reaches a preset value, the algorithm model training is determined to be complete and used as the defect detection model.

[0150] The algorithm model may include multiple models, each corresponding to a defect type. The training image is an original image of a defective display panel attached to a U-shaped film. The training image can be divided into three types of defect sub-images: training sub-images with puncture wounds, training sub-images with foreign objects, and training sub-images with bubbles. In other words, the corresponding algorithm model is trained using each of the three types of defect sub-images. The trained algorithm model is then used to test the verification image to obtain verification test results to optimize the algorithm model. When the accuracy of the verification test results reaches a preset value, the algorithm model training is determined to be complete and is used as the defect detection model.

[0151] The types of defects to be detected may include puncture defects, foreign matter defects, and bubble defects.

[0152] Furthermore, before training the algorithm model using training images, each defect sub-image can be segmented into display area defect sub-images and wiring area defect sub-images. Defects within these sub-images are then labeled to determine their type, size, and location. Furthermore, given the limited number of images currently collected, in some instances, data augmentation can be performed on each defect sub-image type to increase the number of defects in each category and ensure the accuracy of algorithm training.

[0153] The training detection results obtained by training the algorithm model through training images are compared with the actual detection results, so as to continuously adjust and optimize the algorithm model so that the training detection results are close to the actual detection results. When the accuracy and pass rate of the training detection results meet the standards, the algorithm model can be considered to meet the standards, and the trained algorithm model can be used as a defect detection model for defect detection.

[0154] In this way, the algorithm model can be trained using the training images and the training recognition results to obtain a trained defect detection model, which can be obtained according to the defect detection module.

[0155] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any other combination. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital video disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0156] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0157] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0158] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0159] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A method for detecting defects in a display panel, used for detecting defects in a panel with a U-shaped film laminate, characterized in that: The defect detection method comprises: Acquire an original image of the display panel laminated to the U-shaped film; Preprocessing the original image to obtain a preprocessed image; Performing multi-scale feature extraction on the preprocessed image using a defect detection model to obtain a set of multi-scale first feature maps, wherein the defect detection model is constructed and trained by a deep learning multi-layer convolutional network based on a feature map pyramid network; classifying the panel defects according to the first feature map; The pre-processed image includes a display area pre-processed image and a wiring area pre-processed image; The performing multi-scale feature extraction on the pre-processed image by using the defect detection model to obtain a set of multi-scale first feature maps includes: Performing feature extraction on the preprocessed image through a residual convolutional network to obtain a set of second feature maps with different resolutions; constructing a first feature map pyramid based on the set of second feature maps; Performing upsampling from top to bottom according to the first feature map pyramid to generate the set of multi-scale first feature maps with different resolutions; The set of multi-scale first feature maps forms a second feature map pyramid corresponding to the first feature map pyramid, and the upsampling from top to bottom according to the first feature map pyramid to generate the set of multi-scale first feature maps with different resolutions includes: Determine a first feature map at the top of the second feature map pyramid according to the second feature map at the top of the first feature map pyramid; Fusing the upsampling result of the first feature map of the previous layer of the second feature map pyramid with the second feature map of the current layer of the first feature map pyramid to obtain the first feature map of the current layer; The step of fusing a result of upsampling the first feature map of a layer above the second feature map pyramid with the second feature map of a current layer of the first feature map pyramid to obtain the first feature map of the current layer includes: Performing convolution processing on the second feature map of the current layer of the first feature map pyramid to reduce the number of convolution kernels of the second feature map; Upsampling the first feature map of a layer above the second feature map pyramid to obtain a current upsampled feature map, so that the current upsampled feature map and the second feature map of the current layer have the same resolution; The second feature map after convolution processing and the current up-sampled feature map are pixel-superimposed to obtain the first feature map of the current layer.

2. The defect detection method according to claim 1, wherein: The display panel is provided with a partition mark, and the preprocessing of the original image to obtain a preprocessed image includes: determining panel information according to the original image; identifying the partition mark of the display panel in the original image; Dividing the original image into a display area and a wiring area according to the panel information and the partition mark; The display area and the wiring area are pre-processed respectively to obtain the display area pre-processed image and the wiring area pre-processed image.

3. The defect detection method according to claim 2, wherein: The preprocessing of the display area and the wiring area to obtain a display area preprocessed image and a wiring area preprocessed image respectively includes: performing bilateral filtering on the display area to obtain edge complex information of the display area; cropping the display area to remove complex edge information of the display area; Performing image segmentation on the display area after removing the complex edge information to obtain a mask coverage area; The mask covered area is divided into blocks according to a preset size to obtain the display area pre-processed image.

4. The defect detection method according to claim 2, wherein: The preprocessing of the display area and the wiring area to obtain a display area preprocessed image and a wiring area preprocessed image respectively includes: Differentiating the wiring into foreground and background to determine a detection range; trimming the wiring area to remove complex edge information of the wiring area; performing morphological processing and bilateral filtering on the clipped wiring area; The processed wiring area is subjected to image segmentation to obtain a pre-processed image of the wiring area.

5. The defect detection method according to claim 1, wherein: The defect detection method comprises: Establish an algorithm model for the defect type to be detected; Training the algorithm model using training images; Utilizing the trained algorithm model to detect the verification image to obtain a verification detection result to optimize the algorithm model; Repeat the above training steps. When the accuracy of the verification test result reaches a preset value, it is determined that the algorithm model training is completed to serve as the defect detection model.

6. The defect detection method according to claim 1, wherein: The defect detection method comprises: Output the classification results and label them.

7. A display panel defect detection device for detecting defects in U-shaped film lamination panels, characterized in that: The defect detection device is used to implement the defect detection method according to any one of claims 1 to 6, and the detection device includes: An acquisition module, configured to acquire an original image of the display panel attached to the U-shaped film; A processing module, configured to preprocess the original image to obtain a preprocessed image; an extraction module, configured to perform multi-scale feature extraction on the preprocessed image using a defect detection model to obtain a set of multi-scale first feature maps, wherein the defect detection model is constructed and trained by a deep learning multi-layer convolutional network based on a feature map pyramid network; and A classification module is used to classify the panel defects according to the first feature map.

8. An electronic device, characterized in that: The system comprises a processor, a memory and a program, wherein the program is stored in the memory and executed by the processor, and the program includes instructions for executing the defect detection method according to any one of claims 1 to 6.

9. A non-volatile computer-readable storage medium for a computer program, characterized in that: When the computer program is executed by a processor, the processor is caused to perform the defect detection method according to any one of claims 1 to 6.

Images