A comprehensive panel defect detection method and system based on neural network

Abstract

The present invention provides a panel defect comprehensive detection method and system based on a neural network, which relates to the field of defect detection. The method flow is: obtaining a primary sample image through a low-power microscope; inputting the primary sample image into a pre-trained target detection model for defect detection; obtaining a secondary sample image through a high-power microscope; inputting the secondary sample image into a pre-trained target classification model for defect classification; judging whether the defect needs to be repaired based on the classification result, if repair is required, inputting the secondary sample image into a pre-trained semantic segmentation model to extract the edge contour of the image around the defect to obtain a number of edge contour images; and planning the repair path of the defect area image based on the several edge contour images to obtain a defect repair plan. The present invention adopts a combination of deep learning technology and machine vision technology to perform panel defect comprehensive detection, which solves the problem of poor applicability and functionality of existing defect detection.

Description

Technical Field

[0001] The present invention relates to the field of defect detection, and in particular to a neural network-based panel defect comprehensive detection method and system. Background Art

[0002] Many defects will be generated during the panel processing. However, the production process of the entire panel is complex and the production cycle is long. It often takes a long time from the substrate to the completion of production and processing. Therefore, the defects generated in each process stage need to be monitored at all times to prevent the defects from flowing into the next process and causing a reduction in yield.

[0003] Currently, many panel manufacturers have introduced defect detection systems. These systems are usually based on deep learning target detection algorithms that can learn defect samples and establish target detection models for defect detection, ultimately achieving real-time detection of panels in production and timely discovering defects that occur during panel production. However, after defects are detected, it is impossible to determine the severity of the defects, and it is impossible to accurately formulate defect repair plans based on the severity of the defects, resulting in poor applicability and functionality. Summary of the invention

[0004] The present invention provides a neural network-based panel defect comprehensive detection method and system, which solves the problem that the prior art is unable to perform defect level determination and defect repair path formulation based on defect detection, and the defect detection applicability and functionality are poor.

[0005] Acquire a primary panel image through a low-power microscope, and pre-process the primary panel image to obtain a primary sample image;

[0006] Input the first-level sample image into the pre-trained object detection model for defect detection to determine whether the panel has defects. If defects exist, obtain the location information of the defects based on the first-level sample image;

[0007] Based on the position information of the defect, a secondary panel image is obtained through a high-power microscope, and the secondary panel image is preprocessed to obtain a secondary sample image;

[0008] Input the secondary sample image into the pre-trained target classification model for defect classification to obtain defect classification results and defect area images;

[0009] Based on the defect classification result, it is determined whether the defect needs to be repaired. If repair is required, the secondary sample image is input into the pre-trained semantic segmentation model to extract the edge contour of the image around the defect to obtain a number of edge contour images;

[0010] A repair path is planned for the defect area image based on a number of edge contour images to obtain a defect repair solution.

[0011] In the above embodiment, the present invention first uses a target detection model to perform preliminary defect detection on a low-magnification panel image, and can quickly determine whether there are defects at a general level; then uses a target classification model to classify the defects of a high-magnification panel image with defects, and can screen out the defects that need to be repaired according to the classification results; then uses a semantic segmentation model to segment the contour information of the image around the defect; finally, based on the segmented contour information of the image around the defect, a repair template is constructed and a repair plan is formulated; panel defect detection, defect classification and defect repair are fully realized, and the applicability and functionality are strong.

[0012] As some optional implementations of the present application, the process of preprocessing the primary panel image and the secondary panel image is as follows:

[0013] Performing pixel stretching on the primary panel image and the secondary panel image respectively to obtain a pixel-stretched primary panel image and a secondary panel image;

[0014] The pixel-stretched primary panel image and the secondary panel image are binarized respectively to obtain a primary sample image and a secondary sample image.

[0015] In the above embodiment, the present invention can use the Normalize method to perform pixel stretching on the primary panel image and the secondary panel image to achieve contrast enhancement; and use image binarization to convert the grayscale primary panel image and the secondary panel image into black and white primary sample images and secondary sample images to simplify the image data.

[0016] As some optional implementations of the present application, the primary sample image is input into the pre-trained target detection model for defect detection, and the panel is determined to determine whether there is a defect. If there is a defect, the process of obtaining the defect location information based on the primary sample image is as follows:

[0017] Input the first-level sample image into the pre-trained object detection model for defect detection to determine whether there are defects on the surface of the first-level sample image;

[0018] If there is a defect on the surface of the primary sample image, a defect positioning frame including defect position information is acquired based on the primary sample image.

[0019] In the above embodiment, the present invention uses a deep learning-based target detection model to perform defect detection on a low-magnification primary sample image, which can quickly determine whether there are defects in the panel from a general level.

[0020] As some optional implementations of the present application, the process of obtaining the secondary panel image through a high-power microscope based on the defect location information is as follows:

[0021] The lens is debugged based on the defect positioning frame, and the center of the defect positioning frame is used as the center of the lens to magnify the low-power lens into a high-power lens;

[0022] The secondary panel image including the defect corresponding to the defect positioning frame is obtained through a high-power microscope.

[0023] In the above-mentioned embodiment, the present invention performs magnification adjustment based on the defect positioning of the target detection model, and can obtain a secondary panel image with high magnification and containing defects.

[0024] As some optional implementations of the present application, the secondary sample image is input into the pre-trained target classification model for defect classification to obtain the defect classification result and the defect area image in the following process:

[0025] Input the secondary sample image into the pre-trained target classification model for defect classification to obtain a defect target frame including defect category information, defect location information and defect size information; wherein the defect category information includes the defect category and whether the defect needs to be repaired;

[0026] For the secondary sample image corresponding to the defect target frame that does not need to be repaired, the defect level is determined based on the defect position information and defect size information;

[0027] For the secondary sample image corresponding to the defect target frame that needs to be repaired, the secondary sample image is input into the pre-trained semantic segmentation model to extract the edge contour of the image around the defect.

[0028] In the above embodiment, the present invention uses a deep learning-based target classification model to classify defects in high-magnification secondary sample images, and can screen out defects that need to be repaired based on the defect classification results.

[0029] As some optional implementations of the present application, whether the defect needs to be repaired is determined based on the defect classification result. If repair is required, the secondary sample image is input into the pre-trained semantic segmentation model to extract the edge contour of the image around the defect to obtain a number of edge contour images. The process is as follows:

[0030] Based on the defect category information, it is determined whether the defect needs to be repaired. If repair is required, the secondary sample image is input into the pre-trained semantic segmentation model for image contour extraction to obtain several periodic image contours and connected domains around the defect target box;

[0031] A repair template is constructed based on distance information and geometric information of a plurality of periodic image contours around the defect target frame to obtain an edge contour image corresponding to at least one repair template.

[0032] In the above embodiment, the present invention adopts a semantic segmentation model based on deep learning to extract image contours of high-magnification secondary sample images, and constructs a patching template based on the distance information and geometric information of several periodic image contours to obtain an edge contour image corresponding to at least one patching template.

[0033] As some optional implementations of the present application, the process of planning a repair path for a defect area image based on a plurality of edge contour images to obtain a defect repair solution is as follows:

[0034] Acquire a repair template, a repair step length, and a repair quantity based on distance information between periodic image contours around the defect target frame and geometric structure information of the defect target frame;

[0035] Based on the repair template, the repair step size, the repair quantity and the connected domain of the template corresponding to the image contour, at least one repair template is used to perform repair path planning to obtain a defect repair solution.

[0036] In the above embodiment, the present invention obtains a repair template, a repair step and a repair quantity based on the contour image of the segmented defect surrounding image, the distance information between the contour images and the geometric information of the defect, and formulates a defect repair plan using at least one repair template based on the repair template, the repair step and the repair quantity.

[0037] In a second aspect, the present invention provides a panel defect comprehensive detection system based on a neural network, the system comprising:

[0038] A low-magnification sample acquisition unit, wherein the low-magnification sample acquisition unit acquires a primary panel image through a low-magnification microscope and pre-processes the primary panel image to obtain a primary sample image;

[0039] A target defect detection unit, wherein the target defect detection unit is used to input the primary sample image into a pre-trained target detection model for defect detection to determine whether the panel has a defect, and if a defect exists, obtain the position information of the defect based on the primary sample image;

[0040] A high-magnification sample acquisition unit, wherein the high-magnification sample acquisition unit acquires a secondary panel image through a high-magnification microscope based on the position information of the defect, and pre-processes the secondary panel image to obtain a secondary sample image;

[0041] An object category detection unit, wherein the object category detection unit is used to input the secondary sample image into a pre-trained object classification model for defect classification to obtain a defect classification result and a defect area image;

[0042] A target semantic segmentation unit, wherein the target semantic segmentation unit determines whether the defect needs to be repaired based on the defect classification result, and if repair is required, the secondary sample image is input into a pre-trained semantic segmentation model to extract edge contours of images around the defect to obtain a plurality of edge contour images;

[0043] A defect repair formulation unit is used to plan a repair path for a defect area image based on a plurality of edge contour images to obtain a defect repair solution.

[0044] In a third aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the neural network-based comprehensive panel defect detection system when executing the computer program.

[0045] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the neural network-based comprehensive panel defect detection system is implemented.

[0046] The beneficial effects of the present invention are as follows: the present invention first uses a target detection model to perform preliminary defect detection on low-magnification panel images, and can quickly determine whether there are defects at a general level; then uses a target classification model to classify defects in high-magnification panel images with defects, and can screen out defects that need to be repaired based on the classification results; then uses a semantic segmentation model to segment the contour information of the image around the defect; finally, based on the segmented contour information of the image around the defect, a repair template is constructed and a repair plan is formulated; panel defect detection, defect classification and defect repair are fully realized, and the applicability and functionality are strong. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0048] Figure 1 is a schematic diagram of the computer device structure of the hardware operating environment described in the embodiment of the present invention;

[0049] Figure 2 is a flow chart of a comprehensive panel defect detection method according to an embodiment of the present invention;

[0050] Figure 3 is a schematic diagram of a surface defect target frame of a secondary panel image according to an embodiment of the present invention;

[0051] Figure 4 It is a structural block diagram of the panel defect comprehensive detection system described in an embodiment of the present invention. DETAILED DESCRIPTION

[0052] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0053] In order to solve the problem that the existing technology cannot determine the defect level and formulate defect repair paths based on defect detection, and the defect detection applicability and functionality are poor. This application provides a panel defect comprehensive detection method and system based on a neural network. Before introducing the specific technical solution of this application, the hardware operating environment involved in the embodiment of this application is first introduced.

[0054] See also Figure 1 , Figure 1 A schematic diagram of the computer device structure of the hardware operating environment involved in the embodiment of the present application.

[0055] like Figure 1 As shown, the computer device may include: a processor, such as a central processing unit (CPU), a communication bus, a user interface, a network interface, and a memory. Among them, the communication bus is used to realize the connection and communication between these components. The user interface may include a display screen (Display), an input unit such as a keyboard (Keyboard), and optionally, the user interface may also include a standard wired interface and a wireless interface. The network interface may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity interface). The memory may be a high-speed random access memory (Random Access Memory, RAM) memory, or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk memory. The memory may optionally be a storage device independent of the aforementioned processor.

[0056] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the computer device, and may include more or less components than shown in the figure, or combine some components, or arrange the components differently.

[0057] like Figure 1 As shown, the memory as a storage medium may include an operating system, a data storage module, a network communication module, a user interface module and an electronic program module.

[0058] exist Figure 1In the computer device shown, the network interface is mainly used for data communication with the network server; the user interface is mainly used for data interaction with the user; the processor and memory in the computer device of the present application can be set in the computer device, and the computer device calls the computer program product stored in the electronic program module through the processor, and executes the neural network-based comprehensive panel defect detection method provided in the embodiment of the present application.

[0059] Based on the hardware environment of the aforementioned embodiment, the embodiment of the present application provides a comprehensive panel defect detection method based on a neural network, please refer to Figure 2 , Figure 2 The flowchart of the panel defect comprehensive detection method is as follows:

[0060] (1) A primary panel image is acquired through a low-power microscope, and the primary panel image is preprocessed to obtain a primary sample image; wherein the magnification of the low-power microscope is generally 2 times.

[0061] In an embodiment of the present invention, the process of preprocessing the primary panel image is as follows:

[0062] (1.1) Perform pixel stretching on the primary panel image to obtain a pixel stretched primary panel image.

[0063] The pixel stretching formula is:

[0064]

[0065] Wherein, dist represents the pixel value of pixel stretching, src represents the pixel value of the primary panel image, max(src) represents the maximum pixel value of the primary panel image, and min(src) represents the minimum pixel value of the primary panel image. Since the defects of the primary panel image are invisible to the naked eye, and it is impossible to directly use a neural network model to detect defects in the primary panel image, the embodiment of the present invention uses the Normalize standardization method to stretch the pixels of the primary panel image to achieve contrast enhancement; by stretching the pixels of the primary panel image, the pixel value of the primary panel image can be uniformly stretched to between 0 and 255. At the same time, other image enhancement methods can also be used, such as grayscale conversion and frequency domain enhancement.

[0066] (1.2) Binarizing the pixel-stretched primary panel image to obtain a primary sample image. The embodiment of the present invention uses image binarization to convert a grayscale image into a black-and-white image to simplify image data and facilitate subsequent rapid defect detection.

[0067] (2) Inputting the primary sample image into a pre-trained target detection model for defect detection to determine whether there is a defect. If a defect exists, obtaining defect location information based on the primary sample image; wherein the target detection model includes but is not limited to Faster RCNN or YOLO, etc.

[0068] In an embodiment of the present invention, the process of inputting the primary sample image into the pre-trained object detection model for defect detection is as follows:

[0069] (2.1) The first-level sample image is input into the pre-trained target detection model for defect detection to determine whether there are defects on the surface of the first-level sample image.

[0070] (2.2) If there is a defect on the surface of the primary sample image, a defect positioning frame including defect position information is obtained based on the primary sample image.

[0071] In an embodiment of the present invention, before inputting the first-level sample image into the target detection model for defect detection, it is necessary to perform model training on the target detection model, that is, firstly labeling the defect features of the original first-level sample image, and then inputting the labeled first-level sample image into the target detection model, and automatically learning the defect features through the target detection model, so that the target detection model can identify defects and locate defects.

[0072] (3) Based on the defect location information, a secondary panel image is acquired through a high-power microscope, and the secondary panel image is preprocessed to obtain a secondary sample image; wherein the magnification of the high-power microscope is generally 10 times.

[0073] In an embodiment of the present invention, the preprocessing process of the secondary panel image is the same as the preprocessing process of the primary panel image, which mainly includes pixel stretching and image binarization.

[0074] In an embodiment of the present invention, based on the defect location information, the process of obtaining the secondary panel image through a high-power microscope is as follows:

[0075] (3.1) The lens is debugged based on the defect positioning frame, and the center of the defect positioning frame is used as the lens center to magnify the low-power lens into a high-power lens.

[0076] (3.2) Obtaining a secondary panel image including defects corresponding to the defect positioning frame through a high-power lens. The embodiment of the present invention adjusts the magnification based on the defect positioning of the target detection model, and can obtain a secondary panel image with a high magnification and including defects.

[0077] (4) Inputting the secondary sample image into a pre-trained target classification model for defect classification to obtain a defect classification result and a defect area image; the target classification model includes but is not limited to Faster RCNN, etc.

[0078] In an embodiment of the present invention, before the secondary sample image is input into the target classification model for defect classification, it is necessary to perform model training on the target classification model, that is, firstly label the defect category of the original secondary sample image, and then input the labeled secondary sample image into the target classification model, and automatically learn the defect category features through the target classification model, so that the target classification model can identify the defect category.

[0079] In an embodiment of the present invention, the process of inputting the secondary sample image into the pre-trained target classification model for defect classification is as follows:

[0080] (4.1) Input the secondary sample image into the pre-trained target classification model for defect classification to obtain a defect target frame including defect category information, defect location information, and defect size information; see Figure 3 , Figure 3 It is a schematic diagram of the defect target frame on the surface of the secondary panel image, wherein the defect category information includes the defect category and whether the defect needs to be repaired.

[0081] (4.2) For the secondary sample image corresponding to the defect target frame that does not need to be repaired, the defect level is determined based on the defect position information and defect size information; the defect level includes but is not limited to three levels, O, T, and X, which represent slight, medium, and severe levels respectively; that is, if the defect is located at the edge of the panel and the defect size is small, the defect level can be determined as slight; if the defect is located in the middle of the panel or the defect size is large, the defect level can be determined as medium; if the defect is located in the middle of the panel and the defect size is large, the defect level can be determined as severe. Specifically, the embodiment of the present invention does not limit the defect level determination rules, which can be adjusted and set according to actual conditions.

[0082] (4.3) For the secondary sample image corresponding to the defect target frame that needs to be repaired, the secondary sample image is input into the pre-trained semantic segmentation model to extract the edge contour of the image around the defect.

[0083] (5) Determine whether the defect needs to be repaired based on the defect classification result. If repair is required, input the secondary sample image into the pre-trained semantic segmentation model to extract the edge contour of the image around the defect to obtain a plurality of edge contour images. The target classification model includes but is not limited to Mask RCNN, etc.

[0084] In an embodiment of the present invention, before the secondary sample image is input into the semantic segmentation model for edge contour extraction, the semantic segmentation model needs to be trained, that is, the image contour features of the original semantic segmentation model are first labeled, and then the labeled secondary sample image is input into the semantic segmentation model, and the image contour features are automatically learned by the semantic segmentation model, so that the semantic segmentation model can extract image contours.

[0085] In an embodiment of the present invention, the secondary sample image is input into a pre-trained semantic segmentation model to extract edge contours of images around defects to obtain a plurality of edge contour images in the following process:

[0086] (5.1) Determine whether the defect needs to be repaired based on the defect category information. If repair is required, input the secondary sample image into the pre-trained semantic segmentation model for image contour extraction to obtain several periodic image contours and connected domains around the defect target frame; that is, extract the complete image contour within a certain range around the defect target frame and the connected domains between the image contours.

[0087] (5.2) A repair template is constructed based on the distance information and geometric information of several periodic image contours around the defect target frame to obtain an edge contour image corresponding to at least one repair template.

[0088] (6) Based on several edge contour images, a repair path is planned for the defect area image to obtain a defect repair solution.

[0089] In an embodiment of the present invention, the process of planning a repair path for a defective area image based on a plurality of edge contour images is as follows:

[0090] (6.1) Based on the distance information between the periodic image contours around the defect target frame and the geometric structure information of the defect target frame, the patch template, the patch step size and the patch quantity are obtained.

[0091] Specifically, by obtaining the distance information of the periodic image contours around the defect target frame and the length and width information of the defect target frame, the number of periodic images, the number of rows and the number of columns affected by the defect can be estimated. For example, the image contours on the left and right sides of the defect target frame converge at m1, the length of the defect target frame is n1, the image contours on the upper and lower sides of the defect target frame converge at m2, and the width of the defect target frame is n2; then the image contours of the adjacent images with a length of n1 to m1 on the upper or lower side of the defect target frame can be obtained as the repair template, and the upper and lower distances of the adjacent images on the left or right side of the defect target frame are calculated, and the upper and lower distances of the adjacent images are used as the repair step; the number of repair templates that can be placed within the distance range n2 between the upper and lower sides of the defect target frame is calculated, and this is used as the repair quantity.

[0092] (6.2) Based on the patch template, the patch step, the patch quantity, and the connected domain of the template corresponding to the image contour, at least one patch template is used to formulate a patching scheme to obtain a defect patching scheme. That is, the corresponding connected domain is set according to the patch step, the patch quantity, and the patch path of the patch template to obtain the final patching scheme. For example, if the image contour of the upper or lower adjacent image is used as the patch template, the corresponding patching path is vertical; similarly, if the image contour of the left or right adjacent image is used as the patch template, the corresponding patching path is horizontal, thereby generating a targeted patching scheme.

[0093] In summary, the embodiment of the present invention first uses a target detection model to perform preliminary defect detection on low-magnification panel images, which can quickly determine whether there are defects at a general level; then uses a target classification model to classify defects on high-magnification panel images with defects, and can screen out defects that need to be repaired according to the classification results; then uses a semantic segmentation model to segment the contour information of the image around the defect; finally, based on the segmented contour information of the image around the defect, a repair template is constructed and a repair plan is formulated; panel defect detection, defect classification and defect repair are fully realized, and the applicability and functionality are strong.

[0094] In addition, in one embodiment, based on the same inventive concept as the above embodiment, the embodiment of the present invention provides a panel defect comprehensive detection system based on a neural network, and the system corresponds one-to-one with the method of embodiment 1, please refer to Figure 4 , Figure 4 The structure block diagram of the panel defect comprehensive detection system based on neural network is as follows:

[0095] A low-magnification sample acquisition unit, wherein the low-magnification sample acquisition unit acquires a primary panel image through a low-magnification microscope and pre-processes the primary panel image to obtain a primary sample image;

[0096] A target defect detection unit, wherein the target defect detection unit is used to input the primary sample image into a pre-trained target detection model for defect detection to determine whether the panel has a defect, and if a defect exists, obtain the position information of the defect based on the primary sample image;

[0097] A high-magnification sample acquisition unit, wherein the high-magnification sample acquisition unit acquires a secondary panel image through a high-magnification microscope based on the position information of the defect, and pre-processes the secondary panel image to obtain a secondary sample image;

[0098] An object category detection unit, wherein the object category detection unit is used to input the secondary sample image into a pre-trained object classification model for defect classification to obtain a defect classification result and a defect area image;

[0099] A target semantic segmentation unit, wherein the target semantic segmentation unit determines whether the defect needs to be repaired based on the defect classification result, and if repair is required, the secondary sample image is input into a pre-trained semantic segmentation model to extract edge contours of images around the defect to obtain a plurality of edge contour images;

[0100] A defect repair formulation unit is used to plan a repair path for a defect area image based on a plurality of edge contour images to obtain a defect repair solution.

[0101] It should be noted that each unit in the neural network-based panel defect comprehensive detection system in this embodiment corresponds one-to-one to each step in the neural network-based panel defect comprehensive detection method in the aforementioned embodiment. Therefore, the specific implementation method and technical effects achieved by this embodiment can refer to the implementation method of the aforementioned neural network-based panel defect comprehensive detection method, which will not be repeated here.

[0102] In addition, in one embodiment, the present application also provides a computer device, which includes a processor, a memory, and a computer program stored in the memory, and the computer program implements the method in the aforementioned embodiment when executed by the processor.

[0103] In addition, in one embodiment, the present application further provides a computer storage medium, on which a computer program is stored, and the computer program implements the method in the aforementioned embodiment when executed by a processor.

[0104] In some embodiments, the computer readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or various devices including one or any combination of the above memories. The computer may be various computing devices including intelligent terminals and servers.

[0105] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0106] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0107] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0108] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0109] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0110] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a multimedia terminal device (which can be a mobile phone, a computer, a television receiver, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0111] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A comprehensive panel defect detection method based on neural network, characterized in that: The method comprises the following steps: Acquire a primary panel image through a low-power microscope, and pre-process the primary panel image to obtain a primary sample image; Input the first-level sample image into the pre-trained object detection model for defect detection to determine whether the panel has defects. If defects exist, obtain the location information of the defects based on the first-level sample image; Based on the position information of the defect, a secondary panel image is obtained through a high-power microscope, and the secondary panel image is preprocessed to obtain a secondary sample image; Input the secondary sample image into the pre-trained target classification model for defect classification to obtain defect classification results and defect area images; Based on the defect classification result, it is determined whether the defect needs to be repaired. If repair is required, the secondary sample image is input into the pre-trained semantic segmentation model to extract the edge contour of the image around the defect to obtain a number of edge contour images; A repair path is planned for the defect area image based on a number of edge contour images to obtain a defect repair solution.

2. The panel defect comprehensive detection method based on neural network according to claim 1 is characterized in that: The process of preprocessing the primary panel image and the secondary panel image is as follows: Performing pixel stretching on the primary panel image and the secondary panel image respectively to obtain a pixel-stretched primary panel image and a secondary panel image; The pixel-stretched primary panel image and the secondary panel image are binarized respectively to obtain a primary sample image and a secondary sample image.

3. The panel defect comprehensive detection method based on neural network according to claim 1 is characterized in that: The first-level sample image is input into the pre-trained target detection model for defect detection, and the panel is used to determine whether there is a defect. If there is a defect, the process of obtaining the defect location information based on the first-level sample image is as follows: Input the first-level sample image into the pre-trained object detection model for defect detection to determine whether there are defects on the surface of the first-level sample image; If there is a defect on the surface of the primary sample image, a defect positioning frame including defect position information is acquired based on the primary sample image.

4. The panel defect comprehensive detection method based on neural network according to claim 1 is characterized in that: Based on the defect location information, the process of obtaining the secondary panel image through a high-power microscope is as follows: The lens is debugged based on the defect positioning frame, and the center of the defect positioning frame is used as the center of the lens to magnify the low-power lens into a high-power lens; The secondary panel image including the defect corresponding to the defect positioning frame is obtained through a high-power microscope.

5. The method for comprehensive panel defect detection based on neural network according to claim 1, characterized in that: The process of inputting the secondary sample image into the pre-trained target classification model for defect classification to obtain the defect classification result and the defect area image is as follows: Input the secondary sample image into the pre-trained target classification model for defect classification to obtain a defect target frame including defect category information, defect location information and defect size information; wherein the defect category information includes the defect category and whether the defect needs to be repaired; For the secondary sample image corresponding to the defect target frame that does not need to be repaired, the defect level is determined based on the defect position information and defect size information; For the secondary sample image corresponding to the defect target frame that needs to be repaired, the secondary sample image is input into the pre-trained semantic segmentation model to extract the edge contour of the image around the defect.

6. The method for comprehensive panel defect detection based on neural network according to claim 5, characterized in that: Based on the defect classification result, it is determined whether the defect needs to be repaired. If repair is required, the secondary sample image is input into the pre-trained semantic segmentation model to extract the edge contour of the image around the defect to obtain several edge contour images. The process is as follows: Based on the defect category information, it is determined whether the defect needs to be repaired. If repair is required, the secondary sample image is input into the pre-trained semantic segmentation model for image contour extraction to obtain several periodic image contours and connected domains around the defect target box; A repair template is constructed based on distance information and geometric information of a plurality of periodic image contours around the defect target frame to obtain an edge contour image corresponding to at least one repair template.

7. The panel defect comprehensive detection method based on neural network according to claim 6 is characterized in that: The process of planning the repair path of the defect area image based on several edge contour images to obtain the defect repair solution is as follows: Acquire a repair template, a repair step length, and a repair quantity based on distance information between periodic image contours around the defect target frame and geometric structure information of the defect target frame; Based on the repair template, the repair step size, the repair quantity and the connected domain of the template corresponding to the image contour, at least one repair template is used to perform repair path planning to obtain a defect repair solution.

8. A panel defect comprehensive detection system based on neural network, characterized in that: The system comprises: A low-magnification sample acquisition unit, wherein the low-magnification sample acquisition unit acquires a primary panel image through a low-magnification microscope and pre-processes the primary panel image to obtain a primary sample image; A target defect detection unit, wherein the target defect detection unit is used to input the primary sample image into a pre-trained target detection model for defect detection to determine whether the panel has a defect, and if a defect exists, obtain the position information of the defect based on the primary sample image; A high-magnification sample acquisition unit, wherein the high-magnification sample acquisition unit acquires a secondary panel image through a high-magnification microscope based on the position information of the defect, and pre-processes the secondary panel image to obtain a secondary sample image; An object category detection unit, wherein the object category detection unit is used to input the secondary sample image into a pre-trained object classification model for defect classification to obtain a defect classification result and a defect area image; A target semantic segmentation unit, wherein the target semantic segmentation unit determines whether the defect needs to be repaired based on the defect classification result, and if repair is required, the secondary sample image is input into a pre-trained semantic segmentation model to extract edge contours of images around the defect to obtain a plurality of edge contour images; A defect repair formulation unit is used to plan a repair path for a defect area image based on a plurality of edge contour images to obtain a defect repair solution.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the comprehensive panel defect detection method based on neural network described in any one of claims 1-7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for comprehensive panel defect detection based on a neural network as described in any one of claims 1 to 7 is implemented.

Images