Image classification method and device for display substrate

Abstract

An embodiment of the present invention provides an image classification method and device for display substrates, which relates to the field of image processing and is used to solve the problem of time-consuming and low efficiency caused by manual image classification. The image classification method for display substrates includes: obtaining an image to be detected; determining the type of the image to be detected from a first preset type set, the first preset type set including: a first image type, a second image type, and a third image type, wherein images belonging to the first image type are images without defects, images belonging to the second image type are blurred images, and images belonging to the third image type are defective images; when the type of the image to be detected is the third image type, using a first convolutional neural network to determine the defect information of the image to be detected, wherein the defective image is an image of a display substrate with defects, and the defect information includes the defect type of the display substrate on the image to be detected.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a method and device for classifying images of a display substrate. Background Art

[0002] With the continuous development of technology, more and more images need to be classified. Currently, image classification is usually done manually. However, the process of manual image classification often suffers from problems such as long time consumption and low efficiency. Summary of the Invention

[0003] Embodiments of the present invention provide a method and apparatus for classifying images of display substrates, which are used to solve the problems of time-consuming and low efficiency caused by manual image classification of display substrates.

[0004] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions:

[0005] In a first aspect, a method for image classification of a display substrate is provided, the method comprising: acquiring an image to be detected; determining the type of the image to be detected from a first preset type set, the first preset type set comprising: a first image type, a second image type, and a third image type, images belonging to the first image type are images without defects, images belonging to the second image type are blurred images, and images belonging to the third image type are defective images; when the type of the image to be detected is the third image type, using a first convolutional neural network, determining defect information of the image to be detected, the defective image being an image of a display substrate with defects, and the defect information including the defect type of the display substrate on the image to be detected.

[0006] In some embodiments, determining the type of the image to be detected includes: obtaining product information of the display substrate on the image to be detected, the product information including: shape information and / or structural information of the display substrate; matching the product information of the display substrate on the image to be detected with pre-configured product information; if the product information of the display substrate on the image to be detected completely matches the pre-configured product information, determining that the type of the image to be detected is the first image type; if the product information of the display substrate on the image to be detected does not match the pre-configured product information, determining that the type of the image to be detected is the second image type; if the product information of the display substrate on the image to be detected partially matches the pre-configured product information, determining that the type of the image to be detected is the third image type.

[0007] In some embodiments, obtaining product information of the display substrate on the image to be inspected includes: using an image recognition algorithm to obtain shape information and / or structural information of pixels in the display substrate on the image to be inspected.

[0008] In some embodiments, determining the type of the image to be detected includes: binarizing the image to be detected to obtain a black and white image of the image to be detected; if the black and white image of the image to be detected changes periodically, determining that the type of the image to be detected is a first image type; if the black and white image of the image to be detected does not change periodically, determining that the type of the image to be detected is a second image type; if the black and white image portion of the image to be detected changes periodically, determining that the type of the image to be detected is a third image type.

[0009] In some embodiments, determining the type of the image to be detected includes: using a second convolutional neural network to determine the image type of the image to be detected.

[0010] In some embodiments, obtaining the image to be detected includes: scanning an image storage device; determining whether there is a newly added image information storage file in the image storage device, the image information storage file including an image file and an address file; parsing the address file in the image information storage file to obtain the image storage address in the address file; obtaining the image corresponding to the image storage address from the image file as the image to be detected.

[0011] In some embodiments, determining defect information of the image to be inspected using a first convolutional neural network includes: using the first convolutional neural network, determining a defect type of the image to be inspected from a second preset type set, wherein the second preset type set includes at least one defect type. The display substrate image classification method further includes: if the defect type of the image to be inspected does not belong to the second preset type set, outputting the image to be inspected and receiving a first newly created defect type input by a user.

[0012] In some embodiments, the image classification method for a display substrate further includes: adding the first newly created defect type to a second preset type set, and updating the first convolutional neural network.

[0013] In some embodiments, the defect information also includes: a confidence level corresponding to the defect type, where the confidence level corresponding to the defect type is the confidence level that the display substrate on the image to be detected belongs to the defect type; when the confidence level is less than a preset threshold, the image to be detected is output and a re-inspection result input by the user is received; when the re-inspection result indicates that the defect type in the defect information of the image to be detected is incorrect, the defect type in the defect information of the image to be detected is corrected according to the re-inspection result.

[0014] In a second aspect, an image classification device for a display substrate is provided, which includes: an acquisition module and a processing module; the acquisition module is configured to acquire an image to be detected; the processing module is configured to determine the type of the image to be detected acquired by the acquisition module from a first preset type set, the first preset type set including: a first image type, a second image type and a third image type, an image belonging to the first image type is an image without defects, an image belonging to the second image type is a blurred image, and an image belonging to the third image type is a defective image; the processing module is further configured to, when the type of the image to be detected is the third image type, use a first convolutional neural network to determine defect information of the image to be detected, the defective image being an image of a display substrate with defects, and the defect information including the defect type of the display substrate on the image to be detected.

[0015] In a third aspect, an image classification device for a display substrate is provided, which includes: a memory and a processor, wherein the memory is used to store computer instructions, and the processor is used to run the computer instructions so that the display substrate implements the image classification method for the display substrate described in any of the above embodiments.

[0016] In a fourth aspect, a computer-readable storage medium is provided for storing computer instructions, so that when the image classification device for display substrates executes the computer instructions, the image classification method for display substrates described in any of the above embodiments is implemented.

[0017] The display substrate image classification method provided by an embodiment of the present invention can utilize a convolutional neural network (e.g., a first convolutional neural network) to determine the defect type of the display substrate in the image to be inspected. This allows machines to replace manual labor in determining the display substrate defect type. This reduces the time required to determine the display substrate defect type when there are a large number of images to be inspected, thereby improving efficiency and saving labor costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0019] Figure 1 A structural diagram of a computer integrated manufacturing system provided by an embodiment of the present invention;

[0020] Figure 2 A structural diagram of an image classification device for a display substrate provided by an embodiment of the present invention;

[0021] Figure 3 A flow chart of an image classification method for a display substrate provided by an embodiment of the present invention;

[0022] Figure 4A An image to be detected belonging to the first image type provided in an embodiment of the present invention;

[0023] Figure 4B An image to be detected belonging to the second image type provided in an embodiment of the present invention;

[0024] Figure 4C An image to be detected belonging to the third image type provided in an embodiment of the present invention;

[0025] Figure 5 A flowchart of determining defect information of an image to be inspected provided by an embodiment of the present invention;

[0026] Figure 6 A schematic diagram of an image classification method for a display substrate provided by an embodiment of the present invention;

[0027] Figure 7 This is a structural diagram of an image classification device for a display substrate provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0029] Unless the context requires otherwise, throughout the specification and claims, the term "comprise" and its other forms, such as the third person singular form "comprises" and the present participle form "comprising", are to be interpreted as open and inclusive, that is, "including, but not limited to". In the description of the specification, the terms "one embodiment", "some embodiments", "exemplary embodiments", "example", "specific example" or "some examples" are intended to indicate that the particular features, structures, materials or characteristics associated with the embodiment or example are included in at least one embodiment or example of the present disclosure. The schematic representation of the above terms does not necessarily refer to the same embodiment or example. In addition, the particular features, structures, materials or characteristics may be included in any one or more embodiments or examples in any appropriate manner.

[0030] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, unless otherwise specified, "plurality" means two or more.

[0031] When describing some embodiments, the expressions "coupled" and "connected" and their derivatives may be used. For example, when describing some embodiments, the term "connected" may be used to indicate that two or more components are in direct physical or electrical contact with each other. For another example, when describing some embodiments, the term "coupled" may be used to indicate that two or more components are in direct physical or electrical contact. However, the term "coupled" or "communicatively coupled" may also refer to two or more components that are not in direct contact with each other, but still cooperate or interact with each other. The embodiments disclosed herein are not necessarily limited to the contents of this document.

[0032] “At least one of A, B and C” has the same meaning as “at least one of A, B or C” and both include the following combinations of A, B and C: A only, B only, C only, the combination of A and B, the combination of A and C, the combination of B and C, and the combination of A, B and C.

[0033] “A and / or B” includes the following three combinations: A only, B only, and a combination of A and B.

[0034] "Plurality" means at least two.

[0035] The use of "adapted to" or "configured to" herein is intended to be open and inclusive language that does not exclude devices adapted or configured to perform additional tasks or steps.

[0036] Additionally, the use of “based on” is meant to be open and inclusive, as a process, step, calculation, or other action “based on” one or more stated conditions or values ​​may, in practice, be based on additional conditions or values ​​beyond those stated.

[0037] The display device may be an LCD (Liquid Crystal Display) device; it may also be a self-luminous display device, such as an OLED (Organic Light Emitting Diode) display device, a QLED (Quantum Dot Light Emitting Diodes) display device, or a Micro LED (Micro Light-Emitting Diode). For example, the display device includes an array substrate. In an LCD, the array substrate may be a TFT array substrate; in an OLED, QLED, Micro LED, etc., the array substrate may be a driver backplane including pixel circuits.

[0038] In this embodiment, the display substrate may be a product related to the display device, that is, a product obtained after completing at least one process in the production of the display device. For example, in an LCD display device, the display substrate may be a TFT array substrate or an opposing substrate; in a self-luminous display device, the display substrate may be a driver backplane.

[0039] The production process of the display substrate includes multiple steps, and the products obtained from some steps (at least one step) need to be inspected to determine information such as the defect type of the display substrate.

[0040] In this embodiment, the image of the display substrate refers to an image of a product obtained after completing at least one process in the production process of the display substrate; specifically, it may refer to a substrate obtained by forming at least one pattern layer on a base substrate. The pattern layer refers to a pattern formed on the base substrate through a patterning process.

[0041] For example, the process of preparing a display substrate included in an LCD may include: sequentially forming a gate metal pattern layer (including gate lines and TFT gates), an active layer, and a source / drain metal pattern layer (including TFT source electrodes, TFT drain electrodes, and data lines) on a base substrate. The base substrate with the gate metal pattern layer formed thereon is a related product of the display substrate (e.g., referred to as Product A); the base substrate with the gate metal pattern layer and the active layer formed thereon is another related product of the display substrate (e.g., referred to as Product B); the base substrate with the gate metal pattern layer, the active layer, and the source / drain metal pattern layer formed thereon is yet another related product of the display substrate (e.g., referred to as Product C), and so on.

[0042] Taking the detection of defects of related products including display substrates in LCDs as an example, an image acquisition device may be used to capture images of the related products first, and then the defect type of the captured images may be determined.

[0043] like Figure 1 As shown, an embodiment of the present invention provides a Computer Integrated Manufacturing (CIM) system 100. The CIM system may include an image acquisition device 110, an image storage device 120, an image classification device 130 for displaying substrates, and a manufacturing execution device 140.

[0044] The image acquisition device is configured to acquire images of products related to display substrates in LCDs and transmit the acquired images to an image storage device. The image storage device is configured to store the images acquired by the image acquisition device. The display substrate image classification device is configured to classify the images stored in the image storage device and transmit the classification results to the manufacturing execution device. The manufacturing execution device is configured to receive the classification results transmitted by the display substrate image classification device.

[0045] For example, the image acquisition device may be an AOI (Automatic Optical Inspection) device.

[0046] Those skilled in the art should understand that AOI equipment is a testing equipment widely used in industries such as liquid crystal and microelectronics.

[0047] This embodiment does not limit the type of image storage device. For example, the image storage device may be a device with a database. The database may be a relational database that can store picture information of multiple images and the images corresponding thereto. For another example, the image storage device may be a distributed file system (DFS), for example, a distributed file system includes a client and a server. For another example, the image storage device may be a file transfer protocol (FTP), for example, a file transfer protocol includes an FTP server and an FTP client. For another example, the image storage device may be a network attached storage (NAS) device.

[0048] In some embodiments, as Figure 2 As shown, the image classification device 130 for display substrates may include at least one processor 131 and a memory 132 .

[0049] The processor 131 may be one or more general-purpose central processing units (CPUs), microprocessors, application-specific integrated circuits (ASICs), or integrated circuits for controlling program execution in some embodiments of the present disclosure; the CPU may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). A processor 301 herein may refer to one or more devices, circuits, or processing cores for processing data (e.g., computer program instructions).

[0050] The memory 132 can store an operating system and instructions (e.g., computer instructions), and may include, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, or optical storage. The memory 202 stores operating system code.

[0051] For example, the processor 131 reads instructions stored in the memory 132 to cause the display substrate image classification device 130 to implement the display substrate image classification method of the following embodiment to determine the defect type of the display substrate in the image to be inspected. Alternatively, the processor 131 uses internally stored instructions to cause the display substrate image classification device 130 to implement the display substrate image classification method of the following embodiment to determine the defect type of the display substrate in the image to be inspected. When the processor 131 implements the method of the following embodiment by reading instructions stored in the memory 132, the memory 132 stores instructions for implementing the display substrate image classification method provided in this embodiment.

[0052] In other embodiments, Figure 2 The image classification device 130 of the display substrate may further include a receiver 133 and a transmitter 134 .

[0053] The receiver 133 is configured to receive images stored in the image storage device. For example, the receiver 133 can be connected to the routing device via wired or wireless communication to receive images sent by the routing device.

[0054] The transmitter 134 may be connected to the terminal in a wired or wireless communication manner, and is configured to send the defect type of the display substrate on the image to be inspected to the terminal.

[0055] like Figure 3 As shown, an embodiment of the present invention provides a display substrate image classification method. The method can be executed by a CIM system, a display substrate image classification device, or the hardware and / or software in the display substrate image classification device, for example, a combination of a processor and memory, or a computer program called an automated defect classification (ADC) system. The display substrate image classification method includes the following steps S101 to S103.

[0056] S101: Acquire an image to be detected.

[0057] The image to be inspected is an image of a display substrate collected by an AOI device.

[0058] For example, for 25 display substrates included in the same batch (Lot), the AOI equipment will capture images PA1-PA25 of products A1-A25, images PB1-PB25 of products B1-B25, and images PC1-PC25 of products C1-C25.

[0059] In some embodiments, the above S101 can be specifically implemented through the following steps A1 to A4.

[0060] A1. Scanned image storage device.

[0061] For example, after the AOI device completes collecting images of this batch, i.e., after collecting images of PA1 to PA25, PB1 to PB25, and PC1 to PC25, the AOI device saves the collected images in an image file (e.g., a .Qpanel file), saves the image storage addresses corresponding to the images in an address file (e.g., a .Glass file), and saves the image file and the address file in an image information storage file (e.g., a .lot file). The .lot file and the device information of the AOI device are then sent to the image storage device, and the image classification device of the display substrate scans the image storage device at intervals (e.g., 10 minutes).

[0062] It should be noted that this embodiment does not limit the content stored in the .Qpanel file. For example, the .Qpanel file stores images captured by AOI equipment. For another example, the .Qpanel file stores information about the display substrate, such as the model of the display substrate.

[0063] It should be noted that this embodiment does not limit the interval time for the image classification device of the display substrate to scan the image storage device. For example, the scanning interval time can be 5 minutes or 15 minutes.

[0064] A2. Determine whether a newly added image information storage file exists in the image storage device, where the image information storage file includes an image file and an address file.

[0065] Among them, the image file is a file that stores the image to be detected captured by the AOI device, such as the above-mentioned .Qpanel file; the address file is a file that stores the image storage address corresponding to each image to be detected, such as the above-mentioned .Glass file; and the image information storage file is a file that stores the image file and the address file, such as the above-mentioned .lot file.

[0066] A3. Parse the address file in the image information storage file to obtain the image storage address in the address file.

[0067] A4. Obtain the image corresponding to the image storage address from the image file as the image to be detected.

[0068] For example, the display substrate image classification device scans the image storage device every 10 minutes to determine whether a new .lot file exists in the image storage device. If the display substrate image classification device determines that a new .lot file exists in the image storage device, the display substrate image classification device parses the .Glass file stored in the .lot file to obtain the image storage address stored in the .Glass file. Based on the obtained image storage address, the device then retrieves the image corresponding to the image storage address from the .Qpanel file as the image to be detected.

[0069] In this embodiment, the following S102 and S103 may be performed on each image to be detected.

[0070] S102: Determine the type of the image to be detected from a first preset type set.

[0071] Among them, the above-mentioned first preset type set includes: a first image type, a second image type and a third image type. The image belonging to the first image type is an image without defects, the image belonging to the second image type is a blurred image, and the image belonging to the third image type is a defective image.

[0072] For example, a non-defective image is an image of a display substrate without defects, a blurred image is an image captured that is blurred, and a defective image is an image of a display substrate with defects. If the type of image to be inspected is a non-defective image or a blurred image, step S103 below is not performed. For example, the image to be inspected may be deleted, or the image to be inspected and its type may be output together. If the type of image to be inspected is a defective image, step S103 below is performed.

[0073] In some embodiments, the determination of the type of the image to be detected in S102 can be implemented through the following steps B1 to B3.

[0074] B1. Obtain product information of the display substrate on the image to be inspected.

[0075] The above-mentioned product information includes: shape information and / or structure information of the display substrate.

[0076] For example, an image recognition algorithm may be used to obtain product information of the display substrate in the image to be inspected.

[0077] In this embodiment, the algorithm principle of the image recognition algorithm is an algorithm based on Convolutional Neural Networks (CNNs).

[0078] For example, the image to be inspected is input into a convolutional neural network, and the image features are extracted using a deep feature extraction network included in the convolutional neural network, thereby obtaining product information of the display substrate on the image to be inspected.

[0079] In this embodiment, the shape information of the display substrate is shape information of at least one pattern layer formed after the at least one pattern layer is formed on the base substrate. For example, when the display substrate on the image to be detected has a gate metal pattern layer formed thereon, the shape information of the display substrate may be the shape of a gate line.

[0080] The structural information of the display substrate is the connection mode between at least two (or more) pattern layers formed on the base substrate. For example, when the display substrate on the image to be detected is formed with a gate metal pattern layer, an active layer, and a source / drain metal pattern layer, the structural information of the display substrate can be the connection mode between these three. For example, the connection mode can be that the data line and the source electrode of the TFT are connected through a via, or the data line and the source electrode of the TFT are connected in other possible modes.

[0081] In some embodiments, the above step B1 may include: using an image recognition algorithm to obtain shape information and / or structural information of pixels in the display substrate in the image to be detected.

[0082] The pixels in the display substrate are the minimum repeating units distributed along the row and column directions in the display area (Active Area, AA) of the display substrate. If the display substrate is an array substrate included in an LCD, each minimum repeating unit includes a pixel electrode and a thin-film transistor connected to the pixel electrode. If the display substrate is a driving backplane included in a self-luminous display device (such as an OLED), each minimum repeating unit includes a pixel circuit that provides a driving signal to each light-emitting unit.

[0083] Taking the display substrate as an array substrate in an LCD as an example, the pixel shape information is the shape of the pixel electrode in each minimum repeating unit. For example, the shape of the pixel electrode can be the shape of a single-domain pixel electrode or the shape of a multi-domain (for example, two-domain) pixel electrode.

[0084] The structural information of the pixel is the connection mode between the pixel electrode and the thin film transistor TFT in each minimum repeating unit. For example, the pixel electrode and the thin film transistor TFT are connected through a via hole.

[0085] B2. Match the product information of the display substrate on the image to be inspected with pre-configured product information.

[0086] The pre-configured product information is product information of a standard product (a product without defects). For example, the image classification device for a display substrate is pre-configured with product information of products related to the display substrate. For example, the shape information and / or structural information of a display substrate having a gate metal pattern layer, the shape information and / or structural information of a display substrate having a gate metal pattern layer and an active layer, the shape information and / or structural information of a display substrate having a gate metal pattern layer, an active layer, and a source / drain metal pattern layer, and the shape information and / or structural information of pixels in the display substrate.

[0087] For example, a convolutional neural network is first used to learn the pre-configured product information of the display substrate with a gate metal pattern layer stored in the image classification device of the display substrate. That is to say, the convolutional neural network is first made to remember the pre-configured product information of the display substrate with a gate metal pattern layer, and then the product information of the display substrate with a gate metal pattern layer on the image to be detected is compared with the pre-configured product information to see whether they are the same. If they are the same, the two match; if they are not the same, the two do not match.

[0088] To give another example, a convolutional neural network is first used to learn the shape information and / or structural information of the pixels in the preconfigured display substrate stored in the image classification device of the display substrate. That is to say, the convolutional neural network is first made to remember the shape information and / or structural information of the pixels in the preconfigured display substrate, and then the shape information and / or structural information of the pixels on the image to be detected is compared with the shape information of the preconfigured pixels to see whether they are the same. If they are roughly the same, the two match; if they are not the same, the two do not match.

[0089] The matching results include: the product information of the display substrate on the image to be detected completely matches, completely does not match, and partially matches the pre-configured product information.

[0090] For example, a complete match is that the shape information and structural information of the preconfigured product are exactly the same as the shape information and structural information of the display substrate on the image to be detected; a complete mismatch is that the shape information and structural information of the preconfigured product are completely different from the shape information and structural information of the display substrate on the image to be detected; a partial match is that the shape information of the preconfigured product is the same as the shape information of the display substrate on the image to be detected, but the structural information is different; or, a partial match is that the structural information of the preconfigured product is the same as the structural information of the display substrate on the image to be detected, but the shape information is different.

[0091] Those skilled in the art should understand that Convolutional Neural Networks (CNNs) consists of a five-layer structure consisting of an input layer, a convolution layer, a pooling layer, a fully connected layer, and an output layer. It is a deep feedforward artificial neural network whose artificial neurons can respond to surrounding units within a certain coverage area. It has excellent performance in large-scale image processing and has been applied in the field of image recognition.

[0092] For example, when the display substrate on the image to be detected has gate lines formed thereon, the shape of the gate lines is matched with the shape information of the pre-configured gate lines. If the shapes of the two are exactly the same, it can be considered that the product information of the two is completely matched; if the shapes of the two are completely different, it can be considered that the product information of the two is completely mismatched.

[0093] For another example, in a case where the display substrate on the image to be detected is formed with a data line and a source electrode of a TFT, the structural information (such as the connection method) between the data line and the source electrode of the TFT is matched with the connection method between the pre-configured data line and the source electrode of the TFT. If the data line and the source electrode of the TFT on the image to be detected are connected through a via hole, while the pre-configured data line and the source electrode of the TFT are not connected through a via hole, it can be considered that the structural information of the two is completely mismatched.

[0094] As another example, in the case where the display substrate is an array substrate in an LCD, taking the pixel shape information as the shape information of all pixels in the array substrate as an example, the shape information of all pixels in the array substrate is matched with the shape information of pre-configured pixels. If the shape information of all pixels in the display substrate on the image to be detected is exactly the same as the shape information of the pre-configured pixels, then the two are considered to be completely matched; if the shape information of some pixels in the display substrate on the image to be detected can be matched with the shape information of the pre-configured pixels, and there is a difference between the shape of another part of the pixels on the image to be detected and the shape of the pre-configured pixels when matched (for example, there is a missing part, there is a foreign object (particle), etc.), then the two are considered to be partially matched; if the shape information of all pixels in the display substrate on the image to be detected is completely different from the shape information of the pre-configured pixels, then the two are considered to be completely mismatched.

[0095] As another example, in the case where the display substrate is an array substrate in an LCD, taking the shape information of a pixel in the array substrate as an example, the shape information of the pixel in the array substrate is matched with the shape information of the pre-configured pixel. If the shape information of the pixel in the display substrate on the image to be detected is exactly the same as the shape information of the pre-configured pixel, then the two are considered to be completely matched; if the shape information of the pixel in the display substrate on the image to be detected can partially match the shape information of the pre-configured pixel, and there is a difference between the shape of the pixel on the image to be detected and the shape of the pre-configured pixel when they are matched (for example, there is a missing part, there is a foreign object (particle), etc.), then the two are considered to be partially matched; if the shape information of the pixel in the display substrate on the image to be detected is completely different from the shape information of the pre-configured pixel, then the two are considered to be completely mismatched.

[0096] B3. Determine the type of the image to be detected based on the matching results.

[0097] For example, if the product information of the display substrate on the image to be inspected completely matches the pre-configured product information, the type of the image to be inspected is determined to be the first image type (ie, an image without defects).

[0098] For another example, if the product information of the product on the image to be detected does not match the pre-configured product information at all, the type of the image to be detected is determined to be the second image type (ie, a blurred image).

[0099] For another example, if the product information of the product on the image to be inspected partially matches the pre-configured product information, the type of the image to be inspected is determined to be the third image type (ie, defective image).

[0100] In other embodiments, the determination of the type of the image to be detected in S102 can be implemented through the following steps C1 to C2.

[0101] C1. Binarize the image to be detected to obtain a black and white image of the image to be detected.

[0102] Those skilled in the art should understand that binarization processing is processing the grayscale values ​​of points on the image to 0 or 255. That is, binarization processing of the image to be detected can make the entire image to be detected appear in obvious black and white effect.

[0103] In some embodiments, after the image to be detected is binarized, it can be further reduced, for example, to one-fourth the size of the original image. After the reduction, the reduced image is then subjected to subsequent processing. Since the image to be detected is reduced, the amount of data processing required for subsequent steps (e.g., step C2) can be reduced, thereby improving the processing efficiency of step C2.

[0104] It should be noted that this embodiment does not limit the reduction ratio. For example, the image to be detected can be reduced to one-fourth of the original image, or the image to be detected can be reduced to one-half of the original image.

[0105] C2. Determine the type of image to be detected based on the black and white image.

[0106] In some embodiments, if the black and white image of the image to be inspected changes periodically, the type of the image to be inspected is determined to be the first image type (ie, an image without defects).

[0107] In this embodiment, after the image to be detected is binarized, each minimal repeating unit is processed into a black image, and the gate lines and data lines connected to each minimal repeating unit are processed into a white image.

[0108] like Figure 4A As shown, the periodic variation of the black and white images of the image to be detected can be understood as follows: along the row direction, the black image of each minimal repeating unit and the white images of the gate and data lines connected to each minimal repeating unit are repeatedly arranged. In other words, along the row direction, the black image of a minimal repeating unit and the white images of the gate and data lines connected to the minimal repeating unit are the same as the black images and the white images of the gate and data lines connected to the minimal repeating unit.

[0109] In other embodiments, if the black and white image portion of the image to be detected does not change periodically, the type of the image to be detected is determined to be the second image type (ie, a blurred image).

[0110] In this embodiment, Figure 4B As shown, the fact that the black and white images of the image to be detected do not change periodically can be understood as follows: along the row direction, the blackscale image of each minimal repeating unit and the whitescale images of the gate and data lines connected to each minimal repeating unit are not repeatedly arranged. In other words, along the row direction, the blackscale image of a minimal repeating unit and the whitescale images of the gate and data lines connected to the minimal repeating unit are different from the blackscale images and the whitescale images of the gate and data lines connected to the minimal repeating unit.

[0111] In some other embodiments, if the black and white image portion of the image to be detected changes periodically, the type of the image to be detected is determined to be the third image type (ie, defect image).

[0112] In this embodiment, the fact that the black and white images of the image to be detected do not change periodically can be understood as follows: along the row direction, the black images of a portion of the minimal repeating units, and the white images of the gate lines and data lines connected to this portion of the minimal repeating units are repeatedly arranged, while the black images of another portion of the minimal repeating units, and the white images of the gate lines and data lines connected to this portion of the minimal repeating units are not repeatedly arranged. For example, Figure 4C As shown, the image to be inspected has defects.

[0113] In other embodiments, the determination of the type of the image to be detected described in S102 above can be specifically implemented through the following step D1.

[0114] D1. Use the second convolutional neural network to determine the image type of the image to be detected.

[0115] In this embodiment, the second convolutional neural network is a convolutional neural network obtained after training. The training process thereof can be referred to the convolutional neural network training method described below and will not be repeated here.

[0116] For example, the convolutional neural network obtained after training (such as the second convolutional neural network) can remember the features corresponding to each image type, so that after the image to be detected is input into the second convolutional neural network, the second convolutional neural network can be used to determine the image type of the image to be detected.

[0117] S103: When the type of the image to be detected is the third image type, use the first convolutional neural network to determine defect information of the image to be detected.

[0118] The image belonging to the third image type is a defect image, which is an image of a display substrate with defects, and the defect information includes the defect type of the display substrate on the image to be detected.

[0119] In some embodiments, when the type of the image to be detected is the third image type, the image classification device for the display substrate will mark the defect position and defect size of the image to be detected based on the defect information of the image to be detected.

[0120] For example, Figure 5As shown, the image to be detected is decomposed into three layers of RGB layers, which are then input into the first convolutional neural network. The image depth feature extraction network included in the first convolutional neural network is used to perform multi-layer network convolution operations to extract image features and form a network feature map; the defect feature extraction network performs segmentation and pooling on the image depth network feature map and then extracts defect features to extract image defect features; the defect classification network classifies the defect features of the image defect area and determines whether this area has defects; the defect result is output through the output end of the fully connected layer, and the output result includes two parts: the defect classification result and the position of the defect area (coordinate information, width and height of the defect area, etc.).

[0121] In this embodiment, the image classification device for the display substrate also stores the priorities corresponding to each defect type. When the image classification device for the display substrate determines that there are two types of defects on the display substrate in the image to be detected, the image classification device for the display substrate will output the defect type corresponding to the higher priority according to the priority level.

[0122] In this embodiment, a convolutional neural network can be trained and used to determine the defect type of the display substrate in the image to be inspected. The convolutional neural network training process is described in the convolutional neural network training method below and is not further described here.

[0123] In some embodiments, the use of the first convolutional neural network in S103 above to determine defect information of the image to be detected may specifically include: using the first convolutional neural network to determine the defect type of the image to be detected from the second preset type set when the defect type of the image to be detected belongs to the second preset type set; and outputting the image to be detected and receiving a first new defect type input by the user when the defect type of the image to be detected does not belong to the second preset type set.

[0124] The second preset type set includes at least one defect type. For example, the defect type can be Remain, Open, and Short. The Open defect type can be further divided into Gate Open, SD Open, and PVX Open.

[0125] For example, when the second preset type set includes Remain, Open and short, and the defect type of the image to be detected is determined to be PT (Particle), since it does not belong to the defect type included in the second preset type set, the first convolutional neural network will output the image to be detected, and use manual image judgment to determine the defect type of the image to be detected. After manually determining the defect type of the image to be detected (i.e., the first newly created defect type), the defect type of the image to be detected determined manually is input into the first convolutional neural network.

[0126] In some embodiments, after the display substrate image classification device receives the first newly created defect type input by the user, the display substrate image classification method provided by the embodiment of the present invention further includes the following step 2.

[0127] Step 2: Add the first newly created defect type to the second preset type set and update the first convolutional neural network.

[0128] For example, after the user determines the first newly created defect type, the first newly created defect type is input into the first convolutional neural network to update the first convolutional neural network.

[0129] In some embodiments, the defect information further includes: a confidence level corresponding to the defect type, where the confidence level corresponding to the defect type is a confidence level that the display substrate in the image to be inspected belongs to the defect type.

[0130] For example, after determining the defect type of the display substrate in the image to be inspected, the first convolutional neural network also outputs a confidence level that the display substrate in the image to be inspected belongs to the determined defect type. The confidence level represents the accuracy of the first convolutional neural network's determination that the display substrate in the image to be inspected belongs to the determined defect type.

[0131] In some embodiments, if the confidence level is less than a preset threshold, the image classification device for the display substrate outputs the image to be inspected and receives a re-inspection result input by the user. If the re-inspection result indicates that the defect type in the defect information of the image to be inspected is incorrect, the defect type in the defect information of the image to be inspected is corrected based on the re-inspection result.

[0132] For example, Figure 6As shown, if the confidence level corresponding to the defect type in the defect information of the image to be inspected is less than a preset threshold, the first convolutional neural network will output the image to be inspected and manually determine the defect type of the image to be inspected (i.e., manual re-inspection). After the defect type is manually determined (i.e., the re-inspection result), the re-inspection result is input into the first convolutional neural network. If the re-inspection result is different from the defect type of the image to be inspected output by the first convolutional neural network, the first convolutional neural network will correct the defect type in the defect information of the image to be inspected based on the re-inspection result.

[0133] It should be noted that this embodiment does not limit the value of the preset threshold. For example, the preset threshold can be 0.6 or 0.7.

[0134] In some embodiments, as Figure 6 As shown, after the image classification device 130 of the display substrate determines the type of the image to be detected, the image classification device 130 of the display substrate sends the type of the image to be detected to the manufacturing execution device 140. After receiving the type of the image to be detected, the manufacturing execution device 140 will first lock the received image to be detected, and then in this hold state, according to the type of the received image to be detected, determine which station to flow the batch of images to be detected to. For example, if the type of the image to be detected is determined to be the first image type (i.e., an image without defects), the image to be detected is flowed to the next process station; if the type of the image to be detected is determined to be the third image type (i.e., a defective image), and it is determined that the defect type of the image to be detected belongs to the defect type included in the second preset type set, the image to be detected is flowed to the repair station corresponding to each defect type to repair the display substrate on the image to be detected. After repairing the display substrate on the defective image, the manufacturing execution device will feedback the repair information (e.g., the repair result) to the image classification device of the display substrate.

[0135] For example, when the images to be inspected are images PA1 to PA25 of products A1 to A25, and the types of images PA1 to PA10 are images without defects, and the types of images PA11 to PA25 are defective images, the manufacturing execution device will first lock these images after receiving them, and then, in the locked state, flow images PA1 to PA10 to the next station and flow images P11 to P25 to the maintenance station.

[0136] In other embodiments, Figure 6 As shown, the image classification device 130 of the display substrate will also send the report and warning information to relevant personnel by email. The report can display the classification results in a table, chart or other format.

[0137] In some embodiments, when the second preset type set includes the Remain, Open, and Short defect types, each defect type has a different priority. The higher the occurrence rate of each defect type and the more severe the defect type's impact on the yield of the display substrate, the higher its priority. For example, the priority order of the three defect types Remain, Open, and Short is: Open > Remain > Short.

[0138] This embodiment provides a convolutional neural network training method, which may include the following steps E1 to E7.

[0139] E1. Obtain the image to be learned.

[0140] The above-mentioned image to be learned is an image that is manually judged to belong to a certain defect type.

[0141] In this embodiment, each defect type has a different priority (i.e., the occurrence rate of each defect type). The convolutional neural network can first obtain the images to be learned corresponding to the defect type with a high priority. In this way, the convolutional neural network can prioritize the defect type with a high occurrence rate.

[0142] For example, we first use manual image judgment to select 1,000 defect images of the Open type and input them into the convolutional neural network.

[0143] E2. Divide the images to be learned into training set and validation set.

[0144] The images in the training set and the validation set are different from each other, and the ratio of the training set to the validation set is usually 4:1.

[0145] For example, when the images to be learned include 1000 images, the number of images to be learned in the training set may be 800, and the number of images to be learned in the validation set may be 200.

[0146] It should be noted that this embodiment does not limit the number of images included in the training set and the validation set. For example, the number of images to be learned in the training set can be greater than 500, and the remaining images can be used as the validation set.

[0147] E3. Use the images to be learned in the training set to train the third convolutional neural network to obtain the first convolutional neural network.

[0148] For example, the third convolutional neural network uses the to-be-learned images in the training set to train the third convolutional neural network, thereby obtaining a trained first convolutional neural network.

[0149] For example, the image to be learned is input into the third convolutional neural network, the weights of the various parameters of the third convolutional neural network are the first weight values, and the third convolutional neural network outputs a classification result; the classification result is then evaluated using the type of image to be learned input by the user to obtain an evaluation result; according to the evaluation result, the weights of the various parameters of the third convolutional neural network are set to the second weight value, thereby obtaining an updated third convolutional neural network.

[0150] In some embodiments, the third convolutional neural network may include 5 convolutional layers, 5 pooling layers, and 2 fully connected layers. The connection mode of the preset convolutional neural network may be: first convolutional layer, first pooling layer, second convolutional layer, second pooling layer, third convolutional layer, third pooling layer, fourth convolutional layer, fourth pooling layer, fifth convolutional layer, fifth pooling layer, first fully connected layer, and second fully connected layer.

[0151] E4. Use the first convolutional neural network to determine the defect type of each image to be learned in the validation set.

[0152] E5. Obtain the defect type of each image to be learned in the verification set input by the user.

[0153] For example, the user judges the images to be learned in the validation set to determine the defect types of these images to be detected.

[0154] E6. Calculate the accuracy. TP (True Positive) is the number of images that are predicted to be positive and are actually positive; FP (False Positive) is the number of images that are predicted to be positive and are actually negative.

[0155] For example, taking the defect type as Remain defect, TP is the number of images whose defect type is Remain defect determined by the user and whose defect type is also Remain defect predicted by the first convolutional neural network; FP is the number of images whose defect type is Remain defect predicted by the first convolutional neural network but whose actual defect type determined by the user is not Remain defect.

[0156] E7. Calculate recall

[0157] Among them, FN (False Negative) is the number of images predicted to be negative but actually positive.

[0158] For example, taking the defect type as Remain defect as an example, FN is the number of images whose defect type predicted by the first convolutional neural network is not Remain defect, but the actual defect type of the image determined by the user is Remain defect.

[0159] In this embodiment, if the accuracy and recall rates reach the preset values, the trained convolutional neural network can be used to classify images. Correspondingly, if the accuracy and recall rates do not reach the preset values, the convolutional neural network will continue to be trained until the accuracy and recall rates reach the preset values.

[0160] It should be noted that this embodiment does not limit the value of the preset value. For example, the preset value may be 0.7. For another example, the preset value may be 0.8.

[0161] like Figure 7 As shown, this embodiment provides an image classification device 600 for a display substrate, and the image classification device 600 for a display substrate may include an acquisition module 601 and a processing module 602. The acquisition module 601 is configured to acquire an image to be detected. The processing module 602 is configured to determine the type of the image to be detected acquired by the acquisition module 601 from a first preset type set, and the first preset type set includes: a first image type, a second image type, and a third image type. An image belonging to the first image type is an image without defects, an image belonging to the second image type is a blurred image, and an image belonging to the third image type is a defective image. The processing module 602 is also configured to use a first convolutional neural network to determine the defect information of the image to be detected when the type of the image to be detected is the third image type. The defective image is an image of a display substrate with defects, and the defect information includes the defect type of the display substrate on the image to be detected.

[0162] In some embodiments, the processing module 602 is specifically configured to obtain product information of the display substrate on the image to be detected, and the product information includes: shape information and / or structural information of the display substrate; the processing module 602 is specifically configured to match the product information of the display substrate on the image to be detected with the preconfigured product information; if the product information of the display substrate on the image to be detected completely matches the preconfigured product information, the type of the image to be detected is determined to be the first image type; if the product information of the display substrate on the image to be detected completely does not match the preconfigured product information, the type of the image to be detected is determined to be the second image type; if the product information of the display substrate on the image to be detected partially matches the preconfigured product information, the type of the image to be detected is determined to be the third image type.

[0163] In some embodiments, the processing module 602 is specifically configured to obtain shape information and / or structural information of pixels in the display substrate in the image to be detected by using an image recognition algorithm.

[0164] In some embodiments, the processing module 602 is specifically configured to perform binarization processing on the image to be detected to obtain a black and white image of the image to be detected; if the black and white image of the image to be detected changes periodically, the type of the image to be detected is determined to be the first image type; if the black and white image of the image to be detected does not change periodically, the type of the image to be detected is determined to be the second image type; if the black and white image portion of the image to be detected changes periodically, the type of the image to be detected is determined to be the third image type.

[0165] In some embodiments, the processing module 602 is specifically configured to use a second convolutional neural network to determine the image type of the image to be detected.

[0166] In some embodiments, the acquisition module 601 is specifically configured to scan an image storage device; determine whether there is a newly added image information storage file in the image storage device, and the image information storage file includes an image file and an address file; parse the address file in the image information storage file to obtain the image storage address in the address file; obtain the image corresponding to the image storage address from the image file as the image to be detected.

[0167] In some embodiments, processing module 602 is specifically configured to use a first convolutional neural network to determine a defect type of the image to be inspected from a second preset type set, where the second preset type set includes at least one defect type. Processing module 602 is further configured to, if the defect type of the image to be inspected does not belong to the second preset type set, output the image to be inspected and receive a first newly created defect type input by a user.

[0168] In some embodiments, the image classification device for a display substrate provided in this embodiment further includes: an updating module, which is configured to add the first newly created defect type to the second preset type set and update the first convolutional neural network.

[0169] In some embodiments, the defect information also includes a confidence level corresponding to the defect type, where the confidence level corresponding to the defect type indicates the confidence level that the display substrate in the image to be inspected is of the defect type. Processing module 602 is further configured to output the image to be inspected if the confidence level is less than a preset threshold and receive a re-inspection result input by the user.

[0170] The image classification device for display substrates provided in this embodiment further includes: a correction module, which is configured to correct the defect type in the defect information of the image to be detected according to the re-inspection result when the re-inspection result indicates that the defect type in the defect information of the image to be detected is incorrect.

[0171] An embodiment of the present invention also provides an image classification device for a display substrate, such as a server. The image classification device for a display substrate may include a memory and a processor. The memory is used to store computer instructions, and the processor is used to run computer instructions, so that the image classification device for a display substrate implements the image classification method for a display substrate described in the above embodiment.

[0172] An embodiment of the present invention further provides a computer-readable storage medium for storing computer instructions, so that when a display substrate image classification device (e.g., a server) executes the computer instructions, the display substrate image classification method described in any of the above embodiments is implemented. The computer-readable storage medium may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0173] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for classifying images of a display substrate, characterized in that: include: Acquire an image to be detected; the image to be detected is an image of a display substrate obtained by forming at least one pattern layer on a base substrate; Performing binarization processing on the image to be detected to obtain a black and white image of the image to be detected; If the black and white image of the image to be detected changes periodically, determining that the type of the image to be detected is the first image type; If the black and white image of the image to be detected does not change periodically, determining that the type of the image to be detected is the second image type; If the black and white image portion of the image to be detected changes periodically, it is determined that the type of the image to be detected is the third image type; an image belonging to the first image type is an image without defects, an image belonging to the second image type is a blurred image, and an image belonging to the third image type is an image with defects; When the type of the image to be detected is the third image type, the first convolutional neural network is used to determine the defect information of the image to be detected, the defect image is an image of a display substrate with defects, and the defect information includes the defect type of the display substrate on the image to be detected.

2. The image classification method of a display substrate according to claim 1, wherein: Determining the type of the image to be detected includes: Acquiring product information of the display substrate on the image to be detected, wherein the product information includes: shape information and / or structure information of the display substrate; Matching the product information of the display substrate on the image to be inspected with pre-configured product information; If the product information of the display substrate on the image to be detected completely matches the pre-configured product information, determining that the type of the image to be detected is the first image type; If the product information of the display substrate on the image to be detected does not match the pre-configured product information at all, determining that the type of the image to be detected is the second image type; If the product information of the display substrate on the image to be detected partially matches the pre-configured product information, it is determined that the type of the image to be detected is the third image type.

3. The image classification method of a display substrate according to claim 2, wherein: Acquiring product information of the display substrate on the image to be inspected includes: The shape information and / or structure information of the pixels in the display substrate on the image to be detected is obtained by using an image recognition algorithm.

4. The image classification method of a display substrate according to claim 1, wherein: The acquiring of the image to be detected comprises: Scanned image storage device; Determining whether a newly added image information storage file exists in the image storage device, wherein the image information storage file includes an image file and an address file; Parsing the address file in the image information storage file to obtain the image storage address in the address file; The image corresponding to the image storage address is obtained from the image file as the image to be detected.

5. The image classification method of a display substrate according to claim 1, wherein: Determining defect information of the image to be detected using a first convolutional neural network includes: using a first convolutional neural network, and determining, if the defect type of the image to be detected belongs to a second preset type set, the defect type of the image to be detected from the second preset type set, the second preset type set including at least one defect type; In a case where the defect type of the image to be detected does not belong to the second preset type set, the image to be detected is output, and a first newly created defect type input by a user is received.

6. The image classification method for a display substrate according to any one of claims 1 to 5, characterized in that: Also includes: Add the first newly created defect type to the second preset type set, and update the first convolutional neural network.

7. The image classification method of a display substrate according to claim 6, characterized in that: The defect information further includes: a confidence level corresponding to a defect type, where the confidence level corresponding to the defect type is a confidence level that the display substrate on the image to be inspected belongs to the defect type; When the confidence level is less than a preset threshold, outputting the image to be detected and receiving a re-inspection result input by the user; In a case where the re-inspection result indicates that the defect type in the defect information of the image to be inspected is incorrect, the defect type in the defect information of the image to be inspected is corrected according to the re-inspection result.

8. An image classification device for a display substrate, characterized in that: Includes acquisition module and processing module; The acquisition module is configured to acquire an image to be detected; the image to be detected is an image of a display substrate obtained by forming at least one pattern layer on a base substrate; The processing module is configured to perform binarization processing on the image to be detected to obtain a black and white image of the image to be detected; if the black and white image of the image to be detected shows periodic changes, then the type of the image to be detected is determined to be a first image type; if the black and white image of the image to be detected does not show periodic changes, then the type of the image to be detected is determined to be a second image type; if the black and white image portion of the image to be detected shows periodic changes, then the type of the image to be detected is determined to be a third image type; The images belonging to the first image type are images without defects, the images belonging to the second image type are blurred images, and the images belonging to the third image type are images with defects; The processing module is further configured to, when the type of the image to be detected is the third image type, use a first convolutional neural network to determine defect information of the image to be detected, the defective image being an image of a display substrate with defects, and the defect information including the defect type of the display substrate on the image to be detected.

9. An image classification device for a display substrate, characterized in that: include: A memory and a processor, wherein the memory is used to store computer instructions, and the processor is used to execute the computer instructions, so that the display substrate implements the image classification method for the display substrate according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that Used to store computer instructions so that when the image classification device for display substrates executes the computer instructions, the image classification method for display substrates according to any one of claims 1 to 7 is implemented.

Images