Image recognition model training method and device, and display substrate detection method

Abstract

Provided is a training method of an image recognition model for detecting a defect region in an image of a display substrate, the display substrate comprising a display area and a connection area, the method comprising: obtaining a first training sample comprising images of n display substrates; dividing each of the images of the n display substrates into a first sub-image and a second sub-image, the first sub-image being an image of the display area, and the second sub-image being an image of the connection area; inputting the first training sample into the image recognition model, the first training sample comprising the first sub-image and the second sub-image; and adjusting at least one feature parameter of the image recognition model to reduce a difference between an output value of the model and a training value of the first training sample. The model comprises a first parameter and a second parameter, the first parameter and the second parameter being respectively related to a grayscale difference between a normal region and a defect region in the first sub-image and the second sub-image, and the at least one feature parameter comprises at least one of the first parameter and the second parameter.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular, to a training method, apparatus, device, non-transitory computer-readable storage medium and program product for an image recognition model, and a detection method for a display substrate. Background Art

[0002] OLED (Organic Light-Emitting Diode) is a current-type organic light-emitting device, which emits light through the injection and recombination of carriers, and the luminous intensity is proportional to the injected current. OLED display devices are a type of display device that uses luminous OLEDs to display images and other information. OLED display devices have characteristics such as low power consumption, high brightness, and high response speed. During the process of manufacturing OLED display devices, defects may occur, such as foreign matter, bubbles, punctures, etc. Therefore, it is necessary to detect these defects more accurately to help optimize the process flow of the display device and improve the product yield of the display device.

[0003] The above information disclosed in this section is only for understanding the background of the technical concept of the present disclosure and therefore the above information may contain information that does not constitute the prior art. Summary of the Invention

[0004] In view of the above problems, embodiments of the present disclosure provide a method, apparatus, device, non-transitory computer-readable storage medium, and program product for training an image recognition model, and a method for detecting a display substrate.

[0005] According to a first aspect of the present disclosure, a method for training an image recognition model is provided, wherein the image recognition model is used to detect defective areas in an image of a display substrate, the display substrate including a display area and a connection area. The method comprises:

[0006] Acquire a first training sample, wherein the first training sample includes n images of a display substrate, where n is a positive integer greater than or equal to 1;

[0007] Dividing each of the images of the n display substrates into a first sub-image and a second sub-image, wherein the first sub-image is an image of a display area of ​​the display substrate, and the second sub-image is an image of a connection area of ​​the display substrate;

[0008] Inputting the first training sample into the image recognition model, wherein the first training sample includes a first sub-image and a second sub-image; and

[0009] adjusting at least one characteristic parameter of the image recognition model to reduce the difference between an output value of the image recognition model and a training value of the first training sample,

[0010] In which, the image recognition model includes a first parameter and a second parameter, the first parameter is related to the grayscale difference between the normal area and the defective area in the first sub-image, the second parameter is related to the grayscale difference between the normal area and the defective area in the second sub-image, and the at least one feature parameter includes at least one of the first parameter and the second parameter.

[0011] According to some exemplary embodiments, the method further comprises:

[0012] Acquire a second training sample, wherein the second training sample includes m images of the display substrate, where m is a positive integer greater than or equal to 1, and the m images of the display substrate are images that are missed by the image recognition model; and

[0013] The second training sample is input into the image recognition model to optimize the image recognition model.

[0014] According to some exemplary embodiments, the at least one characteristic parameter further includes a boundary threshold parameter, wherein the boundary threshold parameter is used to represent a boundary of a detection area of ​​the display substrate in an image of the display substrate, and the detection area includes a display area and a connection area of ​​the display substrate.

[0015] According to some exemplary embodiments, the at least one characteristic parameter further includes a position parameter of the region of interest, wherein the position parameter includes a coordinate position of at least one vertex of the region of interest in the image of the display substrate and a length and a width of the region of interest.

[0016] According to some exemplary embodiments, the first parameter is calculated according to the following formula:

[0017] P=a×R×OFS,

[0018] Wherein, P is the first parameter, a is a constant, R is the mean square error ratio of each defect in the image of the display substrate, and OFS is the difference between each defect in the image block.

[0019] According to some exemplary embodiments, the second parameter is calculated according to the following formula:

[0020] Q=QM1 / QM2,

[0021] Wherein, Q is the second parameter, QM1 is the grayscale threshold of the defect in the connection area, and QM2 is the grayscale threshold of the background of the connection area.

[0022] According to some exemplary embodiments, the display substrate further includes a plurality of first positioning marks provided in the display area and a plurality of second positioning marks provided in the connection area, and dividing each of the images of the n display substrates into the first sub-image and the second sub-image includes:

[0023] determining positions of the plurality of first alignment marks and the plurality of second alignment marks in an image of the display substrate; and

[0024] The image of the display substrate is divided into a first sub-image and a second sub-image according to positions of the plurality of first positioning marks and the plurality of second positioning marks in the image of the display substrate.

[0025] According to some exemplary embodiments, before inputting the first training sample into the image recognition model, the method further includes: segmenting the first sub-image into multiple image blocks, and segmenting the second sub-image into multiple image blocks according to a grayscale value difference threshold, wherein the at least one feature parameter includes the grayscale value difference threshold, and wherein the first training sample includes multiple image blocks of the first sub-image and multiple image blocks of the second sub-image.

[0026] According to some exemplary embodiments, the method further includes: adjusting the grayscale value difference threshold according to a category of the defect.

[0027] According to some exemplary embodiments, inputting the second training sample into the image recognition model to optimize the image recognition model includes:

[0028] inputting the second training sample into the image recognition model;

[0029] Adjusting the training parameters of the image recognition model to optimize the image recognition model,

[0030] The training parameters include at least one of the following items: the number of training samples input to the image recognition model each time, and the number of cycles during the training process.

[0031] According to some exemplary embodiments, before inputting the second training sample into the image recognition model, the method further includes:

[0032] classifying the m display substrate images in the second training sample according to reasons for missed detection; and

[0033] Different marks are performed on the missed defects in the images of the m display substrates according to categories of the images of the m display substrates.

[0034] According to some exemplary embodiments, the method further includes: comparing the grayscale value of the missed defect with the grayscale value of the normal area, and comparing the grayscale value of the missed defect with the grayscale value of the defective area according to the category of the images of the m display substrates; and adjusting the training parameters of the image recognition model according to the result of the comparison, wherein the training parameters include a grayscale threshold for image segmentation.

[0035] According to a second aspect of the present disclosure, a method for detecting a display substrate is provided, the method comprising:

[0036] acquiring an image of the display substrate;

[0037] Inputting the image of the display substrate into an image recognition model, wherein the image recognition model is obtained by the above-mentioned training method; and

[0038] A detection result of a defect in the display substrate is determined according to an output result of the image recognition model.

[0039] According to a third aspect of the present disclosure, there is provided a training device for an image recognition model, wherein the image recognition model is used to detect defective areas in an image of a display substrate, wherein the display substrate includes a display area and a connection area, wherein the training device includes:

[0040] an acquisition module, configured to acquire a first training sample, wherein the first training sample includes n images of a display substrate, where n is a positive integer greater than or equal to 1;

[0041] a dividing module, configured to divide each of the images of the n display substrates into a first sub-image and a second sub-image, wherein the first sub-image is an image of a display area of ​​the display substrate, and the second sub-image is an image of a connection area of ​​the display substrate;

[0042] an input module, configured to input the first training sample into the image recognition model, wherein the first training sample includes a first sub-image and a second sub-image; and

[0043] an adjustment module, configured to adjust at least one characteristic parameter of the image recognition model to reduce a difference between an output value of the image recognition model and a training value of the first training sample,

[0044] In which, the image recognition model includes a first parameter and a second parameter, the first parameter is related to the grayscale difference between the normal area and the defective area in the first sub-image, the second parameter is related to the grayscale difference between the normal area and the defective area in the second sub-image, and the at least one feature parameter includes at least one of the first parameter and the second parameter.

[0045] According to a fourth aspect of the present disclosure, there is provided an electronic device, including:

[0046] one or more processors;

[0047] a storage device for storing one or more programs,

[0048] When the one or more programs are executed by the one or more processors, the one or more processors execute the above method.

[0049] According to a fifth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, on which executable instructions are stored. When the instructions are executed by a processor, the processor executes the above method.

[0050] According to a sixth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the above method when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0052] Figure 1 It is a schematic diagram of the FPN model;

[0053] Figure 2 is a schematic plan view of a display substrate according to an embodiment of the present disclosure;

[0054] Figure 3 is a flowchart of a method for training an image recognition model according to an embodiment of the present disclosure;

[0055] Figure 4 schematically illustrating a portion of a boundary of a detection area of ​​the display substrate in an image of the display substrate;

[0056] Figure 5 schematically illustrating a region of interest on an image of the display substrate;

[0057] Figure 6 is a partial flow chart of a method for training an image recognition model according to an embodiment of the present disclosure;

[0058] Figure 7 is a partial flow chart of a method for training an image recognition model according to an embodiment of the present disclosure;

[0059] Figure 8 is a flow chart of a method for detecting a display substrate according to an embodiment of the present disclosure;

[0060] Figure 9is a detailed flow chart of a method for inspecting a display substrate according to some exemplary embodiments of the present disclosure;

[0061] Figure 10 A block diagram schematically illustrates a structure of a training device for an image recognition model according to an embodiment of the present disclosure; and

[0062] Figure 11 A block diagram of an electronic device suitable for implementing a training method for an image recognition model or a detection method for a display substrate according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0063] The technical solution of the present disclosure is further described in detail below through examples and in conjunction with the accompanying drawings. In the specification, the same or similar reference numerals indicate the same or similar components. The following description of the embodiments of the present disclosure with reference to the accompanying drawings is intended to explain the overall inventive concept of the present disclosure and should not be construed as limiting the present disclosure.

[0064] Furthermore, in the following detailed description, for ease of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present disclosure. However, it will be apparent that one or more embodiments can be practiced without these specific details.

[0065] It should be noted that although the terms "first," "second," etc. may be used herein to describe various parts, components, elements, regions, layers, and / or portions, these parts, components, elements, regions, layers, and / or portions should not be limited by these terms. Rather, these terms are used to distinguish one part, component, element, region, layer, and / or portion from another. Thus, for example, the first part, first member, first element, first region, first layer, and / or first portion discussed below may be referred to as a second part, second member, second element, second region, second layer, and / or second portion without departing from the teachings of the present disclosure.

[0066] For ease of description, spatially relative terms, such as "upper," "lower," "left," "right," etc., may be used herein to describe the relationship of one element or feature to another element or feature as shown in the figures. It should be understood that the spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if the device in the figures is turned over, elements described as "below" or "beneath" other elements or features would then be oriented "above" or "over" the other elements or features.

[0067] As used herein, the terms "substantially," "about," "approximately," "roughly," and other similar terms are used as terms of approximation rather than as terms of degree, and are intended to account for the inherent deviations in measured or calculated values ​​that would be recognized by one of ordinary skill in the art. To account for factors such as process fluctuations, measurement problems, and errors associated with the measurement of a particular quantity (i.e., limitations of the measurement system), "about" or "approximately" as used herein are inclusive of the stated value and mean within an acceptable range of deviation for the particular value as determined by one of ordinary skill in the art. For example, "approximately" can mean within one or more standard deviations, or within ±30%, ±20%, ±10%, ±5% of the stated value.

[0068] In this article, Open CV, or simply CV, refers to a cross-platform computer vision and machine learning software library released under the Apache 2.0 license (open source), running on Linux, Windows, Android, and Mac OS. It consists of a series of C functions and a small number of C++ classes, and provides interfaces for languages ​​like Python, Ruby, and MATLAB, implementing many common algorithms for image processing and computer vision.

[0069] An embodiment of the present disclosure provides a method for training an image recognition model, wherein the image recognition model is used to detect defective areas in an image of a display substrate, wherein the display substrate includes a display area and a connection area. The method includes: obtaining a first training sample, wherein the first training sample includes n images of the display substrate, where n is a positive integer greater than or equal to 1; dividing each of the n images of the display substrate into a first sub-image and a second sub-image, wherein the first sub-image is an image of the display area of ​​the display substrate, and the second sub-image is an image of the connection area of ​​the display substrate; inputting the first training sample into the image recognition model, wherein the first training sample includes the first sub-image and the second sub-image; and adjusting at least one feature parameter of the image recognition model to reduce the difference between an output value of the image recognition model and a training value of the first training sample, wherein the image recognition model includes a first parameter and a second parameter, wherein the first parameter is related to a grayscale difference between a normal area and a defective area in the first sub-image, and the second parameter is related to a grayscale difference between the normal area and the defective area in the second sub-image, and the at least one feature parameter includes at least one of the first parameter and the second parameter. In the embodiments of the present disclosure, during the training process of the image recognition model, the image of the display substrate is divided into a display area (i.e., the AA area) and a connection area (i.e., the PAD area). Parameters related to the grayscale of the display area and the connection area are set and adjusted to achieve the purpose of optimizing the model. In this way, the trained image recognition model can more accurately detect images of the display substrate and more accurately identify defects in the display substrate.

[0070] For example, the display device of an OLED display device includes an organic light-emitting material, which is easily affected by water and oxygen and usually requires thin film encapsulation. In the manufacturing process flow of the display device, the following process steps are included: tearing off the back film from the lower surface of the cut display substrate and attaching a U-shaped film, that is, the U-Lami process. During the U-Lami process, defects such as "foreign matter, bubbles, punctures" (also called badness) may appear, and these defects will appear in the entire area of ​​the display substrate. In an embodiment of the present disclosure, during the U-Lami process, an image acquisition device is set to take a picture of the display substrate to obtain an image of the display substrate, and then the image of the display substrate is recognized using an image recognition model to detect various defects that may appear in the display substrate.

[0071] For example, the image recognition model may employ a deep learning model such as an artificial neural network (ANN), a convolutional neural network (CNN), or a feature pyramid network (FPN). Herein, the embodiment of the present disclosure is described using the feature pyramid network (FPN) as an example, but it should be understood that the embodiment of the present disclosure is not limited to the feature pyramid network (FPN).

[0072] Figure 1 This is a schematic diagram of the FPN model. During image recognition, low-level features have less semantic information but accurate target locations, while high-level features have richer semantic information but rougher target locations.

[0073] Reference Figure 1 ,In the FPN deep learning algorithm, there is a bottom-up circuit, a top-down circuit, and multiple lateral ,connections.

[0074] In the bottom-up circuit, the bottom-up process is the forward propagation process of the neural network. The feature map will become smaller and smaller after being calculated by the convolution kernel. For example, for ResNets, the feature activation output is the output of the last residual block (i.e., residual block) of each stage. In the top-down circuit, the top-down process is to upsample (upsampling) the more abstract and semantically stronger high-level feature maps. The horizontal connection is to merge (merge) the upsampling result with the feature map of the same size generated from the bottom up. The two layers of features in the horizontal connection are the same in spatial size, so that the detailed information of the bottom layer can be used to locate the details. The low-resolution feature map is upsampled by 2 times, and then the upsampled map is merged with the corresponding bottom-up map by element-wise addition. This process is iterative until the final resolution map is generated.

[0075] exist Figure 1 In the example shown, high-level features are upsampled by a factor of 2 and then combined with the corresponding features of the previous layer. The previous layer must pass through a 1×1 convolution kernel before it can be used. The purpose is to change the number of channels, which should be the same as the number of channels of the next layer. The combination method is to add elements.

[0076] Figure 2 : is a schematic plan view of a display substrate according to an embodiment of the present disclosure. Figure 2 The display substrate 10 includes a display area (i.e., AA area) 11 and a connection area (i.e., PAD area) 12. The display substrate 10 may further include a plurality of positioning marks provided thereon. For the convenience of description, the positioning mark provided in the display area 11 is described as a first positioning mark 13, and the positioning mark provided in the connection area 12 is described as a second positioning mark 14.

[0077] In an embodiment of the present disclosure, an algorithm is used to partition and mark captured images of a display substrate (e.g., into display and connection areas). Defects are then detected based on grayscale differences from normal areas. For example, these defects include foreign matter, bubbles, puncture wounds, and other types. In an embodiment of the present disclosure, during the training process of the image recognition model, the model can be improved through parameter sensitivity adjustment and model optimization.

[0078] Regarding parameter sensitivity adjustment, after the image recognition model is constructed, it is combined with a large number of detected and classified defect images to mark and train the defect features in the display area and the connection area, analyze the grayscale differences between the normal area and the defect area, and optimize a large number of feature parameters related to the image recognition model. For example, the feature parameters may include algorithm parameters and ROI parameters (region of interest). Among them, the algorithm parameters may include the display area AI and CV (computer vision) algorithm switches, display area defect parameters, the positioning reliability of the display area positioning mark, the mean square error of foreign body punctures in the display area, the connection area background grayscale threshold, the threshold for the poor and bangs distance detection range boundary during CV detection, etc. The ROI parameters may include the fine-tuning of the ROI area after selecting the ROI area, the coordinate position of the upper left corner of the upper left corner area selection box, the length and width of the selection box, etc.

[0079] As for the model optimization, the missed images can be classified, and then the missed images can be input into the image recognition model again according to the classification, and the image recognition model can be trained again to optimize the image recognition model.

[0080] Figure 3 1 is a flow chart of a method for training an image recognition model according to an embodiment of the present disclosure. The image recognition model is used to detect defective areas in an image of a display substrate, wherein the display substrate includes a display area and a connection area. Figure 3 As shown, the method 300 may include steps S310, S320, S330, and S340. According to some embodiments of the present disclosure, some steps of the method 300 may be performed individually or in combination, and may be performed in parallel or sequentially, and are not limited to Figure 3 The specific operation sequence shown.

[0081] In step S310 , a first training sample is obtained, wherein the first training sample includes n images of a display substrate, where n is a positive integer greater than or equal to 1.

[0082] It should be understood that in this article, the term "training sample," also known as a training set, is used to fit a model and train the classification model by setting the classifier's parameters. For training samples, their classification results are pre-labeled, and these pre-labeled classification results are called training values.

[0083] In step S320, each of the n images of the display substrates is divided into a first sub-image and a second sub-image, wherein the first sub-image is an image of the display area of ​​the display substrate, and the second sub-image is an image of the connection area of ​​the display substrate.

[0084] For example, refer to Figure 2 , dividing each of the images of the n display substrates into a first sub-image and a second sub-image includes: determining the positions of the multiple first positioning marks 13 and the multiple second positioning marks 14 in the image of the display substrate; and dividing the image of the display substrate into a first sub-image and a second sub-image according to the positions of the multiple first positioning marks 13 and the multiple second positioning marks 14 in the image of the display substrate.

[0085] For example, multiple first positioning marks 13 are located at the edges of the display area. By connecting the multiple first positioning marks 13, the display area can be framed and determined, thereby obtaining a first sub-image. Similarly, multiple second positioning marks 14 are located at the edges of the connection area. By connecting the multiple second positioning marks 14, the connection area can be framed and determined, thereby obtaining a second sub-image.

[0086] In step S330 , the first training sample is input into the image recognition model, wherein the first training sample includes a first sub-image and a second sub-image.

[0087] In step S340, at least one characteristic parameter of the image recognition model is adjusted to reduce the difference between the output value of the image recognition model and the training value of the first training sample.

[0088] It should be understood that the training value of the first training sample may represent information about defects in an image of a display substrate that is pre-marked (eg, manually), such as the category of the defect, the location of the defect, and the like.

[0089] For example, the image recognition model includes a first parameter and a second parameter, the first parameter is related to the grayscale difference between the normal area and the defective area in the first sub-image, the second parameter is related to the grayscale difference between the normal area and the defective area in the second sub-image, and the at least one feature parameter is at least one of the first parameter and the second parameter.

[0090] In the embodiments of the present disclosure, during the training process of the image recognition model, the image of the display substrate is divided into a display area (i.e., the AA area) and a connection area (i.e., the PAD area). Parameters (e.g., a first parameter and a second parameter) related to the grayscale of the display area and the connection area are set, and these parameters are adjusted to achieve the purpose of optimizing the model. In this way, the trained image recognition model can more accurately detect images of the display substrate and more accurately identify defects in the display substrate.

[0091] For example, the first parameter is calculated according to the following formula:

[0092] P=a×R×OFS,

[0093] Where P is the first parameter, a is a constant, R is the grayscale mean square error ratio of each defect in the image of the display substrate, and OFS is the grayscale difference of each defect in the image block, such as the difference between CV edge blocks or AA area segmentation of punctures, foreign objects, or bubbles. For example, the first parameter P can be understood as the grayscale difference between a normal area and a defective area.

[0094] Image analysis using computer vision typically involves two steps: preprocessing and object detection. Image segmentation bridges the gap between these two. Here, CV edge blocks represent the edge blocks formed when segmenting an image using CV (computer vision) techniques.

[0095] In this exemplary embodiment, the grayscale difference between the normal area and the defect area is adjusted to improve the sensitivity of the model to defect detection. For example, the smaller the P value, the higher the sensitivity of the model to defect detection.

[0096] For example, the second parameter is calculated according to the following formula:

[0097] Q=QM l / QM2,

[0098] Wherein, Q is the second parameter, QM1 is the grayscale threshold of the defect in the connection area, and QM2 is the grayscale threshold of the background of the connection area. For example, the second parameter Q can be understood as a background threshold weight factor.

[0099] For example, in an embodiment of the present disclosure, the connection area can be divided into foreground and background. In the process of model training, through data encoding and decoding technology, the connection area is used to identify the characteristics of the defective type using a deep learning multi-layer convolutional network (such as an FPN network), and then the Otsu method binarization method is used in Open CV to further identify the defectiveness of the connection area using grayscale values. For example, the pixels of the image can be divided into 0 to 255 partitions (i.e., bins), and the number of bins falling on each pixel is counted so that the number of pixels in each bin is divided by the total number of pixels by about 0.9, so that the grayscale image is better separated from the binary image, and the grayscale mean in the connection area is determined. If the grayscale mean is less than the grayscale threshold of the background of the connection area, it is determined to be a defect.

[0100] In this exemplary embodiment, the background threshold weight factor is adjusted through a large number of training samples to improve the sensitivity of the model to defect detection.

[0101] In the embodiments of the present disclosure, the algorithm parameters may also be adjusted by adjusting various thresholds.

[0102] For example, the at least one characteristic parameter may further include a boundary threshold parameter, wherein the boundary threshold parameter is used to represent a boundary of the detection area of ​​the display substrate in the image of the display substrate.

[0103] In the embodiments of the present disclosure, after acquiring a raw image of the display substrate, the detection area of ​​the display substrate must first be determined. For example, complex edge information can be cropped from the raw image of the display substrate to prevent positioning interference and algorithm over-detection. Then, within the determined detection area, the detection area can be divided into a display area and a connection area based on markers such as alignment marks. Based on this, model training and recognition can be performed based on the different characteristics of the display and connection areas to improve the efficiency and accuracy of model training.

[0104] Figure 4 Schematically shows a portion of the boundary of the detection area of ​​the display substrate in the image of the display substrate. Figure 4 Marker 41 represents the boundary automatically located by the positioning marker, and mark 42 represents the adjusted boundary. By adjusting this boundary, the detection distance between defects and the upper boundary of the display substrate, i.e., the boundary threshold, can be adjusted so that the adjusted boundary includes the detection range of the display substrate, thereby avoiding the phenomenon of missing defects on the outer boundary.

[0105] For example, the at least one characteristic parameter may further include a position parameter of the region of interest, wherein the position parameter includes a coordinate position of at least one vertex of the region of interest in the image of the display substrate and a length and a width of the region of interest.

[0106] Figure 5 A region of interest on the image of the display substrate is schematically shown. Figure 5 By adjusting the coordinate position of the vertex of a region selection box 51 (eg, the upper left region selection box) at its upper left corner, as well as the length and width of the upper left region selection box, the position of the region of interest can be precisely located.

[0107] Image analysis using computer vision techniques typically involves two steps: preprocessing and object detection. Image segmentation bridges the gap between these two steps. Image segmentation simplifies or alters the image representation, making it easier to analyze. Before image segmentation, image preprocessing can include binarization. Currently, there are numerous methods for image binarization, with thresholding being the most common.

[0108] For example, a global threshold can be used to binarize the entire image.

[0109] In the embodiments of the present disclosure, different thresholds can be used for the display area and the connection area to perform binarization on the image. It should be noted that in this document, expressions such as grayscale value threshold and grayscale value difference threshold all refer to the thresholds used in the image binarization process.

[0110] For example, before inputting the first training sample into the image recognition model, the method may further include: segmenting the first sub-image into a plurality of image blocks, and segmenting the second sub-image into a plurality of image blocks, based on a grayscale value difference threshold, wherein the at least one feature parameter includes the grayscale value difference threshold. For example, the first training sample includes a plurality of image blocks of the first sub-image and a plurality of image blocks of the second sub-image.

[0111] In the embodiments of the present disclosure, a grayscale difference threshold can be determined based on a classic binarization threshold determination method, such as the Otsu method, and then the image is binarized based on the determined grayscale difference threshold. Image segmentation is then performed on the binarized image, i.e., segmented into multiple image blocks. Based on this, the image blocks are input as training samples into the image recognition model for model training. In this way, the efficiency and accuracy of image recognition model training can be improved.

[0112] For example, the method further includes: adjusting the grayscale value difference threshold according to the category of the defect.

[0113] In the embodiment of the present disclosure, by adjusting the gray value difference threshold, the influence of uneven image formation can be reduced and the phenomenon of unclear detection of defective parts can be eliminated.

[0114] Figure 6 FIG. 1 is a partial flow chart of a method for training an image recognition model according to an embodiment of the present disclosure. Figure 6 As shown, the method 300 may further include steps S610 and S620.

[0115] In step S610 , a second training sample is obtained, wherein the second training sample includes m images of display substrates, where m is a positive integer greater than or equal to 1, and the m images of display substrates are images that are missed by the image recognition model.

[0116] In step S620, the second training sample is input into the image recognition model to optimize the image recognition model.

[0117] For example, step S620 may include: inputting the second training sample into the image recognition model; adjusting the training parameters of the image recognition model to optimize the image recognition model, wherein the training parameters include at least one of the following items: the number of training samples input into the image recognition model each time, and the number of cycles during the training process.

[0118] In an embodiment of the present disclosure, the image recognition model can be optimized using the second training sample. Specifically, the optimization can be performed based on the original FPN algorithm. For example, the training parameters can be adjusted, and the training parameters may include training batches (i.e., batches) and the number of cycles (i.e., epochs). Batch refers to the batch that is input into the image recognition model for training each time, and batch size refers to the number of training samples in each batch. Selecting an appropriate batch size can enable the model to achieve the best balance between memory capacity and memory efficiency, thereby optimizing the performance and speed of the model.

[0119] Optionally, before step S610, the images in the second training sample may be classified and preprocessed.

[0120] Figure 7 FIG. 1 is a partial flow chart of a method for training an image recognition model according to an embodiment of the present disclosure. Figure 7 As shown, the method 300 may further include steps S710 and S720.

[0121] In step S710, the m display substrate images in the second training sample are classified based on the reasons for missed detection. That is, step S710 is the classification step described above. For example, the reasons for missed detection of the missed images can be analyzed to determine the categories of the missed detection reasons. For example, the categories of missed detection reasons may include: unstable imaging equipment, resulting in unrecognizable images; dirty display substrate surface; slight puncture wounds, etc.

[0122] In step S720, the missed defects in the m display substrate images are marked differently based on their categories. For example, each missed defect category can be marked with a line segment or rectangle. For example, for "unrecognizable image," a segment can be marked; for "surface stain," a rectangle can be marked; and for "minor puncture wounds," a triangle can be marked.

[0123] For example, the method 300 may further include: comparing the grayscale values ​​of the missed defects with the grayscale values ​​of the normal regions, and comparing the grayscale values ​​of the missed defects with the grayscale values ​​of the defective regions, based on the categories of the m display substrate images; and adjusting the training parameters of the image recognition model based on the comparison results, wherein the training parameters include a grayscale threshold for image segmentation. For example, for missed defects such as faint puncture wounds, the grayscale values ​​of the region containing the faint puncture wound in the image can be compared with the grayscale values ​​of the normal regions, and the grayscale values ​​of the missed defects can be compared with the grayscale values ​​of the regions containing detected puncture wounds in the image. Through such comparisons, the grayscale values ​​of the missed defects can be accurately determined. Thus, when determining a grayscale value difference threshold using a classic binarization threshold determination method, such as Otsu binarization, the threshold can be adjusted based on the comparison results, so that the threshold determination more accurately reflects the grayscale value difference between the normal regions and the defective regions (including the missed defective regions). This enables more accurate image segmentation and image recognition.

[0124] Figure 8 : is a flow chart of a method for detecting a display substrate according to an embodiment of the present disclosure. The method for detecting a display substrate can utilize the image recognition model obtained by the above training method to detect defects in the display substrate. By using the above image recognition model, the accuracy of detection can be improved, thereby avoiding the situation where defects in the display substrate are missed. Figure 8 As shown, the detection method 800 may further include steps S810, S820 and S830.

[0125] In step S810, an image of the display substrate is acquired.

[0126] In step S820 , the image of the display substrate is input into the image recognition model, wherein the image recognition model is obtained according to the training method of the above embodiment.

[0127] In step S830 , a detection result of a defect in the display substrate is determined according to an output result of the image recognition model.

[0128] Figure 9 FIG. 1 is a detailed flow chart of a method for detecting a display substrate according to some exemplary embodiments of the present disclosure. Figure 9 As shown, the detection method 900 may further include steps S910 to S980.

[0129] In step S910 , an image of the display substrate is acquired.

[0130] In step S920, the boundary of the detection area of ​​the display substrate in the image of the display substrate is determined. For example, an edge feature enhancement algorithm may be used to determine the boundary of the detection area of ​​the display substrate in the image of the display substrate.

[0131] In step S930 , the display area and the connection area are roughly positioned according to the multiple positioning marks.

[0132] In step S940 , the display area and the connection area are precisely positioned according to the plurality of positioning marks.

[0133] In step S950, the image of the display substrate is divided into blocks. For example, an existing image segmentation algorithm can be used to divide the image into small blocks, each of which can include 512×512 pixels. For example, if the portion is less than 512 pixels, it can be overlapped with the adjacent area.

[0134] In step S960, the image blocks of the image of the display substrate are input into the image recognition model, wherein the image recognition model is obtained according to the training method of the above embodiment; and the image recognition model is used to detect defects.

[0135] In step S970 , the output result of the image recognition model is further screened using Open CV to determine defects in the display substrate.

[0136] In step S980, the determined defects in the display substrate are output.

[0137] Based on the above-mentioned training method of the image recognition model, the embodiment of the present disclosure also provides a training device for the image recognition model. Figure 10 The device is described in detail.

[0138] Figure 10 The structural block diagram of the image recognition model training device according to an embodiment of the present disclosure is schematically shown.

[0139] like Figure 10As shown, the image recognition model training device 800 of this embodiment includes an acquisition module 810 , a division module 820 , an input module 830 and an adjustment module 840 .

[0140] The acquisition module 810 is used to acquire a first training sample, wherein the first training sample includes n images of display substrates, where n is a positive integer greater than or equal to 1. In one embodiment, the acquisition module 810 can be used to execute step S310 described above, which will not be repeated here.

[0141] The segmentation module 820 is configured to segment each of the n display substrate images into a first sub-image and a second sub-image, where the first sub-image is an image of the display area of ​​the display substrate, and the second sub-image is an image of the connection area of ​​the display substrate. In one embodiment, the segmentation module 820 may be configured to execute step S320 described above, and will not be further described here.

[0142] The input module 830 is used to input the first training sample into the image recognition model. In one embodiment, the input module 830 can be used to execute the step S330 described above, which will not be described in detail here.

[0143] The adjustment module 840 is used to adjust at least one feature parameter of the image recognition model to reduce the difference between the output value of the image recognition model and the training value of the first training sample. In one embodiment, the division module 840 can be used to perform step S340 described above, which is not repeated here.

[0144] For example, the image recognition model includes a first parameter and a second parameter, the first parameter is related to the grayscale difference between the normal area and the defective area in the first sub-image, the second parameter is related to the grayscale difference between the normal area and the defective area in the second sub-image, and the at least one feature parameter includes at least one of the first parameter and the second parameter.

[0145] According to an embodiment of the present disclosure, any multiple modules among the acquisition module 810, the partitioning module 820, the input module 830, and the adjustment module 840 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the acquisition module 810, the partitioning module 820, the input module 830, and the adjustment module 840 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation modes of software, hardware, and firmware, or in an appropriate combination of any of them. Alternatively, at least one of the acquisition module 810, the partitioning module 820, the input module 830, and the adjustment module 840 can be at least partially implemented as a computer program module, which can perform the corresponding function when the computer program module is executed.

[0146] Figure 11 A block diagram of an electronic device suitable for implementing a training method for an image recognition model or a detection method for a display substrate according to an embodiment of the present disclosure is schematically shown.

[0147] like Figure 11 As shown, the electronic device 900 according to an embodiment of the present disclosure includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage part 908 into a random access memory (RAM) 903. The processor 901 may, for example, include a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include an onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0148] Various programs and data required for the operation of the electronic device 900 are stored in the RAM 903. The processor 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. The processor 901 executes the various operations of the method flow according to the embodiment of the present disclosure by executing the programs in the ROM 902 and / or the RAM 903. It should be noted that the programs may also be stored in one or more memories other than the ROM 902 and the RAM 903. The processor 901 may also execute the various operations of the method flow according to the embodiment of the present disclosure by executing the programs stored in the one or more memories.

[0149] According to an embodiment of the present disclosure, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to the bus 904. The electronic device 900 may further include one or more of the following components connected to the I / O interface 905: an input portion 906 including a keyboard, a mouse, etc.; an output portion 907 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage portion 908 including a hard disk, etc.; and a communication portion 909 including a network interface card such as a LAN card or a modem. The communication portion 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed in the drive 910 as needed, so that a computer program read therefrom can be installed into the storage portion 908 as needed.

[0150] The present disclosure also provides a non-transitory computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently without being incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the methods according to the embodiments of the present disclosure.

[0151] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, such as but not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 902 and / or RAM 903 described above and / or one or more memories other than ROM 902 and RAM 903.

[0152] The embodiments of the present disclosure also include a computer program product, which includes a computer program containing program code for executing the method shown in the flowchart. When the computer program product is executed in a computer system, the program code is used to cause the computer system to implement the item recommendation method provided by the embodiments of the present disclosure.

[0153] The computer program executes the above functions defined in the system / device of the embodiment of the present disclosure when the processor 901 executes the computer program. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0154] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 909, and / or installed from a removable medium 911. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0155] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 909, and / or installed from a removable medium 911. When the computer program is executed by the processor 901, the above-described functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the systems, devices, means, modules, units, etc. described above can be implemented by computer program modules.

[0156] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0157] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0158] Those skilled in the art will appreciate that the features described in the various embodiments and / or claims of this disclosure may be combined and / or coupled in various ways, even if such combinations and / or couplings are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure may be combined and / or coupled in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or couplings are intended to fall within the scope of this disclosure.

[0159] The embodiments of the present disclosure are described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be used in combination to advantage. The scope of the present disclosure is defined by the appended claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A method for training an image recognition model for detecting defective areas in an image of a display substrate, wherein the display substrate includes a display area and a connection area, wherein: The method comprises: Acquire a first training sample, wherein the first training sample includes n images of the display substrate, where n is a positive integer greater than or equal to 1; Dividing each of the images of the n display substrates into a first sub-image and a second sub-image, wherein the first sub-image is an image of a display area of ​​the display substrate, and the second sub-image is an image of a connection area of ​​the display substrate; Inputting the first training sample into the image recognition model, wherein the first training sample includes a first sub-image and a second sub-image; and adjusting at least one characteristic parameter of the image recognition model to reduce the difference between an output value of the image recognition model and a training value of the first training sample, The image recognition model includes a first parameter and a second parameter, the first parameter is related to the grayscale difference between the normal area and the defective area in the first sub-image, the second parameter is related to the grayscale difference between the normal area and the defective area in the second sub-image, and the at least one feature parameter includes at least one of the first parameter and the second parameter; The display substrate further includes a plurality of first positioning marks provided in the display area and a plurality of second positioning marks provided in the connection area, and dividing each of the images of the n display substrates into a first sub-image and a second sub-image includes: determining positions of the plurality of first alignment marks and the plurality of second alignment marks in an image of the display substrate; and The image of the display substrate is divided into a first sub-image and a second sub-image according to positions of the plurality of first positioning marks and the plurality of second positioning marks in the image of the display substrate.

2. The method according to claim 1, wherein The method further comprises: Acquire a second training sample, wherein the second training sample includes m images of the display substrate, where m is a positive integer greater than or equal to 1, and the m images of the display substrate are images that are missed by the image recognition model; and The second training sample is input into the image recognition model to optimize the image recognition model.

3. The method according to claim 1 or 2, wherein: The at least one characteristic parameter further includes a boundary threshold parameter, wherein the boundary threshold parameter is used to represent a boundary of a detection area of ​​the display substrate in the image of the display substrate, the detection area including a display area and a connection area of ​​the display substrate.

4. The method according to claim 1 or 2, wherein: The at least one characteristic parameter further includes a position parameter of the region of interest, wherein the position parameter includes a coordinate position of at least one vertex of the region of interest in the image of the display substrate and a length and a width of the region of interest.

5. The method according to claim 1 or 2, wherein: The first parameter is calculated according to the following formula: P=a×R×OFS, Wherein, P is the first parameter, a is a constant, R is the grayscale mean square error ratio of each defect in the image of the display substrate, and OFS is the grayscale difference value of the defect in each image block.

6. The method according to claim 1 or 2, wherein: The second parameter is calculated according to the following formula: Q=QM1 / QM2, Wherein, Q is the second parameter, QM1 is the grayscale threshold of the defect in the connection area, and QM2 is the grayscale threshold of the background of the connection area.

7. The method according to claim 1 or 2, wherein: Before inputting the first training sample into the image recognition model, the method further includes: dividing the first sub-image into a plurality of image blocks, and dividing the second sub-image into a plurality of image blocks according to a grayscale value difference threshold, wherein the at least one feature parameter includes the grayscale value difference threshold, The first training sample includes a plurality of image blocks of the first sub-image and a plurality of image blocks of the second sub-image.

8. The method according to claim 7, wherein: The method further includes adjusting the grayscale value difference threshold according to the category of the defect.

9. The method according to claim 2, wherein: Inputting the second training sample into the image recognition model to optimize the image recognition model includes: inputting the second training sample into the image recognition model; Adjusting the training parameters of the image recognition model to optimize the image recognition model, The training parameters include at least one of the following items: the number of training samples input to the image recognition model each time, and the number of cycles during the training process.

10. The method according to claim 9, wherein: Before inputting the second training sample into the image recognition model, the method further includes: classifying the m display substrate images in the second training sample according to reasons for missed detection; and Different marks are performed on the missed defects in the images of the m display substrates according to categories of the images of the m display substrates.

11. The method according to claim 9 or 10, wherein: The method further comprises: Comparing the grayscale value of the missed defect with the grayscale value of the normal area, and comparing the grayscale value of the missed defect with the grayscale value of the defect area according to the categories of the m display substrate images; and According to the comparison result, the training parameters of the image recognition model are adjusted, wherein the training parameters include a grayscale threshold for image segmentation.

12. A method for detecting a display substrate, characterized in that: The detection method comprises: acquiring an image of the display substrate; inputting the image of the display substrate into an image recognition model, wherein the image recognition model is obtained by the training method according to any one of claims 1 to 11; and A detection result of a defect in the display substrate is determined according to an output result of the image recognition model.

13. A training device for an image recognition model, wherein the image recognition model is used to detect defective areas in an image of a display substrate, wherein the display substrate includes a display area and a connection area, wherein: The training device comprises: an acquisition module, configured to acquire a first training sample, wherein the first training sample includes n images of a display substrate, where n is a positive integer greater than or equal to 1; a dividing module, configured to divide each of the images of the n display substrates into a first sub-image and a second sub-image, wherein the first sub-image is an image of a display area of ​​the display substrate, and the second sub-image is an image of a connection area of ​​the display substrate; an input module, configured to input the first training sample into the image recognition model, wherein the first training sample includes a first sub-image and a second sub-image; and an adjustment module, configured to adjust at least one characteristic parameter of the image recognition model to reduce a difference between an output value of the image recognition model and a training value of the first training sample, The image recognition model includes a first parameter and a second parameter, the first parameter is related to the grayscale difference between the normal area and the defective area in the first sub-image, the second parameter is related to the grayscale difference between the normal area and the defective area in the second sub-image, and the at least one feature parameter includes at least one of the first parameter and the second parameter; The display substrate further includes a plurality of first positioning marks provided in the display area and a plurality of second positioning marks provided in the connection area, and dividing each of the images of the n display substrates into a first sub-image and a second sub-image includes: determining positions of the plurality of first alignment marks and the plurality of second alignment marks in an image of the display substrate; and The image of the display substrate is divided into a first sub-image and a second sub-image according to positions of the plurality of first positioning marks and the plurality of second positioning marks in the image of the display substrate.

14. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to execute the method according to any one of claims 1 to 12.

15. A non-transitory computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to perform the method according to any one of claims 1 to 12.

16. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

Images