A method for detecting defects of a substrate and an AOI defect detection device

Abstract

The application discloses a kind of methods and AOI defect detection equipment related to substrate defect detection, the method comprises: the substrate picture to be detected is respectively input into first model and second model and carries out defect detection;First model includes first training set, second model includes second training set, first training set includes initial defect data and defect expansion data, and second training set includes initial defect data;The substrate picture to be detected includes multiple defects;Calling first model and second model identifies target defect respectively, obtains first detection result and second detection result;If first detection result and second detection result are identical, then complete the defect detection of target defect, and target defect is set as first weight, and target defect is added to first training set and second training set with first weight respectively;First weight indicates that the probability of target defect as defect sample is added to first training set and second training set.The application can reduce the workload of artificial defect labeling.

Description

Technical Field

[0001] This application relates to the field of defect detection technology in inkjet printing scenarios for displays, specifically to a defect detection method and AOI defect detection equipment for substrates. Background Technology

[0002] Currently, in the field of inkjet printing of substrates, substrates need to undergo optical vision inspection to identify defects. Machine learning models are generally used to detect defects in substrate images. The training set for these models often uses manual annotation, meaning the type and location of defects are manually labeled.

[0003] However, in the inspection of large-size substrates, manually marking defects is labor-intensive, time-consuming, and the accuracy of marking is unstable.

[0004] Therefore, to address the issue of the large workload associated with manually marking defects, a defect detection method involving the substrate and AOI defect detection equipment are needed. Summary of the Invention

[0005] This application provides a defect detection method and AOI defect detection equipment for substrates, which can reduce the workload of manually marking defects.

[0006] The first aspect of this application discloses a defect detection method for a substrate. The defect detection method includes: inputting an image of the substrate to be detected into a first model and a second model for defect detection; the first model includes a first training set, and the second model includes a second training set. The first training set includes initial defect data and defect expansion data, and the second training set includes initial defect data; the image of the substrate to be detected includes multiple defects, and the target defect is any one of the multiple defects; calling the first model and the second model to identify the target defect, respectively, to obtain a first detection result and a second detection result; if the first detection result is the same as the second detection result, the defect detection of the target defect is completed, and the target defect is set as a first weight. The target defect is then added to the first training set and the second training set with the first weight, respectively, so that the first model can be trained based on the first training set and the second model can be trained based on the second training set. The first weight represents the probability that the target defect is added as a defect sample to the first training set and the second training set.

[0007] In the above scheme, two models simultaneously detect defects on the substrate image. If the detection results of the two models are consistent, it indicates that both models have identified the target defect, and manual annotation of the target defect is not required. Furthermore, when both models identify the target defect, it means that the target defect is a known defect for both models, and there is no need to add too many of this type of defect to their respective training sets; therefore, a first weight is set to avoid the number of known defects in their respective training sets being too large or occupying too large a proportion.

[0008] In one possible implementation, the defect detection method further includes: if a first detection result indicates that a target defect has been identified and a second detection result indicates that a target defect has not been identified, adding the target defect to a first training set with a second weight and adding it to a second training set with a third weight; wherein the second weight represents the probability that the target defect is added to the first training set as a defect sample, the third weight represents the number of defect samples generated based on the target defect, and the second weight is less than the third weight; and marking the target defect to facilitate subsequent manual detection operations on the target defect.

[0009] The above scheme aims to explain a situation where the first and second detection results differ. The first detection result indicates that the target defect has been identified, meaning it is a known defect for the first model. Therefore, the number of samples with this defect in the first training set of the first model is reduced, and a second weight (less than 1) is assigned to this target defect. Conversely, the second detection result indicates that the target defect has not been identified, meaning it is an unknown defect for the second model. In this case, the number and proportion of samples with this defect can be increased in the second training set of the second model, and a third weight (greater than 1) is assigned.

[0010] In one possible implementation, the defect detection method further includes: if the first detection result indicates that the target defect is not identified and the second detection result indicates that the target defect has been identified, adding the target defect to the first training set with a fourth weight and adding it to the second training set with a fifth weight; the fifth weight represents the probability that the target defect is added to the first training set as a defect sample, the fourth weight represents the number of defect samples generated based on the target defect, and the fifth weight is less than the fourth weight; and marking the target defect to facilitate subsequent manual detection operations on the target defect.

[0011] The above scheme aims to explain another scenario where the first and second detection results are different. In this case, the first detection result indicates that the target defect was not identified and the second detection result indicates that the target defect was identified, versus the first detection result indicating that the target defect was identified and the second detection result indicating that the target defect was not identified. These are two scenarios where the first and second detection results are different.

[0012] In one possible implementation, the detection result of the target defect includes the location of the defect, the type of the defect, and the confidence level of the defect; wherein, the first detection result is the same as the second detection result, specifically including: the location of the target defect and the type of the target defect are the same, and the confidence level of the target defect in both the first detection result and the second detection result is greater than a preset confidence level.

[0013] The above scheme aims to disclose the contents of the target defect detection results. The location of the defect is the coordinate position of the target defect in the image of the substrate to be inspected. For example, by constructing a coordinate system with the upper left corner of the image of the substrate to be inspected as the origin, the coordinates of any point on the image can be obtained. The location of the defect is the location of the defect box, which can be defined by the coordinates of the two vertices on the diagonal of the defect box (the upper left corner and the lower right corner of the defect box).

[0014] In one possible implementation, the first detection result indicating that the target defect was not identified includes any of the following: the confidence level of the target defect is less than the preset first model confidence level of the first model; the confidence level of the target defect is greater than or equal to the preset first model confidence level of the first model, but less than the preset confidence level; the second detection result indicating that the target defect was not identified includes any of the following: the confidence level of the target defect is less than the preset second model confidence level of the second model; the confidence level of the target defect is greater than or equal to the preset second model confidence level of the second model, but less than the preset confidence level.

[0015] The above scheme aims to explain the specific content of unidentified target defects. The first model has a preset first model confidence level; that is, the first model will only confirm the effective identification of a defect and retain it if the confidence level of a defect is greater than or equal to the first model confidence level. If the confidence level of a defect is less than the first model confidence level, the first model will filter out the defect, meaning the defect is not identified. Furthermore, in this specification, both the first and second models are used to identify the target defect (i.e., the same defect) simultaneously. By setting a preset confidence level, the consistency and accuracy of defect identification by the two models are ensured. Therefore, if the target defect is greater than or equal to the preset first model confidence level, but less than the preset confidence level, it will still be filtered out by the first model, confirming that the defect has not been identified.

[0016] In one possible implementation, the defect types in the substrate image to be inspected include dirt, dots, lines, dot mura, and line mura; wherein, the number of defect types in the initial defect data of the first model is a first number, and the number of defect types in the initial defect data of the second model is a second number, and the first number is greater than the second number.

[0017] The above scheme discloses several common defect types. The second model has fewer defect types than the first model. The second model focuses on being small and precise (the training set has fewer defect types, fewer parameters, and the defect data is more scenario-specific; for example, if the substrate is before inkjet printing, the second model only needs three defect types: dirt, dots, and lines; if the substrate is after inkjet printing, the second model only needs two defect types: dot mura and line mura; of course, there are other inkjet printing scenarios, which are not listed here). The first model focuses on being large and comprehensive (applicable to various inkjet printing stages or scenarios of the substrate, including various defect types), serving as the "teacher network model" for training the second model, and the second model serving as the "student network model" for the first model.

[0018] In one possible implementation, the defect augmentation data is obtained by augmenting the initial defect data using a preset defect augmentation method; wherein the preset defect augmentation method includes one or more of random cropping, grayscale variation, top-bottom-left-right mirroring variation, size variation, and random rotation.

[0019] The above scheme discloses several methods for defect expansion; since there are few defect samples, the defect expansion method can be used to expand the number of defect samples.

[0020] In one possible implementation, the first weight represents the probability that a target defect is added as a defect sample to the first training set and the second training set, specifically including any one of the following: when there are multiple target defects, the first weight represents the probability of selecting from multiple target defects as a defect sample to be added to the first training set and the second training set; when there is a single target defect, the first weight represents the probability that a single target defect is added as a defect sample to the first training set and the second training set.

[0021] The above scheme aims to further clarify the meaning of the first weight. If there are multiple target defects, it means that after the first model and the second model identify all the target defects that they can recognize, they will be processed into a unified defect sample set. If there is only one target defect, it means that after the first model and the second model identify each target defect, they will be processed into a defect sample set.

[0022] In one possible implementation, the third weight represents the number of defect samples generated based on the target defect, specifically including: the third weight is greater than 1, and the third weight represents the number of defect samples that are expanded based on the target defect using a preset defect expansion method.

[0023] The above scheme aims to illustrate the expansion of the defect count for unidentified target defects. There is no specific limit to the number of defects to be expanded; the more the better, and it can be set according to actual needs.

[0024] The second aspect of this application discloses an AOI defect detection device, characterized in that it includes a processor, a memory, a user interface, and a network interface. The memory is used to store instructions, the user interface and the network interface are both used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to enable the AOI defect detection device to perform a defect detection method involving a substrate as described in any of the above.

[0025] The beneficial effects of this application include: By simultaneously detecting defects on the substrate image using two models, if the detection results of the two models are consistent, it indicates that both models have identified the target defect, and there is no need for manual annotation of the target defect. The first detection result indicates that the target defect has not been identified and the second detection result indicates that the target defect has been identified. This is one of two different situations where the first detection result indicates that the target defect has been identified and the second detection result indicates that the target defect has not been identified. The first model and the second model are used to identify the target defect (i.e. the same defect) simultaneously. By setting a preset confidence level, the consistency and accuracy of the two models in identifying the defect are ensured. Therefore, if the target defect is greater than or equal to the preset confidence level of the first model, it will still be filtered out by the first model if it is less than the preset confidence level, thus confirming that the defect has not been identified. The first model is comprehensive and applicable to various inkjet printing stages or scenarios on the substrate, including various defect types. It serves as the "teacher network model" for training the second model, while the second model serves as the "student network model" for the first model. Attached Figure Description

[0026] Figure 1 This is a schematic flowchart of a defect detection method for a substrate disclosed in this application. Figure 2 This is a schematic diagram of a defect detection method for a substrate disclosed in this application specification; Figure 3 This is a schematic diagram of another defect detection method for a substrate disclosed in this application. Figure 4 This is a schematic diagram of another defect detection method for a substrate disclosed in this application specification; Figure 5 This is a schematic diagram of another defect detection method for a substrate disclosed in this application specification; Figure 6 This is a schematic diagram of the structure of an AOI defect detection device disclosed in this application. Detailed Implementation

[0027] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0028] In the description of the embodiments of this application, the words "for example" or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design that is described as "for example" or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design options. Rather, the use of the words "for example" or "for instance" is intended to present the relevant concepts in a specific manner.

[0029] In the description of the embodiments of this application, the term "multiple" means two or more. For example, multiple systems means two or more systems, and multiple screen terminals means two or more screen terminals. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0030] This specification discloses a method for detecting defects in a substrate, such as... Figure 1 As shown. The defect detection method includes steps S101-S103.

[0031] Step S101: Input the image of the substrate to be detected into the first model and the second model respectively for defect detection; the first model includes a first training set, the second model includes a second training set, the first training set includes initial defect data and defect expansion data, and the second training set includes initial defect data; the image of the substrate to be detected includes multiple defects, and the target defect is any one of the multiple defects.

[0032] Step S102: Call the first model and the second model to identify the target defects respectively, and obtain the first detection result and the second detection result.

[0033] Step S103: If the first detection result is the same as the second detection result, the defect detection of the target defect is completed, and the target defect is set as the first weight. The target defect is added to the first training set and the second training set respectively with the first weight, so that the first model can be trained according to the first training set and the second model can be trained according to the second training set. The first weight represents the probability that the target defect is added to the first training set and the second training set as a defect sample.

[0034] At this point, two models simultaneously detect defects on the substrate image. If the detection results of the two models are consistent, it indicates that both models have identified the target defect, and manual annotation of the target defect is unnecessary. Furthermore, when both models identify the target defect, it means that the target defect is a known defect for both models, and there is no need to add too many of this type of defect to their respective training sets; therefore, a first weight is set to avoid the number of known defects in their respective training sets being too large or occupying too large a proportion.

[0035] Furthermore, the number of parameters in the first model (e.g., millions) is far greater than that in the second model (tens of thousands). The sample data of the first model includes initial defect data (i.e., all known defect data) and corresponding defect augmentation data; the second model only includes initial defect data. The first model focuses on the accuracy of target defect identification (meeting the requirement of accurate identification of all defects), while the second model focuses on identification efficiency (i.e., meeting the identification efficiency requirements of the production line). In this way, accuracy and efficiency are balanced, allowing the first model to be continuously improved and the second model to be continuously trained using the first model. In practical applications, the detection results of the second model can be directly used as the detection results of defects in the substrate image to be inspected.

[0036] like Figure 2 The path shown indicates that the first detection result of the first model and the second detection result of the second model have the same detection results (the location, type, and confidence level of the target defect are the same). In this case, manual annotation is unnecessary, reducing the workload of manually annotating defect samples. Simultaneously, the first training set of the first model and the second training set of the second model are added with the first weight.

[0037] also, Figure 5 The document also illustrates two scenarios where the first and second detection results differ. These are also two situations requiring manual annotation of target defects, and generally, these two scenarios account for a small percentage. For example, if there are 100 defects in the substrate image, the first model identifies 99, and the second model identifies 90 (all 90 defects have been identified by the first model); then manual annotation is only required for 9 defects. Defects not identified by either the first or second model (such as the one defect in the example) will be discovered during subsequent random or periodic manual inspections. Figure 5 The path in the text shows Figure 3 and Figure 4 In this approach, lower weights are assigned to target defects already identified by the model, while higher weights are assigned to target defects not identified by the model. This method continuously refines the first and second models during the training set in the model application phase.

[0038] In one example, such as Figure 3 As shown. The defect detection method further includes steps S301-S302.

[0039] Step S301: If the first detection result indicates that the target defect has been identified and the second detection result indicates that the target defect has not been identified, the target defect is added to the first training set with the second weight and to the second training set with the third weight; wherein, the second weight represents the probability that the target defect is added to the first training set as a defect sample, the third weight represents the number of defect samples generated based on the target defect, and the second weight is less than the third weight.

[0040] Step S302: Mark the target defect so that it can be manually inspected later.

[0041] Now, let's explain a scenario where the first and second detection results differ. The first detection result indicates that the target defect has been identified, meaning it's a known defect for the first model. Therefore, the number of samples with this defect in the first training set of the first model is reduced, and a second weight (less than 1) is assigned to this target defect. Conversely, the second detection result indicates that the target defect has not been identified, meaning it's an unknown defect for the second model. In this case, the number and proportion of samples with this defect in the second training set of the second model are increased, and a third weight (greater than 1) is assigned.

[0042] Furthermore, the second weight can also be 0, meaning that known defects identified by the first model do not need to be added to the first training set as defect samples. However, defect identification by the model includes identifying the location, type, and confidence level of the defect. Since the confidence level of a defect is generally not 1, defects identified by the model can still be added to the training set as defect samples to continuously enrich the content of the training set. Therefore, the second weight can generally be set between 0 and 0.5, rather than being set to 0.

[0043] Furthermore, the defects of the first and second training sets in the initial case are mainly defects from manual annotation and corresponding augmentation defects; the number of defective samples is limited, and defective samples need to be continuously accumulated in real-world scenarios.

[0044] It should be noted that manually performing the detection operation on the target defect is the same as manually annotating the target defect; the location and type of the target defect are marked manually.

[0045] In one example, such as Figure 4 As shown. The defect detection method further includes steps S401-S402.

[0046] Step S401: If the first detection result indicates that the target defect has not been identified and the second detection result indicates that the target defect has been identified, the target defect is added to the first training set with the fourth weight and to the second training set with the fifth weight; the fifth weight represents the probability that the target defect is added to the first training set as a defect sample, the fourth weight represents the number of defect samples generated based on the target defect, and the fifth weight is less than the fourth weight.

[0047] Step S402: Mark the target defect so that it can be manually inspected later.

[0048] This example illustrates another scenario where the first and second detection results differ. In this case, the first detection result indicates that the target defect was not identified, while the second detection result indicates that the target defect was identified. This is one of two scenarios where the first detection result indicates that the target defect was identified, and the second detection result indicates that the target defect was not identified.

[0049] Similarly, the first detection result indicates that the target defect was not identified, meaning that the target defect is unknown to the first model. Therefore, the number and proportion of this defect in the first training set of the first model can be increased, and a fourth weight (greater than 1) can be set. Meanwhile, the second detection result indicates that the target defect has been identified, meaning that the target defect is known to the second model. Therefore, the number of samples with this defect in the second training set of the second model can be reduced, and a fifth weight (less than 1) can be set for this target defect.

[0050] Furthermore, the fifth weight can also be 0, meaning that known defects identified by the second model do not need to be added to the second training set as defect samples. However, defect identification by the model includes identifying the location, type, and confidence level of the defect. Since the confidence level of a defect is generally not 1, defects identified by the model can still be added to the training set as defect samples to continuously enrich the content of the training set. Therefore, the fifth weight can generally be set between 0 and 0.5, rather than being set to 0.

[0051] The first weight in this specification is less than 1, and the discussion is similar to that of the second and fifth weights mentioned above, so it will not be repeated here.

[0052] In one example, the detection results of the target defect include the location of the defect, the type of the defect, and the confidence level of the defect; wherein, the first detection result is the same as the second detection result, specifically including: the location of the target defect and the type of the target defect are the same, and the confidence level of the target defect in both the first detection result and the second detection result is greater than the preset confidence level.

[0053] At this point, what information is included in the results of the target defect detection? The location of the defect is its coordinate position in the image of the substrate to be inspected. For example, by constructing a coordinate system with the upper left corner of the image of the substrate to be inspected as the origin, the coordinates of any point on the image can be obtained. The location of the defect is the location of the defect box, which can be defined by the coordinates of the two vertices on the diagonal of the defect box (the upper left corner and the lower right corner of the defect box).

[0054] Furthermore, "identical defect location" means that the ratio of the intersection to the union of two defect boxes (the defect boxes corresponding to the first and second detection results) is greater than a preset threshold (e.g., 0.8). In this case, the defects corresponding to the two defect boxes are considered to be the same defect. Moreover, the confidence levels of the target defect in both the first and second detection results must be greater than or equal to a preset confidence level for the confidence levels of the target defect in the first and second detection results to be considered identical.

[0055] For example: if the confidence value of the target defect in the first test result is 0.8 and the confidence value of the target defect in the second test result is 0.85, if the preset confidence level is 0.8, it is confirmed that the confidence levels of the target defect in the first test result and the second test result are the same; if the preset confidence level is 0.85, it is confirmed that the confidence levels of the target defect in the first test result and the second test result are not the same, that is, the first test result and the second test result are also different.

[0056] In this specification, the first and second test results must simultaneously meet the above three conditions (the location and type of the target defect are the same, and the confidence level of the target defect in both the first and second test results is greater than the preset confidence level) for the two test results to be determined to be the same; if any one condition is not met, the two test results are confirmed to be different.

[0057] In one example, a first detection result indicating that a target defect was not identified includes any of the following: the confidence level of the target defect is less than the preset confidence level of the first model; or the confidence level of the target defect is greater than or equal to the preset confidence level of the first model, but less than the preset confidence level. A second detection result indicating that a target defect was not identified includes any of the following: the confidence level of the target defect is less than the preset confidence level of the second model; or the confidence level of the target defect is greater than or equal to the preset confidence level of the second model, but less than the preset confidence level.

[0058] At this point, the specific details regarding unidentified target defects are explained. The first model has a preset first model confidence level. This means that the first model will only confirm a valid identification of a defect and retain it if the confidence level is greater than or equal to the first model confidence level; otherwise, it will filter out the defect, meaning the defect is not identified. Furthermore, this specification uses both the first and second models to simultaneously identify the target defect (i.e., the same defect). By setting preset confidence levels, the consistency and accuracy of defect identification by the two models are ensured. Therefore, if the target defect's confidence level is greater than or equal to the preset first model confidence level, but less than the preset confidence level, it will still be filtered out by the first model, confirming that the defect has not been identified.

[0059] Similarly, the identification and non-identification of target defects by the second model are similar to those by the first model, and will not be repeated here.

[0060] Furthermore, there are no restrictions on the values ​​of the first model confidence score and the preset confidence score; they can be set according to actual needs. Generally, the preset confidence score is greater than the first model confidence score and the second model confidence score. Model confidence score (i.e., the first model confidence score and the second model confidence score) is the probability of identifying the target defect relative to the models (first model and second model) used to identify it. Preset confidence score coordinates the consistency and accuracy of defect identification between the two models, and can continuously improve the identification accuracy of the second model. Therefore, for both cases where the target defect is not identified and when it is identified, both model confidence score and preset confidence score must be considered simultaneously.

[0061] If a defect's confidence level is greater than or equal to the second model's confidence level, but less than the preset confidence level, and this defect is considered identified, then the second model will lose its ability to improve its ability to identify this defect. This is because the weight given to adding an identified defect as a defect sample to the training set is less than the weight given to adding it as a defect sample when the defect is unidentified.

[0062] In one example, the defect types in the substrate image to be detected include dirt, dots, lines, dot mura, and line mura; wherein, the number of defect types in the initial defect data of the first model is the first number, and the number of defect types in the initial defect data of the second model is the second number, and the first number is greater than the second number.

[0063] At this point, several common defect types were disclosed. The second model has fewer defect types than the first model. The second model focuses on being small and precise (the training set has fewer defect types, fewer parameters, and the defect data is more scenario-specific; for example, if the substrate is in a scenario before inkjet printing, the second model only needs three defect types: dirt, dots, and lines; if the substrate is in a scenario after inkjet printing, the second model only needs two defect types: dot mura and line mura; of course, there are other inkjet printing scenarios, which will not be listed here). The first model focuses on being large and comprehensive (applicable to various inkjet printing stages or scenarios of the substrate, including various defect types), serving as the "teacher network model" for training the second model, while the second model serves as the "student network model" for the first model.

[0064] For example: The first model could be the YOLO-X model, characterized by more network parameters, stronger learning ability, and better performance; it generally has no time limit (i.e., high accuracy, but lower recognition efficiency). The second model could be a network model such as YOLO-Nano, SSD (Single Shot MultiBox Detector), or Faster R-CNN, with fewer parameters, suitable for time-limited scenarios in inkjet printing (high recognition efficiency), but with relatively lower accuracy. This manual does not specify the particular model for the first and second models; you can choose according to your actual needs.

[0065] In one example, the defect augmentation data is obtained by augmenting the initial defect data using a preset defect augmentation method; wherein the preset defect augmentation method includes one or more of random cropping, grayscale variation, top-bottom-left-right mirroring variation, size variation, and random rotation.

[0066] This example demonstrates several ways to augment defects; since there are few defect samples, augmentation can be used to increase the number of defect samples.

[0067] At this point, random cropping refers to randomly cropping the defect image (the area corresponding to the defect location, not the substrate image); grayscale variation refers to arbitrarily adjusting the grayscale of the defect image (within the range of 0-255); vertical and horizontal mirroring refers to randomly mirroring the defect image; size variation refers to obtaining a defect image of any size relative to the defect location; and random rotation refers to rotating the defect image at any angle.

[0068] In one example, the first weight represents the probability that a target defect is added as a defect sample to the first training set and the second training set, specifically including any of the following: when there are multiple target defects, the first weight represents the probability of selecting from multiple target defects as a defect sample to be added to the first training set and the second training set; when there is a single target defect, the first weight represents the probability that a single target defect is added as a defect sample to the first training set and the second training set.

[0069] This example further illustrates the meaning of the first weight. If there are multiple target defects, it means that after the first and second models identify all the target defects that they can recognize, they will be processed into a unified defect sample set. If there is only one target defect, it means that after the first and second models identify a single target defect, they will be processed into a defect sample set.

[0070] In other words, when there are multiple target defects, that is, there are multiple target defects on the substrate image to be detected (these target defects can be identified by the first model and the second model at the same time, and the first model and the second model process these target defects in a unified manner after identification), a small part (the proportion is the probability) is selected from the multiple target defects to enter the training set (for example, 3 out of 10 target defects are selected as defect samples to enter the training set); when there is only one target defect, that is, the target defect is processed one by one (that is, the first model and the second model process each target defect once they are identified, instead of the target defects in the substrate image to be detected being identified by the first model and the second model and then processed in a unified manner after being counted), the probability of the target defect entering the defect sample is less than 1 (for example, the probability is 0.5, that is, the target defect has only a 0.5 probability of entering the defect sample as a defect sample into the training set).

[0071] In one example, the third weight represents the number of defect samples generated based on the target defect. Specifically, the third weight is greater than 1, and the third weight represents the number of defect samples that are expanded based on the target defect using a preset defect expansion method.

[0072] At this point, it indicates that the number of defect samples has been expanded for the unidentified target defect. There is no specific limit to the number of defects to be expanded for this target defect; the more the better, and it can be set according to actual needs.

[0073] This specification also discloses an AOI defect detection device, including a processor, a memory, a user interface, and a network interface. The memory is used to store instructions, and both the user interface and the network interface are used to communicate with other devices. The processor is used to execute the instructions stored in the memory so that the AOI defect detection device performs a defect detection method involving a substrate as described in any of the above.

[0074] Specifically, a defect detection method involving a substrate is provided. The defect detection method includes: inputting an image of the substrate to be detected into a first model and a second model for defect detection; the first model includes a first training set, and the second model includes a second training set. The first training set includes initial defect data and defect expansion data, and the second training set includes initial defect data; the image of the substrate to be detected includes multiple defects, and the target defect is any one of the multiple defects; the first model and the second model are called to identify the target defect respectively, and a first detection result and a second detection result are obtained; if the first detection result and the second detection result are the same, the defect detection of the target defect is completed, and the target defect is set as a first weight. The target defect is then added to the first training set and the second training set with the first weight, so that the first model can be trained based on the first training set and the second model can be trained based on the second training set. The first weight represents the probability that the target defect is added as a defect sample to the first training set and the second training set.

[0075] In one example, the defect detection method further includes: if a first detection result indicates that a target defect has been identified and a second detection result indicates that a target defect has not been identified, adding the target defect to a first training set with a second weight and to a second training set with a third weight; wherein the second weight represents the probability that the target defect is added to the first training set as a defect sample, the third weight represents the number of defect samples generated based on the target defect, and the second weight is less than the third weight; and marking the target defect to facilitate subsequent manual detection operations on the target defect.

[0076] In one example, the defect detection method further includes: if the first detection result indicates that the target defect is not identified and the second detection result indicates that the target defect has been identified, adding the target defect to the first training set with a fourth weight and adding it to the second training set with a fifth weight; the fifth weight represents the probability that the target defect is added to the first training set as a defect sample, the fourth weight represents the number of defect samples generated based on the target defect, and the fifth weight is less than the fourth weight; and marking the target defect to facilitate subsequent manual detection operations on the target defect.

[0077] In one example, the detection results of the target defect include the location of the defect, the type of the defect, and the confidence level of the defect; wherein, the first detection result is the same as the second detection result, specifically including: the location of the target defect and the type of the target defect are the same, and the confidence level of the target defect in both the first detection result and the second detection result is greater than the preset confidence level.

[0078] In one example, a first detection result indicating that a target defect was not identified includes any of the following: the confidence level of the target defect is less than the preset confidence level of the first model; or the confidence level of the target defect is greater than or equal to the preset confidence level of the first model, but less than the preset confidence level. A second detection result indicating that a target defect was not identified includes any of the following: the confidence level of the target defect is less than the preset confidence level of the second model; or the confidence level of the target defect is greater than or equal to the preset confidence level of the second model, but less than the preset confidence level.

[0079] In one example, the defect types in the substrate image to be detected include dirt, dots, lines, dot mura, and line mura; wherein, the number of defect types in the initial defect data of the first model is the first number, and the number of defect types in the initial defect data of the second model is the second number, and the first number is greater than the second number.

[0080] In one example, the defect augmentation data is obtained by augmenting the initial defect data using a preset defect augmentation method; wherein the preset defect augmentation method includes one or more of random cropping, grayscale variation, top-bottom-left-right mirroring variation, size variation, and random rotation.

[0081] In one example, the first weight represents the probability that a target defect is added as a defect sample to the first training set and the second training set, specifically including any of the following: when there are multiple target defects, the first weight represents the probability of selecting from multiple target defects as a defect sample to be added to the first training set and the second training set; when there is a single target defect, the first weight represents the probability that a single target defect is added as a defect sample to the first training set and the second training set.

[0082] In one example, the third weight represents the number of defect samples generated based on the target defect. Specifically, the third weight is greater than 1, and the third weight represents the number of defect samples that are expanded based on the target defect using a preset defect expansion method.

[0083] It should be noted that the above embodiments of the apparatus are only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0084] This specification also discloses a computer-readable storage medium storing instructions that, when executed, perform the method described above.

[0085] This embodiment also discloses an electronic device, which may be an AOI defect detection device, to perform the above-described method. (Refer to...) Figure 6 The electronic device may include: at least one processor 601, at least one communication bus 602, display 603, network interface 604, and at least one memory 605.

[0086] The communication bus 602 is used to enable communication between these components.

[0087] The display 603 may include a display screen and a camera.

[0088] The network interface 604 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0089] The processor 601 may include one or more processing cores. The processor 601 connects to various parts of the server using various interfaces and lines, and performs various server functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 605, and by calling data stored in the memory 605. Optionally, the processor 601 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 601 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 601 and may be implemented as a separate chip.

[0090] The memory 605 may include random access memory (RAM) or read-only memory. Optionally, the memory 605 may include a non-transitory computer-readable storage medium. The memory 605 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 605 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), instructions for implementing the various method embodiments described above, etc.; the data storage area may store data involved in the various method embodiments described above, etc. Optionally, the memory 605 may also be at least one storage device located remotely from the aforementioned processor 601. As shown in the figure, the memory 605, as a computer storage medium, may include an operating system, a network communication module, and application programs for a display module.

[0091] exist Figure 6 In the electronic device shown, the display 603 is mainly used to provide an input interface for the user and to obtain the user input data; while the processor 601 can be used to call the application program stored in the memory 605. When executed by one or more processors 601, the electronic device performs one or more methods as described in the above embodiments.

[0092] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0093] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0094] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the shown or discussed mutual couplings or direct couplings or communication connections may be through some service interfaces; indirect couplings or communication connections between apparatuses or units may be electrical or other forms.

[0095] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0096] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0097] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory 605 and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory 605 includes various media capable of storing program code, such as a USB flash drive, external hard drive, magnetic disk, or optical disk.

[0098] The above description is merely an exemplary embodiment of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Those skilled in the art will readily conceive of other embodiments of this disclosure upon considering the specification and the disclosure of practical truth. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described in this disclosure. The specification and embodiments are considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

Claims

1. A method for detecting defects in a substrate, characterized in that, The defect detection method includes: The substrate image to be detected is input into the first model and the second model respectively for defect detection; the first model includes a first training set, the second model includes a second training set, the first training set includes initial defect data and defect expansion data, and the second training set includes initial defect data; the substrate image to be detected includes multiple defects, and the target defect is any one of the multiple defects; The first and second models are invoked to identify the target defects, resulting in the first and second detection results. If the first detection result is the same as the second detection result, the defect detection of the target defect is completed, and the target defect is set as the first weight. The target defect is added to the first training set and the second training set respectively with the first weight, so that the first model can be trained according to the first training set and the second model can be trained according to the second training set. The first weight represents the probability that the target defect is added to the first training set and the second training set as a defect sample.

2. The defect detection method according to claim 1, characterized in that, The defect detection method further includes: If the first detection result indicates that the target defect has been identified and the second detection result indicates that the target defect has not been identified, the target defect is added to the first training set with a second weight and to the second training set with a third weight; wherein, the second weight represents the probability that the target defect is added to the first training set as a defect sample, the third weight represents the number of defect samples generated based on the target defect, and the second weight is less than the third weight. The target defects are marked so that they can be manually inspected later.

3. The defect detection method according to claim 1, characterized in that, The defect detection method further includes: If the first detection result indicates that the target defect has not been identified and the second detection result indicates that the target defect has been identified, the target defect is added to the first training set with the fourth weight and to the second training set with the fifth weight; the fifth weight represents the probability that the target defect is added to the first training set as a defect sample, the fourth weight represents the number of defect samples generated based on the target defect, and the fifth weight is less than the fourth weight. The target defects are marked so that they can be manually inspected later.

4. The defect detection method according to any one of claims 1-3, characterized in that, The detection results of the target defect include the location of the defect, the type of the defect, and the confidence level of the defect; among which, The first detection result is the same as the second detection result, specifically including: the location and type of the target defect are the same, and the confidence level of the target defect in both the first and second detection results is greater than the preset confidence level.

5. The defect detection method according to claim 4, characterized in that, The first detection result indicates that the target defect was not identified, including any of the following: the confidence level of the target defect is less than the preset confidence level of the first model; The confidence level of the target defect is greater than or equal to the preset confidence level of the first model, but less than the preset confidence level; The second detection result indicates that the target defect was not identified, including any of the following: the confidence level of the target defect is less than the preset confidence level of the second model; The confidence level of the target defect is greater than or equal to the preset confidence level of the second model, but less than the preset confidence level.

6. The defect detection method according to claim 1, characterized in that, The defect types in the substrate images to be inspected include dirt, dots, lines, dot mura, and line mura; among them, The number of defect types in the initial defect data of the first model is the first number, and the number of defect types in the initial defect data of the second model is the second number. The first number is greater than the second number.

7. The defect detection method according to any one of claims 1-3, characterized in that, The defect augmentation data is obtained by expanding the initial defect data using a preset defect augmentation method; among which, The preset defect expansion methods include one or more of the following: random cropping, grayscale variation, top-bottom and left-right mirroring variation, size variation, and random rotation.

8. The defect detection method according to any one of claims 1-3, characterized in that, The first weight represents the probability that the target defect will be added to the first and second training sets as a defect sample, and specifically includes any of the following: When there are multiple target defects, the first weight represents the probability of selecting a defect sample from the multiple target defects and adding it to the first training set and the second training set; When there is only one target defect, the first weight represents the probability that the single target defect is included as a defect sample in the first training set and the second training set.

9. The defect detection method according to claim 2, characterized in that, The third weight represents the number of defect samples generated based on the target defect, specifically including: The third weight is greater than 1, and the third weight represents the number of defect samples that are expanded to the target defect using a preset defect expansion method.

10. An AOI defect detection device, characterized in that, It includes a processor, a memory, a user interface, and a network interface. The memory is used to store instructions, and the user interface and network interface are used to communicate with other devices. The processor is used to execute the instructions stored in the memory so that the AOI defect detection device can perform a defect detection method involving a substrate as described in any one of claims 1-9 above.

Images