Detection performance evaluation program, detection performance evaluation method, and detection performance evaluation device

Abstract

A conventional detection performance evaluation method has the problem of inability to evaluate a defect detection device with high reliability. The detection performance evaluation program according to the present invention performs: a comparison process (S4) in which a determination reference image, which is obtained by an operator performing annotation processing on an image for evaluation so as to clearly indicate a defective region, and a detection result image are acquired and the determination reference image is superimposed on and compared to the detection result image; a labeling process (S5) in which any one of a correct report label TP, a false report label FP, and a failed report label FN is affixed to a defect detection region and a defect existing region on the basis of the result of the comparison process (S4) and a preset determination condition; and a detection accuracy calculation process (S6) in which a detection accuracy index value, which is an index value of detection performance of a defect detection device, is calculated on the basis of the number of detection result images subjected to the comparison process (S4) and the number of the correct report labels TP, the false report labels FP, and the failed report labels FN.

Description

Detection performance evaluation program, detection performance evaluation method, and detection performance evaluation apparatus

[0001] The present invention relates to a detection performance evaluation program, a detection performance evaluation method, and a detection performance evaluation apparatus, and more particularly to a detection performance evaluation program, a detection performance evaluation method, and a detection performance evaluation apparatus for evaluating the detection performance of a defect detection apparatus that determines whether an item is good or bad based on a product image acquired from an item to be inspected.

[0002] In the manufacturing process of producing products, defect inspections are conducted at various points to check for defects in the products. While these defect inspections are sometimes performed visually, in recent years, visual inspections using image analysis have become common. An example of such a visual inspection device using image analysis is disclosed in Patent Document 1.

[0003] The appearance inspection device described in Patent Document 1 includes at least a shape measurement device that measures the shape of a welding portion of a workpiece, which is a welded object, and a shape evaluation device that determines the presence and type of welding defects at the welding portion. The shape evaluation device includes a preprocessing unit that converts the measurement result of the shape measurement device into image data and converts the image data into a predetermined format, a first storage unit that stores at least an appearance inspection model for evaluating the shape of the welding portion, a determination unit that evaluates the image data converted by the preprocessing unit using the appearance inspection model and determines the presence and type of welding defects at the welding portion, and an output unit that outputs the determination result of the determination unit. The appearance inspection model includes a data acquisition unit that acquires image data of the welding portion, a storage unit that stores at least a plurality of learning data generated based on the image data with annotations and the appearance inspection model, a data division unit that divides each of the plurality of learning data into a predetermined size, and a learning unit that performs machine learning on the appearance inspection model read from the storage unit based on each of the plurality of learning data divided by the data division unit and the image data acquired from the data acquisition unit and divided into the predetermined size by the data division unit to generate a plurality of learned models, and an evaluation unit that evaluates the plurality of learned models according to a predetermined evaluation criterion and selects the learned model with the best performance. The annotation for the image data is a model generated or updated using a learning support system that identifies welding defect locations in the image data and labels the types of welding defects at the welding defect locations.

[0004] International Publication No. 2023 / 074183

[0005] In the appearance inspection device described in Patent Document 1, the evaluation of the learned model used for defect detection is performed based on the level of the defect detection rate. However, in appearance inspection, even if a defect is detected, the position of the detected defect location may deviate from the position of the defect location of the actual inspection target product. Due to this, there is a problem that the reliability of the inspection result cannot be said to be sufficient if only the goodness or badness of the inspection accuracy is evaluated simply by the level of the detection rate.

[0006] The detection performance evaluation program according to the present invention evaluates the performance of a defect detection device that takes an evaluation image obtained by photographing an item to be inspected as input and outputs a detection result image that includes a portion that differs from the good product image obtained by photographing the item to be inspected as a good product as a defect detection region, and is executed in a calculation unit included in a defect detection system that includes the defect detection device, and includes a comparison process that obtains a judgment criterion image obtained by annotating the evaluation image so that an operator indicates the defect region that should be determined to be a defect in the item to be inspected, and the detection result image, and compares the judgment criterion image and the detection result image by overlaying them, and the result of the comparison process Based on the result and a predetermined judgment condition, a labeling process is performed to attach one of the following labels to the defect detection area and the defect presence area: a correct label, a false label, or a misreport label; and a detection accuracy calculation process is performed to calculate a detection accuracy index value, which is an index value of the detection performance of the defect detection device, based on the number of correct labels, false labels, and misreport labels. The judgment condition is that a correct label is attached if the defect detection area overlaps the defect presence area by at least one pixel; a false label is attached if the defect presence area does not contain even one pixel of the defect detection area; and a misreport label is attached if the defect presence area does not contain even one pixel of the defect detection area.

[0007] The detection performance evaluation method according to the present invention is a method for evaluating the detection performance in a defect detection system having a defect detection device that takes an evaluation image obtained by photographing an item to be inspected as input and outputs a detection result image that includes a portion that differs from a good product image obtained by photographing the item to be inspected as a good product as a defect detection region, and the method involves obtaining a judgment criterion image obtained by annotating the evaluation image so that an operator indicates the defect region that should be determined to be a defect in the item to be inspected, and the detection result image, and then performing a comparison process in which the judgment criterion image and the detection result image are superimposed and compared, and the defect is determined based on the result of the comparison process and a preset judgment condition. The following processes are performed automatically using a computing device: a labeling process in which one of a correct report label, a false report label, or a failed report label is attached to the correct detection area and the faulty area; and a detection accuracy calculation process in which a detection accuracy index value, which is an indicator value of the detection performance of the fault detection device, is calculated based on the number of correct report labels, false report labels, and failed report labels. The determination conditions are that a correct report label is attached if the faulty area overlaps the faulty area by at least one pixel; a false report label is attached if the faulty area does not contain even one pixel of the faulty area; and a failed report label is attached if the faulty area does not contain even one pixel of the faulty area.

[0008] The detection performance evaluation device according to the present invention is a detection performance evaluation device that evaluates the performance of a defect detection device, which takes an evaluation image obtained by photographing an item to be inspected as input and outputs a detection result image that includes a portion that differs from a good product image obtained by photographing the item to be inspected as a good product, as a defect detection device, and acquires a judgment criterion image obtained by annotating the evaluation image so that an operator indicates the defect area that should be determined to be a defect in the item to be inspected, and the detection result image, and compares the judgment criterion image and the detection result image by overlaying them, and based on the result of the comparison processing device and a preset judgment condition, The device includes a labeling processing unit that attaches one of the following labels to the defect detection area and the defect presence area: a correct label, a false label, and a misreport label; and a detection accuracy calculation processing unit that calculates a detection accuracy index value, which is an index value of the detection performance of the defect detection device, based on the number of correct labels, false labels, and misreport labels. The determination condition is that a correct label is attached if the defect detection area overlaps the defect presence area by at least one pixel; a false label is attached if the defect presence area does not contain even one pixel of the defect detection area; and a misreport label is attached if the defect presence area does not contain even one pixel of the defect detection area.

[0009] The detection performance evaluation program, detection performance evaluation method, and detection performance evaluation apparatus according to the present invention calculate a detection accuracy index value based on whether or not the defect detection area overlaps with the defect presence area.

[0010] The detection performance evaluation program, detection performance evaluation method, and detection performance evaluation apparatus according to the present invention can improve the reliability of a fault detection device.

[0011] This is a diagram illustrating the difference in judgment results based on the difference between the evaluation image and the detection result image according to Embodiment 1. This is a schematic diagram of the defect detection system according to Embodiment 1. This is a flowchart illustrating the operation of the defect detection system according to Embodiment 1. This is a table illustrating examples of judgment conditions applied to the defect detection device according to Embodiment 1. This is a schematic diagram of the defect detection system according to Embodiment 2. This is a flowchart illustrating the operation of the defect detection system according to Embodiment 2. This is a schematic diagram of the defect detection system according to Embodiment 3. This is a flowchart illustrating the operation of the defect detection system according to Embodiment 3. This is a schematic diagram of the defect detection system according to Embodiment 4. This is a flowchart illustrating the operation of the defect detection system according to Embodiment 4. This is a schematic diagram of the defect detection system according to Embodiment 5. This is a flowchart illustrating the operation of the defect detection system according to Embodiment 5.

[0012] For clarity of explanation, the following descriptions and drawings have been omitted and simplified as appropriate. Furthermore, each element shown in the drawings as a functional block performing various processes can be composed of a CPU (Central Processing Unit), memory, and other circuits in hardware terms, and implemented in software terms by programs loaded into memory. Therefore, it will be understood by those skilled in the art that these functional blocks can be implemented in various ways using hardware alone, software alone, or a combination thereof, and are not limited to any one of these. In addition, the same elements are denoted by the same reference numerals in each drawing, and redundant explanations have been omitted where necessary.

[0013] Furthermore, the program described above includes, when loaded into a computer, a set of instructions (or software code) for causing the computer to perform one or more of the functions described in the embodiments. The program may be stored in a non-temporary computer-readable medium or a physical storage medium. Examples, but not limited to, include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technologies, CD-ROM, digital versatile disc (DVD), Blu-ray® disc or other optical disc storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices. The program may be transmitted over a temporary computer-readable medium or a communication medium. Examples, but not limited to, include temporary computer-readable medium or a communication medium that includes electrically, optically, acoustically, or otherwise propagating signals.

[0014] Embodiment 1 The defect detection system 1 according to Embodiment 1 includes a defect detection device 22 that detects defective parts of an item to be inspected by image diagnosis. In inspections using the defect detection system 1, the inspection accuracy of the defect detection device 22 has a significant impact on productivity. In particular, even if the defect detection device 22 detects that there are defective parts in an item to be inspected, the detected location of the defective part may differ from the actual defective part of the item to be inspected, and false correct answers may be included. If there are false correct answers in such detection results, it cannot be said that the defect detection device 22 is operating correctly, and its reliability is questionable.

[0015] Therefore, Figure 1 shows a diagram illustrating the difference in judgment results based on the difference between the evaluation image and the detection result image according to Embodiment 1. Figure 1 shows five detection examples (for example, detection examples I to V). Also in Figure 1, the input image in which the inspection target OBJ was captured and the inspection result image output by the defect detection device 22 are shown. The input image shows a defective product image in which the inspection target OBJ having a defect presence area IR was captured, and a good product image in which the inspection target OBJ without a defect presence area IR was captured. The detection result image shows the inspection target OBJ, the defect presence area IR attached to the inspection target OBJ, and the defect detection area DE which is the defect area detected by the defect detection device 22.

[0016] In the detection performance evaluation device 10 described below, a positive label TP, a false alarm label FP, or a missed alarm label FN is assigned to the defect-existing region IR and the defect-detection region DE, respectively. Specifically, a positive label TP is assigned when the defect-existing region IR overlaps with the defect-detection region DE by at least one pixel (Detection Example I). A false alarm label FP is assigned when the defect-existing region IR does not contain even one pixel of the defect-existing region DE (Detection Examples II and IV). The false alarm label FP is assigned to the defect-detection region DE that appears in a region without defects. A missed alarm label FN is assigned when the defect-existing region IR does not contain even one pixel of the defect-detection region DE (Detection Examples II and III). The missed alarm label FN is assigned to the defect-existing region IR that was not detected as a defect-detection region DE. Furthermore, as shown in the detection example V of Figure 1, if a detection result image is obtained in which the defect detection area DE is not included for a good product that does not include the defect presence area IR, this will not be taken into consideration when calculating the detection accuracy index value in the detection performance evaluation device 10 according to Embodiment 1.

[0017] The detection performance evaluation device 10 according to Embodiment 1 calculates a detection accuracy index value, which is an index value of the detection performance of the defect detection device, based on the number of correct labels TP, false labels FP, and missing labels TN. Specifically, the detection accuracy index value calculated by the detection performance evaluation device 10 includes at least one of the false alarm rate, the missing alarm rate, and the accuracy rate. The false alarm rate is calculated as FP / (TP+FP), the missing alarm rate is calculated as FN / (TP+FN), and the accuracy rate is calculated as TP / (TP+FP+FN). Below, an example of a defect detection system 1 including the detection performance evaluation device 10 will be described in detail.

[0018] Figure 2 shows a schematic diagram of the defect detection system 1 according to Embodiment 1. As shown in Figure 2, the defect detection system 1 according to Embodiment 1 includes a defect detection device 22 and a detection performance evaluation device 10. An evaluation image 20 is provided to the defect detection device 22 to obtain a detection result image 23. An operator performs annotation processing on the evaluation image 20 to generate a judgment criterion image 21. The judgment criterion image 21 includes images of defective products to which the operator has tagged the defect-existing region IR, which is determined to be a defective area in the image of the product to be inspected included in the evaluation image 20, to indicate that it is a defect-existing region IR. Note that the evaluation image 20, judgment criterion image 21, and detection result image 23 include multiple images. Furthermore, the evaluation image 20 and judgment criterion image 21 include both images of defective products and images of good products.

[0019] The defect detection device 22 may be, for example, artificial intelligence (AI) that detects the defect-existing region IR on the evaluation image 20 as the defect detection region DE by applying learned parameters, or it may be a device that detects the defect-existing region IR as the defect detection region DE using a predetermined function set in advance. The detection result image 23 is, for example, an image with feature vectors mapped. When the feature vectors of a region considered normal are used as reference feature vectors, the difference between the reference feature vectors and the region judged as normal is small, while the difference between the reference feature vectors and the defect detection region DE is large.

[0020] The detection performance evaluation device 10 takes an evaluation image 20 obtained by photographing an item to be inspected as input and outputs a detection result image that includes the portion that differs from the good product image obtained by photographing an item to be inspected that is judged to be good as a defect detection region DE. The detection performance evaluation device 10 can be implemented with dedicated hardware that implements the functions of each functional block described below, or it can be implemented with a detection performance evaluation program executed on a computer including a calculation unit. Below, as an example, an example of running the detection performance evaluation program on a computer will be described. The functional blocks described below are blocks of functions obtained by executing the program.

[0021] The detection performance evaluation device 10 includes a comparison processing unit 11, a labeling processing unit 12, and a detection accuracy calculation unit 13. The comparison processing unit 11 obtains a judgment criterion image 21 and a detection result image 23, which are obtained by annotating the evaluation image 20 so that the operator indicates the defective region IR, and performs a comparison process by overlaying the judgment criterion image 21 and the detection result image 23 for comparison.

[0022] The labeling processing unit 12 performs a labeling process to attach one of the following labels to the defect detection area DE and the defect presence area IR, based on the results of the comparison process and the predetermined judgment conditions: a correct report label TP, a false report label FP, or a failed report label FN. In this embodiment 1, the judgment conditions applied to the labeling processing unit 12 are as follows: a correct report label TP is attached if the defect detection area DE overlaps with the defect presence area IR by at least one pixel; a false report label FP is attached if the defect presence area DE does not contain any pixels of the defect presence area IR; and a failed report label FN is attached if the defect presence area IR does not contain any pixels of the defect detection area DE.

[0023] The detection accuracy calculation unit 13 calculates a detection accuracy index value, which is an index value of the detection performance of the fault detection device, based on the number of correct labels TP, false alarm labels FP, and missing alarm labels FN. Here, the detection accuracy index value includes at least one of the false alarm rate, the missing alarm rate, and the correct detection rate. The false alarm rate is the value obtained by dividing the number of false alarm labels FP by the total number of correct alarm labels TP and false alarm labels FN (FP / (TP+FP)). The missing alarm rate is the value obtained by dividing the number of missing alarm labels FN by the total number of correct alarm labels TP and missing alarm labels FN (FN / (TP+FN)). The correct detection rate is the value obtained by dividing the number of correct alarm labels TP by the total number of correct alarm labels TP, false alarm labels, and missing alarm labels FN (TP / (TP+FP+FN)).

[0024] Next, the operation of the defect detection system 1 according to Embodiment 1 will be described. Figure 3 shows a flowchart illustrating the operation of the defect detection system 1 according to Embodiment 1. As shown in Figure 3, in Embodiment 1, when evaluating the detection performance, first, multiple evaluation images 20 are obtained by photographing each of the multiple items to be inspected (Step S1). Then, multiple judgment criterion images 21 are generated from each of the multiple evaluation images 20 by operator operation (Step S2).

[0025] Next, multiple evaluation images 20 are input one by one to the defect detection device 22 to obtain multiple detection result images 23 corresponding to each of the multiple evaluation images 20 (step S3). Then, the comparison processing unit 11 is used to perform an overlay process on the multiple judgment criterion images 21 and the multiple detection result images 23 for each corresponding image (step S4).

[0026] Next, the labeling processing unit 12 is used to attach one of the following labels—a correct report label TP, a false report label FP, or a failed report label FN—to the defect detection area DE and the defect presence area IR based on the superposition result and the judgment conditions (step S5). Here, Figure 4 shows a table illustrating an example of the judgment conditions applied to the defect detection device according to Embodiment 1, and the judgment conditions will be explained with reference to Figure 4.

[0027] Figure 4 shows an example of the judgment conditions. In the example shown in Figure 4, if the defect area IR and the defect detection area DE overlap by even one pixel, the correct label TP is applied to both the defect area IR and the defect detection area DE. If the defect area IR and the defect detection area DE do not overlap by even one pixel, the false alarm label FN is applied to the defect area IR and the false alarm label TP is applied to the defect detection area DE. If only the defect area IR is present in the evaluation image and the detection result image, the false alarm label FN is applied to the defect area IR. If only the defect detection area DE is present in the evaluation image and the detection result image, the false alarm label FP is applied to the defect detection area DE. By applying labels to the defect detection area DE and the defect area IR based on these judgment conditions, the detection performance evaluation device 10 facilitates the subsequent calculation of the accuracy rate.

[0028] Referring to Figure 3, after the labeling process in step S5 is completed, the defect detection system 1 calculates a detection accuracy index value using the detection accuracy calculation unit 13 (step S6), and then terminates the detection performance evaluation process. This detection accuracy index value includes at least one of the false alarm rate, the failure rate, and the accuracy rate.

[0029] As described above, by using the detection performance evaluation device 10 according to Embodiment 1, it becomes possible to calculate a detection accuracy index value indicating the level of detection accuracy based on information that does not include false correct answers, which are judged as correct even if the positions of the defect presence area IR and the defect detection area DE are misaligned. Then, by evaluating the detection performance of the defect detection device 22 based on such a detection accuracy index value, it becomes possible to improve the reliability of the detection performance of the defect detection device 22.

[0030] Furthermore, in the judgment conditions applied to the detection performance evaluation device 10, a positive label is assigned if at least a portion of the defect-existing region IR in the judgment criterion image 21 and the defect-detection region DE in the detection result image 23 overlap. The range of the defect-existing region IR assigned by the operator's annotation process and the range of the defect-detection region DE detected by the defect detection device 22 do not always coincide. In such cases, if the condition were to assign a positive label only when the defect-existing region IR in the judgment criterion image 21 and the defect-detection region DE in the detection result image 23 completely match, the judgment would become unfairly strict, resulting in a problem where the detection performance evaluation of the defect detection device 22 would not be valid. However, in the detection performance evaluation device 10, by assigning a positive label if at least a portion of the defect-existing region IR in the judgment criterion image 21 and the defect-detection region DE in the detection result image 23 overlap, it is possible to avoid such an unfairly strict evaluation criterion.

[0031] Embodiment 2 Embodiment 2 describes an example of a process for improving the accuracy rate. In the description of Embodiment 2, components that are the same as those described in Embodiment 1 are denoted by the same reference numerals as in Embodiment 1 and their descriptions are omitted.

[0032] Figure 5 shows a schematic diagram of the defect detection system 2 according to Embodiment 2. If the defect detection device 22 uses artificial intelligence technology to which learned parameters are applied, it is conceivable to retrain and update the learned parameters applied to the defect detection device 22 using the evaluation image 20 and judgment criterion image 21 used in the detection performance evaluation. Therefore, in the defect detection system 2 shown in Figure 5, a learning device 24 is added to the defect detection system 1. In the defect detection system 2 according to Embodiment 2, retraining is performed using the learning device 24 until the detection accuracy index value exceeds a certain standard (for example, a threshold). Retraining can be performed by operator instruction or by the system automatically. The detection accuracy index value to be compared with the threshold only needs to be at least one of the false alarm rate, missed alarm rate, or correct detection rate, and can be appropriately selected according to the system specifications. Furthermore, the fact that any of the false alarm rate, missed alarm rate, or correct detection rate can be used as the detection accuracy index value to be compared with the threshold is the same in other embodiments described later.

[0033] Next, Figure 6 shows a flowchart illustrating the operation of the defect detection system 2 according to Embodiment 2. As shown in Figure 6, in the defect detection system 2 according to Embodiment 2, steps S11 and S12 are added to the operation of the defect detection system 1 according to Embodiment 1 shown in Figure 3. In step S11, it is determined whether the detection accuracy index value calculated by the detection accuracy calculation unit 13 in step S6 is equal to or greater than a preset threshold. If the detection accuracy index value is less than the threshold in step S11, parameter retraining is performed using the learning device 24 (step S12), and the processes of steps S3 to S6 are performed again. The retraining process in step S12 is performed until the detection accuracy index value in step S11 is equal to or greater than the threshold.

[0034] In the defect detection system 2 according to Embodiment 2, the defect detection device 22 includes trained parameters that have been learned so that the detection result image output when an evaluation image is input via machine learning indicates a defect detection area DE at the position of the defect presence area IR shown in the judgment criterion image. Furthermore, it is preferable that the detection accuracy calculation unit 13 presents the operator with the change in the detection accuracy index value before and after the update when the trained parameters are updated.

[0035] As described above, the defect detection system 2 according to Embodiment 2 makes it possible to easily realize a defect detection device 22 to which learned parameters that satisfy a certain standard for detection accuracy index values ​​are applied.

[0036] Embodiment 3 Embodiment 3 describes another example of processing for improving the detection accuracy index value. In the description of Embodiment 3, components that are the same as those described in Embodiment 1 are denoted by the same reference numerals as in Embodiment 1 and their descriptions are omitted.

[0037] Figure 7 shows a schematic diagram of the defect detection system 3 according to Embodiment 3. As shown in Figure 7, the defect detection system 3 is modified from the defect detection system 1 by adding a preprocessing unit 25. The preprocessing unit 25 is positioned in front of the defect detection device 22 and applies one of several preprocessing methods (for example, preprocessing methods [1] to [N], where N is an integer indicating the processing method number) to the evaluation image 20 and provides the preprocessed evaluation image 20 to the defect detection device 22. In Embodiment 3, the detection accuracy calculation unit 13 presents the operator with the detection accuracy index value for each preprocessing method. This makes it easy for the operator to select the method with the highest detection accuracy index value as the preprocessing method to be applied by the preprocessing unit 25 when operating the defect detection system 3.

[0038] Figure 8 shows a flowchart illustrating the operation of the defect detection system 3 according to Embodiment 3. As shown in Figure 8, the operation of the defect detection system 3 according to Embodiment 3 is the same as the defect detection system 1 shown in Figure 3, with the addition of steps S21 to S25. Step S21 is a process performed after step S2, in which the variable n is initialized to 1. Step S22 is a process performed after step S21, in which the preprocessing method [n] is applied to a plurality of evaluation images 20 and then provided to the defect detection device 22. Then, in steps S3 to S6, the detection accuracy index value when the preprocessing method [n] is applied is calculated. After that, in steps S23 and S24, the defect detection system 3 calculates the detection accuracy index value for each preprocessing method while increasing the variable n by 1 each time. Then, in the defect detection system 3, referring to the detection accuracy index value for each preprocessing method presented by the detection accuracy calculation unit 13, the preprocessing method with the highest detection accuracy index value is determined to be the preprocessing method when operating the system (step S25).

[0039] The preprocessing unit 25 may perform image processing such as sharpening and contrast adjustment, and multiple preprocessing methods with varying degrees of each image processing may be prepared. Changing this preprocessing method changes how easily the defect detection device 22 can recognize the defect-existing IR region. Therefore, by evaluating which of the multiple preprocessing methods yields the highest detection accuracy index and selecting the preprocessing method with the highest detection accuracy index during system operation, it is possible to improve the defect detection accuracy of the defect detection system 3.

[0040] Embodiment 4 Embodiment 4 describes another example of processing for improving the detection accuracy index value. In the description of Embodiment 4, components that are the same as those described in Embodiment 1 are denoted by the same reference numerals as in Embodiment 1 and their descriptions are omitted.

[0041] Figure 9 shows a schematic diagram of the defect detection system 4 according to Embodiment 4. As shown in Figure 9, the defect detection system 4 is a defect detection system 1 with the addition of a post-processing unit 26. The post-processing unit 26 is located downstream of the defect detection device and applies one of several post-processing methods (for example, pre-processing methods [1] to [N], where N is an integer indicating the processing method number) to the detection result image output by the defect detection device 22, and provides the post-processed evaluation image 20 to the detection performance evaluation device 10. In Embodiment 4, the detection accuracy calculation unit 13 presents the operator with the detection accuracy index value for each post-processing method. This makes it easy for the operator to select the method with the highest detection accuracy index value as the post-processing method to be applied by the post-processing unit 26 when operating the defect detection system 4.

[0042] Figure 10 shows a flowchart illustrating the operation of the defect detection system 4 according to Embodiment 4. As shown in Figure 10, the operation of the defect detection system 4 according to Embodiment 4 is the same as the defect detection system 1 shown in Figure 3, with the addition of steps S31 to S35. Step S31 is a process performed after step S2, in which the variable n is initialized to 1. Step S32 is a process performed after step S31, in which the post-processing method [n] is applied to a plurality of evaluation images output by the defect detection device 22 and then provided to the detection performance evaluation device 10. Then, in steps S3 to S6, the detection accuracy index value when the post-processing method [n] is applied is calculated. After that, in steps S33 and S34, the defect detection system 4 calculates the detection accuracy index value for each post-processing method while increasing the variable n by 1 each time. Then, in the defect detection system 4, referring to the detection accuracy index value for each post-processing method presented by the detection accuracy calculation unit 13, the post-processing method with the highest detection accuracy index value is determined as the pre-processing method when operating the system (step S35).

[0043] In the post-processing unit 26, for example, it is conceivable to perform image processing such as smoothing processing of the detection feature amount generated in the defect detection device 22 or binarization processing, and to prepare a plurality of post-processing methods with adjusted degrees of each processing. By changing this post-processing method, the range and size of the defect detection area DE change. Therefore, by evaluating which of the plurality of post-processing methods results in the highest detection accuracy index value and selecting the post-processing method with the highest detection accuracy index value during system operation, it becomes possible to improve the defect detection accuracy of the defect detection system 4.

[0044] Embodiment 5 In Embodiment 5, another example of the processing for improving the detection accuracy index value will be described. In the description of Embodiment 5, the same components as those described in Embodiment 1 are denoted by the same reference numerals as in Embodiment 1, and the description thereof is omitted.

[0045] FIG. 11 shows a schematic diagram of a defect detection system 5 according to Embodiment 5. The evaluation image 20 is generated by photographing the inspection target product. In Embodiment 5, however, an evaluation image 30 including a plurality of sets of evaluation images (for example, evaluation image sets OBJ_img_set[1] to [N]) with different shooting conditions such as the shooting angle of the inspection target product, the angle and intensity of illumination, etc. is prepared. Also, in Embodiment 5, a reference image 31 including a plurality of sets of reference images (for example, reference image sets ma_img_set[1] to [N]) corresponding to the plurality of sets of evaluation images, and a detection result image 32 including a plurality of sets of detection result images (for example, detection result image sets a_img_set[1] to [N]) corresponding to the plurality of sets of evaluation images are generated. Then, in the defect detection system 5 according to Embodiment 5, the detection accuracy index value is calculated for each shooting condition, and the shooting condition that results in the highest detection accuracy index value is determined.

[0046] Therefore, FIG. 12 shows a flowchart for explaining the operation of the defect detection system 5 according to Embodiment 5. As shown in FIG. 12, in the defect detection system 5 according to Embodiment 5, first, the variable n is initialized to 1 (step S40). Subsequently, each of the plurality of inspection target products is photographed under the photographing condition [n], and an evaluation image set [n] is obtained. Subsequently, the operator generates a reference image set [n] using the evaluation image set [n] (step S42). Also, the evaluation images included in the evaluation image set [n] are input one by one to the defect detection device 22, and a result image set [n] corresponding to the evaluation image set [n] is obtained.

[0047] After that, a labeling process corresponding to steps S4 and S5 in FIG. 3 is performed (step S44). Specifically, in step S44, for the reference image set [n] and the detection result image set [n], each corresponding image is compared, and one of the positive label TP, false positive label FP, and false negative label FN is attached to the defect detection region DE and the defect existence region IR. Then, in the defect detection system 5, a detection accuracy index value corresponding to the photographing condition [n] is calculated based on the label attached in step S44 (step S6). After that, the processes of steps S41 to S46 are repeated until the detection accuracy index values for all the preset photographing conditions (photographing conditions [1] to [N]) are calculated. After that, the photographing condition [n] with the highest detection accuracy index value among the detection accuracy index values corresponding to the photographing conditions [1] to [N] is determined as the photographing method to be used during system operation (step S47).

[0048] From the above description, according to the defect detection system 5 according to Embodiment 5, system operation under the photographing condition with the highest detection accuracy index value becomes easy.

[0049] Note that the present invention is not limited to the above embodiments, and can be appropriately modified without departing from the gist.

[0050] This application claims priority based on Japanese Patent Application No. 2024-156646 filed on September 10, 2024, and incorporates the entire disclosure thereof herein.

[0051] 1-5 Defect detection system 10 Detection performance evaluation device 11 Comparison processing unit 12 Labeling processing unit 13 Detection accuracy calculation unit 20, 30 Evaluation images 21, 31 Judgment criterion images 22 Defect detection device 23, 32 Detection result images 24 Learning device 25 Pre-processing unit 26 Post-processing unit OBJ Inspection target item IR Defect presence area DE Defect detection area

Claims

1. A detection performance evaluation program is performed in a calculation unit included in a defect detection system that includes the defect detection device. The program evaluates the performance of a defect detection device that takes an evaluation image obtained by photographing an item to be inspected as input and outputs a detection result image that includes a portion that differs from the good product image obtained by photographing the item to be inspected as a good product as a defect detection region, and includes: a comparison process that obtains a judgment criterion image obtained by annotating the evaluation image so that an operator indicates the defect area that should be judged as a defect in the item to be inspected, and the detection result image, and compares the judgment criterion image and the detection result image by overlaying them; a labeling process that attaches one of a correct label, a false label, or a misreport label to the defect detection region and the defect area based on the result of the comparison process and a pre-set judgment condition; and a detection accuracy calculation process that calculates a detection accuracy index value, which is an index value of the detection performance of the defect detection device, based on the number of correct labels, false labels, and misreport labels, wherein the judgment condition is that a correct label is attached when the defect detection region overlaps the defect area by even one pixel. A detection performance evaluation program that, under the conditions that, if the defect detection area does not contain even one pixel of the defect area, a false alarm label is applied, and if the defect area does not contain even one pixel of the defect detection area, a misreport label is applied.

2. The detection performance evaluation program according to claim 1, wherein the detection accuracy index value includes at least one of: a false alarm rate obtained by dividing the number of false alarm labels by the total number of correct alarm labels and the number of false alarm labels; a false alarm rate obtained by dividing the number of false alarm labels by the total number of correct alarm labels and the number of false alarm labels; and a correct alarm rate obtained by dividing the number of correct alarm labels by the total number of correct alarm labels, false alarm labels, and false alarm labels.

3. The defect detection device includes trained parameters that have been trained by machine learning so that the detection result image output when the evaluation image is input indicates the defect detection area at the position of the evaluation image corresponding to the position of the defect presence area shown in the judgment criterion image, and the detection performance evaluation program presents the operator with the change in the detection accuracy index value before and after the update when the trained parameters are updated.

4. The defect detection system comprises at least one of the following: a pre-processing unit located before the defect detection device, which applies one of a plurality of pre-processing methods to the evaluation image and provides the pre-processed evaluation image to the defect detection device; and a post-processing unit located after the defect detection device, which applies one of a plurality of post-processing methods to the detection result image output by the defect detection device and provides the post-processed evaluation image to the detection performance evaluation program, wherein the detection performance evaluation program presents to the operator at least one of the detection accuracy index value for each pre-processing method and the detection accuracy index value for each post-processing method.

5. The detection performance evaluation program according to claim 1, wherein the evaluation image includes a plurality of evaluation image sets obtained by changing the shooting conditions for the same product to be inspected, and the detection performance evaluation program presents the operator with the detection accuracy index value for each evaluation image set.

6. A method for evaluating the detection performance of a defect detection system having a defect detection device that takes an evaluation image obtained by photographing an item to be inspected as input and outputs a detection result image that includes a portion that differs from the good product image obtained by photographing the item to be inspected as a good product as a defect detection region, the method comprising: acquiring a judgment criterion image obtained by annotating the evaluation image so that an operator indicates the defect area that should be judged as a defect in the item to be inspected, and the detection result image, and comparing the judgment criterion image and the detection result image by overlaying them; labeling the defect detection region and the defect area based on the result of the comparison process and a predetermined judgment condition, by attaching one of a correct label, a false label, or a misprint label to the defect detection region and the defect area; and calculating a detection accuracy index value, which is an index value of the detection performance of the defect detection device, based on the number of correct labels, false labels, and misprint labels, the method comprising automatic processing using a calculation device, wherein the judgment condition is that a correct label is attached when the defect detection region overlaps the defect area by even one pixel, A detection performance evaluation method that includes the condition that a false alarm label is applied if the defect detection area does not contain even one pixel of the defect area, and a misreport label is applied if the defect area does not contain even one pixel of the defect detection area.

7. A detection performance evaluation device for evaluating the performance of a defect detection device that takes an evaluation image obtained by photographing an item to be inspected as input and outputs a detection result image that includes a portion that differs from the good product image obtained by photographing the item to be inspected as a good product as a defect detection region, comprising: a comparison processing unit that obtains a judgment criterion image obtained by annotating the evaluation image so that an operator indicates the defect area that should be judged as a defect in the item to be inspected, and the detection result image, and compares the judgment criterion image and the detection result image by overlaying them; a labeling processing unit that attaches one of a positive report label, a false report label, or a misreport label to the defect detection region and the defect area based on the result of the comparison processing unit and a preset judgment condition; and a detection accuracy calculation processing unit that calculates a detection accuracy index value, which is an index value of the detection performance of the defect detection device, based on the number of positive report labels, false report labels, and misreport labels, wherein the judgment condition is to attach a positive report label when the defect detection region overlaps the defect area by even one pixel, A detection performance evaluation device that, under the conditions that a false alarm label is applied if the defect detection area does not contain even one pixel of the defect area, and a misreport label is applied if the defect area does not contain even one pixel of the defect detection area.

Images