Detection performance evaluation program, detection performance evaluation method, and detection performance evaluation device

Abstract

Conventional methods for evaluating detection performance have had the problem that they are unable to provide highly reliable evaluation of defect detection devices. [Solution] The detection performance evaluation program of the present invention performs a comparison process (S4) in which a judgment reference image obtained by an operator annotating an evaluation image to clearly indicate defective areas and a detection result image are acquired, and the judgment reference image and the detection result image are superimposed and compared; a labeling process (S5) in which one of a positive alarm label TP, a false alarm label FP, and a failure label FN is assigned to the defect detection area and the defect existence area based on the result of the comparison process (S4) and preset judgment conditions; and a detection accuracy calculation process (S6) in which a detection accuracy index value, which is an index value for the detection performance of the defect detection device, is calculated based on the number of detection result images that have been subjected to the comparison process (S4) and the number of positive alarm labels TP, false alarm labels FP, and failure labels FN.

Description

[Technical Field]

[0001] The present invention relates to a detection performance evaluation program, a detection performance evaluation method, and a detection performance evaluation device, and more particularly to a detection performance evaluation program, a detection performance evaluation method, and a detection performance evaluation device that evaluate the detection performance of a defect detection device that determines the quality of an inspected product based on a product image acquired from the inspected product. [Background technology]

[0002] In the manufacturing process of producing products, defect inspections are performed at various times to check whether or not there are any defects in the produced products. This defect inspection is sometimes performed visually, but in recent years, appearance inspections using image analysis are often performed. Therefore, an example of such an appearance inspection device using image analysis is disclosed in Patent Document 1.

[0003] The visual inspection device described in Patent Document 1 includes at least a shape measurement device that measures the shape of a welded portion of a workpiece, which is an object to be welded, and a shape evaluation device that determines the presence and type of welding defects at the welded portion. The shape evaluation device includes at least a preprocessing unit that converts the measurement results of the shape measurement device into image data and converts the image data into a predetermined format, a first storage unit that stores at least a visual inspection model for evaluating the shape of the welded portion, a determination unit that evaluates the image data converted by the preprocessing unit using the visual inspection model to determine the presence and type of welding defects at the welded portion, and an output unit that outputs the determination result of the determination unit. The visual inspection model includes a data acquisition unit that acquires image data of the welded portion, and a plurality of learning data generated based on the image data to which annotations have been added. The visual inspection system includes at least a memory unit that stores at least training data and the visual inspection model; a data division unit that divides the plurality of training data into predetermined sizes; a learning unit that performs machine learning on the visual inspection model read out from the memory unit based on the plurality of training data divided by the data division unit and the image data acquired from the data acquisition unit and divided into the predetermined sizes by the data division unit, thereby generating a plurality of trained models; and an evaluation unit that evaluates the plurality of trained models in accordance with predetermined evaluation criteria and selects the trained model with the best performance, and the annotations added to the image data are models generated or updated using a learning support system that identifies defective welds in the image data and labels the defective welds with the type of defective weld. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2023 / 074183 Summary of the Invention [Problem to be solved by the invention]

[0005] In the visual inspection device described in Patent Document 1, the trained model used for defect detection is evaluated based on the defect detection rate. However, in visual inspection, even if a defect is detected, the position of the detected defect may differ from the position of the defect on the actual product being inspected. For this reason, there is a problem in that the reliability of the inspection results cannot be said to be sufficient if the inspection accuracy is evaluated solely based on the detection rate. [Means for solving the problem]

[0006] The detection performance evaluation program according to the present invention evaluates the performance of a defect detection device that receives an evaluation image obtained by photographing an inspection target product as an input, and outputs a detection result image that includes, as a defect detection area, a portion that differs from a good product image obtained by photographing the inspection target product that is judged to be a good product, and is executed by a calculation unit included in a defect detection system that includes the defect detection device. The program includes a comparison process that acquires a judgment reference image obtained by annotating the evaluation image by an operator so as to clearly indicate a defect existence area that should be judged to be a defect in the inspection target product, and the detection result image, and compares the judgment reference image and the detection result image by superimposing them on each other, and a comparison process that evaluates the results of the comparison process. a labeling process for labeling the defect detection area and the defect existence area with one of a true alarm label, a false alarm label, and a false alarm label based on the results of the labeling process and a preset judgment condition; and a detection accuracy calculation process for calculating a detection accuracy index value, which is an index value of the detection performance of the defect detection device, based on the number of true alarm labels, false alarm labels, and false alarm labels. The judgment condition is such that a true alarm label is assigned when the defect detection area overlaps with the defect existence area by even one pixel, a false alarm label is assigned when the defect detection area does not include even one pixel of the defect existence area, and a false alarm label is assigned when the defect existence area does not include even one pixel of the defect detection area.

[0007] The detection performance evaluation method according to the present invention is a detection performance evaluation method in a defect detection system having a defect detection device that receives an evaluation image obtained by photographing an inspection target product and outputs a detection result image including, as a defect detection area, a portion that differs from a good product image obtained by photographing the inspection target product that is judged to be a good product, and the method includes a comparison process in which an operator annotates the evaluation image to clearly indicate a defect existence area in the inspection target product that should be judged to be a defect, obtains a judgment reference image and the detection result image, and compares the judgment reference image and the detection result image by superimposing them on each other, and determines the defect based on the result of the comparison process and a preset judgment condition. A labeling process of assigning one of a true alarm label, a false alarm label, and a false alarm label to the correct detection area and the defect-existing area, and a detection accuracy calculation process of calculating a detection accuracy index value, which is an index value of the detection performance of the defect detection device, based on the number of true alarm labels, false alarm labels, and false alarm labels, are performed automatically using a computing device, and the judgment conditions are as follows: a true alarm label is assigned if the defect detection area overlaps with the defect-existing area by even one pixel, a false alarm label is assigned if the defect detection area does not contain even one pixel of the defect-existing area, and a false alarm label is assigned if the defect-existing area does not contain even one pixel of the defect detection area.

[0008] The detection performance evaluation device according to the present invention is a detection performance evaluation device that evaluates the performance of a defect detection device that receives an evaluation image obtained by photographing an inspection target product as an input, and outputs a detection result image that includes, as a defect detection, a portion that differs from a good product image obtained by photographing the inspection target product that is judged to be a good product, and that acquires a judgment reference image obtained by an operator annotating the evaluation image so as to clearly indicate a defect-existing region that should be judged to be a defect in the inspection target product, and the detection result image, and compares the judgment reference image and the detection result image by superimposing them on each other; and based on the result of the comparison processing unit and preset judgment conditions, The defect detection device has a labeling processing unit that assigns one of a true report label, a false report label, and a false report label to the defect detection area and the defect existence area, and a detection accuracy calculation processing unit that calculates a detection accuracy index value that is an index value of the detection performance of the defect detection device based on the number of true report labels, false report labels, and false report labels, and the judgment conditions are that a true report label is assigned if the defect detection area overlaps with the defect existence area by even one pixel, a false report label is assigned if the defect detection area does not contain even one pixel of the defect existence area, and a false report label is assigned if the defect existence area does not contain even one pixel of the defect detection area.

[0009] The detection performance evaluation program, detection performance evaluation method, and detection performance evaluation device according to the present invention calculate a detection accuracy index value based on whether or not a defect detection region overlaps with a defect existence region. [Effects of the Invention]

[0010] According to the detection performance evaluation program, the detection performance evaluation method, and the detection performance evaluation device of the present invention, the reliability of the defect detection device can be improved. [Brief explanation of the drawings]

[0011] [Figure 1] 10A and 10B are diagrams illustrating a difference in a determination result based on a difference between an evaluation image and a detection result image according to the first embodiment. [Figure 2]1 is a schematic diagram of a defect detection system according to a first embodiment; [Figure 3] 4 is a flowchart illustrating an operation of the defect detection system according to the first embodiment. [Figure 4] 4 is a table illustrating an example of a determination condition applied to the defect detection device according to the first embodiment. [Figure 5] FIG. 10 is a schematic diagram of a defect detection system according to a second embodiment. [Figure 6] 10 is a flowchart illustrating the operation of the defect detection system according to the second embodiment. [Figure 7] FIG. 10 is a schematic diagram of a defect detection system according to a third embodiment. [Figure 8] 10 is a flowchart illustrating the operation of the defect detection system according to the third embodiment. [Figure 9] FIG. 10 is a schematic diagram of a defect detection system according to a fourth embodiment. [Figure 10] 10 is a flowchart illustrating the operation of the defect detection system according to the fourth embodiment. [Figure 11] FIG. 10 is a schematic diagram of a defect detection system according to a fifth embodiment. [Figure 12] 13 is a flowchart illustrating the operation of the defect detection system according to the fifth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] For clarity of explanation, the following description and drawings have been omitted and simplified as appropriate. Furthermore, each element shown in the drawings as a functional block performing various processes can be configured in hardware with a CPU (Central Processing Unit), memory, and other circuits, and in software with a program loaded into memory, etc. Therefore, those skilled in the art will understand that these functional blocks can be realized in various forms using only hardware, only software, or a combination thereof, and are not limited to any one of these. In addition, the same elements are designated by the same reference numerals in each drawing, and redundant explanations are omitted as necessary.

[0013] Furthermore, the above-described program includes a set of instructions (or software code) that, when loaded into a computer, causes the computer to perform one or more functions described in the embodiments. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable media or tangible storage media include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray (registered trademark) disc or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may also be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, transitory computer-readable media or communication media include electrical, optical, acoustic, or other forms of propagated signals.

[0014] Embodiment 1 The defect detection system 1 according to the first embodiment includes a defect detection device 22 that detects defective parts in an inspection target product by image diagnosis. In inspections using the defect detection system 1, the inspection accuracy of the defect detection device 22 has a significant impact on productivity. In particular, even if the defect detection device 22 detects the presence of a defective part in an inspection target product, the detection result may include false positives, where the position of the detected defective part is different from the actual defective part in the inspection target product. If such a detection result includes false positives, it cannot be said with certainty that the defect detection device 22 is operating correctly, and doubts arise about its reliability.

[0015] FIG. 1 illustrates the difference in judgment results based on the difference between the evaluation image and the detection result image according to the first embodiment. Five detection examples (e.g., detection examples I to V) are shown in FIG. 1. Also, FIG. 1 illustrates an input image of an inspection object OBJ and an inspection result image output by the defect detection device 22. The input image includes a defective product image of an inspection object OBJ having a defect existence region IR, and a non-defective product image of an inspection object OBJ that does not include the defect existence region IR. The detection result image includes the inspection object OBJ, the defect existence region IR on the inspection object OBJ, and a defect detection region DE, which is a defect region detected by the defect detection device 22.

[0016] The detection performance evaluation device 10 described below assigns one of a true report label TP, a false report label FP, and a false report label FN to the defect existence region IR and the defect detection region DE, respectively. Specifically, the true report label TP is assigned when the defect detection region DE overlaps the defect existence region IR by even one pixel (detection example I). The false report label FP is assigned when the defect detection region DE does not contain even one pixel of the defect existence region IR (detection examples II and IV). The false report label FP is assigned to a defect detection region DE that appears in an area without a defect. The false report label FN is assigned when the defect existence region IR does not contain even one pixel of the defect detection region DE (detection examples II and III). The false report label FN is assigned to a defect existence region IR that was not detected as the defect detection region DE. Note that when a detection result image that does not contain the defect detection region DE is obtained for a non-defective product that does not contain the defect existence region IR, as in detection example V of FIG. 1, the detection performance evaluation device 10 according to the first embodiment does not take this into consideration when calculating the detection accuracy index value.

[0017] Then, the detection performance evaluation device 10 according to the first embodiment calculates a detection accuracy index value, which is an index value of the detection performance of the defect detection device, based on the number of correct report labels TP, false report labels FP, and false report labels TN. Specifically, the detection accuracy index value calculated by the detection performance evaluation device 10 includes at least one of the false report rate, false report rate, and accuracy rate. The false report rate is calculated as FP / (TP+FP), the false report rate is calculated as FN / (TP+FN), and the accuracy rate is calculated as TP / (TP+FP+FN). An example of a defect detection system 1 including the detection performance evaluation device 10 will be described in detail below.

[0018] FIG. 2 shows a schematic diagram of the defect detection system 1 according to the first embodiment. As shown in FIG. 2, the defect detection system 1 according to the first embodiment includes a defect detection device 22 and a detection performance evaluation device 10. An evaluation image 20 is provided to the defect detection device 22 to obtain a detection result image 23. An operator performs annotation processing on the evaluation image 20 to generate a judgment reference image 21. The judgment reference image 21 includes a defective product image in which the operator has tagged a defect existence region IR, which is determined to be a defective part in the image of the inspection target product included in the evaluation image 20, with an indication that the defect existence region IR is a defect existence region. The evaluation image 20, the judgment reference image 21, and the detection result image 23 each include a plurality of images. The evaluation image 20 and the judgment reference image 21 include both defective product images and non-defective product images.

[0019] The defect detection device 22 may be, for example, an artificial intelligence (AI) that applies learned parameters to detect defect areas IR on the evaluation image 20 as defect detection areas DE, or may be a device that detects defect areas IR as defect detection areas DE using a predetermined function that has been set in advance. The detection result image 23 is, for example, an image onto which feature amounts are mapped. When the feature amounts of areas considered to be normal are used as reference feature amounts, the difference between these feature amounts and the reference feature amounts is small in areas determined to be normal, and large in areas considered to be normal.

[0020] The detection performance evaluation device 10 evaluates the performance of a defect detection device that receives an evaluation image 20 obtained by photographing an inspection target product and outputs a detection result image that includes a defect detection area DE, which is a difference from a good-product image obtained by photographing an inspection target product that is deemed to be good. The detection performance evaluation device 10 can be realized by dedicated hardware that realizes the functions of each functional block described below, or by a detection performance evaluation program executed on a computer including a calculation unit. Below, an example of executing the detection performance evaluation program on a computer will be described. The functional blocks described below are blocks that represent functions obtained by executing the program.

[0021] The detection performance evaluation device 10 has a comparison processing unit 11, a labeling processing unit 12, and a detection accuracy calculation unit 13. The comparison processing unit 11 acquires a judgment reference image 21, which is obtained by an operator annotating an evaluation image 20 to clearly indicate a defect existence region IR, and a detection result image 23, and performs a comparison process in which the judgment reference image 21 and the detection result image 23 are superimposed and compared.

[0022] The labeling processing unit 12 performs labeling processing to assign one of a true alarm label TP, a false alarm label FP, and a false alarm label FN to the defect detection region DE and the defect existence region IR based on the result of the comparison processing and preset judgment conditions. Here, the judgment conditions applied to the labeling processing unit 12 in the first embodiment are as follows: if the defect detection region DE overlaps the defect existence region IR by even one pixel, the true alarm label TP is assigned; if the defect detection region DE does not contain even one pixel of the defect existence region IR, the false alarm label FP is assigned; and if the defect existence region IR does not contain even one pixel of the defect detection region DE, the false alarm label FN is assigned.

[0023] The detection accuracy calculation unit 13 calculates a detection accuracy index value, which is an index value of the detection performance of the defect detection device, based on the number of correct reports TP, false reports FP, and false reports FN. Here, the detection accuracy index value includes at least one of the false report rate, false reports rate, and accuracy rate. The false report rate is the value (FP / (TP+FP)) obtained by dividing the number of false reports FP by the total number of correct reports TP and false reports FN. The false reports rate is the value (FN / (TP+FN)) obtained by dividing the number of false reports FN by the total number of correct reports TP and false reports FN. The accuracy rate is the value (TP / (TP+FP+FN)) obtained by dividing the number of correct reports TP by the total number of correct reports TP, false reports, and false reports FN.

[0024] Next, the operation of the defect detection system 1 according to the first embodiment will be described. Fig. 3 shows a flowchart illustrating the operation of the defect detection system 1 according to the first embodiment. As shown in Fig. 3, in the first embodiment, when evaluating detection performance, first, a plurality of inspection target products are photographed to obtain a plurality of evaluation images 20 (step S1). Then, a plurality of judgment reference images 21 are generated from each of the plurality of evaluation images 20 by an operator's operation (step S2).

[0025] Next, the plurality of evaluation images 20 are input one by one into the defect detection device 22 to obtain a plurality of detection result images 23 corresponding to the plurality of evaluation images 20 (step S3). After that, the comparison processing unit 11 performs a superposition process for each of the plurality of reference images 21 and the plurality of detection result images 23 corresponding to each other (step S4).

[0026] Next, using the labeling processing unit 12, one of a positive report label TP, a false report label FP, and a negative report label FN is assigned to the defect detection area DE and the defect existence area IR based on the overlapping result and the judgment conditions (step S5). Here, a table explaining examples of judgment conditions applied to the defect detection device according to the first embodiment is shown in FIG. 4, and the judgment conditions will be explained with reference to FIG.

[0027] 4 shows an example of the judgment conditions. In the example shown in FIG. 4, if the defect existence region IR and the defect detection region DE overlap even by one pixel, a correct report label TP is assigned to both the defect existence region IR and the defect detection region DE. If the defect existence region IR and the defect detection region DE do not overlap even by one pixel, a false report label FN is assigned to the defect existence region IR and a false report label TP is assigned to the defect detection region DE. If only the defect existence region IR is present in the evaluation image and the detection result image, a false report label FN is assigned to the defect existence region IR. If only the defect detection region DE is present in the evaluation image and the detection result image, a false report label FP is assigned to the defect detection region DE. By assigning labels to the defect detection region DE and the defect existence region IR based on these judgment conditions, the detection performance evaluation device 10 makes it easier to calculate the accuracy rate thereafter.

[0028] 3, after the labeling process in step S5 is completed, the defect detection system 1 calculates a detection accuracy index value using the detection accuracy calculation unit 13 (step S6), and then terminates the detection performance evaluation process. The detection accuracy index value includes at least one of the false alarm rate, the missed alarm rate, and the accuracy rate.

[0029] As explained above, by using the detection performance evaluation device 10 according to the first embodiment, it is possible to calculate a detection accuracy index value that indicates the level of detection accuracy based on information that does not include false positives that are determined to be correct even when the positions of the defect existence region IR and the defect detection region DE are misaligned. Then, by evaluating the detection performance of the defect detection device 22 based on such a detection accuracy index value, it is possible to improve the reliability of the detection performance of the defect detection device 22.

[0030] The detection performance evaluation device 10 applies a judgment condition that a positive report label is assigned if at least a portion of the defect existence region IR in the judgment reference image 21 overlaps with the defect detection region DE in the detection result image 23. The range of the defect existence region IR assigned by the operator in the annotation process and the range of the defect detection region DE detected by the defect detection device 22 often do not necessarily match. In such cases, if the condition were to assign a positive report label only when the defect existence region IR in the judgment reference image 21 and the defect detection region DE in the detection result image 23 completely match, the judgment would be unduly strict, resulting in a problem in which the detection performance evaluation of the defect detection device 22 is not valid. However, the detection performance evaluation device 10 applies a condition that a positive report label is assigned if at least a portion of the defect existence region IR in the judgment reference image 21 overlaps with the defect detection region DE in the detection result image 23, thereby avoiding such unduly strict evaluation criteria.

[0031] Embodiment 2 In the second embodiment, an example of a process for improving the accuracy rate will be described. In the description of the second embodiment, the same components as those described in the first embodiment will be assigned the same reference numerals as those in the first embodiment, and the description thereof will be omitted.

[0032] FIG. 5 shows a schematic diagram of a defect detection system 2 according to a second embodiment. When the defect detection device 22 uses artificial intelligence technology to which learned parameters are applied, it is conceivable to re-learn and update the learned parameters applied to the defect detection device 22 using the evaluation image 20 and the reference image 21 used in the detection performance evaluation. Therefore, the defect detection system 2 shown in FIG. 5 adds a learning device 24 to the defect detection system 1. The defect detection system 2 according to the second embodiment performs re-learning using the learning device 24 until the detection accuracy index value exceeds a certain standard (e.g., a threshold value). Re-learning may be performed by an operator or automatically by the system. The detection accuracy index value to be compared with the threshold value may be at least one of the false alarm rate, the false alarm rate, and the accuracy rate, and may be selected appropriately depending on the system specifications. The same applies to the other embodiments described later, in that any of the false alarm rate, the false alarm rate, and the accuracy rate may be used as the detection accuracy index value to be compared with the threshold value.

[0033] Next, Fig. 6 shows a flowchart illustrating the operation of the fault detection system 2 according to the second embodiment. As shown in Fig. 6, in the fault detection system 2 according to the second embodiment, steps S11 and S12 are added to the operation of the fault detection system 1 according to the first embodiment shown in Fig. 3. In step S11, it is determined whether the detection accuracy index value calculated by the detection accuracy calculation unit 13 in step S6 is equal to or greater than a preset threshold. If the detection accuracy index value is less than the threshold in step S11, parameters are re-learned using the learning device 24 (step S12), and the processes of steps S3 to S6 are performed again. The re-learning process of step S12 is performed until the detection accuracy index value in step S11 is equal to or greater than the threshold.

[0034] In the defect detection system 2 according to the second embodiment, the defect detection device 22 includes learned parameters that have been learned by machine learning so that a detection result image output when an evaluation image is input shows a defect detection area DE at the position of the defect existence area IR shown in the judgment reference image. When the learned parameters are updated, the detection accuracy calculation unit 13 preferably presents the change in the detection accuracy index value before and after the update to the operator.

[0035] As explained above, in the defect detection system 2 according to the second embodiment, it is possible to easily realize the defect detection device 22 to which the learned parameters whose detection accuracy index value satisfies a certain standard are applied.

[0036] Embodiment 3 In the third embodiment, another example of a process for improving the detection accuracy index value will be described. In the description of the third embodiment, the same components as those described in the first embodiment will be assigned the same reference numerals as those in the first embodiment, and the description thereof will be omitted.

[0037] FIG. 7 shows a schematic diagram of a defect detection system 3 according to a third embodiment. As shown in FIG. 7, the defect detection system 3 is obtained by adding a preprocessing unit 25 to the defect detection system 1. The preprocessing unit 25 is disposed before the defect detection device 22, and applies one of a plurality of preprocessing methods (for example, preprocessing methods [1] to [N], where N is an integer indicating the number of the processing method) to the evaluation image 20, and provides the preprocessed evaluation image 20 to the defect detection device 22. In the third embodiment, the detection accuracy calculation unit 13 presents the detection accuracy index value for each preprocessing method to the operator. This makes it easy for the operator to select the method with the highest detection accuracy index value as the preprocessing method to be applied by the preprocessing unit 25 when operating the defect detection system 3.

[0038] FIG. 8 is a flowchart illustrating the operation of the defect detection system 3 according to the third embodiment. As shown in FIG. 8, the operation of the defect detection system 3 according to the third embodiment is obtained by adding steps S21 to S25 to the operation of the defect detection system 1 shown in FIG. 3. Step S21 is a process performed after step S2, in which a variable n is initialized to 1. Step S22 is a process performed after step S21, in which a preprocessing method [n] is applied to a plurality of evaluation images 20 and the results are provided to the defect detection device 22. Then, steps S3 to S6 calculate a detection accuracy index value when the preprocessing method [n] is applied. Thereafter, in steps S23 and S24, the defect detection system 3 calculates a detection accuracy index value for each preprocessing method while incrementing the variable n by 1. Then, in step S25, the defect detection system 3 refers to the detection accuracy index value for each preprocessing method presented by the detection accuracy calculation unit 13, and determines the preprocessing method with the highest detection accuracy index value as the preprocessing method to be used when operating the system.

[0039] The preprocessing unit 25 may perform image processing such as adjusting image sharpness and contrast, and multiple preprocessing methods may be prepared by adjusting the degree of each image processing. Changing the preprocessing method changes the ease with which the defect detection device 22 can recognize the defect existence region IR. Therefore, by evaluating which of the multiple preprocessing methods has the highest detection accuracy index value and selecting the preprocessing method with the highest detection accuracy index value during system operation, it is possible to improve the defect detection accuracy of the defect detection system 3.

[0040] Embodiment 4 In the fourth embodiment, another example of a process for improving the detection accuracy index value will be described. In the description of the fourth embodiment, the same components as those described in the first embodiment will be assigned the same reference numerals as those in the first embodiment, and the description thereof will be omitted.

[0041] FIG. 9 shows a schematic diagram of a defect detection system 4 according to a fourth embodiment. As shown in FIG. 9, the defect detection system 4 is obtained by adding a post-processing unit 26 to the defect detection system 1. The post-processing unit 26 is disposed downstream of the defect detection device, and applies one of a plurality of post-processing methods (for example, pre-processing methods [1] to [N], where N is an integer indicating the number of the processing method) to the detection result image output by the defect detection device 22, and provides the post-processed evaluation image 20 to the detection performance evaluation device 10. In the fourth embodiment, the detection accuracy calculation unit 13 presents the detection accuracy index value for each post-processing method to the operator. This makes it easy for the operator to select the method with the highest detection accuracy index value as the post-processing method to be applied by the post-processing unit 26 when operating the defect detection system 4.

[0042] FIG. 10 shows a flowchart illustrating the operation of the defect detection system 4 according to the fourth embodiment. As shown in FIG. 10, the operation of the defect detection system 4 according to the fourth embodiment is the same as that of the defect detection system 1 shown in FIG. 3, except that steps S31 to S35 are added. Step S31 is a process performed after step S2, in which a variable n is initialized to 1. Step S32 is a process performed after step S31, in which a post-processing method [n] is applied to a plurality of evaluation images output by the defect detection device 22 and the results are provided to the detection performance evaluation device 10. Then, steps S3 to S6 calculate a detection accuracy index value when the post-processing method [n] is applied. Thereafter, in steps S33 and S34, the defect detection system 4 calculates a detection accuracy index value for each post-processing method while incrementing the variable n by 1. Then, in step S35, the defect detection system 4 refers to the detection accuracy index value for each post-processing method presented by the detection accuracy calculation unit 13, and determines the post-processing method with the highest detection accuracy index value as the pre-processing method for operating the system.

[0043] The post-processing unit 26 may perform image processing such as smoothing of the detection feature quantities generated by the defect detection device 22 or binarization, and multiple post-processing methods may be prepared with different levels of each process. Changing the post-processing method changes the range and size of the defect detection area DE. Therefore, it is possible to improve the defect detection accuracy of the defect detection system 4 by evaluating which of the multiple post-processing methods produces the highest detection accuracy index value and selecting the post-processing method with the highest detection accuracy index value during system operation.

[0044] Fifth embodiment In the fifth embodiment, another example of a process for improving the detection accuracy index value will be described. In the description of the fifth embodiment, the same components as those described in the first embodiment will be assigned the same reference numerals as those in the first embodiment, and the description thereof will be omitted.

[0045] FIG. 11 shows a schematic diagram of a defect detection system 5 according to a fifth embodiment. An evaluation image 20 is generated by photographing an inspection target product. In the fifth embodiment, an evaluation image 30 is prepared, which includes a plurality of evaluation image sets (e.g., evaluation image sets OBJ_img_set[1] to [N]) obtained by changing photographing conditions such as the photographing angle of the inspection target product and the angle and intensity of illumination. In addition, in the fifth embodiment, a determination reference image 31 is generated, which includes a plurality of determination reference image sets (e.g., determination reference image sets ma_img_set[1] to [N]) corresponding to the plurality of evaluation image sets, and a detection result image 32 is generated, which includes a plurality of detection result image sets (e.g., detection result image sets a_img_set[1] to [N]) corresponding to the plurality of evaluation image sets. Then, the defect detection system 5 according to the fifth embodiment calculates a detection accuracy index value for each photographing condition and determines the photographing condition that results in the highest detection accuracy index value.

[0046] 12 shows a flowchart illustrating the operation of the defect detection system 5 according to the fifth embodiment. As shown in FIG. 12, in the defect detection system 5 according to the fifth embodiment, first, a variable n is initialized to 1 (step S40), and then, each of a plurality of inspection target products is photographed under a photographing condition [n] to obtain an evaluation image set [n]. Next, an operator uses the evaluation image set [n] to generate a judgment reference image set [n] (step S42). In addition, the evaluation images included in the evaluation image set [n] are input one by one to the defect detection device 22 to obtain an output image set [n] corresponding to the evaluation image set [n].

[0047] 3 is then performed (step S44). Specifically, in step S44, the reference image set [n] and the detection result image set [n] are compared for each corresponding image, and one of a positive alarm label TP, a false alarm label FP, and a false alarm label FN is assigned to the defect detection area DE and the defect existence area IR. Then, the defect detection system 5 calculates a detection accuracy index value corresponding to the imaging condition [n] based on the labels assigned in step S44 (step S6). Thereafter, the processes of steps S41 to S46 are repeated until detection accuracy index values ​​for all of the preset imaging conditions (imaging conditions [1] to [N]) are calculated. Thereafter, the imaging condition [n] with the highest detection accuracy index value among the detection accuracy index values ​​corresponding to the imaging conditions [1] to [N] is determined as the imaging method to be used during system operation (step S47).

[0048] As explained above, the defect detection system 5 according to the fifth embodiment facilitates system operation under the imaging conditions that result in the highest detection accuracy index value.

[0049] The present invention is not limited to the above-described embodiment, and can be modified as appropriate within the scope of the invention. [Explanation of symbols]

[0050] 1~5 Defect detection system 10. Detection performance evaluation device 11 Comparison processing section 12 Labeling Processing Unit 13. Detection accuracy calculation unit 20, 30 Evaluation images 21, 31 Criterion images 22 Defect detection device 23, 32 Detection result images 24 Learning Device 25 Pretreatment section 26 Post-processing section OBJ Inspection item IR defect area DE defect detection area

Claims

1. A detection performance evaluation program executed by a calculation unit included in a defect detection system including the defect detection device, which evaluates the performance of a defect detection device that receives an evaluation image obtained by photographing an inspection target product as an input and outputs a detection result image including a defect detection area that is different from a good product image obtained by photographing the inspection target product that is judged to be a good product, and which comprises: a comparison process in which an operator performs an annotation process on the evaluation image to clearly indicate defect-existing areas in the inspection target product, and the detection result image are acquired, and the determination reference image and the detection result image are superimposed and compared; a labeling process of labeling the defect detection region and the defect existence region with one of a true alarm label, a false alarm label, and a false alarm label based on the result of the comparison process and a predetermined judgment condition; a detection accuracy calculation process for calculating a detection accuracy index value, which is an index value of the detection performance of the defect detection device, based on the number of correct report labels, the number of false report labels, and the number of missing report labels; The determination condition is: If the defect detection area overlaps the defect existence area by even one pixel, a positive report label is attached to the defect existence area and the defect detection area; If the defect detection region does not include even one pixel of the defect existence region, a false alarm label is attached to the defect existence region; a condition that a failure label is attached to the defect detection region when the defect existence region does not include even one pixel of the defect detection region; The detection accuracy index value is a false alarm rate obtained by dividing the number of false alarm labels by the total number of true alarm labels and the number of false alarm labels; a failure rate obtained by dividing the number of the failure labels by the total number of the correct labels and the failure labels; a correct report rate obtained by dividing the number of correct reports by the total number of correct reports, false reports, and missing reports; A detection performance evaluation program including at least one of the above.

2. the defect detection device includes learned parameters that have been learned by machine learning so that the detection result image that is output when the evaluation image is input shows the defect detection area at a position on the evaluation image that corresponds to the position of the defect existence area shown in the judgment reference image; The detection performance evaluation program according to claim 1 , wherein when the learned parameters are updated, the detection performance evaluation program presents to an operator a change in the detection accuracy index value before and after the update.

3. The fault detection system comprises: a pre-processing unit that is arranged before the defect detection device and applies any one of a plurality of pre-processing methods to the evaluation image and provides the pre-processed evaluation image to the defect detection device; a post-processing unit that is arranged downstream of the defect detection device and applies one of a plurality of post-processing methods to the detection result image output by the defect detection device, and provides the post-processed evaluation image to the detection performance evaluation program; and The detection performance evaluation program according to claim 1 , wherein the detection performance evaluation program presents to an operator at least one of the detection accuracy index value for each of the pre-processing methods and the detection accuracy index value for each of the post-processing methods.

4. The evaluation images include a plurality of evaluation image sets obtained by changing the photographing conditions of the same inspection object, The detection performance evaluation program according to claim 1 , wherein the detection performance evaluation program presents the detection accuracy index value for each of the evaluation image sets to an operator.

5. A method for evaluating detection performance in a defect detection system having a defect detection device that receives an evaluation image obtained by photographing an inspection target product as an input and outputs a detection result image including a defect detection area that is different from a good-product image obtained by photographing the inspection target product that is determined to be a good product, a comparison process in which an operator performs an annotation process on the evaluation image to clearly indicate defect-existing areas in the inspection target product, and the detection result image are acquired, and the determination reference image and the detection result image are superimposed and compared; a labeling process of labeling the defect detection region and the defect existence region with one of a true alarm label, a false alarm label, and a false alarm label based on the result of the comparison process and a predetermined judgment condition; a detection accuracy calculation process for calculating a detection accuracy index value, which is an index value of the detection performance of the defect detection device, based on the number of correct report labels, the number of false report labels, and the number of unreported labels, by automatic processing using a computing device; The determination condition is: If the defect detection area overlaps the defect existence area by even one pixel, a positive report label is attached to the defect existence area and the defect detection area; If the defect detection region does not include even one pixel of the defect existence region, a false alarm label is attached to the defect existence region; a condition that a failure label is attached to the defect detection region when the defect existence region does not include even one pixel of the defect detection region; The detection accuracy index value is a false alarm rate obtained by dividing the number of false alarm labels by the total number of true alarm labels and the number of false alarm labels; a failure rate obtained by dividing the number of the failure labels by the total number of the correct labels and the failure labels; a correct report rate obtained by dividing the number of correct reports by the total number of correct reports, false reports, and missing reports; A detection performance evaluation method including at least one of the above.

6. A detection performance evaluation device for evaluating the performance of a defect detection device that receives an evaluation image obtained by photographing an inspection target product as an input and outputs a detection result image including a defect detection area that is different from a good product image obtained by photographing the inspection target product that is judged to be a good product, a comparison processing unit that acquires a determination reference image obtained by an operator annotating the evaluation image so as to clearly indicate defect-existing areas that should be determined to be defective in the inspection target product, and the detection result image, and superimposes and compares the determination reference image and the detection result image; a labeling processing unit that assigns one of a correct report label, a false report label, and a failed report label to the defect detection region and the defect existence region based on a result of the comparison processing unit and a predetermined judgment condition; a detection accuracy calculation processing unit that calculates a detection accuracy index value, which is an index value of the detection performance of the defect detection device, based on the number of the correct report labels, the false report labels, and the missing report labels; The determination condition is: If the defect detection area overlaps the defect existence area by even one pixel, a positive report label is attached to the defect existence area and the defect detection area; If the defect detection region does not include even one pixel of the defect existence region, a false alarm label is attached to the defect existence region; a condition that a failure label is attached to the defect detection region when the defect existence region does not include even one pixel of the defect detection region; The detection accuracy index value is a false alarm rate obtained by dividing the number of false alarm labels by the total number of true alarm labels and the number of false alarm labels; a failure rate obtained by dividing the number of the failure labels by the total number of the correct labels and the failure labels; a correct report rate obtained by dividing the number of correct reports by the total number of correct reports, false reports, and missing reports; A detection performance evaluation device including at least one of the above.

Images