Trained model selection apparatus

Abstract

To propose a method to appropriately select an optimal trained model from a plurality of trained models.SOLUTION: A trained model selection apparatus comprises: an internal storage area for calculation that stores a plurality of types of evaluation indexes for a plurality of trained models; and a calculation processing unit that executes a performance score calculation step for calculating a performance score, which is the sum of products of the plurality of types of evaluation indexes and weights for the plurality of trained models, for a plurality of types of weight setting patterns having a common total value of the weights, a candidate model identification step for identifying, as a candidate model, the trained model having the highest performance score calculated in the performance score calculation step for each of the plurality of types of weight setting patterns, and an optimal model selection step for selecting the trained model that is most frequently identified as the candidate model in the candidate model identification step as an optimal trained model.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present disclosure relates to a learned model selection device that selects an optimal learned model from a plurality of learned models.

Background Art

[0002] Patent Document 1 discloses that, for a recognition algorithm that performs image recognition on an object to be recognized, evaluation scores of a plurality of types of evaluation items are calculated, and the recognition algorithm with the highest comprehensive evaluation score based on these evaluation scores is selected as the optimal algorithm. It is also disclosed that weights are assigned to each evaluation item when calculating the comprehensive evaluation score.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in Patent Document 1, the selection of the algorithm is based only on the comprehensive evaluation score calculated with one pattern of weighting of the evaluation items, so there are cases where the algorithm cannot be appropriately selected.

[0005] Such a problem also occurs similarly when the selection method of Patent Document 1 is adopted for the selection of the learned model.

[0006] The present disclosure has been made in view of such a point, and an object thereof is to propose a method for appropriately selecting an optimal learned model from a plurality of learned models.

Means for Solving the Problems

[0007] To achieve the above object, the present disclosure provides a learned model selection device that selects an optimal learned model from a plurality of learned models. The device includes an evaluation index storage unit that stores, for the plurality of learned models, a plurality of types of evaluation indexes calculated based on output data when predetermined image data is input and annotation data of the predetermined image data; a performance score calculation step that calculates, for a plurality of types of weight setting patterns with a common total value of weights, a performance score that is the sum of products of the plurality of types of evaluation indexes and weights for the plurality of learned models; a candidate model identification step that identifies, for each of the plurality of types of weight setting patterns, a learned model with the highest performance score calculated in the performance score calculation step as a candidate model; and a processing unit that executes an optimal model selection step of selecting, as an optimal learned model, a learned model that is identified as a candidate model most frequently in the candidate model identification step.

[0008] Accordingly, an optimal learned model can be selected from a plurality of learned models based on performance scores calculated using a plurality of types of weight setting patterns.

Advantages of the Invention

[0009] According to the present disclosure, an optimal learned model can be selected from a plurality of learned models based on performance scores calculated using a plurality of types of weight setting patterns.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Modes for Carrying Out the Invention

[0011] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. The following description of the preferred embodiments is merely illustrative in nature and is in no way intended to limit the present invention, its applications, or its uses.

[0012] FIG. 1 shows a learned model selection device 1 according to an embodiment of the present disclosure. This learned model selection device 1 includes an image processing unit 10, a calculation processing unit 20, a calculation internal memory area 30 as an evaluation index storage unit, a data storage 40, an input / output internal memory area 50, an input device 60, and an output device 70. The input / output internal memory area 50 has first to sixth areas 51 to 56. The input device 60 is composed of, for example, a keyboard 61 and a mouse 62. The output device 70 is composed of, for example, a liquid crystal display.

[0013] The image processing unit 10 includes a data acquisition unit 11, a data preparation unit 12, a learning unit 13, a data inspection unit 14, and an image processing internal memory area 15. The data acquisition unit 11, the data preparation unit 12, the learning unit 13, and the data inspection unit 14 are composed of a GPU (Graphics Processing Unit).

[0014] The data acquisition unit 11 acquires acquired image data of a 3D image including around a welding location. The welding location is a location where a welding mark is formed, for example, a welding bead of laser welding or an indentation by a welding gun of resistance welding. The data acquisition unit 11 receives the acquired image data from an imaging device (not shown), processes received data from an imaging device (not shown) to acquire it, or creates it according to a user input. The data acquisition unit 11 acquires acquired image data for learning and acquired image data for evaluation. The data acquisition unit 11 stores the acquired image data in the data storage 40 as an image data file 42.

[0015] The data preparation unit 12 prepares learning image data and annotation data for the learning image data based on the acquired learning image data acquired by the data acquisition unit 11. Further, the data preparation unit 12 prepares annotation data for the acquired evaluation image data based on the acquired evaluation image data acquired by the data acquisition unit 11. These annotation data specify the presence or absence of welding defects as detection targets in the image data, the rectangular region (bounding box) as the target region including the welding defects in the image based on the image data, and the type of welding defects within the rectangular region (for example, holes, pits, spatter, undercut, protrusions, etc.). The annotation data is specified by the user's input to the input device 60 with the image based on the acquired learning image data output to the output device 70. Specifically, the image is output to the output device 70, and the user surrounds and specifies the rectangular region including the welding defects in the image, and assigns the type name of the welding defects to the rectangular region.

[0016] The data preparation unit 12 may use the acquired learning image data as the learning image data as it is, or may perform data augmentation processing based on the acquired learning image data to generate more learning image data. The data preparation unit 12 stores the prepared learning image data in the data storage 40 as an image data file 42. Further, the data preparation unit 12 stores the prepared annotation data in the data storage 40 as an annotation data file 43.

[0017] The learning unit 13 reads the learning image data prepared by the data preparation unit 12 and their annotation data from the data storage 40. Then, based on the read learning image data and annotation data, the learning unit 13 generates a plurality of learned models. The learning unit 13 is implemented with a known inference engine, for example, YOLO which is an object detection algorithm. For example, the learning unit 13 generates a learned model (weight file) every time the number of iterations of repeating learning reaches a predetermined number. Then, the learning unit 13 stores the generated learned model in the data storage 40 as a learned model file 44. The input data of the learned model is image data. The output data of the learned model identifies the presence or absence of welding defects in the image data, the rectangular region (bounding box) as the target region including the welding defects as the detection target in the image based on the image data, and the type of welding defects (for example, holes, pits, spatter, undercut, protrusions, etc.) within the rectangular region.

[0018] The data inspection unit 14 reads the acquired image data for evaluation acquired by the data acquisition unit 11 and the plurality of learned models generated by the learning unit 13 from the data storage 40. Then, the data inspection unit 14 inputs the acquired image data for evaluation into the plurality of learned models to obtain the output data of the plurality of learned models, and stores it in the data storage 40 as a defect detection result file 45. The data inspection unit 14 sets multiple types of yolo thresholds and acquires the output data of the plurality of learned models for each yolo threshold. Specifically, the data inspection unit 14 sets the yolo threshold to 0.01 and 0.25 and acquires the output data of the plurality of learned models.

[0019] The calculation processing unit 20 is composed of, for example, a CPU (Central Processing Unit). The calculation processing unit 20 has a detection result evaluation unit 21 and an AI model selection unit 22. Hereinafter, the operation of the calculation processing unit 20 will be described with reference to the flowchart of FIG. 2. The operation shown in the flowchart of FIG. 2 is executed in a state where the output data of a plurality of learned models are stored in the data storage 40 as a defect detection result file 45.

[0020] First, in S101, the detection result evaluation unit 21 reads the annotation data file 43 and the defect detection result file 45, and stores the output data when the acquisition image data for evaluation is input to a plurality of learned models and the annotation data of the acquisition image data for evaluation in the third and fifth regions 53 and 55 of the input / output internal storage area 50. Then, based on the output data when the acquisition image data for evaluation is input to these plurality of learned models and the annotation data of the acquisition image data for evaluation, the detection result evaluation unit 21 calculates the total number of correct detections, the total number of false detections, the total number of undetected detections, and the accuracy of the detection position (Intersection over Union). Here, the total number of correct detections is the number of welding defects included in the welding defects specified by the annotation data among the welding defects specified by the output data of the learned model. Also, the total number of false detections is the number of welding defects not included in the welding defects specified by the annotation data among the welding defects specified by the output data of the learned model. Also, the total number of undetected detections is the number of welding defects not included in the welding defects specified by the output data of the learned model among the welding defects specified by the annotation data. Also, the accuracy of the detection position is the area of the region included in at least one of the rectangular regions specified by the annotation data and the rectangular region specified by the output data of the learned model, divided by the area of the region that is the rectangular region specified by the annotation data and is also the rectangular region specified by the output data of the learned model.

[0021] Next, in S102, the detection result evaluation unit 21 stores the total number of positive detections, the total number of false detections, the total number of undetected detections, and the accuracy of the detection positions calculated in S101 in the internal storage area 30 for calculation. Next, the detection result evaluation unit 21 calculates the precision rate (Precision) and the recall rate (Recall) based on the stored total number of positive detections, the total number of false detections, and the total number of undetected detections. Then, the detection result evaluation unit 21 stores three types of evaluation indicators, namely the precision rate, the recall rate, and the accuracy of the detection positions, in the internal storage area 30 for calculation for each learned model.

[0022] These precision rate, recall rate, and accuracy of the detection positions are represented by the following formulas (1) to (3).

[0023] Here, let the precision rate be Pr, the recall rate be Re, the accuracy of the detection position be IoU, the total number of the above positive detections be TP, the total number of the above false detections be FP, the total number of the above undetected detections be FN, the area of the region included in at least one of the rectangular regions specified by the annotation data and the rectangular region specified by the output data of the learned model be S1, the rectangular region specified by the annotation data, and the area of the region that is the rectangular region specified by the output data of the learned model be S2.

[0024] Pr = TP / (TP + FP) ···(1) Re = TP / (TP + FN) ···(2) IoU = S2 / S1 ···(3) Next, in S201, the detection result evaluation unit 21 reads a plurality of types of weight setting patterns stored as the operation setting file 41 in the data storage 40. The total value of the weights of these plurality of types of weight setting patterns is common. The detection result evaluation unit 21 stores the read plurality of types of weight setting patterns in the first area 51 of the internal storage area 50 for input / output. Then, the AI model selection unit 22 reads three types of evaluation indicators for the plurality of learned models calculated by the detection result evaluation unit 21 from the internal storage area 30 for calculation. Then, the AI model selection unit 22 calculates a performance score, which is the sum of the products of the three types of evaluation indicators and the weights for the plurality of learned models, for the plurality of types of weight setting patterns stored in the first area 51. The performance score for each learned model can be calculated by the following formulas (4) to (6). Here, the performance score for each learned model is Score, the yolo threshold is, and the performance score for each learned model when the yolo threshold is 0.01 is Score 0.01 , and the performance score for each learned model when the yolo threshold is 0.25 is Score 0.25 . Let the weight of the positive detection rate be w_p, the weight of the detection rate be w_r, and the weight of the accuracy of the detection position be w_iou. Let the positive detection rate Pr when the yolo threshold is 0.01 be the positive detection rate Pr 0.01 , the detection rate Re be Re 0.01 , and the accuracy of the detection position IoU be IoU 0.01 . Let the positive detection rate Pr when the yolo threshold is 0.25 be the positive detection rate Pr 0.25 , the detection rate Re be Re 0.25 , and the accuracy of the detection position IoU be IoU 0.25 . Let the weight of the performance score when the yolo threshold is 0.01 be w 0.01 , and the weight of the performance score when the yolo threshold is 0.25 be w 0.25 .

[0025] Score 0.01 =Pr 0.01 *w_p+Re 0.01 *w_r+IoU 0.01*w_iou ···(4) Score 0.25 =Pr 0.25 *w_p+Re 0.25 *w_r+IoU 0.25 *w_iou ···(5) Score=Score 0.01 *w 0.01 +Score 0.25 *w 0.25 ···(6) Then, the AI model selection unit 22 stores a table showing the weight setting pattern, the file name of the learned model, and the performance score in each row in the internal storage area 30 for calculation as a performance score table. FIG. 3 illustrates the performance score table. In this FIG. 3, precision_priority indicates the weight of the positive detection rate (w_p), recall_priority indicates the weight of the detection rate (w_r), and iou_priority indicates the weight of the accuracy of the detection position (w_iou). The first row of FIG. 3 shows that when the weight of the positive detection rate is 20%, the weight of the detection rate is 70%, the weight of the accuracy of the detection position is 10%, and the yolo threshold is 0.01, the performance score (Score 0.01 ) with weight (w 0.01 ) being 0.5 and yolo threshold being 0.25, the performance score (Score 0.25 ) with weight (w 0.25 ) being 0.5, for the learned model with file name ai_multi_gpu_11304, the performance score (Score) is 73.20135. The second row of FIG. 3 shows that when the weight of the positive detection rate is 20%, the weight of the detection rate is 65%, the weight of the accuracy of the detection position is 15%, and the yolo threshold is 0.01, for each learned model, the weight of the performance score is 0.5, and when the yolo threshold is 0.25, for each learned model, the weight of the performance score is 0.5, the performance score of the learned model with file name ai_multi_gpu_11304 is 71.41648. The same applies to the third row and subsequent rows of FIG. 3.

[0026] Next, in S202, the AI model selection unit 22 groups the rows of the performance score table so that the data of the common weight setting patterns form common groups. Then, the AI model selection unit 22 extracts the row with the maximum performance score for each group, that is, for each weight setting pattern, and stores it in the calculation internal storage area 30 as the score table of the candidate models. That is, the AI model selection unit 22 identifies, for each of the plurality of types of weight setting patterns, the learned model with the highest performance score calculated in S201 as the candidate model.

[0027] Next, in S203, the AI model selection unit 22 groups the rows of the score table of the candidate models so that the data of the common learned models form common groups. Then, the AI model selection unit 22 identifies the number of rows in each group, that is, the number of weight setting patterns for which each candidate model was selected. The AI model selection unit 22 stores, in the calculation internal storage area 30 as the candidate model selection count table, a table indicating, for each row, the file name of the learned model included in the score table of the candidate models and the number of rows (the number of times selected) of the group to which the learned model belongs.

[0028] Next, in S204, the AI model selection unit 22 sorts the rows of the candidate model selection count table in descending order of the number of rows (the number of times selected). Then, the AI model selection unit 22 stores the file name of the learned model in the first row of the sorted table in the calculation internal storage area 30 as the optimal model file. That is, the AI model selection unit 22 selects the learned model with the most occurrences as the candidate model in S202 as the optimal learned model. Then, the total number of positive detections, the total number of false detections, the total number of non-detections, and the positive detection rate Pr, detection rate Re, and detection position accuracy IoU for each type of welding defect corresponding to the optimal model file are read from the calculation internal storage area 30 and stored in the data storage 40 as the evaluation result file 46 in table form.

[0029] For example, in S202, assume that 13 learned models with file names ai_multi_gpu_11304, ai_multi_gpu_8504, ai_multi_gpu_11192, ai_multi_gpu_10184, ai_multi_gpu_11864, ai_multi_gpu_10744, ai_multi_gpu_10856, ai_single _gpu_300, ai_multi_gpu_11976, ai_multi_gpu_6712, ai_multi_gpu_10072, ai_multi_gpu_11752, and ai_single_gpu_1000 are identified. And assume that the number of times the learned models of ai_multi_gpu_11304, ai_multi_gpu_8504, ai_multi_gpu_11192, ai_multi_gpu_10184, ai_multi_gpu_11864, ai_multi_gpu_10744, ai_multi_gpu_10856, ai_single _gpu_300, ai_multi_gpu_11976, ai_multi_gpu_6712, ai_multi_gpu_10072, ai_multi_gpu_11752, and ai_single_gpu_1000 are identified as candidate models in S202 are, in order, 255 times, 116 times, 72 times, 68 times, 49 times, 39 times, 37 times, 25 times, 20 times, 9 times, 1 time, 1 time, and 1 time. In this case, in S204, the AI model selection unit 22 selects the learned model of ai_multi_gpu_11304, which has the largest number of times of being identified as a candidate model in S202, as the optimal learned model.

[0030] Thus, according to this embodiment, an optimal learned model can be selected from a plurality of learned models based on the performance scores calculated using a plurality of types of weight setting patterns.

[0031] In addition, in S201, the detection result evaluation unit 21 may output a plurality of types of weight setting patterns stored as an operation setting file 41 in the data storage 40 to the output device 70. Then, the detection result evaluation unit 21 may receive an input from the user input device 60 for selecting a part of the plurality of types of weight setting patterns from these plurality of types of weight setting patterns. Further, the detection result evaluation unit 21 determines a part of the plurality of types of weight setting patterns among the plurality of types of weight setting patterns stored as the operation setting file 41 in the data storage 40 based on the above input of the user, and may save only the part of the plurality of types of weight setting patterns in the first region 51. Thereby, the performance scores for the plurality of learned models can be calculated by the AI model selection unit 22 only for the plurality of types of weight setting patterns determined based on the above input of the user. Therefore, by adopting only the weight setting patterns with a large weight for the evaluation index emphasized by the user, the user's selection criteria can be reflected in the selection result of the optimal learned model.

[0032] In addition, in the present embodiment, the detection target is a welding defect, but the present invention can also be applied when the detection target is other than a welding defect, such as a person.

[0033] In addition, in the present embodiment, the target area including the welding defect is a rectangular area, but it may be an area with a shape other than a rectangle.

[0034] In addition, in the present embodiment, the plurality of types of evaluation indexes stored in the internal storage area 30 for calculation and used for calculating the performance score are three types of evaluation indexes, namely, the true positive rate, the detection rate, and the accuracy of the detection position. However, as long as they are a plurality of types of evaluation indexes, they may be two types or four or more types of evaluation indexes. Further, the plurality of types of evaluation indexes stored in the internal storage area 30 for calculation and used for calculating the performance score may include only one or two of the true positive rate, the detection rate, and the accuracy of the detection position.

Industrial Applicability

[0035] The learned model selection device of the present disclosure is useful because it can appropriately select an optimal learned model from a plurality of learned models.

Explanation of Signs

[0036] 1 Learned model selection device 20 Calculation processing unit 30 Internal storage area for calculation (evaluation index storage unit)

Claims

1. A learned model selection device that selects an optimal learned model from a plurality of learned models, comprising: an evaluation index storage unit that stores, for the plurality of learned models, a plurality of types of evaluation indices calculated based on output data when predetermined image data is input and annotation data of the predetermined image data; a performance score calculation step of calculating, for a plurality of types of weight setting patterns with a common total value of weights, a performance score that is the sum of the products of the plurality of types of evaluation indices and weights for the plurality of learned models; a candidate model identification step of identifying, for each of the plurality of types of weight setting patterns, the learned model with the highest performance score calculated in the performance score calculation step as a candidate model; and a processing unit that executes an optimal model selection step of selecting, as an optimal learned model, the learned model that is identified as a candidate model most frequently in the candidate model identification step.

2. The learned model selection device according to Claim 1, wherein the output data of the learned model identifies a target region including a detection target in an image based on the image data.

3. The learned model selection device according to Claim 2, wherein the number of detection targets included in the detection targets identified by the annotation data among the detection targets identified by the output data of the learned model is TP; the number of detection targets not included in the detection targets identified by the annotation data among the detection targets identified by the output data of the learned model is FP; the number of detection targets not included in the detection targets identified by the output data of the learned model among the detection targets identified by the annotation data is FN; the area of a region included in at least one of the target region identified by the annotation data and the target region identified by the output data is S1; when the area of the region that is the target region identified by the annotation data and is also the target region identified by the output data is S2 The learning model selection device is characterized in that the plurality of types of evaluation indicators include at least one of the positive detection rate Pr represented by the following formula (1), the detection rate Re represented by the following formula (2), and the accuracy IoU of the detection position represented by the following formula (3). Pr = TP / (TP + FP) ・・・(1) Re = TP / (TP + FN) ・・・(2) IoU = S2 / S1 ・・・(3)

4. In the learned model selection device according to claim 1, the processing unit is characterized in that it determines the plurality of types of weight setting patterns based on user input.

Images