Image recognizing model evaluating device, image recognizing model evaluating method and program

Abstract

To provide an image recognizing model evaluating device, an image recognizing model evaluating method and a program capable of appropriately evaluating an image recognizing model.SOLUTION: An image recognizing model evaluating device according to the present disclosure includes: an obtaining unit that obtains evaluation image data applied for evaluating an image recognizing model; an inference processing unit that executes processes of detecting, based on the evaluation image data, an object present in the evaluation image data, and of associating the detected object with a label which represents the kind of the object; an evaluation value calculating unit that calculates, for each detected object, the evaluation value of an image recognition process; and a display control unit that causes information including the evaluation value of the image recognition process for each object to be displayed.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present disclosure relates to an image recognition model evaluation device, an image recognition model evaluation method, and a program.

Background Art

[0002] Image recognition processing technology for recognizing an object shown in an image using machine learning technology and performing processing for classifying by type has been advancing. When applying image recognition processing technology using machine learning technology to a product, it has been necessary to evaluate a learned model and examine the direction of further learning and the validity of the model to be used.

[0003]

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, since the model evaluation method described in Patent Document 1 above represents the performance of the target model by numerical data and evaluates it using a graph or the like, the result of the inference made by the model with respect to the image cannot be evaluated on the image data.

[0006] ​In view of the above problems, an object of the present disclosure is to provide an image recognition model evaluation apparatus, an image recognition model evaluation method, and a program that can appropriately evaluate an image recognition model.

Means for Solving the Problems

[0007] In order to solve the above-described problems and achieve the object, an image recognition model evaluation apparatus according to the present disclosure includes an acquisition unit that acquires evaluation image data used for evaluating an image recognition model, and based on the evaluation image data, detects an object depicted in the evaluation image data, and an inference processing unit that executes a process of associating a label indicating the type of the detected object with the detected object, an evaluation value calculation unit that calculates an evaluation value of the image recognition process for each object detected by the inference processing unit, and a display control unit that displays information including the evaluation value of the image recognition process for each object.

[0008] In order to solve the above-described problems and achieve the object, an image recognition model evaluation method according to the present disclosure includes a step of acquiring evaluation image data used for evaluating an image recognition model, a step of detecting an object depicted in the evaluation image data based on the evaluation image data and executing a process of associating a label indicating the type of the detected object with the detected object, a step of calculating an evaluation value of the image recognition process for each detected object, and a step of displaying information including the evaluation value of the image recognition process for each object.

[0009] In order to solve the above-described problems and achieve the object, a program according to the present disclosure causes a computer to execute a step of acquiring evaluation image data used for evaluating an image recognition model, a step of detecting an object depicted in the evaluation image data based on the evaluation image data and executing a process of associating a label indicating the type of the detected object with the detected object, a step of calculating an evaluation value of the image recognition process for each detected object, and a step of displaying information including the evaluation value of the image recognition process for each object.

Advantages of the Invention

[0010] According to the present disclosure, an image recognition model evaluation apparatus, an image recognition model evaluation method, and a program capable of appropriately evaluating an image recognition model can be provided.

Brief Description of Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Mode for Carrying Out the Invention

[0012] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the present disclosure is not limited by the embodiments described below.

[0013] (Outline of Image Recognition Model Evaluation Apparatus) First, with reference to FIG. 1, the outline of the image recognition model evaluation apparatus according to the present disclosure will be described. FIG. 1 is a diagram for explaining the outline of image recognition model evaluation according to the present disclosure. As shown in FIG. 1, in the image recognition model evaluation apparatus 100 according to the present disclosure, a target model, which is an image recognition model to be evaluated, is used as inference processing software, and an evaluation image data set is input to the target model to output an inference result label. Then, the image recognition model evaluation apparatus 100 inputs the inference result label and the correct label to the evaluation value calculation software to calculate a plurality of evaluation values (evaluation value (1), evaluation value (2), evaluation value (3), etc.). The image recognition model evaluation apparatus 100 displays the evaluation values calculated by the evaluation value calculation software using a GUI (Graphical User Interface).

[0014] Based on the evaluation results of the image recognition model displayed by the GUI, the user performs the evaluation work of the image recognition model. At this time, for the evaluation image data by the GUI, the position coordinates of the recognized object, the size of the object, the type (class) of the object, and the confidence score are superimposed and displayed.

[0015] In the above-described evaluation operation, each time the user sets the thresholds for the IoU (Intersection Over Union) and the confidence score, which will be described later, the image recognition model evaluation apparatus 100 performs an inference process on the evaluation dataset using the inference processing software, outputs the true positive rate and the false positive rate according to the inference processing result, and the user evaluates the validity of the image recognition model based on this.

[0016] Note that the evaluation by the user is not necessarily limited to performing a uniform evaluation based on a unique numerical value. The true positive rate and the false positive rate in a specific situation may be emphasized and used for tuning the image recognition model, etc. For example, in a situation with a high risk of an accident and a situation where false detection is not a problem, the evaluation may be performed according to an evaluation criterion assuming that the importance of the true positive rate and the false positive rate is different.

[0017] As described above, the image recognition model evaluation apparatus 100 according to the present disclosure is an apparatus for evaluating an image recognition model by inputting an evaluation image dataset into the image recognition model and performing an evaluation of the image recognition model based on the inference result of the image recognition model.

[0018] (Configuration of the Image Recognition Model Evaluation Apparatus) Next, the configuration of the image recognition model evaluation apparatus according to the present disclosure will be described with reference to FIG. 2. FIG. 2 is a diagram showing a configuration example of the image recognition model evaluation apparatus according to the present disclosure. As shown in FIG. 2, the image recognition model evaluation apparatus 100 according to the present disclosure includes a communication unit 110, a storage unit 120, a control unit 130, an input unit 140, and a display unit 150. These configurations will be described in order below.

[0019] The communication unit 110 is responsible for transmitting and receiving various information, etc., to and from an external device by wire or wirelessly. In the case of wire, for example, it may be provided with interfaces such as a USB terminal and a wired LAN terminal. In the case of wireless, it may be realized by a wireless LAN defined in IEEE802.11, Bluetooth (registered trademark), Wi-Fi (registered trademark), etc.

[0020] Note that the image recognition model evaluation device 100 according to the present disclosure includes a communication unit 110, but it is preferably used as a stand-alone device that executes processing independently without being connected to other devices. Thereby, the outflow of information to external devices can be suppressed. Of course, it may be connected to other devices via the communication unit 110 to execute processing.

[0021] The storage unit 120 is a storage device that stores various types of information. The storage unit 120 includes a main storage device and an auxiliary storage device. The main storage device may be realized by a semiconductor memory element such as a RAM (Random Access Memory), a ROM (Read Only Memory), or a flash memory. The auxiliary storage device may be realized by, for example, a hard disk, an SSD (Solid State Drive), an optical disk, or the like.

[0022] As shown in FIG. 3, the storage unit 120 includes a learning image data storage unit 121, an evaluation image data storage unit 122, an inference processing result storage unit 123, and a learned model storage unit 124. Hereinafter, the information stored in these configurations will be described in order.

[0023] The learning image data storage unit 121 stores information related to the image data for learning the image recognition model. Here, an example of the information stored in the learning image data storage unit 121 will be described with reference to FIG. 3. FIG. 3 is a diagram showing an example of the information stored in the learning image data storage unit of the image recognition model evaluation device according to the present disclosure.

[0024] As shown in FIG. 3, the learning image data storage unit 121 stores information related to the items of "learning image data ID" and "learning image data".

[0025] The "learning image data ID" is an identifier that identifies the image data for training the image recognition model, and is represented by a character string, number, etc. The "learning image data" is the learning image data identified by the "learning image data ID", and may be, for example, image data in a file format such as JPEG (Joint Photographic Experts Group) or TIFF (Tag Image File Format).

[0026] That is, in FIG. 3, an example is shown in which the learning image data "IMGDT#1" identified by the learning image data ID "IMGID#1" is stored.

[0027] Note that the information stored in the learning image data storage unit 121 is not limited to the information related to the items of "learning image data ID" and "learning image data", and information related to any other learning image data may be stored.

[0028] The evaluation image data storage unit 122 stores information related to the image data for evaluating the image recognition model. Here, an example of the information stored in the evaluation image data storage unit 122 will be described with reference to FIG. 4. FIG. 4 is a diagram showing an example of the information stored in the evaluation image data storage unit of the image recognition model evaluation apparatus according to the present disclosure.

[0029] As shown in FIG. 4, the evaluation image data storage unit 122 stores information related to the items of "evaluation image data ID" and "evaluation image data".

[0030] The "evaluation image data ID" is an identifier that identifies the image data for evaluating the image recognition model, and is represented by a character string, number, etc. The "evaluation image data" is the evaluation image data identified by the "evaluation image data ID", and may be, for example, image data in a file format such as JPEG or TIFF.

[0031] That is, in FIG. 4, an example is shown in which evaluation image data "EVIMGDT#1" identified by the evaluation image data ID "EVIMGID#1" is stored.

[0032] Note that the information stored in the evaluation image data storage unit 122 is not limited to the information related to the items of "evaluation image data ID" and "evaluation image data", and information related to any other evaluation image data may be stored.

[0033] The inference processing result storage unit 123 stores information related to the result of the inference processing by the image recognition model. Here, an example of the information stored in the inference processing result storage unit 123 will be described with reference to FIG. 5. FIG. 5 is a diagram showing an example of the information stored in the inference processing result storage unit of the image recognition model evaluation apparatus according to the present disclosure.

[0034] As shown in FIG. 5, the inference processing result storage unit 123 stores information related to the items of "evaluation image ID", "object ID", "class", "position coordinates", "size", and "confidence score".

[0035] The "evaluation image ID" is an identifier for identifying the evaluation image, and is represented by a character string, a number, or the like. The "object ID" is an identifier for identifying an object detected and identified by performing image recognition processing on the image identified by the "evaluation image ID", and is represented by a character string, a number, or the like. The "class" is information related to the class indicating the type of the object associated with the object identified by the "object ID". The "position coordinates" is information related to the position coordinates in the evaluation image ID of the object identified by the "object ID". The "size" is information related to the size of the object identified by the "object ID". The "confidence score" is information related to the certainty of the class associated with the object identified by the "object ID".

[0036] That is, in FIG. 5, when inference processing was executed on the evaluation image identified by the evaluation image ID "IMGID#1", objects identified by object IDs "OBID#1-1", "OBID#1-2", etc. were detected, the class of the object identified by the object ID "OBID#1-1" was classified as "person", the position coordinates of the said object were "(x #1-1 ,y #1-1 )", the size of the said object was "(Δx #1-1 ,Δy #1-1 )", and the confidence score of the said object was "SCR#1-1", etc. are shown as an example.

[0037] Note that the information stored in the inference processing result storage unit 123 is not limited to the information related to the items of "evaluation image ID", "detected / identified object ID", "class", "position coordinates", "size", "confidence score", and information related to the results of inference processing by other arbitrary image recognition models may be stored.

[0038] The learned model storage unit 124 stores information related to the image recognition model of the learned model. Here, an example of the information stored in the learned model storage unit 124 will be described with reference to FIG. 6. FIG. 6 is a diagram showing an example of the information stored in the learned model storage unit of the image recognition model evaluation device according to the present disclosure.

[0039] As shown in FIG. 6, the learned model storage unit 124 stores information related to the items of "model ID" and "model data".

[0040] The "model ID" is an identifier that identifies a trained model and is represented by a character string or a number. The "model data" is the data of a trained model, and the model may be a neural network or the like. The "model data" includes, for example, various types of information such as connection information on how the nodes included in each of the plurality of layers constituting the neural network are connected to each other, and connection coefficients that are multiplied by the numerical values input and output between the connected nodes.

[0041] That is, in FIG. 6, an example is shown in which the model data "MDT#1" of the trained model identified by the model ID "M#1" is stored.

[0042] Note that the information stored in the trained model storage unit 124 is not limited to the information related to the items of "model ID" and "model data", and other arbitrary information related to the trained model may be stored.

[0043] Next, returning to FIG. 2, the control unit 130 will be described. The control unit 130 is a controller that executes various arithmetic processes and processes for realizing functions. The control unit 130 is realized by executing various programs stored in the storage unit 120 using the RAM as a work area by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like. Further, the control unit 130 may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0044] As shown in FIG. 2, the control unit 130 includes an acquisition unit 131, an inference processing unit 132, an evaluation value calculation unit 133, an inference result storage unit 134, a reception unit 135, and a display control unit 136. These configurations will be described in order below.

[0045] The acquisition unit 131 acquires evaluation image data used for evaluating the image recognition model. The evaluation image data is image data in which an object to be detected by the inference processing unit 132 (image recognition model) described later is captured. The acquisition unit 131 also acquires, together with the evaluation image data, the correct answer data associated with the evaluation image data. The correct answer data is data indicating the object shown in the evaluation image data, and in this embodiment, it is data indicating the class (type) of the object, the position (position coordinates) of the object, and the size of the object. Note that the correct answer data is set in advance.

[0046] Specifically, the acquisition unit 131 receives a designation of evaluation image data that is the target of image recognition processing from the user via the input unit 140, and reads the designated evaluation image data from the evaluation image data storage unit 122, thereby acquiring the evaluation image data and the correct answer data used for evaluating the image recognition model. Note that the designation of the evaluation image data from the user may be received by inputting the file name of the evaluation image data, clicking on the thumbnail of the image, or the like.

[0047] The inference processing unit 132 detects an object shown in the evaluation image data based on the evaluation image data, and executes a process of associating a label indicating the type of the object with the detected object. In this embodiment, the inference processing unit 132 outputs, by this process, the class (type) of the object shown in the evaluation image data, the position (position coordinates) of the object, and the size of the object.

[0048] The inference processing unit 132 executes these processes using an image recognition model. As the image recognition model, a pre-trained model that has learned the features of objects appearing in various image data may be used, and for example, it may be realized by an object detection model such as CNN (Convolutional Neural Network), R-CNN (Reginal Convolutional Neural Network), Fast R-CNN, SSD (Single Shot Multibox Detector), YOLO (You Only Look Once), DETR (Detection Transformer).

[0049] The evaluation value calculation unit 133 calculates an evaluation value of the image recognition process for each object detected by the inference processing unit 132. Specifically, the evaluation value calculation unit 133 calculates the evaluation value as described below.

[0050] For example, the evaluation value calculation unit 133 calculates, as the evaluation value, a confidence score indicating the certainty of the label associated with the object detected by the inference processing unit 132. Here, the confidence score is a value indicating the probability of being classified into the label calculated by the image recognition model. The evaluation value calculation unit 133 calculates the value of the discrimination probability calculated by the image recognition model as the confidence score.

[0051] In addition, the evaluation value calculation unit 133 calculates, as the evaluation value, a common union ratio index indicating the degree of overlap between the region of the object detected by the inference processing unit 132 and the region of the object in the correct data. Here, the common union ratio index can also be called IoU (Intersection over Union), and it is an evaluation value indicating the degree of overlap between the bounding box A that is the correct answer and the bounding box B indicating the detection location of the object detected by the image recognition process. Let the area of the bounding box A that is the correct answer be Aa and the area of the bounding box B indicating the detection location of the object detected by the image recognition process be Bb, then it is calculated by the following formula (1).

[0052]

Mathematics

[0053] Further, the evaluation value calculation unit 133 calculates, as evaluation values, a correct detection rate indicating the probability that the inference processing unit 132 can detect an object included in the correct data of the evaluation image data, and a false detection rate indicating the probability that the inference processing unit 132 falsely detects an object not included in the correct data. The correct detection rate may be calculated by any method. In the present embodiment, depending on the number of objects in the correct data, the correct detection rate is a value obtained by dividing the number of correct detections indicating the number of labels satisfying the condition that both the common union ratio index and the confidence score are equal to or greater than an arbitrary threshold by the number of objects in the correct data. The false detection rate may be calculated by any method. In the present embodiment, depending on the number of objects detected by the inference processing unit 132, the false detection rate is a value obtained by dividing the number of false detections indicating the number of labels satisfying the condition that at least one of the common union ratio index and the confidence score is less than an arbitrary threshold by the number of objects detected by the inference processing unit 132. Here, assuming that the correct detection rate is CRR, the number of correct detections is NCR, and the number of objects is NO, the correct detection rate CRR is calculated by the following formula (2). Also, assuming that the false detection rate is FRR, the number of false detections is NFR, and the total number of detections is NTR, the false detection rate FRR is calculated by the following formula (3).

[0054]

Mathematics

Mathematics

[0055] Here, the number of objects is the total number of correct labels. The number of correct detections is the number of labels satisfying the condition that both the IoU and the confidence score are equal to or greater than an arbitrary threshold received from the user. The total number of detections is the total number of inference result labels. The number of false detections is the number of labels satisfying the condition that at least one of the IoU and the confidence score is less than an arbitrary threshold received from the user.

[0056] Note that when the evaluation value calculation unit 133 receives the input of the threshold of the intersection over union (IoU) and the threshold of the confidence score from the reception unit 135 described later, it recalculates the true positive rate and the false positive rate using the received threshold of the intersection over union (IoU) and the threshold of the confidence score, and calculates them as evaluation values. Thereby, based on the adjustment of the threshold of the intersection over union (IoU) and the threshold of the confidence score by the user, the true positive rate and the false positive rate for each of these thresholds can be evaluated.

[0057] After the inference processing unit 132 executes inference processing on the evaluation image data, the inference result storage unit 134 aggregates and stores the inference processing results of the inference processing unit 132 in tabular form for each object detected for each evaluation image data. Specifically, the inference result storage unit 134 may aggregate and store information as shown in FIG. 5 in tabular form in the inference processing result storage unit 123. That is, for the object ID of the object detected by the inference processing unit 132 for each evaluation image data, the class, position coordinates, size, and confidence score are associated and aggregated and stored in tabular form. Note that the inference result storage unit 134 is not limited to the information described above, and may aggregate information related to the inference processing results of the inference processing unit 132 in tabular form and store it in the inference processing result storage unit 123.

[0058] The reception unit 135 receives the input of various information from the user. Specifically, the reception unit 135 receives the input of various information from the user via the input unit 140.

[0059] For example, the reception unit 135 receives an input of a threshold value of the intersection over union (IoU) and a threshold value of the confidence score from the user. The reception unit 135 may give a command to the display control unit 136 to cause the display unit 150 to display information as shown in FIG. 7, and receive an input of a threshold value of the intersection over union (IoU) and a threshold value of the confidence score from the user. FIG. 7 is a diagram showing an example of a reception screen for various threshold values by the reception unit of the image recognition model evaluation apparatus according to the present disclosure. As shown in FIG. 7, the reception unit 135 may display an input screen for a threshold value of the intersection over union (IoU) and a threshold value of the confidence score, and receive these from the user.

[0060] In addition, the reception unit 135 receives conditions of the true positive rate and the false positive rate from the user. For example, the reception unit 135 may give a command to the display control unit 136 to cause the display unit 150 to display information as shown in FIG. 8, and receive conditions of the true positive rate and the false positive rate from the user. FIG. 8 is a diagram showing an example of a reception screen for conditions of the true positive rate and the false positive rate by the reception unit of the image recognition model evaluation apparatus according to the present disclosure. As shown in FIG. 8, the reception unit 135 may display an input screen for a lower limit value and an upper limit value of the true positive rate (CRR), and a lower limit value and an upper limit value of the false positive rate (FRR), and receive these from the user.

[0061] The display control unit 136 gives a control command to cause the display unit 150 to display various information. Specifically, the display control unit 136 causes the display unit 150 to display information as described below.

[0062] For example, the display control unit 136 displays information including the evaluation value of the image recognition process for each object. The display control unit 136 superimposes and displays information indicating positive detection, false detection, and missed detection locations on the evaluation image. FIG. 9 is a diagram showing an example of the result of the image recognition process by the image recognition model evaluation apparatus according to the present disclosure. In FIG. 9, the dotted bounding boxes FB1 and FB2 indicate false detections, and the solid bounding box CB indicates a positive detection. Note that in FIG. 8, although the missed detection locations are not shown, the missed detection locations may be displayed in a manner distinguishable from these other than the dotted and solid lines.

[0063] In addition, the display control unit 136 displays information regarding the inference processing result by the inference processing unit 132. Specifically, the display control unit 136 displays the common union ratio index of the objects detected by the inference processing unit 132 and the common union ratio index and the confidence score of the objects for which both the common union ratio index and the confidence score exceed the threshold value. FIG. 10 is a diagram showing an example of the maximum IoU with respect to the correct label and the confidence score as the analysis result of the inference processing result by the image recognition model evaluation apparatus according to the present disclosure. The correct label NO shown in FIG. 10 corresponds to the total number of objects shown in the evaluation image data. As shown in FIG. 10, the maximum IoU value and the confidence score of the object are displayed for the entire objects shown in the evaluation image data. Thereby, it is possible to grasp the characteristics of the objects that are likely to be accurately recognized by the image recognition model. Therefore, it can be used to consider the direction of collecting learning data for re-learning the image recognition model for improving the recognition accuracy and the like.

[0064] In addition, the display control unit 136 may display, for the object, both the common union ratio index of the object detected by the inference processing unit 132 and the common union ratio index and the confidence score of the object for which both exceed the threshold value. FIG. 11 is a diagram showing an example of the maximum IoU for an object and the confidence score as an analysis result of the inference processing result by the image recognition model evaluation apparatus according to the present disclosure. The object NO shown in FIG. 11 corresponds to the total number of objects recognized as being shown in the evaluation image data. As shown in FIG. 10, the maximum IoU value and the confidence score of the object are displayed for the entire object recognized as being shown in the evaluation image data. Thereby, it is possible to grasp the characteristics of the object that are likely to be accurately recognized by the image recognition model. Therefore, it can be used to consider the direction of collecting learning data for re-learning the image recognition model for improving the recognition accuracy and the like.

[0065] In addition, the display control unit 136 lists the true positive rate and the false positive rate, extracts an image showing the true positive rate and the false positive rate under specific conditions using the sorting and filtering functions, and displays information about the image. FIG. 12 is a diagram showing an example of information displayed by the display control unit of the image recognition model evaluation apparatus according to the present disclosure. As shown in FIG. 12, the display control unit 136 displays the image file name, the true positive rate, and the false positive rate side by side on the display unit 150. As shown in FIG. 12, the true positive rate (CRR) and the false positive rate (FRR) for the entire image file may be displayed at the top. Then, below that, the true positive rate received by the reception unit 135, the image file name of the image file that satisfies the conditions of the true positive rate and the false positive rate, the true positive rate, and the false positive rate may be displayed side by side.

[0066] The input unit 140 receives various operation information from the user. For example, the input unit 140 may receive various operations from the operator via the display surface (for example, the display unit 150) by a touch panel. Further, the input unit 140 may receive various operations from the user by various buttons, a keyboard, or a mouse.

[0067] The display unit 150 displays various types of information. The display unit 150 displays various types of information according to the instructions of the display control unit 136. The display unit 150 may display, for example, a GUI (Graphical User Interface) for receiving operations related to various processes from the user. The display unit 150 may be realized by a liquid crystal display, an organic EL (Electro Luminescence) display, a micro LED (Light Emitting Diode) display, or the like. Further, the display unit 150 may be a touch panel of various types such as a capacitance type.

[0068] According to the image recognition model evaluation apparatus 100 described above, it is possible to extract and analyze an image of an evaluation value under specific conditions. Further, by collecting images similar to an image with a high true detection rate and re-training the model to recognize it as a true detection, it is possible to improve the true detection rate. Also, by collecting images similar to an image with a high false detection rate and re-training the model to recognize it as a false detection, it is possible to reduce the false detection rate. At this time, regarding the displayed image, since the user can determine whether the image is in a situation where false detection does not pose a problem, it is possible to avoid the overall image recognition performance from being biased due to learning that excessively reduces the false detection rate.

[0069] Further, by using the image recognition model evaluation apparatus 100 without connecting it to an external network, the risk of data leakage to the outside can be reduced. Therefore, the user can evaluate the image recognition model without worrying about data leakage.

[0070] (Regarding the image recognition model evaluation method) Next, the image recognition model evaluation method according to the present disclosure will be described with reference to FIG. 13. FIG. 13 is a flowchart showing the flow of the image recognition model evaluation method according to the present disclosure. The image recognition model evaluation method according to the present disclosure will be described along the flow shown in FIG. 13.

[0071] First, the image recognition model evaluation apparatus 100 acquires evaluation image data associated with correct labels (step S101). Next, the image recognition model evaluation apparatus 100 executes an inference process on the evaluation image data using the target model (step S102). Next, the image recognition model evaluation apparatus 100 calculates an evaluation value based on the inference process result (step S103). Next, the image recognition model evaluation apparatus 100 uses the GUI to display information regarding the inference process result including the evaluation value (step S104). Next, the image recognition model evaluation apparatus 100 receives an input of the IoU and the threshold of the confidence score from the user via the input unit 140 (step S105). Next, the image recognition model evaluation apparatus 100 returns to step S103 and calculates the evaluation value using the received IoU and the threshold of the confidence score. Note that in step S105, when the image recognition model evaluation apparatus 100 receives an input of information indicating the end of evaluation from the user, the image recognition model evaluation apparatus 100 ends the process.

[0072] According to this, it is possible to calculate the evaluation value of the image recognition process for each detected object, which is the result of the inference process for the evaluation image data. Therefore, it is possible to provide an image recognition model evaluation method and a program that can appropriately evaluate the image recognition model.

[0073] (Configuration and Effect) The image recognition model evaluation apparatus 100 according to the present disclosure includes an acquisition unit 131 that acquires evaluation image data used for evaluating the image recognition model, an inference process unit 132 that detects an object shown in the evaluation image data based on the evaluation image data and executes a process of associating a label indicating the type of the detected object with the detected object, an evaluation value calculation unit 133 that calculates an evaluation value of the image recognition process for each detected object, and a display control unit 136 that displays information including the evaluation value of the image recognition process for each object.

[0074] According to this configuration, an evaluation value of the image recognition process can be calculated for each detected object that is the result of the inference process on the evaluation image data. Therefore, it is possible to provide an image recognition model evaluation apparatus 100 that can appropriately evaluate the image recognition model.

[0075] The evaluation value calculation unit 133 of the image recognition model evaluation apparatus 100 according to the present disclosure calculates, as the evaluation value, a confidence score indicating the certainty of the label associated with the object detected by the inference processing unit 132, and a common union ratio index indicating the degree of overlap between the area of the object detected by the inference processing unit 132 and the area of the object in the correct answer data.

[0076] According to this configuration, a confidence score can be calculated and displayed as an evaluation value of the image recognition model. Therefore, the user can evaluate the image recognition model based on it. Therefore, it is possible to provide an image recognition model evaluation apparatus 100 that can appropriately evaluate the image recognition model.

[0077] The evaluation value calculation unit 133 of the image recognition model evaluation apparatus 100 according to the present disclosure calculates, as the evaluation value, a true positive rate indicating a value obtained by dividing the number of labels satisfying the condition that both the common union ratio index and the confidence score are equal to or greater than an arbitrary threshold by the number of objects in the correct answer data, and a false positive rate indicating a value obtained by dividing the number of labels satisfying the condition that at least one of the common union ratio index and the confidence score is less than an arbitrary threshold by the number of objects detected by the inference processing unit 132.

[0078] According to this configuration, the true positive rate and the false positive rate can be calculated and displayed as the evaluation value of the image recognition model. Therefore, the user can evaluate the image recognition model based on it. Therefore, it is possible to provide an image recognition model evaluation apparatus 100 that can appropriately evaluate the image recognition model.

[0079] The image recognition model evaluation device 100 according to the present disclosure further includes an inference result storage unit 134 that, after the inference processing unit 132 executes inference processing on the evaluation image data, summarizes and stores the inference processing results of the inference processing unit 132 in a tabular format for each object detected for each evaluation image data.

[0080] According to this configuration, the inference processing results can be summarized and stored in a tabular format. Therefore, every time the threshold value is changed, the trouble of re-executing the inference processing can be saved. Thus, it is possible to provide the image recognition model evaluation device 100 that can appropriately evaluate the image recognition model.

[0081] The image recognition model evaluation device 100 according to the present disclosure further includes a reception unit 135 that receives input of various information from a user. The reception unit 135 receives input of the threshold value of the common union ratio index and the threshold value of the confidence score from the user, and the display control unit 136 displays the common union ratio index and the confidence score of the objects in which both the common union ratio index and the confidence score of the objects detected by the inference processing unit 132 exceed the threshold value.

[0082] According to this configuration, it is possible to receive input of the threshold value of the common union ratio index and the threshold value of the confidence score from the user, and display the common union ratio index and the confidence score of the objects in which both the common union ratio index and the confidence score exceed the threshold value among the detected objects. Therefore, for the image recognition model to be evaluated, it is possible to grasp the characteristics of the objects that are easy to recognize and use them for formulating the preparation policy of the learning data. Thus, it is possible to provide the image recognition model evaluation device 100 that can appropriately evaluate the image recognition model.

[0083] The image recognition model evaluation device 100 according to the present disclosure further includes a reception unit 135 that receives input of various types of information from a user. The reception unit 135 receives conditions of a true detection rate and a false detection rate from the user, and the display control unit 136 extracts image data that satisfies the conditions based on the conditions of the true detection rate and the false detection rate received from the user, and displays the file name of the image data that satisfies the conditions.

[0084] According to this configuration, it is possible to receive conditions of a true detection rate and a false detection rate from a user, extract image data that satisfies the conditions, and display the file name of the image data that satisfies the conditions. Therefore, it is possible to grasp the characteristics of objects that are easily detected correctly and the characteristics of objects that are easily detected falsely, and use them for formulating a preparation policy for learning data. Therefore, it is possible to provide an image recognition model evaluation device 100 that can appropriately evaluate an image recognition model.

[0085] The image recognition model evaluation method according to the present disclosure includes steps of: acquiring evaluation image data to be used for evaluating an image recognition model; based on the evaluation image data, detecting an object depicted in the evaluation image data and performing a process of associating a label indicating the type of the object with the detected object; calculating an evaluation value of the image recognition process for each detected object; and displaying information including the evaluation value of the image recognition process for each object.

[0086] According to this configuration, it is possible to calculate an evaluation value of the image recognition process for each detected object, which is the result of the inference process for the evaluation image data. Therefore, it is possible to provide an image recognition model method that can appropriately evaluate an image recognition model.

[0087] The program according to the present disclosure causes a computer to execute steps of: acquiring evaluation image data used for evaluating an image recognition model; detecting an object depicted in the evaluation image data based on the evaluation image data, and executing a process of associating a label indicating the type of the object with the detected object; calculating an evaluation value of the image recognition process for each detected object; and displaying information including the evaluation value of the image recognition process for each object.

[0088] According to this configuration, it is possible to calculate an evaluation value of the image recognition process for each detected object, which is the result of the inference process for the evaluation image data. Therefore, it is possible to provide a program that can appropriately evaluate the image recognition model.

[0089] As described above, the embodiments of the present disclosure have been described, but the embodiments are not limited by the content of these embodiments. Further, the above-described components include those that can be easily assumed by those skilled in the art, those that are substantially the same, and those within the so-called equivalent range. Furthermore, the above-described components can be combined as appropriate. Furthermore, various omissions, substitutions, or changes of the components can be made without departing from the gist of the above-described embodiments.

Description of Reference Numerals

[0090] 100 Image Recognition Model Evaluation Apparatus 110 Communication Unit 120 Storage Unit 121 Learning Image Data Storage Unit 122 Evaluation Image Data Storage Unit 123 Inference Process Result Storage Unit 124 Learned Model Storage Unit 130 Control Unit 131 Acquisition Unit 132 Inference Process Unit 133 Evaluation Value Calculation Unit 134 Inference Result Storage Unit 135 Reception Unit 136 Display Control Unit 140 Input Unit 150 display unit

Claims

1. An acquisition unit that acquires evaluation image data used for evaluating an image recognition model; An inference processing unit that, based on the evaluation image data, detects an object depicted in the evaluation image data and executes a process of associating a label indicating the type of the detected object; An evaluation value calculation unit that calculates an evaluation value of the image recognition process for each object detected by the inference processing unit; A display control unit that displays information including the evaluation value of the image recognition process for each object; and An image recognition model evaluation apparatus.

2. The evaluation value calculation unit calculates, as the evaluation value, a confidence score indicating the certainty of the label associated with the object detected by the inference processing unit, and a common union ratio index indicating the degree of overlap between the area of the object detected by the inference processing unit and the area of the object in the correct answer data. The image recognition model evaluation apparatus according to claim 1.

3. The evaluation value calculation unit calculates, as the evaluation value, a true positive rate, which is a value obtained by dividing the number of correctly detected labels, where both the common union ratio index and the confidence score are equal to or greater than an arbitrary threshold, by the number of objects in the correct answer data, and a false positive rate, which is a value obtained by dividing the number of misdetected labels, where at least one of the common union ratio index and the confidence score is less than the threshold, by the number of objects detected by the inference processing unit. The image recognition model evaluation apparatus according to claim 2.

4. After the inference processing unit executes inference processing on the evaluation image data, the inference processing unit further includes an inference result storage unit that stores the inference processing results of the inference processing unit in a tabular format for each object detected for each evaluation image data. The image recognition model evaluation apparatus according to claim 1 or 2.

5. Further comprising a reception unit that receives input of various information from a user, The reception unit receives input of the threshold of the common union ratio index and the threshold of the confidence score from the user, The display control unit displays the common union ratio index and the confidence score of the object whose both the common union ratio index and the confidence score of the object detected by the inference processing unit exceed the threshold. The image recognition model evaluation apparatus according to claim 3.

6. further comprising a reception unit that receives input of various information from a user; the reception unit receives from the user the conditions of the true detection rate and the false detection rate; the display control unit extracts image data that satisfies the conditions received from the user based on the conditions of the true detection rate and the false detection rate received from the user, and displays the file name of the image data that satisfies the conditions; The image recognition model evaluation device according to claim 3.

7. a step of obtaining evaluation image data used for evaluating an image recognition model; a step of detecting an object shown in the evaluation image data based on the evaluation image data, and executing a process of associating a label indicating the type of the detected object with the detected object; a step of calculating an evaluation value of the image recognition process for each detected object; a step of displaying information including the evaluation value of the image recognition process for each object; An image recognition model evaluation method.

8. a step of obtaining evaluation image data used for evaluating an image recognition model; a step of detecting an object shown in the evaluation image data based on the evaluation image data, and executing a process of associating a label indicating the type of the detected object with the detected object; a step of calculating an evaluation value of the image recognition process for each detected object; a step of displaying information including the evaluation value of the image recognition process for each object; A program for causing a computer to execute.

Images