Information processing system, its control method, and program

Abstract

Even if an AI performs well on an evaluation dataset, it may not achieve the same level of accuracy in real-world applications. In such cases, confidence in the AI ​​diminishes, highlighting the need for a mechanism to develop reliable AI. [Solution] In model comparison analysis mode, the CPU 201 of the client terminal 101 acquires the inference results of the first AI model and the inference results of the second AI model for each of the multiple images (S1002-S1005), and controls the system to compare and display the inference results of each AI model for each image in order of the image in which the number of correctly detected objects among the inference results of each AI model increased (S1006, S1009, S1010).

Description

Technical Field

[0001] The present invention relates to an information processing system, a control method thereof, and a program.

Background Art

[0002] In recent years, AI generated by deep learning or machine learning using learning data has been used in various fields. When determining whether AI is practical, it is common to prepare an evaluation dataset and measure the accuracy against it. However, the quality of AI may not be reliable based solely on the performance on the evaluation dataset. Patent Document 1 discloses a technique for visualizing a prediction situation in order to grasp the factors that cause an error between a prediction and an actual result.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Even if the accuracy for the evaluation dataset is high, high accuracy may not be achieved when actually operating in the field. In such a case, the reliability of AI decreases, so a mechanism for developing reliable AI is necessary.

[0005] Therefore, an object of the present invention is to provide a mechanism capable of grasping the reliability of AI.

Means for Solving the Problems

[0006] The present invention is characterized by comprising: acquisition means for acquiring the inference result of a first AI model for detecting an object and the inference result of a second AI model for detecting an object for each of a plurality of images; counting means for counting the number of cases in the inference result of each AI model for each image; and display control means for controlling the comparison and display of the inference results of each AI model for each image in order of the image in which the number of correctly detected cases among the inference results of each AI model increases. [Effects of the Invention]

[0007] According to the present invention, it becomes possible to provide a mechanism that allows for the assessment of the reliability of AI. [Brief explanation of the drawing]

[0008] [Figure 1] A diagram illustrating the configuration of an AI-based system representing one embodiment of the present invention. [Figure 2] A diagram showing an example of the hardware configuration of the client terminal in this embodiment. [Figure 3] A diagram showing an example of the functional configuration of the client terminal in this embodiment. [Figure 4] A flowchart showing an example of the annotation quality inspection process in this embodiment. [Figure 5] A flowchart showing an example of the rectangular enclosure inspection process in this embodiment. [Figure 6] A flowchart showing an example of the label error detection process in this embodiment. [Figure 7] A flowchart showing an example of the AI ​​model learning and evaluation process in this embodiment. [Figure 8] A flowchart showing an example of the AI ​​quality inspection process in this embodiment. [Figure 9] A flowchart showing an example of the process for confirming the detection results of this embodiment. [Figure 10] A flowchart showing an example of the comparison process of the AI ​​model in this embodiment. [Figure 11]A flowchart showing an example of accuracy processing according to the size of the rectangle in this embodiment. [Figure 12] A flowchart showing an example of the relationship processing between the amount of learning data and accuracy in this embodiment. [Figure 13] A diagram showing an example of the screen displayed when the program of this embodiment is started. [Figure 14] A diagram showing an example of the annotation quality inspection screen (inspection of the way of enclosing a rectangle) in this embodiment. [Figure 15] A diagram for explaining the inspection of the way of enclosing a rectangle in this embodiment. [Figure 16] A diagram for explaining the inspection of the way of enclosing a rectangle in this embodiment. [Figure 17] A diagram for explaining the inspection of the way of enclosing a rectangle in this embodiment. [Figure 18] A diagram for explaining the inspection of the way of enclosing a rectangle in this embodiment. [Figure 19] A diagram for explaining the inspection of the way of enclosing a rectangle in this embodiment. [Figure 20] A diagram showing an example of the annotation quality inspection screen (inspection of label errors) in this embodiment. [Figure 21] A diagram for explaining the inspection of label errors in this embodiment. [Figure 22] A diagram for explaining the inspection of label errors in this embodiment. [Figure 23] A diagram showing an example of the AI model learning / evaluation screen in this embodiment. [Figure 24] A diagram for explaining the AI model learning / evaluation in this embodiment. [Figure 25] A diagram for explaining the AI model learning / evaluation in this embodiment. [Figure 26] A diagram showing an example of the AI quality inspection screen (confirmation of detection results) in this embodiment. [Figure 27] A diagram showing an example of the detailed image display of the preview image for confirming the detection results in this embodiment. [Figure 28] A diagram showing an example of the detailed image display of the preview image for confirming the detection results in this embodiment. [Figure 29]This figure shows an example of the AI ​​quality inspection screen (AI model comparison) of this embodiment. [Figure 30] This figure shows an example of the AI ​​quality inspection screen (AI model comparison) of this embodiment. [Figure 31] This figure shows an example of the AI ​​quality inspection screen (accuracy according to the size of the rectangle) of this embodiment. [Figure 32] A diagram showing an example of a list of ranges for the size of the rectangle in this embodiment. [Figure 33] This figure shows an example of the AI ​​quality inspection screen (relationship between training data volume and accuracy) of this embodiment. [Modes for carrying out the invention]

[0009] Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the drawings. Figure 1 is a diagram illustrating the configuration of a system using AI (a trained model generated by deep learning / machine learning, also called an AI model) that represents one embodiment of the present invention.

[0010] In the system of this embodiment, the client terminal 101 is connected to the network 100. The client terminal is, for example, a personal computer (hereinafter referred to as "PC"). The client terminal 101 performs the main processing for confirming the detection results of objects by AI and understanding their reliability.

[0011] Network 100 can take the form of a wired LAN, wireless LAN, USB, etc., depending on the physical interface of the client terminal 101. A server 102 may be placed on network 100. Server 102 may be implemented by a single computer or by multiple computers. For example, server 102 may be implemented using cloud computing technology. The client terminal 101 may read data from the server 102.

[0012] Figure 2 is a block diagram showing an example of the hardware configuration of client terminal 101. As shown in Figure 2, the client terminal 101 is connected via the system bus 204 to a CPU (Central Processing Unit) 201, ROM (Read Only Memory) 202, RAM (Random Access Memory) 203, input controller 205, video controller 206, memory controller 207, and communication I / F controller 208.

[0013] CPU201 provides comprehensive control over all devices and controllers connected to the system bus 204. The ROM 202 or external memory 211 holds the BIOS (Basic Input / Output System) and OS (Operating System), which are control programs executed by the CPU 201, as well as computer-readable and executable programs and various necessary data (including data tables) for realizing the various information processing in this embodiment.

[0014] RAM203 functions as the main memory, work area, etc., of CPU201. The CPU 201 loads the necessary programs and other data for processing from the ROM 202 or external memory 211 into the RAM 203, and then executes the loaded programs to perform various operations.

[0015] The input controller 205 controls input from input devices such as a keyboard 209 or a pointing device such as a mouse (not shown). If the input device is a touch panel, the user can give various instructions by pressing (touching with a finger, etc.) icons, cursors, or buttons displayed on the touch panel. The touch panel may also be a multi-touch screen or other touch panel capable of detecting the positions of multiple fingers touching it.

[0016] The video controller 206 controls the display to an external output device such as the display 210. The display may include the display of a laptop PC integrated with the main unit. The external output device is not limited to a display; for example, it may be a projector. Furthermore, if the external output device is a device that can accept the aforementioned touch operation, it also provides the function of an input device.

[0017] The video controller 206 can control the video memory (VRAM) for display control, and can utilize a portion of the RAM 203 as the video memory area, or it can provide a separate dedicated video memory.

[0018] The memory controller 207 controls access to the external memory 211. The external memory can include external storage devices (such as HDDs (Hard Disk Drives) or SSDs (Solid State Drives)) that store boot programs, various applications, font data, user files, editing files, and other data, as well as flexible disks (FDs), or CompactFlash® memory connected to a PCMCIA card slot via an adapter.

[0019] The communication interface controller 208 connects to and communicates with external devices via a network and performs communication control processing over the network. For example, it can communicate using TCP / IP, telephone lines such as ISDN, and mobile communication lines such as cell phones.

[0020] The CPU 201 enables display on the display 210 by, for example, performing the process of expanding (rasterizing) outline fonts into the display information area in RAM 203. The CPU 201 also enables user input via a mouse cursor (not shown) on the display 210.

[0021] Furthermore, the server 102 may have the same hardware configuration as the client terminal 101, but it does not need to have a keyboard 209 or a display 210.

[0022] Next, we will explain the functional configuration of the client terminal 101 using Figure 3. Figure 3 shows an example of the functional configuration of the client terminal 101 in this embodiment. Each functional configuration shown in Figure 3 corresponds to a function realized by the CPU 201 of the client terminal 101 loading programs, etc., from ROM 202 or external memory 211 into RAM 203 and executing them as needed.

[0023] The input receiving unit 300 receives instructions input by the user through the input controller 205.

[0024] The data analysis unit 310 acquires data stored in the database 340 and file storage 350 in response to instructions received from the input reception unit 300, and processes it for inspection and evaluation by the annotation quality inspection unit 320, the AI ​​learning and evaluation unit 330, and the AI ​​quality inspection unit 360.

[0025] The annotation quality inspection unit 320 inspects the way the annotations in the training dataset are enclosed in rectangles and checks for label errors. The AI ​​learning and evaluation unit 330 trains an AI model using a training dataset and evaluates the trained AI model using an evaluation dataset. The AI ​​Quality Inspection Unit 360 performs processing to present the reliability of the AI, which has been trained and evaluated by the AI ​​Learning and Evaluation Unit 330, in a way that makes it easy to understand.

[0026] The result output unit 370 displays the inspection results from the annotation quality inspection unit 320 and the AI ​​quality inspection unit 360 on the display of the client terminal 101. In this embodiment, the display destination is the display of the client terminal 101, but it is not limited to this.

[0027] Database 340 and file storage 350 store training datasets, evaluation datasets, AI training and evaluation results, lists of ranges of rectangular sizes, etc., as described later. Database 340 and file storage 350 store data in the external memory 211 of the client terminal 101, for example, but the data may also be stored in the server 102. In this embodiment, the CPU 201 of the client terminal 101 retrieves data, etc., from database 340 and file storage 350 and uses it for annotation quality inspection, AI training and evaluation, AI quality inspection, etc.

[0028] As described above, the functional configuration shown in Figure 3 constitutes the information processing system of the present invention, and some or all of these may be provided on the server 102. With this configuration, by accessing the server 102 from a web browser or dedicated application program running on a personal computer, tablet terminal, smartphone, etc., the functions shown in Figure 3 can be provided from the server 102.

[0029] Next, the processes executed by the client terminal 101 in this embodiment will be explained using flowcharts from Figures 4 to 12.

[0030] When the CPU 201 of the client terminal 101 executes the program of this embodiment, it displays a screen on the display 210 as shown in Figure 13 and accepts user input. Figure 13 shows an example of a screen displayed when the program of this embodiment is started.

[0031] In a screen like the one shown in Figure 13, when the user presses [Screen Switching] 1001, the screen switching menu 1002 is displayed, allowing the user to select one of the following screens to switch to: "Annotation Quality Inspection," "AI Learning and Evaluation," or "AI Quality Inspection." Specifically, selecting "Annotation Quality Inspection" switches to the Annotation Quality Inspection screen 2000 shown in Figure 14. Selecting "AI Learning and Evaluation" switches to the AI ​​Learning and Evaluation screen 2200 shown in Figure 23. Selecting "AI Quality Inspection" again switches to the AI ​​Quality Inspection screen 2300 shown in Figure 26. Furthermore, screen switching is possible from any of these Annotation Quality Inspection screens 2000, AI Learning and Evaluation screen 2200, or AI Quality Inspection screen 2300 using the screen switching menu 1002. Note that the startup screen shown in Figure 13 may be any of the Annotation Quality Inspection screen 2000, the AI ​​Learning / Evaluation screen 2200, or the AI ​​Quality Inspection screen 2300, or a different startup screen may be provided. Furthermore, although an example of these screens being displayed within a single application is described, they may also be implemented in multiple applications. For example, the Annotation Quality Inspection screen 2000 and the AI ​​Quality Inspection screen 2300 may be controlled to switch between screens within a single application, while the AI ​​Learning / Evaluation screen 2200 may be displayed in a separate application.

[0032] [Annotation Quality Inspection Process] When the CPU 201 of client terminal 101 detects that "Annotation Quality Inspection" has been selected in the screen switching menu 1002, it switches the screen display to the annotation quality inspection screen 2000 as shown in Figure 14 and starts the process shown in the flowchart of Figure 4.

[0033] Figure 4 is a flowchart showing an example of the annotation quality inspection process in this embodiment. The processing in this flowchart is performed by the input reception unit 300, the annotation quality inspection unit 320, and the result output unit 370 of the client terminal 101. Specifically, it is achieved by the CPU 201 of the client terminal 101 loading a program stored in the ROM 202 or external memory 211 into the RAM 203 and executing it.

[0034] In step S401, the CPU 201 of the client terminal 101 accepts input from the user regarding the inspection mode on the annotation quality inspection screen 2000. The inspection mode selection unit 2001 of the annotation quality inspection screen 2000 allows the user to select either "Inspection of Rectangle Enclosure Method" or "Inspection of Label Errors" as the inspection mode.

[0035] Figures 14 and 20 show examples of the annotation quality inspection screen 2000. Figure 14 corresponds to the case where "Inspection of Rectangle Enclosure" is selected as the inspection mode, and Figure 20 corresponds to the case where "Inspection of Label Errors" is selected as the inspection mode. The screen changes when the inspection mode is switched. When an inspection mode is entered in the inspection mode selection section 2001 of the annotation quality inspection screen 2000, the CPU 201 proceeds to step S402.

[0036] In step S402, the CPU 201 determines whether the input inspection mode is "rectangular enclosure inspection". If the input inspection mode is "rectangular enclosure inspection" (Yes in S402), the CPU201 switches to the rectangular enclosure inspection screen 2000-1 and proceeds to step S403.

[0037] In step S403, the CPU201 performs a rectangle enclosure check. Details of the rectangle enclosure check are shown in Figure 5, which will be described later. Once the rectangle enclosure check in S403 is completed, the CPU201 proceeds to step S406. In this case, in step S406, the CPU 201 displays the result of the rectangle enclosure inspection on the display 210.

[0038] On the other hand, if the input inspection mode is not "rectangular enclosure inspection" (i.e., No in S402), CPU201 proceeds to step S404.

[0039] In step S404, the CPU201 determines whether the input inspection mode is "label error inspection". If the entered inspection mode is "Label Error Inspection" (Yes in S404), CPU201 switches to the label error inspection screen 2000-2 and proceeds to step S405.

[0040] In step S405, CPU201 performs a label error check. Details of the label error check are shown in Figure 6, which will be described later. Once the label error check in S405 is completed, CPU201 proceeds to step S406. In this case, in step S406, the CPU 201 displays the results of the label error check on the display 210.

[0041] On the other hand, if the input inspection mode is not "Label Error Inspection" (i.e., No in S404), CPU201 proceeds to step S406. In this case, in step S406, the CPU 201 displays a prompt on the display 210, such as an indication prompting the input of the test mode. After processing S406, CPU201 terminates the processing of this flowchart.

[0042] <Checking the way rectangles are enclosed> In the "rectangular enclosure inspection" of this embodiment, object detection is performed on images included in the dataset to be inspected (Figures 15 and 16) using an object detection AI model (Figure 17). Furthermore, the detected rectangles are associated with the annotation rectangles corresponding to the images to be inspected (Figure 18). Then, the overlap ratio (hereinafter referred to as "overlap degree") between the associated detected rectangles and annotation rectangles is calculated (Figure 19). Finally, improvements are suggested for annotation rectangles whose overlap degree is below a threshold.

[0043] The following section will explain the rectangular enclosure inspection in more detail using Figures 15 to 19. Figures 15 to 19 are diagrams illustrating the inspection of how rectangles are enclosed.

[0044] Figure 15 shows an example of a dataset containing images to be examined. The dataset is stored, for example, in database 340 or file storage 350 as shown in Figure 3.

[0045] As shown in Figure 15, each dataset is assigned a unique identifier (dataset ID). A dataset includes an image folder and an annotation file folder. The image folder contains multiple image files. The annotation folder contains annotation files for the image files within the image folder.

[0046] Annotation refers to the process of attaching information, such as tags, to each piece of data. In this embodiment, each image file stored in the image folder is given information, including a rectangle and a class name, indicating where in the image it represents what is located. This information is stored as an annotation file in the annotation folder. Each image file in the image folder and the annotation file in the annotation folder are linked, for example, by a string included in the file name.

[0047] Figure 16(a) shows an example of an annotation file. As shown in Figure 16(a) 2050, the annotation file contains information such as the rectangle ID, the rectangle's position coordinates, and the class name.

[0048] Figure 16(b) shows an example of a list of detected rectangles by an object detection AI model. As shown in Figure 16(b) 2070, the detected rectangle list contains information such as the rectangle ID, rectangle position coordinates, class name, and confidence level of the detected rectangles detected by the object detection AI model.

[0049] Figure 17(a) shows an example of image 2051, which was the subject of the examination. Figure 17(b) shows an example of the detection result when object detection is performed on the image 2051 to be inspected using the object detection AI 2052. Here, it is assumed that an object was detected, as indicated by the solid line frame 2053.

[0050] Figure 18(a) shows an example of an annotation rectangle 2054 registered in annotation file 2050, which corresponds to the image 2051 being examined. Figure 18(b) shows an example of the correspondence between the detected rectangle and the annotation rectangle. As shown in figures 1 to 3, the detected rectangle (solid line frame) is associated with the annotation rectangle (dashed line frame) located near the detected rectangle.

[0051] In this embodiment, for example, IoU (Intersection over Union) is used to check how the rectangles are enclosed. IoU is an index that represents the degree of overlap between images, and the larger the IoU, the more the images overlap. The maximum value is "1" when the two regions completely overlap, and the minimum value is "0" when they do not overlap at all. In this embodiment, the degree of overlap between the rectangle (solid line frame) detected by the object detection AI model and the rectangle (dashed line frame) in the annotation file is calculated.

[0052] Figure 19 illustrates the method for calculating the degree of overlap. Here, we will refer to the detection rectangle of the object detection AI model shown with a solid line frame as "Rectangle A," and the rectangle of the annotation file shown with a dashed line frame as "Rectangle B." As shown in Figure 19, in this embodiment, the "degree of overlap" is defined as the ratio of the area of ​​the product of the areas enclosed by rectangle A and rectangle B (overlapping area) to the sum of the areas enclosed by rectangle A and rectangle B.

[0053] In the inspection of the rectangular enclosure method of this embodiment, for example, the "overlap degree" as described above is calculated, and if the calculated "overlap degree" is less than the specified "threshold" (or "overlap degree ≤ threshold"), an improvement to the annotation rectangle is suggested. The following explanation will use the flowchart in Figure 5.

[0054] Figure 5 is a flowchart showing an example of the rectangular enclosure inspection process in this embodiment. Specifically, the processing in this flowchart is performed by the input reception unit 300, annotation quality inspection unit 320, and result output unit 370 of the client terminal 101. In other words, it is achieved by the CPU 201 of the client terminal 101 loading a program stored in ROM 202 or external memory 211 into RAM 203 and executing it.

[0055] First, in step S501, the CPU 201 of the client terminal 101 accepts the user's selection input of the dataset ID to be inspected in the dataset ID input section 2002 of the rectangular enclosure inspection screen 2000-1 shown in Figure 14. The dataset ID input section 2002 is a pull-down menu, and the user can select any dataset ID from the pull-down menu, but they may also directly input the dataset ID using a keyboard or the like. When a dataset ID is entered into the dataset ID input unit 2002, the CPU 201 proceeds to step S502.

[0056] In step S502, the CPU 201 of the client terminal 101 receives input from the user for an object detection AI model to be used for inspection at the AI ​​model input section 2003 of the annotation quality inspection screen 2000. For example, the AI ​​model input section 2003 is a pull-down menu, and the user can select one of the object detection AI models from the pull-down menu, or they may directly enter the model name using a keyboard or the like. When an object detection AI model is input to the input unit 2003, the CPU 201 proceeds to step S503.

[0057] In step S504, the CPU 201 of the client terminal 101 receives input from the user for a threshold value of the rectangular overlap in the input section 2004 of the annotation quality inspection screen 2000. When the threshold value for the degree of rectangular overlap is input to the input unit 2004, the CPU 201 proceeds to step S505.

[0058] In step S505, when the CPU 201 of the client terminal 101 detects that the test execution button 2005 has been pressed, it proceeds to step S506.

[0059] In step S506, the CPU 201 of the client terminal 101 retrieves a dataset corresponding to the dataset ID entered in S501 from the database 340 or file storage 350, and retrieves an image folder from the dataset. Furthermore, in step S507, the CPU 201 of the client terminal 101 retrieves the annotation folder from the above dataset.

[0060] Next, in step S508, the CPU 201 of the client terminal 101 repeats the following steps S509 to S517 for the number of image files contained in the image folder acquired in S506.

[0061] First, in step S509, the CPU 201 of the client terminal 101 retrieves an unprocessed image file (hereinafter referred to as "image") from the image folder mentioned above. Next, in step S510, the CPU 201 of the client terminal 101 reads the annotation file corresponding to the image acquired in S509. Next, in step S511, the CPU 201 of the client terminal 101 performs object detection on the image file using the object detection model AI selected in step S502, and obtains a list of detected rectangles as shown in Figure 16(b).

[0062] Next, in step S512, the CPU 201 of the client terminal 101 repeats the following steps S513 to S515 for the number of rectangle information (hereinafter referred to as "annotation rectangles") contained in the annotation file read in S510.

[0063] First, in step S513, the CPU 201 of the client terminal 101 obtains one unprocessed annotation rectangle from the annotation file read in S512.

[0064] Next, in step S514, the CPU 201 of the client terminal 101 searches for and obtains a detected rectangle that is similar to the annotation rectangle obtained in S513 from the rectangle information in the detected rectangle list obtained in S511 (hereinafter referred to as "detected rectangle").

[0065] Next, in step S515, the CPU 201 of the client terminal 101 calculates the degree of overlap between the detected rectangle acquired in S514 and the annotation rectangle acquired in S513.

[0066] Then, the CPU 201 of the client terminal 101 determines whether it has completed the processing in S513 to S515 for all annotation rectangles contained in the annotation file read in S510. If there are still annotation rectangles that have not been processed, it returns to S513. On the other hand, if it has completed processing for all annotation rectangles, the CPU 201 of the client terminal 101 proceeds to step S516.

[0067] In step S516, the CPU 201 of the client terminal 101 lists the annotation rectangles processed in the loop of S512 whose overlap level is less than the threshold entered in S504. If such annotation rectangles can be listed, the CPU 201 increments the number of images that can be improved (initial value 0).

[0068] Next, in step S517, the CPU 201 of the client terminal 101 uses the listing results from S516 to present the results of the rectangle enclosure inspection as shown in Figure 14, 2010. For example, the CPU 201 of the client terminal 101 presents information as shown in Figure 14, 2012 for each of the annotation rectangles listed in S516. Specifically, the CPU 201 of the client terminal 101 presents information such as the rectangle ID (2020), the file name of the corresponding image (2021), the rectangle image (2030), the position of the rectangle relative to the entire corresponding image (2031), the rectangle (2033), the associated detected rectangle (2034), and judgment 2040 (information suggesting improvements (possibility of corrections) regarding the overlap and the way the annotation rectangle is enclosed).

[0069] Next, the CPU 201 of the client terminal 101 determines whether it has completed the processes in S509 to S517 for all images in the image folder acquired in S506. If there are still unprocessed images, it returns to S509. On the other hand, if it has completed processing for all images, the CPU 201 of the client terminal 101 terminates the process in this flowchart. Then, in S406 of Figure 4, the CPU 201 of the client terminal 101 displays the number of images that can be improved, as shown in 2011 of Figure 14.

[0070] As described above, in the rectangle encapsulation inspection of this embodiment, the detected rectangle is compared with the annotation information to determine whether the annotation information satisfies predetermined conditions (for example, the degree of overlap is above a threshold). If the predetermined conditions are not met, it is determined that the annotation information is not set within the appropriate range, and the annotation information that is not set within the appropriate range (inappropriate rectangle encapsulation) is picked out and clearly presented, and the number of images that can be improved and the number of images that can be improved are displayed as the judgment result. This allows the user to easily determine how many images in the dataset can be improved and how to improve them by presenting the correction suggestions. As a result, improvements in the way rectangles are encapsulated in annotations can be expected. In the above embodiment, whether or not the annotation information is set within the appropriate range was determined by the degree of overlap, but it may be determined by other methods.

[0071] <Checking for label errors> Next, we will explain the label error inspection process for S405 in Figure 4. Figures 21 and 22 are diagrams illustrating the inspection for label errors. In the "label error check" of this embodiment, all annotation rectangles corresponding to image files are extracted from the image files included in the dataset to be checked (a dataset like the one in Figure 21(a)) and stored in folders for each class (Figure 21(b)). This process is also performed for other image files in the dataset. Furthermore, an image classification AI is used to learn and infer from each of the extracted images through cross-validation to obtain the confidence level of each image (Figure 22(a)). Furthermore, based on the confidence level information, images that are thought to have label errors are picked out (Figure 22(b)).

[0072] The following explanation will use the flowchart in Figure 6. Figure 6 is a flowchart showing an example of the label error inspection process in this embodiment. Specifically, the processing in this flowchart is performed by the input reception unit 300, annotation quality inspection unit 320, and result output unit 370 of the client terminal 101. In other words, it is achieved by the CPU 201 of the client terminal 101 loading a program stored in ROM 202 or external memory 211 into RAM 203 and executing it.

[0073] First, in step S601, the CPU 201 of the client terminal 101 accepts the user's selection input of the dataset ID to be inspected in the dataset ID input section 2102 of the label error inspection screen 2000-2 shown in Figure 20. The dataset ID input section 2002 is a pull-down menu, and the user can select any dataset ID from the pull-down menu, but they may also directly input the dataset ID using a keyboard or the like. When a dataset ID is entered into the dataset ID input unit 2002, the CPU 201 proceeds to step S602.

[0074] In step S602, when the CPU 201 of the client terminal 101 detects that the test execution button 2005 has been pressed, it proceeds to step S603.

[0075] In step S603, the CPU 201 of the client terminal 101 retrieves a dataset corresponding to the dataset ID entered in S601 from the database 340 or file storage 350, and retrieves an image folder from the dataset. Furthermore, in step S604, the CPU 201 of the client terminal 101 retrieves the annotation folder from the above dataset.

[0076] Next, in step S605, the CPU 201 of the client terminal 101 repeats the process in step S606 for the number of image files contained in the image folder acquired in S603 above.

[0077] In step S606, the CPU 201 of the client terminal 101 retrieves an unprocessed image file (hereinafter referred to as "image") from the image folder, extracts a rectangular portion from the image corresponding to the rectangular information contained in the annotation file corresponding to the image, and saves the extracted rectangular image for each class name (i.e., label) contained in the annotation file. For example, it saves them to external memory 211, etc.

[0078] Next, the CPU 201 of the client terminal 101 determines whether it has completed the process in S606 for all images in the image folder acquired in S603. If there are still unprocessed images, it returns to S606. On the other hand, if it has completed processing for all images, the CPU 201 of the client terminal 101 proceeds to step S607.

[0079] In step S607, the CPU 201 of the client terminal 101 obtains label-specific confidence information based on cross-validation, determining which label each extracted rectangular image corresponds to (which class name assigned by annotation). Specifically, the training dataset is split into training and evaluation sets. Label-specific confidence information is obtained by inferring on the evaluation dataset using an image classification AI trained on the training dataset. Confidence information is obtained for all images in the training dataset by iterating through the training and evaluation datasets in sequence. For example, confidence information for image B is obtained when input to a trained model trained on images A, C, and D, and then confidence information for image A is obtained when input to a trained model trained on images B, C, and D. This process is repeated to obtain confidence information for all images.

[0080] Next, in step S608, the CPU 201 of the client terminal 101 creates a list of rectangular images that appear to have label errors from among the extracted rectangular images, based on the confidence information obtained in S607. Here, images with low confidence levels for the labels (class names assigned by annotation) attached to them (where there are other labels with higher confidence levels) are listed as rectangular images that appear to have label errors. At this time, the label with the highest confidence level among the label-specific confidence levels of the rectangular images that appear to have label errors is associated with the rectangular images that appear to have label errors as the recommended label.

[0081] Next, in step S609, the CPU 201 of the client terminal 101 uses the list created in S607 to present the label error check results as shown in Figure 20, 2110. For example, the CPU 201 of the client terminal 101 presents the number of rectangular images included in the list created in S608 as the number of rectangles that are suspected to have label errors, as shown in Figure 20, 2111. The CPU 201 of the client terminal 101 also presents information for each of the rectangular images in the list that are suspected to have label errors, as shown in Figure 20, 2112. In other words, for each rectangular image that appears to have a label error, the CPU 201 of the client terminal 101 presents information such as the rectangle ID (2120), the corresponding image file name (2121), the rectangular image (2130), the position of the rectangle relative to the entire image file (2131), the judgment 2140 (including the presentation of the label attached by annotation and the presentation of a recommended label), an example of the image with the label attached by annotation (2141), and an example of the image with the recommended (estimated) label (2142). After the processing in S609 described above, the CPU 201 of the client terminal 101 terminates the processing of this flowchart. Alternatively, the CPU 201 of the client terminal 101 may present the information in S609 in S406 of Figure 4.

[0082] As described above, in the label error check of this embodiment, images of the target object are extracted from the images in the dataset and classified according to the label assigned to the target object. Furthermore, the confidence level for each label of the extracted target object is calculated. Then, based on the labels assigned to the target object and the confidence levels for each label, it is determined whether the label assigned to the target object is appropriate or not. In this determination, if the confidence level of the label assigned to the target object is low (for example, lower than a predetermined confidence level, or there is a label with higher confidence), it is determined to be inappropriate. If it is determined that an inappropriate label has been assigned, the label that was determined to be inappropriate and the extracted image of the target object to which the inappropriate label was assigned are displayed. Furthermore, the recommended label (the label with the highest confidence level) is displayed. In addition, images related to the inappropriate label and images related to the recommended label are also displayed. This makes it easy to check for label errors in annotation and to determine the need for label correction. Furthermore, by displaying images related to the inappropriate label and the recommended label, it becomes possible to visually determine which labels should be corrected. In other words, if the judgment result is displayed only in text, it becomes necessary to check what label should be corrected if label names such as "Harvest Period 1" or "Harvest Period 2" are assigned. Therefore, displaying an image makes it easier to determine whether label correction is necessary. As a result, improvements in annotation labeling can be expected.

[0083] [AI model learning and evaluation processing] Figure 7 is a flowchart showing an example of the AI ​​model learning and evaluation process in this embodiment. The processing in this flowchart is executed by the input receiving unit 300, the AI ​​learning and evaluation unit 330, and the result output unit 370 of the client terminal 101. Specifically, it is achieved by the CPU 201 of the client terminal 101 loading a program stored in the ROM 202 or external memory 211 into the RAM 203 and executing it.

[0084] When the CPU 201 of the client terminal 101 detects that "AI Learning and Evaluation" has been selected from the [Screen Switching] 1001 and the screen switching menu 1002 described above, it displays the AI ​​model learning and evaluation screen 2200, as shown in Figure 23, on the display 210 of the client terminal 101 and starts processing the flowchart shown in Figure 7.

[0085] In step S701, the CPU 201 of the client terminal 101 accepts input of learning and evaluation parameters from the user on the AI ​​model learning and evaluation screen 2200. Figure 23 shows an example of the AI ​​model learning and evaluation screen 2200. The AI ​​model learning and evaluation screen 2200 is provided with a parameter input section 2201 for AI learning and evaluation. In the parameter input section 2201 for AI learning and evaluation, it is possible to input parameter values ​​2220 corresponding to the parameter names 2210 used during AI learning and evaluation. For example, parameters such as "batch size," "epoch," and "dataset splitting" can be input.

[0086] Batch size is a parameter that indicates the size of each individual batch when the dataset is divided. The epoch is a parameter that indicates the number of training iterations. The "Dataset Splitting" parameter indicates whether or not to split the dataset. Enter "True" to split the dataset, and "False" to not split it. The input field for the "Dataset Splitting" parameter is a pull-down menu, and the user selects either "True" or "False" from the pull-down menu.

[0087] Once the parameters for AI learning and evaluation are input, CPU201 proceeds to step S702. In step S702, when the CPU 201 of the client terminal 101 detects that the learning / evaluation execution button 2230 has been pressed, it proceeds to step S703.

[0088] In step S703, the CPU 201 of the client terminal 101 determines whether the "dataset splitting" parameter entered in S701 is "True".

[0089] If the "dataset splitting" parameter is set to "True" (i.e., S703 is Yes), the CPU 201 of the client terminal 101 proceeds to step S704.

[0090] In step S704, the CPU 201 of the client terminal 101 divides the entire training dataset based on the parameters entered in S701 and generates the divided training datasets. Here, for example, it is also possible to generate a training dataset that is less than 50% of the original training dataset, and a training dataset that is between 50% and 100% of the original training dataset. In the example in Figure 24(a), a training dataset of 10% (3301) and a training dataset of 50% (3302) are generated. Figure 24(a) illustrates the process of splitting the dataset for training.

[0091] Next, in step S705, the CPU 201 of the client terminal 101 performs the processing in steps S706 to S707 for each of the training datasets generated in S704 (including the entire original training dataset).

[0092] First, in step S706, the CPU 201 of the client terminal 101 obtains one unprocessed training dataset from the training dataset divided in S704, and trains the AI ​​model using the obtained training dataset based on the parameters entered in S701. In the example in Figure 24(a), the AI ​​model is trained using either the divided training dataset 3301, the divided training dataset 3302, or the entire training dataset 3300. The AI ​​generated through this training is managed by an AI job number (identifier) ​​to uniquely identify the AI. The proportion of the training dataset used for training to the entire training dataset is also managed in a way that allows for identification. Furthermore, the AI ​​trained using each of the training datasets divided from the entire training data is managed in association. In the example in Figure 24(a), the AI ​​trained using the entire training dataset, the AI ​​trained using 10% of the training dataset, and the AI ​​trained using 50% of the training dataset are managed in association.

[0093] Furthermore, in step S707, the CPU 201 of the client terminal 101 evaluates the AI ​​model trained in S706. In the example of Figure 24(b), object detection is performed on the evaluation dataset 3330 using AI 3311 trained on the divided training dataset 3301 to calculate the value of the accuracy evaluation index (3321), object detection is performed on the evaluation dataset 3330 using AI 3312 trained on the divided training dataset 3302 to calculate the value of the accuracy evaluation index (3322), or object detection is performed on the evaluation dataset 3330 using AI 3313 trained on the entire training dataset 3300 to calculate the value of the accuracy evaluation index (3323).

[0094] In the AI ​​evaluation phase, the accuracy of the AI's predictions is determined by comparing the object detection results with the annotations included in the evaluation dataset 3330. For example, all cases like the following are identified and counted. Here, we will explain using the detection of "flowers" as an example. TP (True Positive): The AI ​​predicts that the answer is "flower," and that prediction is correct (correct answer). TN (True Negative): When the AI ​​predicts that it is not a "flower," and that prediction is correct. FP (False Positive): When the AI ​​predicts something is a "flower," but that prediction is incorrect (false positive). FN (False Negative): The AI ​​predicts that something is not a "flower," and that prediction is incorrect (missed).

[0095] Furthermore, using the count results mentioned above, we calculate the value of the AI ​​accuracy evaluation index. Examples of AI accuracy evaluation metrics include accuracy, precision, recall, F-score (F1-score), MAE, MSE, RMSE, and correlation coefficient (contribution rate). Here, we will show, as an example, how to calculate accuracy, precision, and recall. Accuracy = (TP+TN) / (TP+FP+TN+FN) Precision = TP / (TP+FP) Recall = TP / (TP+FN)

[0096] The CPU 201 of the client terminal 101 determines whether it has completed the processing in S706 to S707 for all of the training datasets that were divided in S704 (including the entire training dataset before division). If there are still divided training datasets that have not been processed, it returns to S706. On the other hand, if it has completed processing for all of the divided training datasets, the CPU 201 of the client terminal 101 proceeds to step S710.

[0097] Furthermore, in loop 1 of S705 above, if training is performed on 10% of the training dataset, 50% of the training dataset, and the entire training dataset, it is also possible to perform training up to 10% of the training dataset in the first loop, further training in the second loop to perform training up to 50% of the training dataset by adding more training to the training up to 10% of the training dataset, and further training in the third loop to perform training on the entire training dataset by adding more training to the training up to 50% of the training dataset.

[0098] The method for splitting the training dataset is flexible; for example, it can be split randomly, or the balance of each label (the proportion and number of each label in each training dataset after splitting) can be adjusted so that the number of images with each label is uniform. In other words, if images labeled "flower" are concentrated in one training dataset, sufficient accuracy may not be achieved in subsequent steps related to evaluating the AI ​​model. Therefore, the training dataset may be split to adjust the balance of labels. It is also possible to set up notifications if there is a bias in the balance of labels. For example, notifications could be issued if the majority of images in 10% of the training dataset are labeled "flower," or if only 3 images in 50% of the training dataset are labeled "strawberry."

[0099] Furthermore, the label balance of the training dataset can be displayed on the screen for the user to review before splitting. For example, the display could show that 10% of the training dataset contains 10 images labeled "flower" and 15 images labeled "strawberry," making it easily recognizable. Additionally, for each split training dataset, images categorized by label can be displayed in a list, and the system can accept a predetermined operation (e.g., dragging) to move images between the split training datasets. For example, for 10% of the training dataset, images labeled "flower" and images labeled "strawberry" can be displayed in separate lists. The same format can be used for 50% of the training dataset. By accepting an operation on an image labeled "strawberry" in 10% of the training dataset, it can be moved to 50% of the training dataset. Since accepting user operations may affect the splitting ratio (for example, if the training dataset becomes 13% instead of 10% due to image movement), a warning may be issued if the predetermined ratio is not met.

[0100] You may also allow the user to specify the number of divisions. For example, if 10 is specified, the training dataset will be divided into 10 parts.

[0101] On the other hand, if the "dataset splitting" parameter in step S703 is not "True" (i.e., S703 is No), the CPU 201 of the client terminal 101 proceeds to step S708.

[0102] In step S708, the CPU 201 of the client terminal 101 trains the AI ​​model using the entire training dataset based on the parameters entered in S701. For example, as shown in Figure 24(a), the AI ​​model is trained using the entire training dataset 3300.

[0103] Furthermore, in step S709, the CPU 201 of the client terminal 101 evaluates the AI ​​model trained in S708. For example, as shown in Figure 24(b), the AI ​​3313 trained on the entire training dataset 3300 is used to perform object detection on the evaluation dataset 3330 and calculate the value of the accuracy evaluation index (3323). The calculation of the accuracy evaluation index is the same as in the case of splitting described above, so the explanation is omitted.

[0104] After the processing in step S709 described above, the CPU 201 of the client terminal 101 proceeds to step S710.

[0105] In step S710, the CPU 201 of the client terminal 101 saves the AI ​​learning and evaluation output generated by the above process to the database 340 or file storage 350. For example, as AI learning and evaluation output, as shown in Figure 25(a), the trained AI (AI model), a file of parameters set during learning and evaluation, a detection rectangle list file for the evaluation dataset of the trained AI, and an accuracy file for the evaluation dataset are saved.

[0106] Figure 25(b) shows an example of a precision file for an evaluation dataset. The CPU 201 of client terminal 101 saves the values ​​of each accuracy evaluation metric calculated for each class as shown above, as an accuracy file for the evaluation dataset shown in Figure 25(b).

[0107] The following provides a more detailed explanation of the above-mentioned S710. When the dataset is split, in the example in Figure 24, in S710, the trained AIs 3311, 3312, and 3313 are saved as trained AIs. Also, a file consisting of the parameters entered in S701 is saved as a parameter file set during training and evaluation. Furthermore, a list of detected rectangles for the evaluation dataset of the trained AIs is saved as a list of detected rectangles for the evaluation dataset of trained AI 3311, a list of detected rectangles for the evaluation dataset of trained AI 3312, and a list of detected rectangles for the evaluation dataset of trained AI 3313. Finally, an accuracy file for the evaluation dataset is saved as an accuracy file for the evaluation dataset of trained AI 3311, an accuracy file for the evaluation dataset of trained AI 3312, and an accuracy file for the evaluation dataset of trained AI 3313.

[0108] On the other hand, if the dataset is not split, in S710, the trained AI 3313 is saved as the trained AI. Also, a file consisting of the parameters entered in S701 is saved as the parameter file set during training and evaluation. Furthermore, a list of detected rectangles for the evaluation dataset of the trained AI 3313 is saved as a detection rectangle list file for the evaluation dataset of the trained AI. Finally, a precision file for the evaluation dataset of the trained AI 3313 is saved as a precision file for the evaluation dataset.

[0109] Furthermore, information corresponding to the deliverables of the above AI learning and evaluation is stored in database 340 or file storage 350, linked to the AI ​​job number of each AI model. In addition, in S710, information linking the dataset ID of the evaluation dataset used for evaluating the above AI with the AI ​​job number of the IA is also stored in database 340 or file storage 350.

[0110] After the processing in step S710, the CPU 201 of the client terminal 101 terminates the processing of this flowchart. As described above, AI models can be easily trained and evaluated, and the results of the training and evaluation can be saved.

[0111] [AI Quality Inspection Process] Figure 8 is a flowchart showing an example of the AI ​​quality inspection process in this embodiment. The processing in this flowchart is executed by the input reception unit 300, the AI ​​quality inspection unit 360, and the result output unit 370 of the client terminal 101. Specifically, it is achieved by the CPU 201 of the client terminal 101 loading a program stored in the ROM 202 or external memory 211 into the RAM 203 and executing it.

[0112] When the CPU 201 of the client terminal 101 detects that "AI Quality Inspection" has been selected from the [Screen Switching] 1001 and the screen switching menu 1002 described above, it displays the AI ​​Quality Inspection screen 2300, as shown in Figure 26, on the display 210 of the client terminal 101 and starts processing the flowchart shown in Figure 8.

[0113] In step S801, the CPU 201 of the client terminal 101 accepts input from the user regarding the analysis mode on the AI ​​quality inspection screen 2300. The analysis mode selection unit 2301 of the AI ​​quality inspection screen 2300 allows the user to select one of the following analysis modes: "Confirmation of detection results," "Comparison of AI models," "Accuracy according to rectangle size," or "Relationship between training data amount and accuracy." Selecting "Confirmation of detection results" switches to the AI ​​quality inspection screen (Confirmation of detection results screen 2300-1) as shown in Figure 26. Selecting "Comparison of AI models" switches to the AI ​​quality inspection screen (Comparison of AI models screen 2300-2) as shown in Figure 29. Selecting "Accuracy according to rectangle size" switches to the AI ​​quality inspection screen (Accuracy according to rectangle size screen 2300-3) as shown in Figure 31. Selecting "Relationship between training data amount and accuracy" switches to the AI ​​quality inspection screen (Relationship between training data amount and accuracy screen 2300-4) as shown in Figure 33.

[0114] When an analysis mode is input to the analysis mode selection unit 2301, the CPU 201 proceeds to step S802.

[0115] In step S802, the CPU201 determines whether the input analysis mode is "confirmation of detection results". If the entered analysis mode is "Confirm detection results" (Yes in S802), the CPU 201 switches to the detection results confirmation screen 2300-1 and proceeds to step S803.

[0116] In step S803, the CPU 201 performs a detection result verification process. Details of the detection result verification process are shown in Figure 9, which will be described later. Once the detection result verification process in S803 is completed, the CPU 201 proceeds to step S810. In this case, in step S810, the CPU 201 displays the result of the detection result verification on the display 210.

[0117] On the other hand, if the input analysis mode is not "Confirm detection results" (i.e., No in S802), CPU201 proceeds to step S804.

[0118] In step S804, the CPU201 determines whether the input analysis mode is "AI model comparison". If the input analysis mode is "AI model comparison" (Yes in S804), CPU201 switches to the AI ​​model comparison screen 2300-2 and proceeds to step S805.

[0119] In step S805, CPU201 performs a comparison process of AI models. Details of the AI ​​model comparison process are shown in Figure 10, which will be described later. Once the AI ​​model comparison process in S805 is complete, CPU201 proceeds to step S810. In this case, in step S810, the CPU 201 displays the results of the AI ​​model comparison on the display 210.

[0120] On the other hand, if the input analysis mode is not "AI model comparison" (i.e., No in S804), CPU201 proceeds to step S806.

[0121] In step S806, the CPU 201 determines whether the input analysis mode is "accuracy according to the size of the rectangle". If the input analysis mode is "accuracy according to the size of the rectangle" (Yes in S806), the CPU 201 switches to the accuracy according to the size of the rectangle screen 2300-3 and proceeds to step S807.

[0122] In step S807, the CPU 201 performs precision processing according to the size of the rectangle. Details of the precision processing according to the size of the rectangle are shown in Figure 11, which will be described later. Once the precision processing according to the size of the rectangle in S807 is completed, the CPU 201 proceeds to step S810. In this case, in step S810, the CPU 201 displays the result on the display 210 with an accuracy corresponding to the size of the rectangle.

[0123] On the other hand, if the input analysis mode is not "accuracy according to the size of the rectangle" (No in S806), the CPU201 proceeds to step S808.

[0124] In step S808, the CPU201 determines whether the input analysis mode is "relationship between amount of training data and accuracy". If the input analysis mode is "Relationship between training data volume and accuracy" (Yes in S808), CPU201 switches to the "Relationship between training data volume and accuracy" screen 2300-4 and proceeds to step S809.

[0125] In step S809, CPU201 performs processing to determine the relationship between the amount of training data and accuracy. Details of this processing are shown in Figure 12, which will be described later. Once the processing of the relationship between the amount of training data and accuracy in S809 is complete, CPU201 proceeds to step S810. In this case, in step S810, the CPU 201 displays the result of the relationship between the amount of training data and accuracy on the display 210.

[0126] On the other hand, if the input analysis mode is not "relationship between training data volume and accuracy" (No in S808), CPU201 proceeds to step S810. In this case, in step S810, the CPU 201 displays a prompt on the display 210, such as an indication prompting input for analysis mode. After processing by S810, CPU201 terminates the processing of this flowchart.

[0127] <Checking the detection results> Figure 9 is a flowchart showing an example of the process for confirming the detection result of S803 in Figure 8. The processing in this flowchart is performed by the input reception unit 300, the AI ​​quality inspection unit 360, and the result output unit 370 of the client terminal 101. Specifically, this is achieved by the CPU 201 of the client terminal 101 loading a program stored in ROM 202 or external memory 211 into RAM 203 and executing it.

[0128] First, in step S901, the CPU 201 of the client terminal 101 accepts input of an AI job number (also called an AI job ID) from the user at the AI ​​job number input section 2302 of the detection result confirmation screen 2300-1 shown in Figure 26. The AI ​​job number input section 2302 is a pull-down menu, and the user can select and input any AI job number from the pull-down menu, or they can directly input the AI ​​job number using a keyboard or the like. Using the AI ​​job number as a key, data related to the AI ​​identified by that AI job number can be obtained from the database 340 or file storage 350. AI-related data includes, for example, the dataset used for AI evaluation, and the results of AI learning and evaluation as shown in Figure 25.

[0129] When an AI job number is entered in the AI ​​job number input section 2302, the CPU 201 of the client terminal 101 proceeds to step S902.

[0130] In step S902, the CPU 201 of the client terminal 101 uses the AI ​​job number entered in S901 to retrieve an image file (hereinafter referred to as "image") from the database 340 or file storage 350 within the dataset used for the AI ​​evaluation of the AI ​​job number.

[0131] Next, in step S903, the CPU 201 of the client terminal 101 uses the AI ​​job number entered in S901 to retrieve the annotation file (hereinafter referred to as "annotation") from the database 340 or file storage 350 within the dataset used for the AI ​​evaluation of the AI ​​job number.

[0132] Next, in step S904, the CPU 201 of the client terminal 101 uses the AI ​​job number entered in S901 to retrieve a list of detection rectangles within the AI ​​learning and evaluation deliverables associated with that AI job number from the database 340 or file storage 350.

[0133] Next, in step S905, the CPU 201 of the client terminal 101 uses the information acquired in S902 to S904 to display a list on the display 210 of the client terminal 101 as an image preview of the detection results. This list shows the rectangles from the detection rectangle list acquired in S904 plotted on the image acquired in S902 (for example, a rectangle plot image like 2310 in Figure 26).

[0134] Next, in step S906, if the CPU 201 of the client terminal 101 detects that a preview image (rectangular plot image 2310, etc.) displayed in the list in S905 has been pressed, the process proceeds to step S907.

[0135] In step S907, the CPU 201 of the client terminal 101 displays the image details of the preview image pressed in S906 on the display 210 as a pop-up, for example, as shown in 2320 in Figures 27 and 28.

[0136] Figures 27 and 28 show examples of detailed image displays for preview images used to confirm detection results. Figure 27 corresponds to a display mode in which the detected object is marked with a border (rectangle), and Figure 28 corresponds to a display mode in which the detected object (for example, the center position of the detected object (e.g., the center of gravity)) is marked with a circle. In Figures 27 and 28, 2330 corresponds to the preview image pressed at S906 in Figure 9.

[0137] The image detail display 2320 has an object display method selection unit 2340, which allows the user to select a display mode, either marking the detected object with a rectangle or marking the detected object with a circle. In other words, the object display method selection unit 2340 allows the user to select a display method when displaying the detection results of objects contained in an image on the image. If display with a rectangle 2341 is selected, the CPU 201 of the client terminal 101 marks the detected object on the image 2330 with a rectangle, as shown in 2350 of Figure 27. On the other hand, if display with a circle 2342 is selected, the CPU 201 of the client terminal 101 marks the detected object on the image 2330 with a circle, as shown in 2360 of Figure 28.

[0138] When a detected object is selected, the CPU 201 of the client terminal 101 changes only the marking display of the selected detected object (changing from a rectangle to a circle, and from a circle to a rectangle). This makes it possible to change only the markings of some detected objects. Furthermore, the instruction to make this change is not limited to selecting a detected object, but may be done by other means. For example, when a detected object or the rectangle 2350 or circle 2360 that marks it is selected, the CPU 201 of the client terminal 101 displays a submenu. In this submenu, the user can select "Change to circle" or "Change to rectangle". If "Change to circle" is selected in the submenu, the CPU 201 of the client terminal 101 changes only the marking of the selected detected object to a circle. Alternatively, if "Change to rectangle" is selected in the submenu, the CPU 201 of the client terminal 101 may change only the marking of the selected detected object to a rectangle.

[0139] Furthermore, when a marking display such as rectangle 2350 or circle 2360 is selected, the CPU 201 of the client terminal 101 displays a pop-up with detailed information about the detected rectangle corresponding to the selected marking display (such as the detection result class name and prediction confidence shown in Figure 16(b)). In other words, when the detected object marked by the selected marking display is detected by the AI ​​model, the prediction result predicted by the AI ​​model and information on its confidence level are displayed in a pop-up. Moreover, the instruction to display the details is not limited to the selection of a marking display, but may be done by other means. For example, when a detected object or the rectangle 2350 or circle 2360 that marks it is selected, the CPU 201 of the client terminal 101 displays a submenu. In this submenu, "Detailed Display" can be selected. When "Detailed Display" is selected in this submenu, the CPU 201 of the client terminal 101 may display a pop-up with detailed information about the detected rectangle corresponding to the selected detected object (such as the detection result class name and prediction confidence shown in Figure 16(b)).

[0140] Furthermore, by checking "Correct Answer" and "Prediction" in the selection column 2380 of the object to be displayed, the information to be displayed can be changed. For example, when "Correct Answer" is checked, the CPU 201 of the client terminal 101 marks and displays the correctly identified object with a rectangle or a circle. Also, when "Prediction" is checked, the predicted object is marked and displayed with a rectangle or a circle. Although a circle is used as an example, as long as the position of the detected object can be understood, it may be displayed in other forms such as a square or a star.

[0141] As described above, in the confirmation of the detection result of this embodiment, the selection of the display method of the detected object by the AI is accepted, and the detected object is marked and displayed with a rectangle, a circle, or the like in the selected display method. As a result, it is possible to select whether to display the detected object as a rectangle or a circle, and the user can easily confirm the detection result of the AI in a display form that is easy to recognize.

[0142] <Comparison of AI Models> FIG. 10 is a flowchart showing an example of the comparison process of the AI model in S805 of FIG. 8. The process of this flowchart is executed by the input reception unit 300, the AI quality inspection unit 360, and the result output unit 370 of the client terminal 101. That is, it is realized by the CPU 201 of the client terminal 101 loading and executing a program stored in the ROM 202 or the external memory 211 into the RAM 203.

[0143] First, in step S1001, the CPU 201 of the client terminal 101 receives the input of the first AI job number (AI job ID) from the user in the AI job number input unit 2401 of the AI model comparison screen 2300-2 as shown in FIG. 29. Also, in step S1002, the CPU 201 of the client terminal 101 receives the input of the second AI job number (AI job ID) from the user in the AI job number input unit 2402.

[0144] The AI ​​job number input sections 2401 and 2402 are each pull-down menus, allowing the user to select and input an AI job number from one of the menus. Alternatively, the AI ​​job number can be entered directly using a keyboard or other means.

[0145] When the first AI job number is entered in the AI ​​job number input unit 2401 and the second AI job number is entered in the AI ​​job number input unit 2402, the CPU 201 of the client terminal 101 proceeds to step S1003.

[0146] In step S1003, the CPU 201 of the client terminal 101 uses the first AI job number (or second AI job number) entered above to retrieve images from the database 340 or file storage 350 within the dataset used for the AI ​​evaluation of the first and second AI job numbers.

[0147] Next, in step S1004, the CPU 201 of the client terminal 101 uses the first AI job number (or second AI job number) entered above to retrieve annotations from the database 340 or file storage 350 within the dataset used for the AI ​​evaluation of the first and second AI job numbers.

[0148] Next, in step S1005, the CPU 201 of the client terminal 101 uses the first and second AI job numbers entered above to retrieve from the database 340 or file storage 350 the detection rectangle list file within the AI ​​learning and evaluation deliverables associated with the first AI job number, and the detection rectangle list file of the AI ​​learning and evaluation deliverables associated with the second AI job number, respectively.

[0149] Next, in step S1006, the CPU 201 of the client terminal 101 compares the rectangle information of the annotation information acquired in S1004 with the rectangle information in the detection rectangle list file associated with the first AI job number acquired in S1005 to determine the accuracy of the AI ​​prediction, and counts the number of correct answers, false positives, and missed detections for each image and class acquired in S1003, and also counts the total for all classes. Similarly, the CPU 201 of the client terminal 101 compares the rectangle information of the annotation information acquired in S1004 with the rectangle information in the detection rectangle list file associated with the second AI job number acquired in S1005 to determine the accuracy of the AI ​​prediction, and counts the number of correct answers, false positives, and missed detections for each image and class acquired in S1003, and also counts the total for all classes.

[0150] Next, in step S1007, the CPU 201 of the client terminal 101 displays the rectangular plot images (2430, 2431) and the detection count transition table (2440) for each of the first and second AI jobs on the display 210 of the client terminal 101 in a default presentation order (for example, in order of image file names).

[0151] Next, in step S1008, the CPU 201 of the client terminal 101 receives a selection from the user regarding the order in which the results should be presented, in the result presentation order selection unit 2410 as shown in Figure 29. If the "order in which the number of correct answers increases" is selected as the result presentation order in the presentation order selection unit 2410, the CPU 201 of the client terminal 101 proceeds to step S1009.

[0152] Next, in step S1009, the CPU 201 of the client terminal 101 sorts the rectangular plot images and the detection count transition table for each image in order of increasing total correct answers across all classes from the first AI job to the second AI job. Next, in step S1010, the CPU 201 of the client terminal 101 displays the sorted result from S1009 on the display 210. For example, it sorts and displays as shown in Figure 30.

[0153] Furthermore, if the presentation order selection unit 2410 then selects "by image file name" as the order in which the results are presented, the CPU 201 of the client terminal 101 resumes processing from step S1006.

[0154] Furthermore, the display order is not limited to the order of image file names or the order in which the number of correct answers increases; other orders may also be allowed. For example, the display order selection unit 2410 may allow users to select "in order of decreasing number of correct answers," "in order of increasing false positives," "in order of decreasing false positives," "in order of increasing missed answers," or "in order of decreasing missed answers," and the display order may be changed according to this selection. For example, in the case of an AI that detects lesions from examination images such as CT scan images or MRI scan images, the "number of missed answers" is considered more important than the "number of correct answers" or the "number of false positives." In this way, the user can change the display order according to their purpose.

[0155] Furthermore, the AI ​​model comparison screen 2300-2 may be provided with a label selection section, allowing users to narrow down the comparison to specific labels, such as "All Labels," "Flower," "Strawberry," etc. In this case, the CPU 201 of the client terminal 101 will focus on the types of labels selected in the label selection section, calculate correct answers, false positives, missed answers, etc., and control the display to show them in the selected order.

[0156] As described above, in the comparison of AI models in this embodiment, for each of the multiple images in the dataset used for evaluation, the number of correctly detected objects, the number of false positives, the number of missed objects, etc., are calculated for each label (class) for the first AI model and the second AI model based on the inference results of the first AI model and the second AI model (i.e., for each label, the number for each condition in the inference results of each AI model is counted). Then, rectangular plot images, detection count transition tables, etc., are compared and displayed in order of the images in which the number of correctly detected objects increased (or other order is also acceptable). Note that the order in which the inference results of each AI model for each image are displayed can be specified (in order of images in which the number of correctly detected objects increased or decreased, in order of images in which the number of falsely detected objects increased or decreased, in order of images in which the number of missed objects increased or decreased), and the system controls the comparison and display of the inference results of each AI model for each image in the specified order. Furthermore, the order of images can be specified by label specification and the order of images in which the number corresponding to the specified label for each condition in the inference results of each AI model increased or decreased. This allows users to easily compare AI models and view the comparison results in their desired order.

[0157] <Accuracy based on the size of the rectangle> Figure 11 is a flowchart showing an example of precision processing according to the size of the rectangle in S807 of Figure 8. The processing in this flowchart is performed by the input reception unit 300, AI quality inspection unit 360, and result output unit 370 of the client terminal 101. In other words, it is achieved by the CPU 201 of the client terminal 101 loading a program stored in ROM 202 or external memory 211 into RAM 203 and executing it.

[0158] First, in step S1101, the CPU 201 of the client terminal 101 receives input of an AI job number (AI job ID) from the user at the AI ​​job number input section 2501 of the AI ​​quality inspection screen (accuracy screen 2300-3 according to the size of the rectangle) as shown in Figure 31. The AI ​​job number input section 2501 is a pull-down menu, and the user can select and input any AI job number from the pull-down menu, but they may also directly input the AI ​​job number using a keyboard or the like.

[0159] When an AI job number is entered in the input unit 2501, the CPU 201 of the client terminal 101 proceeds to step S1102.

[0160] In step S1102, the CPU 201 of the client terminal 101 uses the AI ​​job number entered in S1101 to retrieve images from the database 340 or file storage 350 within the dataset used for the AI ​​evaluation of the AI ​​job number.

[0161] Next, in step S1103, the CPU 201 of the client terminal 101 uses the AI ​​job number entered in S1101 to retrieve annotations from the database 340 or file storage 350 within the dataset used for the AI ​​evaluation of that AI job number.

[0162] Next, in step S1104, the CPU 201 of the client terminal 101 uses the AI ​​job number entered in S1101 to obtain a detection rectangle list file of AI learning and evaluation results associated with that AI job number from the database 340 or file storage 350.

[0163] Next, in step S1105, the CPU 201 of the client terminal 101 accepts the user's selection of a precision evaluation index in the precision evaluation index selection unit 2510, as shown in Figure 31. The precision evaluation index selection unit 2510 is a pull-down menu, and the user can select one of the precision evaluation indices (here, "Recall" and "Precision," but not limited to these) from the pull-down menu.

[0164] When a precision evaluation index is selected in the precision evaluation index selection unit 2510, the CPU 201 of the client terminal 101 proceeds to step S1106.

[0165] In step S1106, the CPU 201 of the client terminal 101 obtains a list 2550 of ranges with a rectangular size as shown in Figure 32 from the database 340 or file storage 350. Figure 32 shows an example of a list of 2550 ranges of rectangular sizes. The range for the size of the rectangle is defined by the interval start at 2251 and the interval end at 2555.

[0166] Next, in step S1107, the CPU 201 of the client terminal 101 is controlled to execute the processes of steps S1108 to S1112 for each range of rectangle sizes included in the list 2550 of rectangle size ranges obtained in S1106.

[0167] First, in step S1108, the CPU 201 of the client terminal 101 acquires one unprocessed range (hereinafter referred to as the "current range"), and then acquires the rectangle information within the annotation acquired in S1103 that falls within the current range. Next, in step S1109, the CPU 201 of the client terminal 101 acquires the rectangle information within the current range from the rectangle information within the detected rectangle acquired in S1104.

[0168] Next, in step S1110, the CPU 201 of the client terminal 101 uses the annotation information and rectangle information within the current range acquired above, and the rectangle information within the detected rectangle, to calculate the accuracy for each class based on the accuracy evaluation index selected in S1105. That is, it calculates the accuracy for each class in the current range. For example, similar to S707 and S709 in Figure 7, the CPU 201 compares the annotation information and rectangle information within the current range and the rectangle information within the detected rectangle to determine the correctness of the AI's prediction, identifies and counts all cases corresponding to TP, TN, FP, FN, etc., mentioned above for each class, and uses these to calculate the value of the selected accuracy evaluation index for each class.

[0169] Next, in step S1111, the CPU 201 of the client terminal 101 collects images of false detections from the rectangle information within the detected rectangle acquired above. That is, it collects images of false detections in the current range.

[0170] Next, in step S1112, the CPU 201 of the client terminal 101 collects images from the rectangular information within the annotations acquired above that were missed. That is, it collects images that were missed in the current range.

[0171] Then, the CPU 201 of the client terminal 101 determines whether it has completed the processing in S1108 to S1112 for all ranges included in the range list obtained in S1106. If there are still ranges that have not been processed, it returns to S1108. On the other hand, if it has completed processing for all ranges, the CPU 201 of the client terminal 101 proceeds to step S1113.

[0172] In step S1113, the CPU 201 of the client terminal 101 displays the accuracy for each range of the rectangle size on the display 210 as a graph, as shown in Figure 31, 2520.

[0173] Next, in step S1114, the CPU 201 of the client terminal 101 displays the falsely detected images (images where the prediction was incorrect) and the missed images on the display 210 as graphs, as shown in Figures 31, 2530 and 2540, for each range of the rectangle size.

[0174] Furthermore, if the accuracy evaluation index is re-selected in the accuracy evaluation index selection unit 2510, the CPU 201 of the client terminal 101 will resume processing from step S1106.

[0175] Alternatively, an AI quality inspection screen (accuracy screen 2300-3 corresponding to the size of the rectangle) as shown in Figure 31 may be provided with a range selection section for the size of the rectangle ("All", "0~1000", "1000~2000", "2000~3000", ...), and the accuracy evaluation index values, falsely detected images, missed images, etc., may be displayed only within the range of the rectangle size selected here. Furthermore, when displaying accuracy evaluation metrics for each rectangle size (graph display), accuracy evaluation metrics for certain sizes (e.g., 0 to 1000 pixels) will not be displayed.

[0176] As described above, in this embodiment, the accuracy according to the rectangle size is calculated for each rectangle size related to the object to be detected, and images with incorrect detection results (falsely detected images, missed images) are displayed for each rectangle size. Furthermore, the accuracy for each rectangle size is displayed graphically based on the calculation results. When calculating the accuracy, the number of false detections is counted by determining the rectangle size based on the rectangle size related to the inference result. On the other hand, the number of missed images is counted by determining the rectangle size based on the size of the annotation rectangle. In addition, depending on the range of rectangle size selected, incorrect images narrowed down to the selected rectangle size are displayed. For example, if "0~1000" pixels are selected, only incorrect images of "0~1000" pixels are displayed. This makes it easy for the user to check the accuracy according to the rectangle size. For example, generally, accuracy increases as the rectangle size increases, but if this trend is not observed, it may be possible that there is an anomaly in the training data of the selected AI job, which motivates the user to review the training data. Furthermore, it becomes easy to recognize the characteristics of each rectangle size, such as what kind of images are prone to false detection or missed images.

[0177] <Relationship between training data volume and accuracy> Figure 12 is a flowchart showing an example of the processing of the relationship between the amount of training data and accuracy in S809 of Figure 8. The processing in this flowchart is executed by the input reception unit 300, AI quality inspection unit 360, and result output unit 370 of the client terminal 101. In other words, it is realized by the CPU 201 of the client terminal 101 loading a program stored in ROM 202 or external memory 211 into RAM 203 and executing it.

[0178] First, in step S1201, the CPU 201 of the client terminal 101 accepts the input of an AI job number (AI job ID) from the user at the AI ​​job number input section 2601 of the AI ​​quality inspection screen (screen 2300-4 showing the relationship between the amount of training data and accuracy) as shown in Figure 33. The AI ​​job number input section 2601 is a pull-down menu, and the user can select and input any AI job number from the pull-down menu, but they may also directly input the AI ​​job number using a keyboard or the like.

[0179] When an AI job number is entered in the input unit 2601, the CPU 201 of the client terminal 101 proceeds to step S1202.

[0180] In step S1202, the CPU 201 of the client terminal 101 uses the AI ​​job number entered in S1201 to obtain accuracy evaluation metrics for each divided data volume associated with the AI ​​job number from the database 340 or file storage 350. Specifically, it obtains an accuracy file (e.g., Figure 25(b)) for the evaluation dataset within the AI ​​learning and evaluation output (e.g., Figure 25(a)) for each divided data volume, and then obtains the value of the accuracy evaluation metric for each divided data volume from the accuracy file.

[0181] Next, in step S1203, the CPU 201 of the client terminal 101 graphs the values ​​of the accuracy evaluation index for each divided data amount obtained in S1202, as shown in 2610 of Figure 33, and displays them on the display 210.

[0182] As described above, in the relationship between the amount of training data and accuracy in this embodiment, multiple training data sets with different amounts of data are generated from the training data (for example, generating 10% of the original training data and 50% of the original training data), and the accuracy of the AI ​​trained using these training data (10% of the original training data, 50% of the original training data, and the entire original training data) is controlled to be displayed as a graph related to the amount of data trained (percentage of the original training data). That is, the accuracy of the trained model trained on the datasets divided from the original training dataset is calculated for each division ratio, and the accuracy for each calculated division ratio is displayed (the correlation between the percentage of the original training dataset and the accuracy of the AI ​​model trained at that ratio is displayed as a graph). Note that in the training phase, for example, if training is performed in the order of 10% of the training dataset, 50% of the training dataset, and the entire training dataset, training may be performed up to 10% of the training dataset first, then further training may be performed on the training up to 10% of the training dataset to train up to 50%, and then further training may be performed on the training up to 50% to train up to 50%. These features allow users to easily verify the correlation between the amount of training data and the accuracy of the AI ​​model. Furthermore, by dividing the data for training, it becomes possible to indicate how much more training data is needed to achieve the desired accuracy.

[0183] Based on the functions described above, it is possible to provide a mechanism that makes it easy to recognize object detection results by AI. Furthermore, it is possible to provide a mechanism that makes it easy to assess the reliability of the AI. As a result, it is possible to appropriately support AI users and significantly improve the usability when using AI.

[0184] It should be noted that the structure and content of the various data described above are not limited to those mentioned, and it goes without saying that they can be composed of various structures and contents depending on the use and purpose. Although one embodiment has been described above, the present invention can take the form of, for example, a system, apparatus, method, program, or storage medium. Specifically, it may be applied to a system consisting of multiple devices, or to an apparatus consisting of a single device. Furthermore, any configurations combining the above embodiments are also included in the present invention.

[0185] [Other Embodiments] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions. Furthermore, the present invention may be applied to a system consisting of multiple devices or to a device consisting of a single device. The present invention is not limited to the embodiments described above, and various modifications (including organic combinations of each embodiment) are possible based on the spirit of the invention, and these are not excluded from the scope of the invention. That is, all configurations that combine the above-described embodiments and their modified forms are included in the present invention. [Explanation of Symbols]

[0186] 100 Networks 101 Client terminals 102 Servers

Claims

1. An acquisition means for acquiring the inference results of the first AI model for detecting objects and the inference results of the second AI model for detecting objects for each of multiple images, A display control means controls the display to compare and display the inference results of each AI model for each image in order of the images in which the number of correctly detected objects increased among the inference results of each AI model, An information processing system characterized by having the following features.

2. An acquisition means for acquiring the inference results of the first AI model for detecting objects and the inference results of the second AI model for detecting objects for each of multiple images, A specifying means for specifying the order in which images are displayed for each of the AI ​​models, A display control means controls the display to compare and display the inference results of each AI model for each image in the order of the specified images, An information processing system characterized by having the following features.

3. The information processing system according to claim 2, characterized in that the designation means can specify the order of the images in which the number of correctly detected objects among the inference results of each AI model increases.

4. The information processing system according to claim 2, characterized in that the designation means can specify the order of the images in which the number of falsely detected objects increases or decreases among the inference results of each AI model.

5. The information processing system according to claim 2, characterized in that the designation means can specify the order of the images in which the number of missed objects increases or decreases among the inference results of each AI model.

6. The information processing system according to claim 2, further comprising a counting means for each image, which counts the number of each condition in the inference results of each AI model.

7. The information processing system according to claim 6, characterized in that the counting means counts the number of conditions in the inference results of each AI model for each label assigned to the image.

8. The information processing system according to claim 7, characterized in that the designation means can specify the order of the images, including the designation of labels and the order in which the number of images corresponding to the designated labels for each condition in the inference results of each AI model is increased or decreased.

9. An acquisition step to obtain the inference result of the first AI model for detecting objects and the inference result of the second AI model for detecting objects for each of multiple images, A display control step that controls the display to compare and display the inference results of each AI model for each image in order of the number of images in which the number of correctly detected cases increased among the inference results of each AI model, A control method for an information processing system, characterized by having the following features.

10. An acquisition step to obtain the inference result of the first AI model for detecting objects and the inference result of the second AI model for detecting objects for each of multiple images, A specifying step for specifying the order in which images will be displayed for each of the AI ​​models mentioned above, A display control step that controls the display to compare and display the inference results of each AI model for each image in the order of the specified images, A control method for an information processing system, characterized by having the following features.

11. A program for causing at least one computer to function as an information processing system according to any one of claims 1 to 8.

Images