Information processing device, information processing method, and program

Abstract

To streamline the testing of AI models. [Solution] The information processing device according to the embodiment comprises an acquisition unit, an evaluation unit, and an output unit. The acquisition unit acquires a first output result obtained by inputting test data into a model under test, and a second output result obtained by inputting test data into a reference model related to the output of the model under test. The evaluation unit evaluates the first output result based on the relationship between the first output result and the second output result. The output unit outputs the evaluation result from the evaluation unit.

Description

Technical Field

[0001] The embodiments disclosed in the present specification and drawings relate to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] Conventionally, tests of AI (Artificial Intelligence) models use test data and correct answer data associated with the test data. It is generally stated that in the testing of AI models, it is desirable to execute as many test cases as possible. However, since correct answer labels corresponding to test data are generally generated by humans, the workload for creating the correct answer labels may be large. In addition, considerable labor was also required to check the test results.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of Invention

Problems to be Solved by the Invention

[0004] One of the problems to be solved by the embodiments disclosed in the present specification and drawings is to improve the efficiency of AI model testing. However, the problems to be solved by the embodiments disclosed in the present specification and drawings are not limited to the above-mentioned problems. The problems corresponding to the respective effects of the respective configurations described in the embodiments to be described later can also be positioned as other problems.

Means for Solving the Problems

[0005] The information processing device according to the embodiment comprises an acquisition unit, an evaluation unit, and an output unit. The acquisition unit acquires a first output result obtained by inputting test data into a model under test, and a second output result obtained by inputting test data into a reference model associated with the model under test. The evaluation unit evaluates the first output result based on the relationship between the first output result and the second output result. The output unit outputs the evaluation result from the evaluation unit. [Brief explanation of the drawing]

[0006] [Figure 1] Figure 1 shows an example of the overall configuration of the information processing system according to the first embodiment. [Figure 2] Figure 2 shows an example of a medical AI graph database according to the first embodiment. [Figure 3] Figure 3 shows an example of input and output in a test according to the first embodiment. [Figure 4] Figure 4 shows another example of input and output in a test according to the first embodiment. [Figure 5] Figure 5 shows yet another example of input and output in a test according to the first embodiment. [Figure 6] Figure 6 shows an example of output data obtained when the reference model according to the first embodiment segments the liver. [Figure 7] Figure 7 shows an example of output data obtained when the test model according to the first embodiment segments the liver cancer region. [Figure 8] Figure 8 is an image combining the liver region and the liver cancer region from the output data shown in Figures 6 and 7. [Figure 9] Figure 9 is a flowchart showing an example of the flow of a ground truth labelless test using the reference model according to the first embodiment. [Figure 10] Figure 10 shows an example of a medical AI graph database according to the second embodiment. [Figure 11]Figure 11 shows an example of a medical AI graph database according to the third embodiment. [Figure 12] Figure 12 shows an example of a medical AI graph database according to the fourth embodiment. [Figure 13] Figure 13 shows another example of input and output in a test according to the fifth embodiment. [Figure 14] Figure 14 shows another example of input and output in a test according to the sixth embodiment. [Figure 15] Figure 15 shows an example of the overall configuration of the information processing system according to the 13th embodiment. [Figure 16] Figure 16 shows an example of the lifecycle flow of the AI ​​model's continuous functional improvement according to the 13th embodiment. [Modes for carrying out the invention]

[0007] The following describes in detail embodiments of the information processing apparatus, information processing method, and program with reference to the drawings. In the drawings used in the embodiments, substantially identical or similar components are denoted by the same reference numerals. Duplication of explanation may be omitted or simplified.

[0008] (First embodiment) Figure 1 shows an example of the overall configuration of an information processing system S1 according to the first embodiment. As shown in Figure 1, the information processing system S1 includes, for example, an information processing device 100a and a medical information system 300. The information processing system S1 may be installed, for example, in a medical institution such as a hospital. Alternatively, the information processing system S1 may be constructed, for example, as a platform provided by a business operator to a user. The information processing device 100a and the medical information system 300 are connected to each other via a network N, for example, an in-hospital LAN (Local Area Network).

[0009] The medical information system 300 is a system that stores and manages medical information, and includes, for example, one or a plurality of servers or PCs (Personal Computers) and the like. The medical information stored in the medical information system 300 is, for example, clinical data. As specific examples, the clinical data may be a patient's electronic medical record, medical treatment information, interview results, test data, medical image data, diagnosis results, and the like. The medical information system 300 may be, for example, an electronic medical record system, a Hospital Information System (HIS), a Laboratory Information System (LIS), a Radiology Information System (RIS), or a medical image storage apparatus, and the like. In addition, the medical information system 300 may include a plurality of the above systems and apparatuses.

[0010] The information processing apparatus 100a is, for example, a computer such as a server or a PC.

[0011] The information processing apparatus 100a performs testing of an AI (Artificial Intelligence) model. More specifically, the information processing apparatus 100a tests the performance of an AI model to be tested, using a reference AI model and test data to which no correct label is assigned. In the present embodiment, the test performed by the information processing apparatus 100a is also referred to as a Ground-Truth-Less Test using a reference model (GTLT) that uses a reference model. Hereinafter, the AI model is simply referred to as a model.

[0012] The reference model is a model related to the test target model. In the present embodiment, as an example, the reference model is a model that indicates the reference for the output of the test target model. In the correct label-less test using the reference model, instead of using correct labels, the output result from the reference model serves as the reference for evaluating the output result of the test target model. Further, in the correct label-less test using the reference model, the same test data is input to both the reference model and the test target model, and the output result of the test target model based on the test data is compared with the output result of the reference model, thereby evaluating the output result of the test target model.

[0013] First, the hardware configuration of the information processing apparatus 100a will be described. The information processing apparatus 100a includes, for example, a NW (network) interface 110, a storage circuit 120, an input interface 130, a display 140, and a processing circuit 150.

[0014] The NW interface 110 is connected to the processing circuit 150, and controls transmission and communication of various data performed between the information processing apparatus 100a and the medical information system 300. The NW interface 110 is implemented by a network card, a network adapter, a NIC (Network Interface Controller), or the like.

[0015] The storage circuit 120 preliminarily stores various types of information used by the processing circuit 150. The storage circuit 120 also stores various programs. The storage circuit 120 is, for example, a non-volatile storage device such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), or an integrated circuit storage device that stores various types of information. In addition to HDDs, SSDs, and the like, the storage circuit 120 may be a drive device that reads and writes various information with a portable storage medium such as a CD (Compact Disc), a DVD (Digital Versatile Disc), a flash memory, or a semiconductor memory element such as a RAM (Random Access Memory). The storage circuit 120 is an example of a storage unit.

[0016] Furthermore, the memory circuit 120 stores a medical AI graph database (DB) 210, test data 310, a model under test 410, and multiple reference candidate models 420a to 420n. The model under test 410 and the multiple reference candidate models 420a to 420n are collectively referred to as the model group 40. In this embodiment, the models included in the model group 40 are medical AI models related to medicine. Hereafter, if individual reference candidate models 420a to 420n are not specified, they will simply be referred to as reference candidate models 420.

[0017] Test data 310 is the data used to test the model under test 410. Test data 310 is clinical data obtained from, for example, a medical information system 300. Test data 310 does not include the correct label (also called the true value).

[0018] The Medical AI Graph Database (DB) 210 is an example of a graph database that defines the relationships between multiple reference candidate models 420a to 420n included in the model group 40 and the model under test 410. More specifically, the Medical AI Graph Database (DB) 210 is a graph database that defines the relationships between multiple reference candidate models 420a to 420n and the model under test 410. The multiple reference candidate models 420a to 420n are candidates for reference models for testing the model under test 410.

[0019] The medical AI graph database 210 can be created, for example, by a user identifying the relationships between models based on the target areas and tasks of multiple reference candidate models 420a to 420n and the model under test 410 included in the model group 40. Alternatively, the medical AI graph database 210 may be automatically generated based on the target areas and tasks of multiple reference candidate models 420a to 420n and the model under test 410 included in the model group 40, as well as medical ontology and clinical guidelines, etc.

[0020] The medical AI graph database 210 may be generated by another device outside the information processing device 100a and stored in the memory circuit 120, or it may be generated by the information processing device 100a. In addition, the test data 310, the model under test 410, and the multiple reference candidate models 420a to 420n may be stored in the memory circuit 120 in advance, or they may be downloaded from an external source and stored in the memory circuit 120 during test execution.

[0021] The candidate reference models 420a to 420n have the same or similar data format and imaging range of the input test data 310 as the model under test 410. For example, if the model under test 410 uses still image data as input data, the models that can be adopted as reference models are those that use still image data as input data. Also, for example, if the model under test 410 uses medical image data of the subject's abdomen as input data, the models that can be adopted as reference models are those that use medical image data of the subject's abdomen as input data. If the candidate reference models 420a to 420n include models for input data with different data formats and imaging ranges of the input test data 310 of the model under test 410, the reference model selection function 153 will exclude those models from the candidates. Note that multiple medical AI graph databases 210 may be stored in the memory circuit 120 for each data format and imaging range of the input test data 310.

[0022] The data format and imaging range of the test data 310 that can be input to each model are determined based on the training data that each model has learned. In this embodiment, the reference candidate models 420a to 420n and the test target model 410 are assumed to be models that have been trained with training data that is the same or similar in data format and imaging range.

[0023] Figure 2 shows an example of a medical AI graph database 210a according to the first embodiment. The medical AI graph database 210 defines the relationships between multiple models included in the model group 40 using a network-like graph structure with three elements: nodes (v), edges (e), and node properties (P). The structure of the medical AI graph database 210 can adopt a general graph database structure.

[0024] Properties include, for example, name, task, model, and performance.

[0025] `name` is the name of the model, but in this embodiment, the target site of the model is set as the name. The target site of the model is the part of the subject that the model is trying to infer. Target sites include organs, abnormal regions, medical devices, etc. Abnormal regions include cancer, tumors, infarcts, or other lesion sites. In the example shown in Figure 2, `name` is liver, liver cancer, etc. The target site of each model is an example of the first site.

[0026] A task is a task (work) of the model. In the example shown in Figure 2, tasks include detection, segmentation, and grading.

[0027] `model` is the path to the location where the models are stored. In Figure 1, the model group 40 was stored in the memory circuit 120 of the information processing device 100a, but it may also be stored outside of the information processing device 100a.

[0028] Although not shown in Figure 2, the `performance` parameter contains information about the model's reliability. Model reliability is expressed using accuracy metrics such as IoU (Intersection over Union) and AUC (Area Under the Curve). To enable comparisons between models, the reliability of all models included in the model group 40 is pre-evaluated using the same metrics.

[0029] A node is a unit for classifying a model. In the example shown in Figure 2, nodes are defined by the body part or task of the model. For example, the medical AI graph database 210a includes a node v1 for the liver, a node v2 for liver cancer, and a node v3 for liver cancer stages.

[0030] The liver node v1 has two models registered: one with the name "liver" and task "detection" as property p11, and another with the name "liver" and task "segmentation" as property p12. In other words, property p11 represents a model that detects the liver from the input medical image data. Property p12 represents a model that extracts the region where the liver is depicted from the input medical image data.

[0031] Furthermore, in the liver cancer node v2, a model with name = liver cancer and task = detection is registered as property p21, and a model with name = liver cancer and task = segmentation is registered as property p22. In other words, property p21 represents a model that detects liver cancer from the input medical image data. Property p22 represents a model that extracts the region in which liver cancer is depicted from the input medical image data.

[0032] Furthermore, in node v3 for liver cancer staging, a model with name "liver cancer" and task "grading" is registered as property p31. In other words, property p31 indicates a model that identifies the stage of liver cancer from the input medical image data.

[0033] Edges indicate relationships between nodes. Generally, the edges shown in Figure 2 indicate the direction of the relationship between nodes with the direction of the arrow. In the medical AI graph database 210a, nodes to which models whose task content is a higher-level concept belong are positioned higher. For example, edge e1 indicates that the liver node v1 is positioned higher than the liver cancer node v2. Similarly, edge e2 indicates that the liver cancer node v2 is positioned higher than the liver cancer stage node v3.

[0034] Furthermore, a hierarchy of superior and inferior relationships is defined for the properties p associated with each node. For example, property p11, which indicates a model for detecting the liver, is higher in rank than property p12, which indicates a model for extracting the region where the liver is depicted. The more limited the target area, the lower the rank of the model. Also, a model that grades identified target areas is lower in rank than a model that identifies the target area. Moreover, if the target area is the same, the more detailed the task, the lower the rank of the model.

[0035] Furthermore, nodes with properties that are similar in content, such as name and task, will have a closer relationship with each other in the medical AI graph database 210a. For example, the distance between the liver cancer node v2 and the liver node v1 is closer than the distance between the liver cancer stage node v3 and the liver node v1. Details on selecting a reference model using the medical AI graph database 210a will be described later.

[0036] Returning to Figure 1, the input interface 130 is implemented by a mouse, keyboard, pen tablet (combining a stylus and tablet that accept user input), trackball, switch buttons, touchpad (for input operations by touching the operating surface), touchscreen (integrating a display screen and touchpad), non-contact input circuit using an optical sensor, and audio input circuit, etc. The input interface 130 may include multiple devices that accept user operations. The input interface 130 is connected to the processing circuit 150 and converts the input operations received from the user into electrical signals and outputs them to the processing circuit 150. In this specification, the input interface is not limited to those equipped with physical operating components such as a mouse and keyboard. For example, an electrical signal processing circuit that receives electrical signals corresponding to input operations from an external input device provided separately from the device and outputs these electrical signals to the processing circuit 150 is also included as an example of an input interface.

[0037] The display 140 displays various information under the control of the processing circuit 150. For example, the display 140 outputs a screen containing the test results of the model under test 410, or a GUI (Graphical User Interface) for accepting various operations from the user. Specifically, the display 140 is a liquid crystal display or a CRT (Cathode Ray Tube) display, etc. The input interface 130 and the display 140 may be integrated. For example, the input interface 130 and the display 140 may be implemented by a touch panel. The display 140 may also be a display device provided outside the information processing device 100a. The display 140 is an example of a display unit.

[0038] The processing circuit 150 is a processor that reads programs from the memory circuit 120 and executes them to realize functions corresponding to each program. The processing circuit 150 in this embodiment includes a reception function 151, a test information acquisition function 152, a reference model selection function 153, an input function 154, a test result acquisition function 155, an evaluation function 156, and a display control function 157. The reception function 151 is an example of a reception unit. The test information acquisition function 152 is an example of a test information acquisition unit and acquisition unit. The reference model selection function 153 is an example of a selection unit. The input function 154 is an example of an input unit and test execution unit. The test result acquisition function 155 is an example of an acquisition unit, a test result acquisition unit, and a test execution unit. The evaluation function 156 is an example of an evaluation unit and a judgment unit. The display control function 157 is an example of a display control unit and an output unit.

[0039] Here, for example, the processing functions of the processing circuit 150, which are components of the processing circuit 150, such as the reception function 151, test information acquisition function 152, reference model selection function 153, input function 154, test result acquisition function 155, evaluation function 156, and display control function 157, are stored in the memory circuit 120 in the form of programs that can be executed by a computer. The processing circuit 150 is a processor. For example, the processing circuit 150 realizes the functions corresponding to each program by reading the program from the memory circuit 120 and executing it. In other words, the processing circuit 150 in the state in which each program has been read will have the functions shown in the processing circuit 150 of Figure 1. Note that in Figure 1, the processing functions performed by the reception function 151, test information acquisition function 152, reference model selection function 153, input function 154, test result acquisition function 155, evaluation function 156, and display control function 157 are realized by a single processor, but it is also possible to configure the processing circuit 150 by combining multiple independent processors, and each processor realizes the functions by executing a program. Furthermore, although Figure 1 describes a single memory circuit 120 that stores programs corresponding to each processing function, it is also possible to have multiple memory circuits distributed and the processing circuit 150 read the corresponding programs from individual memory circuits.

[0040] The above description illustrates an example in which a "processor" reads and executes programs corresponding to each function from a memory circuit, but the embodiments are not limited to this. The term "processor" refers to circuits such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an Application Specific Integrated Circuit (ASIC), and a programmable logic device (e.g., a Simple Programmable Logic Device (SPLD), a Complex Programmable Logic Device (CPLD), and a Field Programmable Gate Array (FPGA)). If the processor is a CPU, for example, it realizes its functions by reading and executing programs stored in a memory circuit. On the other hand, if the processor is an ASIC, instead of storing the program in the memory circuit 120, the function is directly incorporated as a logic circuit within the processor's circuit. In this embodiment, each processor is not limited to being configured as a single circuit; multiple independent circuits may be combined to form a single processor and realize its functions. Furthermore, the multiple components shown in Figure 1 may be integrated into a single processor to realize its functions.

[0041] The reception function 151 accepts various user operations via the input interface 130. For example, the reception function 151 accepts a user operation to instruct the start of a test. The reception function 151 may also accept a user operation to specify a model 410 to be tested from among the model group 40.

[0042] Furthermore, if the medical AI graph database 210 is generated by the information processing device 100a, the reception function 151 may accept operations by the user to input nodes, edges, and properties of the medical AI graph database 210.

[0043] The test information acquisition function 152 acquires test information related to the test to be performed. The test information includes, for example, information specifying the model under test 410. The test information may also include other test conditions. The information specifying the model under test 410 may be direct identification information of the model under test, or it may be information specifying the target part (name) or task, etc. The test information may be input by the user via the reception function 151, for example. Alternatively, the test information may be transmitted from another information processing device. The test information may also be included as supplementary information in the test data 310.

[0044] The reference model selection function 153 selects a reference model from the reference candidate models 420a to 420n. More specifically, it selects a reference model from the reference candidate models 420a to 420n registered in the medical AI graph database 210 that has a specified relationship with the model under test 410. For example, the reference model selection function 153 selects a reference model from the reference candidate models 420a to 420n registered in the medical AI graph database 210 based on the target area and task of the model under test 410 specified in the test information. A reference model is a model that presents the necessary conditions that the output of the model under test 410 must satisfy. The reference candidate models 420a to 420n are examples of multiple candidate models.

[0045] As a concrete example, the reference model selection function 153 selects a reference model from the candidate reference models 420a to 420n registered in the medical AI graph database 210 based on an evaluation metric (Reference Model Selection Score: RMSS). The reference model defines the conditions that the output results of the model under test must satisfy. In other words, the reference model selection function 153 can determine the constraints for testing through the reference model selection process.

[0046] The reference model selection function 153 calculates the evaluation index RMSS using the following formula (1). In formula (1), v, e, and p represent nodes, edges, and properties in the medical AI graph database 210. In formula (1), Reference Model represents the reference model, and Test Model represents the model under test.

[0047] RMSS expresses the distance between nodes and the similarity of tasks as the distance between models. The reference model selection function 153 selects, for example, the reference model from among the reference candidate models 420 that belong to nodes higher than the node to which the model under test 410 belongs, which has the smallest RMSS value. In other words, the model with the smallest RMSS value among the reference candidate models 420 is the model that is closest to the model under test 410 on the medical AI graph database 210.

[0048]

number

[0049] The following equations (2) to (4) are examples of applying equation (1) to the medical AI graph database 210a shown in Figure 2.

[0050]

number

[0051] For example, as shown in equation (2), suppose the model under test 410 specified in the test information is a model with the name "liver cancer" and the task "segmentation" corresponding to property p22 registered in the medical AI graph database 210a in Figure 2. In this case, the reference model selection function 153 determines which of the two reference candidate models 420 belonging to a node higher than the model under test 410 is closer to the model under test 410.

[0052] Equation (3) above calculates the RMSS, which represents the distance on the medical AI graph database 210a between the model under test 410 and the reference candidate model 420, where the name corresponding to property p11 is liver and the task is detection.

[0053] Furthermore, equation (4) above calculates the RMSS, which represents the distance on the medical AI graph database 210a between the test model 410 and the reference candidate model 420, whose name corresponds to property p12 and whose task is segmentation.

[0054] The reference candidate model 420 corresponding to property p11 has a different task than the model under test 410, but the reference candidate model 420 corresponding to property p12 has the same task as the model under test 410. Therefore, even if the distance between nodes is the same, the RMSS value will be smaller for the reference candidate model 420 corresponding to property p12.

[0055] In this case, the reference model selection function 153 selects the reference candidate model 420 corresponding to property p12 as the reference model.

[0056] Furthermore, the reference model selection function 153 may also select a reference candidate model 420 by taking into account reliability evaluation indicators such as IoU. For example, the reliability of the reference candidate model 420 for liver segmentation corresponding to property p12 may be lower than the threshold. In such a case, if the reliability of the reference candidate model 420 for liver detection corresponding to property p11 is higher than that of the reference candidate model 420 corresponding to property p12, the reference model selection function 153 may select the reference candidate model 420 corresponding to property p11 as the reference model. The reliability threshold for selecting a reference model is not particularly limited, but may be set by the user according to the accuracy required for the test, for example.

[0057] The input function 154 inputs test data 310 to the model under test 410 and the reference model. In this embodiment, the test data 310 input to the model under test 410 and the test data 310 input to the reference model are the same.

[0058] The test result acquisition function 155 acquires output data generated from the input test data 310 from the model under test 410 and the reference model. The output data output from the model under test 410 is an example of the first output result. The output data output from the reference model is an example of the second output result. The output data output from the reference model serves as a standard for evaluating the output data of the model under test 410.

[0059] Figure 3 shows an example of input and output in a test according to the first embodiment. In the example shown in Figure 3, medical image data capturing an area including the liver of the subject is used as test data 310a. Furthermore, the liver in the medical image data, which is test data 310a, shown in Figure 3, is assumed to include liver cancer. Test data 310a does not have correct labels indicating the liver and liver cancer regions. The liver is an example of the first site.

[0060] The test model 410a shown in Figure 3 is a model for segmenting liver cancer (name: liver cancer, task: segmentation).

[0061] The reference model 430a shown in Figure 3 is a model for segmenting the liver (name: liver, task: segmentation).

[0062] Furthermore, in the example shown in Figure 3, the IoU of the reference model 430a is 98%, while the IoU of the model under test 410a is 60%, indicating that the reliability of the reference model 430a is higher than that of the model under test 410a.

[0063] The model under test 410a receives test data 310a as input and outputs output data 61a, which is segmented from the liver cancer region 601a in the test data 310a. The liver cancer region is an example of an abnormal region.

[0064] The reference model 430a receives test data 310a as input and outputs output data 62a, which is segmented from the liver region 602a depicted in the test data 310a. The liver region is an example of a reference region.

[0065] The evaluation function 156 evaluates the first output result based on the relationship between the first and second output results. More specifically, the evaluation function 156 evaluates the first output result of the model under test 410a by comparing the output result of the model under test 410a based on test data 310a with the output result of the reference model 430a based on test data 310a. In other words, the evaluation function 156 evaluates the output result of the model under test 410a based on the output result of the reference model 430a. For example, as shown in equation (5) below, the evaluation function 156 evaluates the first output result as a success if the output from the reference model 430a includes the output from the model under test 410a. Also, the evaluation function 156 evaluates the first output result as a failure if the output from the reference model 430a does not include the output from the model under test 410a. The evaluation function 156 is also called a judgment function because it determines whether the test is a success or a failure.

[0066]

number

[0067] In Figure 3, the evaluation function 156 determines that the test target model 410a and the reference model 430a target the same organ (liver), and as shown in image 70a, the liver cancer targeted by the test target model 410a is contained within the liver targeted by the reference model 430a. In this relationship, if the abnormal region identified in the first output result is contained within the reference region identified in the second output result, the evaluation function 156 evaluates the test for the test target model 410a as successful. In Figure 3, the liver cancer region 601a segmented by the test target model 410a is contained within the liver region 602a segmented by the reference model 430a, so the evaluation function 156 evaluates the test for the test target model 410a as successful.

[0068] Note that image 70a is an example to show the relationship between the output results of the test model 410a and the reference model 430a, and image 70a does not necessarily need to be generated in the actual test.

[0069] Since the test data 310a does not contain correct labels, we will not verify whether the liver cancer region 601a segmented by the test model 410a is actually liver cancer. However, since it was verified that the liver cancer region 601a output by the test model 410a is located within the liver region 602a output by the highly reliable reference model 430a, it can be inferred that the output data 61a of the test model 410a is highly reliable.

[0070] Figure 4 shows another example of input and output in a test according to the first embodiment. In the example shown in Figure 4, the model under test 410a receives test data 310a as input and outputs output data 61b which is segmented to represent the liver cancer region 601b in the test data 310a.

[0071] In Figure 4, as shown in image 70b, the liver cancer region 601b segmented by the test model 410a is not included in the liver region 602a segmented by the reference model 430a. Therefore, the output data 61b, which is the first output result of the test model 410a, is unreliable. In this case, the evaluation function 156 evaluates the test against the test model 410a as a failure.

[0072] Figure 5 shows yet another example of input and output in a test according to the first embodiment. In the example shown in Figure 5, the model under test 410a receives test data 310a as input and outputs output data 61c which is segmented to represent the liver cancer region 601c in the test data 310a.

[0073] In Figure 5, as shown in image 70c, the liver region 602a segmented by the reference model 430a only partially contains the liver cancer region 601c segmented by the test model 410a, and does not encompass the entire region. Therefore, the reliability of the output data 61c of the test model 410a is low. In this case, the evaluation function 156 evaluates the test against the test model 410a as a failure.

[0074] More specifically, the evaluation function 156 may evaluate the output results of the model under test 410 based, for example, on equations (6) and (7) below. The threshold in equation (7) is not particularly limited, but may be set by the user depending on the strictness required for the test.

[0075]

number

[0076] Furthermore, if the reference model 430a outputs data that does not include the target area of ​​the model under test 410, the evaluation function 156 includes in its evaluation whether the outputs of the reference model 430a and the model under test 410 are in contact.

[0077] For example, Figure 6 shows an example of output data 62b obtained by segmenting the liver using the reference model 430a according to the first embodiment. In Figure 6, the reference model 430a is assumed to be a model that has been trained to segment only the normal region of the liver as "liver". Therefore, in the output data 62b shown in Figure 6, unlike the output data 62a shown in Figures 3 to 5, only the liver region 602b excluding liver cancer is segmented.

[0078] Figure 7 shows an example of output data 61d obtained when the test target model 410a according to the first embodiment segments the liver cancer region 601d. Figure 8 is an image 70d combining the liver region 602b and the liver cancer region 601d in the output data 61d shown in Figures 6 and 7. As shown in image 70d, the liver region 602b is located surrounding the liver cancer region 601d, and the outline of the liver cancer region 601d is adjacent to the outline of the liver region 602b. In this case, the evaluation function 156 may evaluate the test result as successful.

[0079] The evaluation function 156 may evaluate the test result as successful if the entire outline of the liver cancer region 601d is in contact with the outline of the liver region 602b, or if the proportion of the entire outline of the liver cancer region 601d that is in contact with the outline of the liver region 602b is greater than or equal to a specified threshold.

[0080] Note that image 70d is an example to show the relationship between the output results of the test model 410a and the reference model 430a, and image 70d does not necessarily need to be generated in the actual test.

[0081] Returning to Figure 1, the display control function 157 outputs the evaluation results from the evaluation function 156. The evaluation results indicate whether the test for the model under test 410a was successful or unsuccessful. For example, the display control function 157 displays the test failure or success on the display 140.

[0082] Furthermore, the display control function 157 may display the reliability of the output results of the test target model 410a on the display 140, based on the evaluation of the test results.

[0083] Furthermore, if multiple tests are performed, the display control function 157 may display the test failure rate or success rate on the display 140.

[0084] Furthermore, the method for outputting the evaluation results is not limited to displaying them on the display 140. For example, the information processing device 100a may be equipped with a transmission function to transmit the evaluation results from the evaluation function 156 to another information processing device. The transmission function is an example of an output unit and a transmission unit.

[0085] Next, we will explain the flow of the correct answer labelless test performed by the information processing device 100 configured as described above.

[0086] Figure 9 is a flowchart showing an example of the flow of a ground truth labelless test using a reference model according to the first embodiment. This flowchart starts, for example, when the reception function 151 receives a test start operation from the user. It is also assumed that before the start of this flowchart, the medical AI graph database 210, test data 310, and model group 40 are stored in the memory circuit 120 of the information processing device 100a.

[0087] First, the test information acquisition function 152 acquires test information related to the test to be performed (S1). If the test information is included as supplementary information in the test data 310, the test information acquisition function 152 may acquire the test information from the test data 310 stored in the memory circuit 120. Alternatively, the test data 310 may not be stored in the memory circuit 120 at the start of this flowchart and may be acquired in the process of S1.

[0088] Then, the reference model selection function 153 selects a reference model 430a from the reference candidate models 420a to 420n registered in the medical AI graph database 210, based on the target area and task of the test target model 410 specified in the test information (S2).

[0089] Then, the input function 154 inputs the test data 310 to the selected reference model 430a and the model under test 410 identified by the test information (S3).

[0090] The test result acquisition function 155 acquires the output results of the reference model 430a and the output results of the model under test 410 (S4).

[0091] The evaluation function 156 determines, for example, whether the output result from the reference model 430a encompasses the output result from the model under test 410, as shown in equation (5) above (S5).

[0092] Then, the evaluation function 156 evaluates the test result of the test model 410 as successful if the output result from the reference model 430a includes the output result from the model under test 410 (S5 "Yes").

[0093] Furthermore, if the output result from the reference model 430a does not include the output result from the model under test 410 (S5 "No"), the evaluation function 156 evaluates the test result of the model under test 410 as a failure (S7).

[0094] Then, the display control function 157 outputs the evaluation result from the evaluation function 156 (S8). At this point, the processing of this flowchart ends. Alternatively, the processing from S1 to S8 may be repeated for a predetermined number of test runs. Or, the processing from S3 to S8 may be executed for a predetermined number of test runs, with the test data 310 changed each time.

[0095] As described above, the information processing device 100a of this embodiment acquires a first output result obtained by inputting test data 310 into the model under test 410 and a second output result obtained by inputting test data 310 into the reference model 430a, and evaluates the first output result based on the relationship between the first output result and the second output result. For this reason, according to the information processing device 100a of this embodiment, testing of the model under test 410 can be performed even if the test data 310 does not have labels, and the testing of AI models can be made more efficient.

[0096] Generally, it is said that it is desirable to test AI models with as much test data as possible in order to find as many discrepancies between the requirements and the implementation results, i.e., bugs as possible. However, since the correct labels corresponding to the test data are typically generated by humans, the workload for creating these labels can be heavy. In particular, when the test data is medical data, the creation of correct labels by experts such as doctors is required, making it difficult to prepare a large number of correct labels. Furthermore, in AI model testing, not only is the preparation of test data required, but AI engineers also had to spend a lot of effort reviewing a large number of test results. In addition, in AI model testing, the difficulty of designing test specifications can be high, resulting in a heavy workload for test design. In contrast, with the information processing device 100a of this embodiment, the evaluation of test results can also be performed automatically, thus reducing the effort required for evaluating test results.

[0097] In this embodiment, the test data 310 is medical image data captured from an area including a first part of the subject (e.g., the liver). The test model 410 receives the medical image data as input and outputs a first output result that identifies an abnormal region (e.g., liver cancer) included in the first part depicted in the medical image data. In this embodiment, the reference model 430a receives the medical image data, which is the test data 310, as input and outputs a second output result that identifies a reference region in which the first part is depicted in the medical image data. The information processing device 100a of this embodiment evaluates the test on the test model 410 as successful if the abnormal region identified in the first output result is contained within the reference region identified in the second output result, or if the outline of the abnormal region touches the outline of the reference region surrounding the abnormal region. Generally, segmenting an organ is less difficult than segmenting an abnormal region within an organ, making it easier to prepare a highly reliable reference model 430a. According to the information processing device 100a of this embodiment, by utilizing a highly reliable reference model 430a, it is possible to automatically perform tests on a difficult test target model 410 without a correct answer label.

[0098] Furthermore, the information processing device 100a of this embodiment selects a reference model 430a from among the multiple reference candidate models 420 registered in the medical AI graph database 210, which defines the relationship between the multiple reference candidate models 420, which are candidates for the reference model 430a, and the model under test 410, that has a specified relationship with the model under test 410. According to the information processing device 100a of this embodiment, by using the medical AI graph database 210, the user does not need to select the reference model 430a, thus reducing the workload for testing.

[0099] Furthermore, the medical AI graph database 210 of this embodiment defines the relationship between multiple reference candidate models 420 and the target area and task of the model under test 410. The information processing device 100a of this embodiment selects a reference model 430a from the multiple reference candidate models 420 registered in the medical AI graph database 210 based on the target area and task of the model under test 410. Therefore, according to the information processing device 100a of this embodiment, an appropriate reference model 430a can be automatically identified to evaluate the first output result of the model under test 410.

[0100] (Second embodiment) In the first embodiment described above, the information processing device 100a selected the reference model 430a from among the reference candidate models 420 belonging to a node higher than the node to which the model under test 410 belongs, the one with the smallest RMSS value. However, if the number of models registered in the medical AI graph database 210 is small, the reference model 430a may be selected using simpler criteria.

[0101] The configuration of the information processing system S1 and the information processing device 100a in this embodiment is the same as in the first embodiment.

[0102] Figure 10 shows an example of a medical AI graph database 210b according to the second embodiment. The medical AI graph database 210b has a simple configuration, including only two nodes: a liver node v101 and a liver cancer node v102. Edge e101 defines the liver node v101 as higher in rank than the liver cancer node v102.

[0103] Furthermore, node v101 is associated with a model whose name is "liver" and task is "segmentation" as property p101. Similarly, node v102 is associated with a model whose name is "liver cancer" and task is "segmentation" as property p121. In other words, this medical AI graph database 210b shows the relationship between the two models.

[0104] In this case, when the reference model selection function 153 of the information processing device 100a is specified as the model under test 410 for property p121, it selects as the reference model 430a a reference candidate model 420 that corresponds to property p101 and is associated with node v101, which is located above node v102 to which property p121 belongs.

[0105] Thus, if the relationship between the model under test 410 and the reference model 430a is uniquely determined by the medical AI graph database 210b, the reference model selection function 153 may determine the reference model 430a without performing the RMSS calculation process.

[0106] In addition to the same effects as the first embodiment, the information processing device 100a of this embodiment can efficiently select a reference model 430a according to the configuration of the medical AI graph database 210b.

[0107] (Third embodiment) In the first embodiment described above, the reference model selection function 153 selected the reference model with the smallest RMSS value from among the reference candidate models 420 belonging to a node higher than the node to which the model under test 410 belongs. In contrast, in this third embodiment, the information processing device 100a selects the reference model 430a by prioritizing the reliability of the model over the higher and lower relationships between nodes in the medical AI graph database 210.

[0108] The configuration of the information processing system S1 and the information processing device 100a in this embodiment is the same as in the first embodiment.

[0109] Figure 11 shows an example of a medical AI graph database 210c according to a third embodiment. The medical AI graph database 210c includes two nodes: a liver node v201 and a liver cancer node v202. Edge e201 defines the liver node v201 as higher in rank than the liver cancer node v202.

[0110] Additionally, node v201 is associated with a model whose name is "liver," task is "segmentation," and IoU is 0.5 (50%) as property p201. Similarly, node v202 is associated with a model whose name is "liver cancer," task is "segmentation," and IoU is 0.9 (90%) as property p221.

[0111] If the model corresponding to property p201 is designated as the test model 410, then there are no models in the medical AI graph database 210c that are higher in rank than the test model 410. However, property p221 has a larger IoU value than property p201, making it more reliable. In such a case, the reference model selection function 153 of the information processing device 100a may select the model corresponding to property p221 as the reference model 430a.

[0112] Furthermore, in this case, since the liver, which is the target site of the model under test 410, encompasses liver cancer, which is the target site of the reference model 430a, the evaluation function 156 of the information processing device 100a evaluates the test of the model under test 410 as successful if the output result from the model under test 410 encompasses the output result from the reference model 430a. Conversely, the evaluation function 156 evaluates the test of the model under test 410 as unsuccessful if the output result from the model under test 410 does not encompass the output result from the reference model 430a.

[0113] In addition to the same advantages as the first embodiment, the information processing device 100a of this embodiment can also select a reference model 430a and perform tests even when there is no higher-end model of the model under test 410.

[0114] Furthermore, the reference model selection function 153, similar to the first embodiment, is based on selecting the reference candidate model 420 with the smallest RMSS value among the reference candidate models 420 belonging to a node higher than the node to which the model under test 410 belongs, as the reference model 430a, but the selection may be changed according to the accuracy of the reference model 430a.

[0115] For example, the reference model selection function 153 identifies the model with the smallest RMSS value among the candidate reference models 420 belonging to a node higher than the node to which the model under test 410 belongs as a provisional reference model. If the reliability of the identified provisional reference model is below a threshold, the reference model selection function 153 may select a model with accuracy greater than the threshold belonging to a node higher than the provisional reference model, and whose reliability is higher than the threshold, as the reference model 430a. The reliability of each model is represented, for example, by the IoU value.

[0116] In this case, the range in which the reference model selection function 153 searches for nodes higher than the provisional reference model is predetermined. The search range is determined, for example, by the distance between nodes, the similarity of the target parts of the model and the input data, the similarity of the tasks, etc.

[0117] Furthermore, the reference model selection function 153 selects the identified provisional reference model as reference model 430a if its reliability is higher than the threshold. The threshold used to determine the selection of reference model 430a may be set by the user, for example, according to the reliability required for the test.

[0118] (Fourth embodiment) In this fourth embodiment, the medical AI graph database 210 defines rules that are applied as output conditions for the model under test 410, in addition to the registration information similar to that in the first embodiment.

[0119] The configuration of the information processing system S1 and the information processing device 100a in this embodiment is the same as in the first embodiment.

[0120] Figure 12 shows an example of a medical AI graph database 210d according to the fourth embodiment. The medical AI graph database 210d includes two nodes: a liver node v301 and a liver cancer node v302. Edge e301 defines the liver node v301 as higher in rank than the liver cancer node v302.

[0121] Furthermore, node v301 is associated with a model whose name is "liver" and task is "segmentation" as property p301. Similarly, node v302 is associated with a model whose name is "liver cancer" and task is "segmentation" as property p322. In addition, node v302 is associated with the rule "The estimated pixel size must be less than or equal to XX" for the model whose name is "liver cancer" and task is "segmentation". In the rule, "estimated pixel size" refers to the pixel size of the image region segmented as the liver cancer area.

[0122] If the model under test, model 410, has property p322 named "liver cancer" and task named "segmentation," then this rule is used as a requirement for the output of model under test 410.

[0123] Furthermore, the reference model selection function 153 selects a model where the property p301 name is "liver" and the task is "segmentation" as the reference model 430a.

[0124] In this case, the evaluation function 156 evaluates the first output result of the model under test 410 based on the set rules, in addition to its relationship with the second output result of the reference model 430a. For example, the evaluation function 156 evaluates the test as successful if the first output result of the model under test 410 is included in the second output result of the reference model 430a and the first output result satisfies the set rules. The evaluation function 156 also evaluates the test as failed if the first output result of the model under test 410 is not included in the second output result of the reference model 430a, or if the first output result does not satisfy the set rules.

[0125] It should be noted that the rules that can be associated with the model are not limited to those shown in Figure 12. For example, if the model targets the medical field, various medical findings may be set as rules.

[0126] Thus, the information processing device 100a of this embodiment sets the output conditions of the model under test 410 not only by the reference model 430a but also by rules, so in addition to the same effects as the first embodiment, it is possible to perform tests with higher accuracy.

[0127] (Fifth embodiment) In the first embodiment described above, a model performing segmentation and detection tasks was used as an example, but the tasks of the models that can be tested are not limited to these. In this fifth embodiment, the testing of a glioma subtype prediction model (regression model) will be described as an example.

[0128] The configuration of the information processing system S1 and the information processing device 100a in this embodiment is the same as in the first embodiment.

[0129] Figure 13 shows another example of input and output in a test according to the fifth embodiment. In the example shown in Figure 13, the model under test 410b is a model that predicts the subtype of glioma, and the reference model 430b is a model that predicts the presence or absence of glioma.

[0130] The test data 310b in Figure 13 is unlabeled data, including biomarkers and / or images. The images included in the test data 310b may be, for example, medical image data showing the diagnostic area of ​​a glioma in a subject, or microscopic image data showing tissue taken from a subject.

[0131] Generally, gliomas are classified into clinicopathological and histological subtypes. Glioma subtypes include oligodendroglioma, astrocytoma, and glioblastoma. These subtypes may also be further subdivided. For example, astrocytoma is further classified into several subtypes, such as IDH wild-type astrocytoma, fibrillary astrocytoma, and hypertrophic astrocytoma. Estimating or identifying the subtype of a glioma is clinically important.

[0132] In the example shown in Figure 13, the AUC values ​​representing the discriminative ability of the test model 410b and the reference model 430b have been calculated in advance. Since the AUC of the reference model 430b is 98%, the reliability of the output results of the reference model 430b is high. On the other hand, since the AUC of the test model 410b is 60%, the reliability of the output results of the test model 410b is lower than that of the reference model 430b.

[0133] The reference model 430b receives the test data 310b as input and outputs an estimated result of whether or not there is a risk of glioma development in the test data 310b.

[0134] The model under test, 410b, receives the test data 310b as input and outputs the estimated subtype of glioma that will occur in the test data 310b.

[0135] In the example shown in Figure 13, the output of the reference model 430b is "no risk of glioma," while the output of the test model 410b is "risk of glioma subtype (IDH wild-type astrocytoma)."

[0136] The baseline model 430b has the capability to predict gliomas, which are a higher-level concept than the subtypes predicted by the model under test 410b. Nevertheless, the output of the baseline model 430b (second output) does not encompass the output of the model under test 410b (first output).

[0137] In this case, the evaluation function 156 determines that there is a difference between the prediction results of the reference model 430b and the model under test 410b, and evaluates the test result as a failure.

[0138] As shown in this embodiment, the information processing device 100a can apply the correct labelless test using the reference model 430b to various types of models, not just the example described in the first embodiment.

[0139] (Sixth embodiment) In the first embodiment described above, it was assumed that the reference model 430a and the model under test 410 had already been trained with training data having the same or similar data format and imaging range. However, in this sixth embodiment, a model trained with training data in a different data format than the model under test 410 may be selected as the reference model.

[0140] The configuration of the information processing system S1 and the information processing device 100a in this embodiment is the same as in the first embodiment.

[0141] Figure 14 shows another example of input and output in a test according to the sixth embodiment. In the example shown in Figure 14, the model under test 410c is a time-series prediction model for predicting heart failure. More specifically, the model under test 410c is a model created by training it with training data that associates time-series vital data with the presence or absence of heart failure one week later.

[0142] Furthermore, the baseline model 430c is a single-phase model for predicting heart failure. More specifically, the baseline model 430c is a model created by training it with training data that correlates single-phase vital data with the presence or absence of heart failure one week later.

[0143] The AUC of the reference model 430c is 0.82 (82%), so the reliability of the output results of the reference model 430c is high. On the other hand, the AUC of the test model 410c is 0.65 (65%), so the reliability of the output results of the test model 410c is lower than that of the reference model 430c.

[0144] Furthermore, the first output result from the test model 410c and the second output result from the reference model 430c in this embodiment are numerical data. More specifically, the test model 410c and the reference model 430c output numerical data representing the probability that the subject of the test data 310c will develop heart failure one week later.

[0145] The evaluation function 156 of this embodiment evaluates the test on the model under test 410c as successful if the difference between the first output result and the second output result is less than or equal to a threshold. The evaluation function 156 also evaluates the test on the model under test 410c as unsuccessful if the difference between the first output result and the second output result is greater than a threshold. The threshold used in evaluating the magnitude of the difference between the first output result and the second output result in this embodiment is an example of a first threshold. The first threshold may be set by the user, for example, according to the required accuracy of the test.

[0146] In the example shown in Figure 14, the second output of the baseline model 430c is "20% risk of heart failure after one week," while the first output of the test model 410c is "60% risk of heart failure after one week."

[0147] Furthermore, the threshold used to evaluate the magnitude of the difference between the first and second output results is, for example, 10%. In the example shown in Figure 14, the difference between the first and second output results is 40%.

[0148] In this case, the evaluation function 156 determines that there is a difference between the prediction results of the reference model 430c and the model under test 410c, and evaluates the test result as a failure.

[0149] Thus, in addition to the same effects as the first embodiment, the information processing device 100a of this embodiment can also apply a correct labelless test using the reference model 430c even when the first output result and the second output result are output as numerical data.

[0150] Furthermore, generally speaking, preparing time-series data without missing values ​​as training data is more difficult than preparing single-phase data. For this reason, generating a high-precision model using time-series data can be more difficult than generating a high-precision model using single-phase data. Therefore, according to the information processing device 100a of this embodiment, in addition to the same effects as the first embodiment, when testing the test target model 410 generated using time-series data, it is possible to facilitate the execution of the test by adopting a high-precision model using single-phase data, which is easier to generate than a high-precision model using single-phase data, as the reference model 430c.

[0151] Note that the ease of preparing training data varies depending on the user's environment for generating the model, so the relationship between time-series data and single-phase data may be reversed from the example shown in Figure 14.

[0152] (Seventh Embodiment) In the first embodiment described above, the output from the reference model 430a was in a relationship where it encompassed the output from the model under test 410, such as the liver and liver cancer. In this eighth embodiment, we will explain as an example the case where the output from the reference model 430a and the output from the model under test 410 are mutually exclusive.

[0153] The configuration of the information processing system S1 and the information processing device 100a in this embodiment is the same as in the first embodiment.

[0154] The test data 310 of this embodiment is medical image data captured from an area including a second and a third area of ​​the subject. For example, the test data 310 is medical image data of the abdomen, captured from an area including the liver and stomach. The liver is an example of the second area. The stomach is an example of the third area. Since the liver and stomach are adjacent and different organs, the area in which the liver is depicted and the area in which the stomach is depicted in the medical image data are mutually exclusive.

[0155] The test model 410 of this embodiment is, for example, a model for segmenting the region of liver cancer. Liver cancer is an example of an abnormal region included in the second site (liver).

[0156] The test model 410 receives medical image data, which is test data 310, as input and outputs a first output result that identifies the liver cancer depicted in the medical image data.

[0157] Furthermore, the reference model 430a of this embodiment is a model that segments organs other than the liver, for example. An organ other than the liver is the stomach.

[0158] The reference model 430a receives medical image data, which is test data 310, as input and outputs a second output result that identifies the reference region in which the stomach is depicted in the medical image data.

[0159] The evaluation function 156 of this embodiment evaluates the test on the model under test 410 as successful if the liver cancer identified in the first output result is not included in the reference region identified in the second output result. Conversely, the evaluation function 156 of this embodiment evaluates the test on the model under test 410 as unsuccessful if the liver cancer identified in the first output result is included in the reference region identified in the second output result.

[0160] Thus, in addition to the same effects as in the first embodiment, the information processing device 100a of this embodiment can also apply a correct answer labelless test using the reference model 430a even when the target part of the reference model 430a and the target part of the model under test 410 are mutually exclusive.

[0161] (Eighth embodiment) In the eighth embodiment described above, an example was described in which the task of the reference model 430a and the model under test 410 is segmentation, and the target parts of the reference model 430a and the target parts of the model under test 410 are mutually exclusive. In this ninth embodiment, a case is described in which the inference results of the reference model 430a and the model under test 410, which are mutually exclusive, are output as numerical data.

[0162] The configuration of the information processing system S1 and the information processing device 100a in this embodiment is the same as in the first embodiment.

[0163] The test subject model 410 of this embodiment outputs the probability of the occurrence of the first event as numerical data. The first event may be, for example, the subject of the test data 310 developing heart failure one week later, as illustrated in the sixth embodiment.

[0164] Furthermore, the reference model 430a of this embodiment outputs numerical data indicating the probability of a second event occurring, which is mutually exclusive with the first event. The second event is, for example, that the subject of test data 310 does not develop heart failure. Alternatively, the second event may be, for example, that the subject of test data 310 is discharged from the hospital after one week with no abnormal test results.

[0165] In this embodiment, the evaluation function 156 evaluates the test on the model under test 410 as successful if the difference between the first output result from the model under test 410 and the second output result from the reference model 430a is greater than or equal to a threshold.

[0166] Furthermore, the evaluation function 156 evaluates the test on the model under test 410 as a failure if the difference between the first output result from the model under test 410 and the second output result from the reference model 430a is less than a threshold. The threshold used to evaluate the magnitude of the difference between the first and second output results in this embodiment is an example of a second threshold. The second threshold may be set by the user, for example, according to the accuracy required for the test.

[0167] Thus, in addition to the same effects as the first embodiment, the information processing device 100a of this embodiment can also apply a ground truth labelless test using the reference model 430a even when the inference results, in which the reference model 430a and the model under test 410 are mutually exclusive, are output as numerical data.

[0168] (Ninth embodiment) In the first embodiment described above, the information processing device 100a performed testing on one model under test 410 using one reference model 430a. In this tenth embodiment, the information processing device 100a performs testing on one model under test 410 using multiple reference models 430a.

[0169] The configuration of the information processing system S1 and the information processing device 100a in this embodiment is the same as in the first embodiment.

[0170] The reference model selection function 153 of the information processing device 100a in this embodiment selects multiple reference models from multiple reference candidate models 420a to 420n registered in the medical AI graph database 210.

[0171] For example, suppose the model under test 410 is a model that segments liver cancer. Also, suppose the test data 310 is medical image data depicting hemangiomas and liver cancer. In this case, the reference model selection function 153 selects two models as reference models, for example, a model that segments the liver and a model that detects hemangiomas.

[0172] The reference model selection function 153 may determine the number of reference models to select based, for example, on information about the target area included in the test information, or on supplementary information of the test data 310. Furthermore, the number of reference models to select is not limited to two, but may be three or more.

[0173] The input function 154 of the information processing device 100a in this embodiment inputs test data 310 to the model under test 410 and two selected reference models.

[0174] The test result acquisition function 155 of the information processing device 100a in this embodiment acquires a first output result based on test data 310 from the model under test 410. The test result acquisition function 155 also acquires two second output results based on test data 310 from two selected reference models.

[0175] The evaluation function 156 of the information processing device 100a in this embodiment evaluates the first output result based on the relationship between the first output result of the model under test 410 and a plurality of second output results output from a plurality of reference models.

[0176] Thus, according to the information processing device 100a of this embodiment, in addition to the same effects as in the first embodiment, by using multiple reference models, the first output result of the test target model 410 can be evaluated even when the test target model 410 or test data 310 includes multiple target sites or lesions.

[0177] (Tenth embodiment) In this eleventh embodiment, the information processing device 100a modifies or adjusts the test verification conditions based on attribute information attached to the test data 310.

[0178] The configuration of the information processing system S1 and the information processing device 100a in this embodiment is the same as in the first embodiment.

[0179] If the test data 310 is medical image data, the accompanying information for the test data 310 may include metadata about the test data 310. For example, if the medical image data is in a format compliant with DICOM (Digital Imaging and Communications in Medicine), metadata can be stored as accompanying information. However, the data format of the test data 310 is not limited to this.

[0180] Metadata includes, for example, attribute information of the subject of the test data 310. In this embodiment, attribute information includes at least one of the following: gender, age, and disease information of the subject related to the test data 310. Disease information includes, for example, the approximate location of the lesion, the stage of the lesion, etc.

[0181] The reference model selection function 153 of the information processing device 100a in this embodiment selects a reference model 430a from a plurality of reference candidate models 420 registered in the medical AI graph database 210, based on attribute information attached to the test data 310.

[0182] For example, if the target area or type of lesion of the test model 410 is affected by the attribute information of the subject, the reference model selection function 153 may select the reference model 430a according to the attribute information attached to the test data 310.

[0183] As a specific example, if the target area or type of lesion targeted by the test model 410 is susceptible to influence on the inference results depending on the gender of the subject, the reference model selection function 153 will, if the test data 310 consists of images of males, select a model trained on male images from among the multiple reference candidate models 420 as the reference model 430a. Similarly, if the test data 310 consists of images of females, the reference model selection function 153 will select a model trained on female images from among the multiple reference candidate models 420 as the reference model 430a.

[0184] Furthermore, the evaluation function 156 of the information processing device 100a in this embodiment may change the evaluation criteria according to the attribute information included in the supplementary information of the test data 310.

[0185] Thus, according to the information processing device 100a of this embodiment, in addition to the same effects as in the first embodiment, the accuracy of the test can be improved by individually adjusting the test conditions based on the attribute information attached to the test data 310.

[0186] (11th embodiment) In this twelfth embodiment, the information processing device 100a analyzes the trend of the test results of the model under test 410 based on the results of multiple tests and provides feedback.

[0187] The configuration of the information processing system S1 and the information processing device 100a in this embodiment is the same as in the first embodiment.

[0188] The evaluation function 156 of the information processing device 100a in this embodiment generates information regarding test failures for the model under test 410 based on a plurality of test data 310 input to the model under test 410 and a plurality of evaluation results for a plurality of first output results output from the model under test 410 based on the input of the plurality of test data 310.

[0189] For example, the evaluation function 156 may extract commonalities in the test data 310 where the evaluation of the first output result of the model under test 410 is "failure," and generate information indicating the failure trend. Furthermore, if there are multiple reference models 430a used for comparison with the model under test 410, the evaluation function 156 may generate information indicating the trend of differences in test results due to differences in the reference models 430a.

[0190] The display control function 157 of the information processing device 100a in this embodiment outputs information about test failures for the model under test 410, which is generated by the evaluation function 156. For example, the display control function 157 may display information indicating the failure trend on the display 140. In addition, the display control function 157 may display a text message on the display 140 recommending that the model under test 410 be retrained based on the information about the test failures for the model under test 410.

[0191] Thus, according to the information processing device 100a of this embodiment, in addition to the same effects as the first embodiment, it is possible to facilitate efficient retraining according to the trend of the test results of the model 410 under test.

[0192] (12th embodiment) In this 13th embodiment, a ground truth labelless test using a reference model is incorporated and utilized in the lifecycle of continuous improvement of the AI ​​model.

[0193] Figure 15 shows an example of the overall configuration of the information processing system S2 according to the 13th embodiment. As shown in Figure 1, the information processing system S2 includes, for example, an information processing device 100b and a medical information system 300. The medical information system 300 is the same as in the first embodiment.

[0194] The hardware configuration of the information processing device 100b in this embodiment is the same as in the first embodiment. The processing circuit 150 of the information processing device 100b in this embodiment includes a reception function 151, a test information acquisition function 152, a reference model selection function 153, an input function 154, a test result acquisition function 155, an evaluation function 156, a display control function 157, and a training function 158. The training function 158 is also called a learning function. The training function 158 is an example of a training unit and a learning unit.

[0195] Figure 16 shows an example of the lifecycle flow for the continuous functional improvement of an AI model according to the 13th embodiment. This lifecycle is provided by the service provider to the user, for example, as a Continuous Integration System that provides monitoring functions for the AI ​​model in operation. This flow, including the AI ​​model construction phase 91 and the AI ​​model operation phase 92, is also called a machine learning pipeline.

[0196] The processes included in the AI ​​model construction phase 91 and the AI ​​model operation phase 92 shown in Figure 16 may all be performed on the information processing device 100b, or some of them may be performed on other information processing devices.

[0197] AI model building phase 91 includes the task of annotating medical data used for training by doctors and other professionals (S101). This task is also called generating correct labels. Medical data to which correct labels have been assigned becomes training data (learning data) 801.

[0198] The training function 158 of the information processing device 100b trains (learns) and validates the model using the training data 801, thereby generating a trained AI model as a training result (S102). The AI ​​model thus trained is stored in the information processing device 100b, for example, as the model group 40 in the first embodiment.

[0199] Then, the reference model selection function 153 of the information processing device 100b selects reference models 430a, 430b, and 430c (hereinafter, when individual reference models 430a, 430b, and 430c are not specified, they are simply referred to as reference model 430) from the model group 40 (S104).

[0200] Furthermore, the input function 154 acquires test data 310 from the medical data 701 and inputs it into the reference model 430 and the model under test 410. The reference model 430 and the model under test 410 perform inference processing based on the input test data 310 (S104).

[0201] The evaluation function 156 of the information processing device 100b evaluates the inference result (first output result) of the model under test 410 based on the inference result (second output result) of the reference model 430 (S105).

[0202] Then, the display control function 157 of the information processing device 100b displays, for example, the confidence level of the inference result of the test target model 410 based on the test results on the display 140 (S106).

[0203] Furthermore, the display control function 157 of the information processing device 100b may output whether or not the test target model 410 needs to be retrained, based on the evaluation results from the evaluation function 156.

[0204] The user performs retraining on the model under test 410 based on the evaluation results from the evaluation function 156 (S107). For example, if the test result is a failure, the test data 310 used in that test is designated as annotation target data 501 to be used as training data for retraining. Note that the test data 310 designated as annotation target data 501 may be automatically extracted by the evaluation function 156 or the training function 158.

[0205] Then, the annotation target data 501 becomes the subject of annotation work by doctors and others in S101, and becomes training data 801 to which correct labels are assigned.

[0206] In this embodiment, a ground truth labelless test using the reference model 430 is incorporated in this way. The test data 310, which results in a test failure, is used as training data 801 in retraining, thereby efficiently improving the accuracy of the model under test 410.

[0207] (Variation 1) In the embodiments described above, a pre-trained model specialized for each individual target part or task was used as the reference model 430, but the reference model 430 is not limited to this. For example, a Foundation Model may be adopted as the reference model 430. An example of a Foundation Model is the Segment Anything Model (general-purpose segmentation model). By adopting a Foundation Model as the reference model 430, it may be possible to save the effort of preparing various reference candidate models 420.

[0208] (Modification 2) In the first embodiment described above, RMSS was used to select the reference model 430, but the formula for calculating RMSS is not limited to the formula shown as number (1). For example, the formula for calculating RMSS may include a term relating to the accuracy of the model. The term relating to the accuracy of the model is, for example, the reciprocal of the accuracy difference between the models. This makes it easier to select a model among the multiple reference candidate models 420 that has a large accuracy difference from the model under test 410 as the reference model 430.

[0209] (Variation 3) In the embodiments described above, the cases in which each model and test data 310 relate to medical use were explained. However, each model and test data is not limited to medical use.

[0210] (Modification 4) In Figure 5 of the first embodiment described above, the evaluation function 156 evaluated the test for the test model 410a as a failure if the liver region 602a segmented by the reference model 430a contained only a portion of the liver cancer region 601c segmented by the test model 410a, and did not encompass the entire region. However, due to tumor hypertrophy, the tumor region may become larger than the original tissue volume. In this case, if the liver region and the liver cancer region overlap to some extent, the test result may be correct even if the liver cancer region exceeds the liver region. For this reason, the evaluation function 156 may evaluate the test as a success if the degree of overlap between the reference region and the abnormal region is greater than or equal to a specified standard. Alternatively, the evaluation function 156 may evaluate the test as a failure if the degree of overlap between the reference region and the abnormal region is less than a specified standard. The criteria for determining the degree of overlap between the reference region and the abnormal region may be the volume of the overlapping portion, or the proportion of the overlapping portion to the reference region or the abnormal region.

[0211] It should be noted that the names of specific organs and diseases used in the descriptions of each embodiment described above are all examples, and the application of each embodiment is not limited to the specific organs and diseases mentioned above.

[0212] The various types of data discussed in this specification are typically digital data.

[0213] According to at least one embodiment described above, the testing of AI models can be made more efficient.

[0214] While several embodiments have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These embodiments can be implemented in a variety of other forms, and various omissions, substitutions, modifications, and combinations of embodiments are possible without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents. [Explanation of Symbols]

[0215] 40 Model Groups Output data 61a~61d,62a,62b Images 70a~70d 91 AI Model Building Phase 92 AI Model Operation Phase 100a, 100b Information Processing Device 110 NW Interfaces 120 Memory circuit 130 Input Interfaces 140 displays 150 Processing Circuits 151 Reception function 152 Test Information Acquisition Function 153. Reference Model Selection Function 154 Input Functions 155 Test Result Acquisition Function 156 Rating function 157 Display control function 158 training functions 210, 210a~210d Medical AI Graph Database 300 Medical Information Systems 310, 310a, 310b, 310c Test Data 410, 410a, 410b, 410c are the models being tested. 420, 420a~420n Standard Candidate Models 430, 430a~430c Standard Model 501 Annotated Data 601a~601d, Liver cancer field 602a, 602b Liver region 701 Medical Data 801 Training Data e1,e2,e101,e201,e301 edge N Network p,p11,p12,p21,p22,p31,p101,p121,p201,p221,p301,p322 Properties S1, S2 Information Processing System v1, v2, v3, v101, v102, v201, v202, v301, v302 nodes

Claims

1. An acquisition unit that acquires a first output result obtained by inputting test data into the model under test, and a second output result obtained by inputting the test data into a reference model related to the model under test, An evaluation unit evaluates the first output result based on the relationship between the first output result and the second output result, An output unit that outputs the evaluation results from the evaluation unit, An information processing device equipped with the following features.

2. The second output result indicates the criteria for evaluating the first output result. The information processing apparatus according to claim 1.

3. The test data input to the model under test and the test data input to the reference model are identical. The evaluation unit evaluates the first output result of the model under test by comparing the first output result of the model under test based on the test data with the first output result of the reference model based on the test data. The information processing apparatus according to claim 1.

4. The aforementioned test data is medical image data captured from an area including the first part of the subject. The test model receives the medical image data as input and outputs a first output result that identifies an abnormal region included in the first part depicted in the medical image data. The reference model receives the medical image data as input and outputs a second output result that identifies a reference region in which the first part is depicted in the medical image data. The evaluation unit evaluates that the test on the model under test is successful if the abnormal region identified in the first output result is contained within the reference region identified in the second output result, if the outer edge of the abnormal region is in contact with the outer edge of the reference region surrounding the abnormal region, or if the degree of overlap between the reference region and the abnormal region is greater than or equal to a specified standard. The information processing apparatus according to claim 1.

5. The aforementioned test data is medical image data captured from an area including the second and third parts of the subject. The second part and the third part are mutually exclusive. The test model receives the medical image data as input and outputs a first output result that identifies an abnormal region included in the second part depicted in the medical image data. The reference model receives the medical image data as input and outputs a second output result that identifies a reference region in which the third part is depicted in the medical image data. The evaluation unit evaluates that the test on the model under test is successful if the abnormal region identified in the first output result is not included within the reference region identified in the second output result. The information processing apparatus according to claim 1.

6. The first output result and the second output result are numerical data. The evaluation unit evaluates that the test on the model under test is successful if the difference between the first output result and the second output result is less than or equal to the first threshold. The information processing apparatus according to claim 1.

7. The aforementioned test model outputs the probability of the first event occurring as numerical data. The aforementioned reference model outputs the probability of a second event occurring, which is mutually exclusive with the first event, as numerical data. The evaluation unit evaluates that the test on the model under test is successful if the difference between the first output result and the second output result is greater than or equal to the second threshold. The information processing apparatus according to claim 1.

8. The system further includes a selection unit that selects a model from among a plurality of candidate models, which are candidates for the reference model, that has a defined relationship with the model under test, from among the plurality of candidate models registered in a graph database that defines the relationship between the plurality of candidate models and the model under test, and selects one that has a defined relationship with the model under test, to be used as the reference model. The information processing apparatus according to claim 1.

9. The graph database defines the relationship between the multiple candidate models and the target parts and tasks of the model under test, The selection unit selects the reference model from the multiple candidate models registered in the graph database based on the target part of the model under test and the task. The information processing apparatus according to claim 8.

10. The graph database further defines rules that are applied as output conditions for the model under test, The evaluation unit further evaluates the first output result based on the rule. The information processing apparatus according to claim 9.

11. The selection unit selects a plurality of reference models from the plurality of candidate models registered in the graph database. The evaluation unit evaluates the first output result based on the relationship between the first output result and a plurality of second output results output from a plurality of reference models. The information processing apparatus according to claim 8.

12. The selection unit selects the standard model from the multiple candidate models registered in the graph database based on the attribute information attached to the test data. The attribute information includes at least one of the following: the gender, age, and disease information of the subject relating to the test data. The information processing apparatus according to claim 8.

13. The evaluation results indicate whether the test was successful or unsuccessful for the model under test. The evaluation unit generates information regarding test failures for the model under test based on a plurality of test data input to the model under test and a plurality of evaluation results for a plurality of first output results output from the model under test based on the input of the plurality of test data. The output unit outputs information regarding the failure of the test for the model under test. The information processing apparatus according to any one of claims 1 to 12.

14. An acquisition step to obtain a first output result obtained by inputting test data into the model under test, and a second output result obtained by inputting the test data into a reference model related to the output of the model under test, An evaluation step in which the first output result is evaluated based on the relationship between the first output result and the second output result, An output step that outputs the evaluation results obtained in the above evaluation step, Information processing methods including

15. An acquisition step to obtain a first output result obtained by inputting test data into the model under test, and a second output result obtained by inputting the test data into a reference model related to the output of the model under test, An evaluation step in which the first output result is evaluated based on the relationship between the first output result and the second output result, An output step that outputs the evaluation results obtained in the above evaluation step, A program that causes a computer to execute something.

Images