Medical information processing device, medical information processing method, and program

Abstract

To appropriately update a learned model.SOLUTION: A medical information processing apparatus of an embodiment has an acquisition unit and a specification unit. The acquisition unit acquires a plurality of sets of verification data including medical data, result data obtained by inputting the medical data to a learned model, and true-false data related to the result data. The specification unit, based on the relationship between the sets of verification data and a first learned model receiving input of the medical data and outputting the result data, specifies a target set of verification data suitable for evaluation of performance required for a learned model receiving input of the medical data and outputting the result data, from the plurality of sets of verification data.SELECTED DRAWING: Figure 2

Description

TECHNICAL FIELD

[0001] The embodiments disclosed in the present specification and drawings relate to a medical information processing apparatus, a medical information processing method, and a program.

BACKGROUND ART

[0002] In recent years, technologies based on machine learning using Artificial Intelligence (AI) have been used in various systems, and such machine learning-based technologies have also been widely applied in the medical industry. Learned models such as analysis applications used for machine learning are updated through additional learning or the like to regularly improve their accuracy.

[0003] When updating a learned model to improve its accuracy, the update direction may not align with user needs or operations, and in some cases, the update of the learned model may even result in deterioration. To avoid cases where the update of the learned model leads to deterioration, it is required to update the learned model in a manner suitable for user needs and operations.

PRIOR ART DOCUMENTS

PATENT DOCUMENTS

[0004]

PATENT DOCUMENT 1

SUMMARY OF THE INVENTION

PROBLEMS TO BE SOLVED BY THE INVENTION

[0005] The problem to be solved by the embodiments disclosed in the present specification and drawings is to enable appropriate updating of a learned model. However, the problem intended to be solved by the embodiments disclosed in the present specification and drawings is not limited to the above-mentioned problem. Problems corresponding to the respective effects of the respective configurations shown in the embodiments described later can also be positioned as other problems.

MEANS FOR SOLVING THE PROBLEMS

[0006] The medical information processing device of the embodiment includes an acquisition unit and a selection unit. The acquisition unit acquires a plurality of validation datasets, which include medical data, result data obtained by inputting the medical data into a trained model, and correct / incorrect data regarding the result data. The selection unit identifies a target validation dataset from among the plurality of validation datasets that is suitable for evaluating the performance required of the trained model that inputs the medical data and outputs the result data, based on the relationship between the first trained model, which takes the medical data as input and outputs the result data, and the validation dataset. [Brief explanation of the drawing]

[0007] [Figure 1] A block diagram showing an example of the configuration of the hospital system 1 of the embodiment. [Figure 2] A block diagram showing an example of the configuration of the medical information processing device 100 according to the embodiment. [Figure 3] A diagram showing an example of the contents of the pre-trained model DB151. [Figure 4] A diagram showing an example of the contents of the validation dataset DB152. [Figure 5] A diagram showing an example of the content of the judgment criteria data. [Figure 6] A diagram showing the number of validation datasets included in validation dataset DB152. [Figure 7] A flowchart showing an example of processing in the medical information processing device 100 of the embodiment. [Figure 8] A flowchart showing an example of processing in the medical information processing device 100 of the embodiment. [Modes for carrying out the invention]

[0008] The medical information processing device, medical information processing method, and program of the embodiment will be described below with reference to the drawings. The medical information processing device of the embodiment is used, for example, as part of a hospital system installed within a hospital.

[0009] Figure 1 is a block diagram showing an example of the configuration of the hospital system 1 of the embodiment. The hospital system 1 of the embodiment includes, for example, a Hospital Information System (HIS) 10, a Radiology Information System (RIS) 20, a medical image diagnostic device (modality) 30, a Picture Archiving and Communication System (PACS) 40, and a medical information processing device 100.

[0010] HIS10 is a computer system that provides operational support within hospitals. Specifically, HIS10 has various subsystems. These subsystems include, for example, an electronic medical record system, a medical accounting system, an appointment scheduling system, a patient registration system, and an admission / discharge management system.

[0011] HIS10 includes, for example, computers such as server devices and client terminals equipped with a processor such as a CPU (Central Processing Unit), memory such as ROM (Read Only Memory) and RAM (Random Access Memory), a display, an input interface, and a communication interface.

[0012] Users input and retrieve patient information using the electronic medical record system included in HIS10. Users issue imaging order requests to HIS10. HIS10 then transfers the order information corresponding to the imaging order to other systems such as RIS20.

[0013] RIS20 is a computer system that provides operational support in the diagnostic imaging department. RIS20 manages the reservation of imaging examination orders in conjunction with HIS10, as well as integrating reservation information with examination equipment and managing examination information. RIS20 includes, for example, a server device and client terminals equipped with a processor such as a CPU, memory such as ROM and RAM, a display, input interfaces, and communication interfaces.

[0014] Modality 30 performs imaging (taking images) according to imaging conditions (imaging protocol) determined, for example, based on imaging examination instructions. Examples of modality 30 include X-ray computed tomography scanners, X-ray diagnostic equipment, magnetic resonance imaging equipment, ultrasound diagnostic equipment, and nuclear medicine diagnostic equipment. Modality 30 is operated by operators such as physicians (radiologists) or radiological technologists. Medical images (image data) generated by imaging with modality 30 are transmitted to PACS 40.

[0015] PACS40 is a computer system that receives medical images transmitted by modality 30 and stores them in a database. PACS40 transmits (transfers) medical images stored in the database in response to requests from clients. PACS40 includes a server computer with a processor such as a CPU, memory such as ROM and RAM, a display, input interfaces, and communication interfaces. Medical images stored in PACS40 are accompanied by supplementary information, such as patient ID, examination ID, and imaging conditions (imaging protocol), in a format compliant with the DICOM (Digital Imaging and Communication in Medicine) standard.

[0016] The configuration of hospital system 1 is not limited to the above. Hospital system 1 may include, for example, a radiographic report generation device. Also, several elements of hospital system 1 may be integrated. For example, HIS10 and RIS20 may be integrated into a single system.

[0017] Figure 2 is a block diagram illustrating an example of the configuration of a medical information processing apparatus 100 according to an embodiment. The medical information processing apparatus 100 includes, for example, a communication interface 110, an input interface 120, a display 130, a processing circuit 140, and a memory 150. The communication interface 110, the input interface 120, and the display 130 in the medical information processing apparatus 100 are provided separately from the communication interface, the input interface, and the display included in the HIS 10, but these may be shared.

[0018] The communication interface 110 communicates with external apparatuses such as the RIS 20, the modality 30, and the PACS 40 via a network NW such as a LAN (Local Area Network). The communication interface 110 includes, for example, a communication interface such as a NIC (Network Interface Card). The network NW may include the Internet, a cellular network, a Wi-Fi network, a WAN (Wide Area Network), or the like instead of or in addition to the LAN.

[0019] The input interface 120 includes, for example, a mouse, a keyboard, a trackball, a switch, a button, a joystick, a touch panel, and the like. The input interface 120 may be, for example, a user interface that receives voice input from a microphone or the like. When the input interface 120 is a touch panel, the input interface 120 may also have the display function of the display 130.

[0020] Note that in the present specification, the input interface is not limited to those provided with physical operation components such as a mouse and a keyboard. For example, an electric signal processing circuit that receives an electric signal corresponding to an input operation from an external input device provided separately from the apparatus and outputs this electric signal to a control circuit is also included in examples of the input interface.

[0021] The input interface 120 receives various input operations from doctors and other medical personnel, converts the received input operations into electrical signals, and outputs them to the processing circuit 140. For example, doctors and other medical personnel perform selection input operations to select a trained model for detecting a specific disease (e.g., cerebral infarction, hereinafter referred to as the "specific disease") in a patient whose medical image data has been captured (hereinafter referred to as the "target patient"). When a selection input operation is performed by a doctor or other medical personnel, the input interface 120 generates selection information corresponding to the input operation. The input interface 120 outputs the generated selection information to the processing circuit 140. Medical image data is an example of medical data.

[0022] For example, when evaluating the performance of a trained model, a physician or other medical professional performs a performance requirement input operation to identify the trained model to be evaluated and the performance required for that trained model. The input interface 120 generates performance requirement information corresponding to the input operation when, for example, a physician or other medical professional performs the performance requirement input operation. The input interface 120 outputs the generated performance requirement information to the processing circuit 140. The evaluation of the performance of the trained model will be explained further later.

[0023] The display 130 displays various types of information. For example, the display 130 displays images generated by the processing circuit 140, or a GUI (Graphical User Interface) for receiving various input operations from the operator. For example, the display 130 may be an LCD (Liquid Crystal Display), a CRT (Cathode Ray Tube) display, or an organic EL (Electro Luminescence) display.

[0024] The processing circuit 140 includes, for example, a determination processing circuit 141 and an evaluation processing circuit 142. The processing circuit 140 realizes these functions, for example, by having a hardware processor (computer) execute a program stored in the memory (storage circuit) 150.

[0025] Hardware processors refer to circuits such as CPUs, GPUs (Graphics Processing Units), Application Specific Integrated Circuits (ASICs), programmable logic devices (e.g., Simple Programmable Logic Devices (SPLDs) or Complex Programmable Logic Devices (CPLDs)), and Field Programmable Gate Arrays (FPGAs).

[0026] Instead of storing the program in memory 150, the system may be configured to directly incorporate the program into the hardware processor's circuitry. In this case, the hardware processor performs its function by reading and executing the program incorporated into the circuitry. The program may be stored in memory 150 beforehand, or it may be stored on a non-temporary storage medium such as a DVD or CD-ROM, and installed from the non-temporary storage medium to memory 150 when the non-temporary storage medium is mounted on the drive device (not shown) of the medical information processing device 100.

[0027] The hardware processor is not limited to being configured as a single circuit; it may be configured as a single hardware processor by combining multiple independent circuits to realize each function. Alternatively, multiple components may be integrated into a single hardware processor to realize each function. The hardware processor and memory in the medical information processing device 100 are provided separately from the hardware processor and memory in the HIS 10, but they may be common to each other.

[0028] Memory 150 stores the trained model database (hereinafter referred to as DB) 151 and the validation dataset DB 152. The trained model DB 151 is a database containing multiple trained models. The validation dataset DB 152 is a database containing multiple validation datasets. A validation dataset is a dataset consisting of medical image data, result data, and ground truth data.

[0029] Figure 3 shows an example of the contents of the trained model DB151. The trained model DB151 contains data from multiple trained models. Each trained model is assigned a model ID and a name. For example, the first model is assigned the model ID "M001".

[0030] The pre-trained models included in the DB151 are models for diagnosing target patients. The pre-trained models determine, for example, whether a specific disease is detected in the target patient (positive or negative). Whether a specific disease is detected in the target patient is determined, for example, by whether the medical image data acquired from the target patient contains images of the specific disease.

[0031] Each pre-trained model has a specific disease assigned to it. Alternatively, a pre-trained model may not have a specific disease assigned to it; instead, it could be a model that identifies the type of disease, for example. When medical image data is input to a pre-trained model, it outputs a result indicating whether or not the image corresponds to a specific disease.

[0032] The trained model is updated using the judgment results obtained by determining whether or not a person has a specific disease using the trained model, and the correct data obtained, for example, from a findings report. The trained model may be updated each time a judgment result is output, or it may be updated after a predetermined number of judgment results have been output.

[0033] Figure 4 shows an example of the contents of the validation dataset DB152. The validation dataset DB152 contains multiple validation datasets. Each validation dataset is assigned a validation ID and a judgment result. The validation dataset DB152 is created for each of the multiple trained models.

[0034] Medical image data is, for example, data of medical images acquired by modality 30 included in hospital system 1. Medical image data may also be data provided by PACS 40, or data provided by an external device installed outside hospital system 1.

[0035] Result data is, for example, data related to diagnostic results, showing the results obtained by inputting medical image data into a trained model. Ground truth data is data indicating the correctness (positive or negative) of a specific disease set in the trained model. Ground truth data is, for example, data based on a physician's findings. Ground truth data is, for example, included in a findings report created by a physician. The judgment result is determined based on the judgment criteria data. Ground truth data is an example of correct / incorrect data.

[0036] Figure 5 shows an example of the content of the judgment criteria data. In the judgment criteria data, when the result data is "positive" and the correct answer data is "positive", it is classified as a true positive (TP), and when the result data is "negative" and the correct answer data is "negative", it is classified as a true negative (NP). In the judgment criteria data, when the result data is "positive" and the correct answer data is "negative", it is classified as a false positive (FP), and when the result data is "negative" and the correct answer data is "positive", it is classified as a false negative (NP).

[0037] The validation dataset DB152 contains multiple validation datasets. Figure 6 shows the number of validation datasets included in the validation dataset DB152. The validation dataset DB152 shown in Figure 6 contains 100 datasets with a judgment result of "TP", 100 datasets with a judgment result of "TN", 20 datasets with a judgment result of "NP", and 20 datasets with a judgment result of "FN". As a result, the accuracy of the trained model created using the validation dataset DB152 is approximately 0.83 (83%).

[0038] The determination processing circuit 141 in the processing circuit 140 is a circuit for determining the presence or absence of a specific disease using a trained model based on medical image data acquired by modality 30, etc. The determination processing circuit 141 includes, for example, a first acquisition function 143 and a determination function 144.

[0039] The first acquisition function 143 acquires medical image data captured by modality 30. The medical image data here is image data used to determine the presence or absence of a specific disease. The medical image data may also be medical image data other than that captured by modality 30.

[0040] The judgment function 144 selects a trained model from the trained model DB 151 to judge the medical image data acquired by the first acquisition function 143, based on the input operation information transmitted by the input interface 120 in response to the input operation of a doctor or other person. The judgment function 144 inputs the medical image data acquired by the first acquisition function 143 into the selected trained model to determine whether or not a specific disease can be found in the target patient.

[0041] The evaluation processing circuit 142 in the processing circuit 140 is a circuit for evaluating the performance of the trained model used by the determination processing circuit 141 to determine whether or not a specific disease is detected in the target patient. The evaluation processing circuit 142 includes, for example, a second acquisition function 145, a specific function 146, and an evaluation function 147.

[0042] The second acquisition function 145 acquires multiple validation datasets, for example, when evaluating the performance of a trained model. The second acquisition function 145 further acquires the performance required for the trained model being evaluated (hereinafter referred to as the model required performance) and the trained model that serves as the standard for performance evaluation (hereinafter referred to as the first trained model). The second acquisition function 145 is an example of an acquisition unit.

[0043] The specific function 146 identifies, for example, a validation dataset suitable for evaluating the model's required performance (hereinafter referred to as the target validation dataset). The required model performance may be, for example, a high probability of being judged as a true positive or true negative (high positive detection rate), or a low probability of being judged as a false positive or false negative (low false positive rate).

[0044] The required model performance here is determined, for example, based on performance requirement information transmitted via the input interface 120. The target validation dataset is, for example, a validation dataset that yields results corresponding to the required model performance. For example, if the required model performance is to increase the probability of true positives, then the target validation dataset may be one in which a large number of validation datasets have a judgment result of true positive "TP", or all of the target validation datasets may be one in which the judgment result of true positive "TP".

[0045] The evaluation function 147, for example, specifies a trained model to be evaluated (hereinafter referred to as the second trained model) and evaluates the second trained model. The second trained model is different from the first trained model. The evaluation function 147 uses the target validation dataset identified by the identification function 146 to evaluate the performance of the second trained model based on the relationship between the first trained model and the target validation dataset, and the relationship between the second trained model and the target validation dataset. The second trained model is, for example, a trained model that has been further trained than the first trained model. The evaluation function 147 is an example of an evaluation unit.

[0046] In the first embodiment of the medical information processing device 100, processing is performed to determine a specific disease in the target patient and to evaluate a trained model used for diagnosis. These processing steps will be described below. First, the processing for determining a specific disease in the target patient will be described. Figures 7 and 8 are flowcharts showing an example of processing in the medical information processing device 100 of the embodiment.

[0047] Figure 7 shows an example of the process for determining a specific disease in a target patient. The process shown in the flowchart in Figure 7 is mainly performed in the determination processing circuit 141, for example. First, the medical information processing device 100 acquires medical image data of the area to be diagnosed in the target patient, for example, the brain if the specific disease is cerebral infarction, in the first acquisition function 143 (step S101).

[0048] Next, the first acquisition function 143 acquires selection information transmitted by a doctor or other medical professional operating the input interface 120 (step S103). The selection information includes information on trained models that the doctor or other medical professional selects from the trained model DB 151 for use in diagnosis.

[0049] Next, the judgment function 144 selects a trained model to be used for diagnosis based on the selection information acquired by the first acquisition function 143 (step S105). Subsequently, the judgment function 144 inputs the medical image data acquired by the first acquisition function 143 into the selected trained model (step S107).

[0050] In the judgment function 144, the trained model outputs result data indicating the judgment result when medical image data is used as input data (step S109). The judgment function 144 displays the judgment result indicated by the result data (e.g., "TP", "FN") on the display 130. In this way, the medical information processing device 100 completes the process shown in Figure 7.

[0051] Next, we will explain the process of evaluating the trained model used for diagnosis. Figure 8 shows an example of the process of evaluating the trained model used for diagnosis. The process shown in the flowchart in Figure 8 is mainly performed in the evaluation processing circuit 142, for example. First, the medical information processing device 100 acquires performance request information transmitted by a doctor or the like when they operate the input interface 120 in the second acquisition function 145 (step S201).

[0052] Next, the specific function 146 specifies the second trained model to be evaluated based on the performance requirement information acquired by the second acquisition function 145 (step S203). Subsequently, the specific function 146 specifies the first trained model based on the specified second trained model (step S205). For example, the specific function 146 specifies the trained model immediately before the second trained model is updated as the first trained model. The specific function 146 may also specify other trained models as the first trained model, such as the trained model before multiple updates of the second trained model were performed, or a known trained model.

[0053] Next, the identification function 146 identifies the target verification dataset based on the performance requirement information obtained by the second acquisition function 145 (step S207). The identification function 146 identifies the target verification dataset according to the performance required by the performance requirement information. For example, if the model performance requirement in the performance requirement information is to increase the probability of true positives, the identification function 146 identifies a large number of verification datasets whose judgment result is true positive "TP" as the target verification dataset, or identifies all of the target verification datasets as verification datasets whose judgment result is true positive "TP".

[0054] Furthermore, if the required model performance is to increase the probability of true negatives, then the target validation dataset should be one in which a large number of validation datasets have a true negative ("TN") result, or all target validation datasets should be one in which the judgment result is true negative ("TN").

[0055] Furthermore, if the model performance requirement is to reduce the likelihood of false positives, then, as the target validation dataset, validation data is generated by changing the ground truth data of the validation dataset whose judgment result is a false positive (FP) to "negative," and the target validation dataset is identified by including this generated validation data.

[0056] Furthermore, if the required model performance is to reduce the likelihood of false negatives, then, as the target validation dataset, validation data is generated by setting the ground truth data of the validation dataset where the judgment result is a false negative ("FN") to "positive," and the target validation dataset is identified by including this generated validation data.

[0057] Next, the evaluation function 147 inputs medical image data contained in each of the multiple target validation datasets into the second trained model in order to evaluate the second trained model using the target validation dataset identified by the identification function 146 (step S209). Subsequently, the second trained model, having received the medical image data as input, outputs result data (hereinafter referred to as second result data) (step S211).

[0058] Next, the evaluation function 147 inputs the medical image data from which the second result data was obtained into the first trained model, and collects the result data output by the first trained model (hereinafter referred to as the first result data) from the validation dataset DB 152. The evaluation function 147 refers to the validation dataset DB 152 and calculates the degree of agreement between the second result data and the ground truth data, and the degree of agreement between the first result data and the ground truth data, respectively. The evaluation function 147 compares the degree of agreement between the second result data and the ground truth data with the degree of agreement between the first result data and the ground truth data (step S213). Based on the results of comparing the degree of agreement between the second result data and the ground truth data and the degree of agreement between the first result data and the ground truth data, the evaluation function 147 evaluates the second trained model (step S215).

[0059] The evaluation function 147 evaluates the second trained model as having better performance than the first trained model if, for example, the degree of agreement between the second result data and the ground truth data is higher than the degree of agreement between the first result data and the ground truth data. The evaluation function 147 evaluates the second trained model as having worse performance than the first trained model if, for example, the degree of agreement between the second result data and the ground truth data is lower than the degree of agreement between the first result data and the ground truth data. In this way, the medical information processing device 100 completes the process shown in Figure 8.

[0060] In the embodiment, the medical information processing device 100, in its evaluation function 147, identifies a target validation dataset to be used when evaluating the model requirement performance of a trained model generated by updating the first trained model, based on the relationship between the first trained model and the target validation dataset. Therefore, it is possible to identify a target validation dataset suitable for evaluating the model requirement performance of a second trained model, thereby enabling the trained model to be updated appropriately.

[0061] In the above embodiment, the medical information processing device 100 includes a determination processing circuit 141 and an evaluation processing circuit 142, and performs determination processing and evaluation processing. However, the medical information processing device 100 may include an evaluation processing circuit 142 without the determination processing circuit 141. In this case, the medical information processing device 100 may receive the determination result of whether or not a patient falls under a specific disease and the verification dataset from an external device.

[0062] Furthermore, in the above embodiment, the medical information processing device 100 is provided as part of the hospital system 1, but the medical information processing device 100 may be provided independently of the hospital system 1. In this case, the medical information processing device 100 may evaluate the second trained model using a validation dataset generated within the hospital, or it may evaluate the second trained model using a validation dataset provided to the hospital from an external source.

[0063] Furthermore, in the above embodiment, the selection of a trained model is performed based on selection information corresponding to the selection input operation by a physician or the like. However, the medical information processing device 100 may perform the selection based on various information such as the type of medical image data.

[0064] Furthermore, in the above embodiment, the correct / incorrect data is correct answer data, but the correct / incorrect data may also be incorrect answer data (incorrect response data), or it may include both correct answer data and incorrect answer data. Also, in the above embodiment, the medical data is medical image data, but the medical data may also be data other than medical image data. For example, the medical data may be medical text data in which diagnostic results are entered.

[0065] According to at least one embodiment described above, the trained model can be appropriately updated by having an acquisition unit that acquires a plurality of validation datasets including medical data, result data obtained by inputting the medical data into a trained model, and correct / incorrect data regarding the result data, and an identification unit that identifies a target validation dataset from among the plurality of validation datasets that is suitable for evaluating the performance required for the trained model that inputs the medical data and outputs the result data, based on the relationship between the first trained model that inputs the medical data and outputs the result data and the validation dataset.

[0066] While several embodiments have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents. [Explanation of Symbols]

[0067] 1…Hospital system 10…Hospital Information System (HIS) 20…Radiology Information System (RIS) 30…Medical imaging equipment (modality) 40…PACS (Picture Archiving and Communication System) 100... Medical information processing device 110...Communication Interface 120... Input Interface 130…Display 140… Processing circuit 141... Judgment processing circuit 142…Evaluation processing circuit 143...First acquisition function 144... Judgment function 145...Second acquisition function 146...Specific function 147…Rating function 150...memory 151... Pre-trained model database 152…Verification dataset DB NW...Network

Claims

1. An acquisition unit that acquires multiple validation datasets, including medical data, result data obtained by inputting the medical data into a trained model, and correct / incorrect data regarding the result data. The system includes a selection unit that identifies a target validation dataset from among a plurality of validation datasets that is suitable for evaluating the performance required for a first trained model that takes the medical data as input and outputs the result data, based on the relationship between the first trained model that takes the medical data as input and outputs the result data, and the validation dataset. Medical information processing device.

2. The system further includes an evaluation unit that evaluates the performance of a second trained model based on the relationship between the first trained model and the target validation dataset, and the relationship between a second trained model, which is different from the first trained model, and the validation dataset, after receiving the medical data as input and outputting the result data. The medical information processing device according to claim 1.

3. The aforementioned result data is data relating to the diagnostic results, The performance required of the trained model includes a low false positive rate in the diagnostic results. The medical information processing device according to claim 1.

4. The identification unit identifies the target verification dataset, including the verification dataset in which the result data shows a positive result and the correct / incorrect data shows a negative result. The medical information processing device according to claim 3.

5. The aforementioned result data is data relating to the diagnostic results, The performance required of the trained model includes a low false negative rate in the diagnostic results. The medical information processing device according to claim 1.

6. The identification unit identifies the target verification dataset, including the verification dataset in which the result data shows a negative result and the correct / incorrect data shows a positive result. The medical information processing device according to claim 5.

7. The aforementioned correction data includes data based on the findings report. The medical information processing device according to claim 1.

8. The aforementioned medical data is medical image data. The medical information processing device according to claim 1.

9. Computers Multiple validation datasets are obtained, each including medical data, result data obtained by inputting the medical data into a trained model, and correct / incorrect data regarding the result data. Based on the relationship between the first trained model, which takes the medical data as input and outputs the result data, and the validation dataset, and the relationship between the second trained model, which takes the medical data as input and outputs the result data and is different from the first trained model, and the validation dataset, a target validation dataset suitable for evaluating the performance required for the trained model that takes the medical data as input and outputs the result data is identified from among the multiple validation datasets. Medical information processing method.

10. On the computer, Multiple validation datasets are obtained, each including medical data, result data obtained by inputting the medical data into a trained model, and correct / incorrect data regarding the result data. Based on the relationship between the first trained model, which takes the medical data as input and outputs the result data, and the validation dataset, and the relationship between the second trained model, which takes the medical data as input and outputs the result data and is different from the first trained model, and the validation dataset, the system identifies a target validation dataset from among the multiple validation datasets that is suitable for evaluating the performance required for the trained model that takes the medical data as input and outputs the result data. program.

Images