Medical information processing device and program

Abstract

To improve prediction accuracy of a machine learning model.SOLUTION: A medical information processing apparatus includes an adding section, a collection section, a first learning section, an acquisition section, and a second learning section. The adding section adds a correct answer label used for training a decision making model, which is used for decision making in medical care, in accordance with an operator's input instruction. The collection section collects status data which is data indicating an operator's status while doing work of adding the correct answer label. The first learning section trains a reliability determination model to which the status data is input and which outputs reliability of the correct answer label on the basis of the status data and the correct answer label. The acquisition section acquires input data on the decision making model. The second learning section trains the decision making model to which the input data is input and which outputs output data representing a result of decision making on the basis of the input data, the correct answer label, and the reliability.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The embodiments disclosed in the present specification and drawings relate to a medical information processing device and a program. [Background technology]

[0002] To improve the efficiency of tasks that require the advanced knowledge of experts, there are high expectations for the semi-to-full automation of these tasks using machine learning models. These tasks lack work standards and rely on individuals to determine their own judgments, making them known as personalized tasks. In the medical field, tasks related to medical decision-making, such as image diagnosis and determining examination protocols, are typical examples of personalized tasks. When using machine learning to support personalized tasks, it can be difficult to accurately assign correct labels due to factors such as the skill of the worker and the circumstances surrounding the assignment of the correct labels, which can result in a deterioration in the prediction accuracy of the machine learning model. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2020-086519 Summary of the Invention [Problem to be solved by the invention]

[0004] One of the problems to be solved by the embodiments disclosed in this specification and the drawings is to improve the prediction accuracy of a machine learning model. However, the problems to be solved by the embodiments disclosed in this specification and the drawings are not limited to the above problem. Problems corresponding to the effects of each configuration shown in the embodiments described below can also be positioned as other problems. [Means for solving the problem]

[0005] A medical information processing device according to an embodiment includes an assignment unit, a collection unit, a first learning unit, an acquisition unit, and a second learning unit. The assignment unit assigns correct labels to be used in training a decision-making model, which is a machine learning model used for medical decision-making, in accordance with input instructions from an operator. The collection unit collects situation data, which is data representing the situation of the operator when assigning the correct labels. The first learning unit trains a reliability determination model, which is a machine learning model that inputs the situation data and outputs the reliability of the correct labels, based on the situation data and the correct labels. The acquisition unit acquires input data for the decision-making model. The second learning unit trains the decision-making model, which inputs the input data and outputs output data, which is data representing the result of the decision-making, based on the input data, the correct labels, and the reliability. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a medical information system according to this embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of the configuration of the medical information processing apparatus illustrated in FIG. [Figure 3] FIG. 3 is a diagram showing the relationship between input and output of the decision-making model in the learning phase and the operational phase. [Figure 4] FIG. 4 is a diagram showing the relationship between the input and output of the reliability determination model in the learning phase and the operational phase. [Figure 5] FIG. 5 is a diagram showing a flow of an example of medical information processing by the medical information processing device. [Figure 6] FIG. 6 is a diagram schematically showing the work of assigning correct labels and the collection of situation data in step S3 of FIG. [Figure 7] FIG. 7 is a diagram showing the relationship between the input and output of the reliability determination model in the learning phase. [Figure 8] FIG. 8 is a diagram illustrating an example of training data. [Figure 9] FIG. 9 is a diagram showing the relationship between the input and output of the trained reliability determination model in an operational phase. [Figure 10]FIG. 10 is a diagram illustrating an example of the reliability database. [Figure 11] FIG. 11 is a diagram illustrating an example of input data. [Figure 12] FIG. 12 is a diagram showing the relationship between the input and output of the examination protocol classification model in the learning phase. [Figure 13] FIG. 13 is a diagram showing the relationship between the input and output of the trained test protocol classification model in the operational phase. [Figure 14] FIG. 14 is a diagram showing the relationship between the input and output of the reliability determination model according to the first modification in the learning phase. [Figure 15] FIG. 15 is a diagram illustrating an example of training data according to the first modification. [Figure 16] FIG. 16 is a diagram showing the relationship between input and output of the reliability determination model according to the second modification in the learning phase. [Figure 17] FIG. 17 is a diagram showing an example of a reliability display screen according to the third modification. [Figure 18] FIG. 18 is a diagram showing an example of the correspondence between the frequency distribution of the reliability and the color value. [Figure 19] FIG. 19 is a diagram showing another example of the correspondence between the frequency distribution of the reliability and the color value. DETAILED DESCRIPTION OF THE INVENTION

[0007] Hereinafter, embodiments of a medical information processing device and a program will be described in detail with reference to the drawings.

[0008] FIG. 1 is a diagram showing an example of the configuration of a medical information processing system 1 according to this embodiment. As shown in FIG. 1, the medical information processing system 1 is a computer system having a medical information processing device 2 and a medical device terminal 3. The medical information processing device 2 and the medical device terminal 3 are connected to each other via wired or wireless communication so that they can communicate information with each other. The medical information processing device 2 is an information processing terminal such as a computer, such as a workstation, that processes medical information. The medical device terminal 3 is an information processing terminal that has a processor, a storage device, an input device, a communication device, and a display device and can send instructions from an operator to the medical information processing device. The medical device terminal 3 can be a desktop computer, a laptop computer, a tablet terminal, a smartphone, or the like.

[0009] Fig. 2 is a diagram showing an example of the configuration of the medical information processing device 2. As shown in Fig. 2, the medical information processing device 2 has a processing circuit 21, a storage device 22, an input device 23, a communication device 24, and a display device 25. The processing circuit 21, the storage device 22, the input device 23, the communication device 24, and the display device 25 are connected via a bus so that signals can be input and output to and from each other.

[0010] The medical information processing device 2 generates a machine learning model used for medical decision-making (hereinafter referred to as a decision-making model) and a machine learning model that determines the reliability of a correct label (hereinafter referred to as a reliability determination model).

[0011] FIG. 3 illustrates the input / output relationship of the decision-making model 31 during the learning phase and the operational phase. As shown in FIG. 3, during the learning phase, the decision-making model 31 is trained based on medical decision-making input data 32, a correct label 33, and a reliability 34 of the correct label 33. The correct label 33 is a correct label assigned in accordance with instructions from an operator via the medical device terminal 3. The reliability 34 is an index value representing the degree of reliability, such as the validity, of the correct label 33. The reliability 34 is used as a weight for the correct label 33. The reliability 34 is output by a reliability judgment model. A combination of the input data 32, the correct label 33, and the reliability 34 is called a training sample. Multiple training samples are collected for various types of input data. A trained decision-making model 35 is generated by training the decision-making model 31 based on the multiple training samples. During the operational phase, the decision-making model 35 inputs medical decision-making input data 32 and outputs output data 36, ​​which is the decision-making result.

[0012] The decision-making model 31 and the trained decision-making model 35 are neural networks having an input layer for inputting input data 32, a hidden layer for converting the input data 32 into output data 36, ​​and an output layer for outputting the output data 36. The number of hidden layers may be one or more. The decision-making model 31 and the trained decision-making model 35 are multi-class classification models that output the probability of each of multiple classes related to the decision-making result as output data 36. For example, in the case of medical decision-making for imaging diagnosis, diseases are classified as classes, and in the case of medical decision-making for determining an examination protocol, examination protocols are classified as classes.

[0013] FIG. 4 illustrates the input / output relationship of the reliability determination model 41 during the learning phase and the operational phase. As shown in FIG. 4, during the learning phase, the reliability determination model 41 is trained based on situation data 42 and the correct label 43. The situation data 42 represents the situation of the worker when assigning the correct label 43. Specifically, the situation data 42 is data related to the worker's operation, gaze, speech, and / or facial expression, reflecting the worker's decision-making process during the assignment. The correct label 43 is the same as the correct label 33 used for the decision-making model 31, and the correct label assigned in accordance with the worker's instructions via the medical device terminal 3 is used. A combination of the situation data 42 and the correct label 43 is called a training sample. Multiple training samples are collected for various input data. A reliability determination model 45 is generated by training the reliability determination model 41 based on the multiple training samples. During the operational phase, the reliability determination model 45 inputs the situation data 42 and outputs the reliability 45 of the correct label.

[0014] The reliability judgment model 41 and the trained reliability judgment model 44 are neural networks having an input layer for inputting situation data 42, a hidden layer for converting the situation data 42 into reliability 45, and an output layer for outputting the reliability 45. The number of hidden layers may be one or more. The reliability judgment model 41 and the trained reliability judgment model 44 are multi-class classification models that output the probability of each of multiple classes related to the decision-making result as reliability 45. The specific tasks of the reliability judgment model 41 and the trained reliability judgment model 44 are set according to the specific tasks of the decision-making model 31 and the trained decision-making model 35. For example, if the task of the decision-making model 31 and the trained decision-making model 35 is medical decision-making for image diagnosis, the task of the reliability judgment model 41 and the trained reliability judgment model 44 is also medical decision-making for image diagnosis, and if the task of the decision-making model 31 and the trained decision-making model 35 is medical decision-making for determining an examination protocol, the task of the reliability judgment model 41 and the trained reliability judgment model 44 is also medical decision-making for determining an examination protocol.

[0015] 2, the medical information processing device 2 has a processing circuit 21, a storage device 22, an input device 23, a communication device 24, and a display device 25. The processing circuit 21, the storage device 22, the input device 23, the communication device 24, and the display device 25 are connected via a bus so as to be able to input and output signals to and from each other.

[0016] The processing circuitry 21 has processors such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The processing circuitry 21 executes a medical information processing program to realize an acquisition function 211, an assignment function 212, a collection function 213, a first learning function 214, a reliability calculation function 215, a second learning function 216, a display control function 217, and the like. Note that each of the functions 211 to 217 is not limited to being realized by a single processing circuit. A processing circuit may be configured by combining multiple independent processors, and each processor may execute a program to realize each of the functions 211 to 217. Furthermore, the functions 211 to 217 may be modular programs that respectively constitute a medical information processing program. These programs are stored in the storage device 22.

[0017] By implementing the acquisition function 211, the processing circuitry 21 acquires various information. For example, the processing circuitry 21 acquires medical decision-making input data. The medical decision-making input data is used to train the decision-making model. The medical decision-making input data can be acquired from a medical information system such as a Hospital Information System (HIS) or a Radiology Information System (RIS).

[0018] By implementing the assignment function 212, the processing circuit 21 assigns a correct label according to an instruction from an operator via the medical device terminal 3. The correct label is used to train a reliability judgment model and a decision-making model.

[0019] The processing circuitry 21 collects various information by implementing the collection function 213. For example, the processing circuitry 21 collects situation data that indicates the situation of the worker when assigning the correct label.

[0020] By implementing the first learning function 214, the processing circuit 21 trains a reliability judgment model that inputs situation data and outputs the reliability of the correct label based on the situation data and the correct label.

[0021] By implementing the reliability calculation function 215, the processing circuit 21 calculates the reliability of the correct label based on the trained reliability determination model.

[0022] By implementing the second learning function 216, the processing circuit 21 trains a decision-making model that inputs medical decision-making input data and outputs output data representing the results of medical decision-making based on the input data, correct answer labels, and confidence levels.

[0023] By implementing the display control function 217, the processing circuitry 21 displays various information on the display device 25 and / or the medical device terminal 3. For example, the processing circuitry 21 displays the reliability obtained by the reliability calculation function 215 or the like on the display device 25 and / or the medical device terminal 3.

[0024] The storage device 22 is a read-only memory (ROM), a random access memory (RAM), a hard disk drive (HDD), a solid state drive (SSD), an integrated circuit storage device, or the like that stores various types of information. In addition to the above storage devices, the storage device 22 may also be a drive device that reads and writes various types of information from portable storage media such as a compact disc (CD), a digital versatile disc (DVD), or a flash memory, or a semiconductor memory element. The storage device 22 may also be located in another computer connected via a network.

[0025] The input device 23 receives various input operations from an operator, converts the received input operations into electrical signals, and outputs the electrical signals to the processing circuit 21. Specifically, the input device 23 is connected to input devices such as a mouse, a keyboard, a trackball, a switch, a button, a joystick, a touchpad, and a touch panel display. The input device 23 outputs electrical signals corresponding to the input operations to the input device to the processing circuit 21. The input device 23 may also be an input device provided in another computer connected via a network or the like.

[0026] The communication device 24 is an interface for transmitting and receiving various information to and from other computers, such as the medical device terminal 3, included in the medical information processing system 1.

[0027] The display device 25 displays various information in accordance with the display control function 217 of the processing circuit 21. For example, a liquid crystal display (LCD), a cathode ray tube (CRT) display, an organic electroluminescence display (OLED), a plasma display, or any other display may be used as the display device 25. A projector may also be used as the display device 25.

[0028] Next, a description will be given of an example of the operation of the medical information processing device 2. In the following description, it is assumed that the task of the decision-making model is to determine an examination protocol. The decision-making model that determines the examination protocol will be called an examination protocol classification model.

[0029] FIG. 5 is a diagram showing an example of the flow of medical information processing by the medical information processing device 2. As shown in FIG. 5, the processing circuitry 21 acquires test data by implementing the acquisition function 211 (step S1). In step S1, the processing circuitry 21 acquires test data related to a patient who is the subject of medical decision-making. In this embodiment, the medical decision-making is to determine an examination protocol, so the test data is acquired as data that can be referenced when determining the examination protocol. The test data is acquired from a hospital information system such as an HIS (Hospital Information System) or an RIS (Radiology Information System). Some or all of the test data is used as input data for medical decision-making.

[0030] When step S1 is performed, the processing circuitry 21 displays a work screen for determining an examination protocol (step S2) by implementing the display control function 217. In step S2, the processing circuitry 21 displays a work screen with a predetermined layout.

[0031] When step S2 is performed, the processing circuitry 21 assigns a correct label by implementing the assignment function 212 (step S3). In step S3, the processing circuitry 21 assigns a correct label on the work screen displayed in step S2 in accordance with instructions from the operator via the input device 23 and / or the medical device terminal 3. The examination protocol is assigned as the correct label. During the correct label assignment work, the processing circuitry 21 collects situation data representing the situation of the operator during the correct label assignment work by implementing the collection function 213. The processing circuitry 21 collects data regarding the operator's operations, gaze, speech, and / or facial expressions as situation data, which reflect the operator's decision-making process during the assignment work.

[0032] FIG. 6 is a diagram schematically illustrating the assignment of correct labels and the collection of situation data in step S3. As shown in FIG. 6, a work screen 61 for assigning correct labels is displayed on the display device 25. The work screen 61 includes a selection area 611 for a target patient for whom an examination protocol is to be determined, a display area 612 for displaying examination data related to the target patient, and a selection area 613 for an examination protocol that is a correct label. The selection area 611 displays a list of candidate patient names, each of which can be selected. FIG. 6 illustrates an example in which "Patient Taro" is selected as the target patient. The display area 612 displays examination data related to the target patient "Patient Taro." The examination data specifically displays order information such as age, gender, presence or absence of contrast agent, imaging range, and diagnosed disease name. The selection area 613 displays a list of examination protocol names or symbols, each of which can be selected. FIG. 6 illustrates that "Protocol B" has been selected as the correct label. Once the examination protocol is selected, a confirm button is pressed via the input device 23.

[0033] When the worker assigns a correct label, the processing circuitry 21 collects an event log 62 obtained via the work screen 61 during the worker's process of determining the correct label. The event log 62 is raw data related to situation data. For example, the event log 62 collects an operation log of the worker's screen operations via the input device 23 and a log of the worker's gaze at the work screen 61.

[0034] As shown in FIG. 6, the event log 62 records the following items in chronological order: date and time, user name, event name, and acquisition location. The date and time is the time when the event occurred. The user name is the name of the worker who assigned the correct label. The event name indicates the event type. The acquisition location indicates the location on the work screen where the event occurred. As an example, user "Doctor Taro" selected "Patient Taro" in the selection area 611 with the mouse at date and time "2021-02-22 16:53:14.854," so the event name "Mouse Click" and acquisition location "Patient Taro" are recorded as the operation log. As another example, user "Doctor Taro" gazed at "Age" in the display area 612 at date and time "2021-02-22 16:53:22.432," so the event name "Gaze" and acquisition location "Age" are recorded as the gaze log. It is preferable to determine whether or not the worker has gazed, based on whether or not the worker's line of sight has stayed on the same display item on the work screen 61 for a predetermined period of time or more. The display item at the point of the worker's line of sight can be calculated based on the correspondence between the position of the worker's eyeballs shown in the image captured by an optical camera provided on the display device 25 or the like, and the position of each item on the work screen 61.

[0035] The event log 62 is recorded for each target patient. Specifically, events from the selection event of the target patient to the assignment event of the correct label, i.e., the pressing event of the confirm button, are collected as one event log 62 for the target patient. Note that an utterance log, which is a log related to the operator's utterances, and / or an expression log, which is a log related to the operator's facial expressions, may also be collected as the event log 62. The utterance log may be collected by performing voice recognition on a voice signal collected by a microphone and converting it into text information. The expression log may be collected by analyzing the facial expressions of the operator captured in images collected by an optical camera.

[0036] When an event log 62 related to the target patient is collected, the processing circuit 21 generates situation data 63 based on the event log 62. The situation data 63 includes, for example, judgment time information and reference items. The judgment time information is information related to the time required to assign a correct label. More specifically, it is the elapsed time from the time the target patient is selected to the time the confirmation button for the examination protocol is pressed. The reference items are information related to the display items in the examination data that the operator gazed at. More specifically, the reference items are information on the acquisition position related to the event name "gazing." The situation data 63 related to the target patient is stored in the storage device 22. In addition, a combination of the situation data 63 related to the target patient and the correct label is stored in the storage device 22 as one training sample.

[0037] The judgment time may be subdivided and recorded according to the reference item. For example, x seconds from patient selection to patient information confirmation, x seconds from patient information confirmation to protocol confirmation, etc. The level of attention to each reference item may also be quantified. For example, 3 seconds was spent on the presence or absence of contrast medium, and 10 seconds was spent on the diagnosed disease name, or the presence or absence of contrast medium was checked (reviewed) three times in one diagnosis.

[0038] After step S3 is performed, the processing circuit 21 determines whether or not to start training of the reliability determination model by implementing the first learning function 214 (step S4). In step S4, the processing circuit 21 determines whether or not the number of collected training samples has reached the number required for training the reliability determination model. If the number of collected training samples is less than the required number, training of the reliability determination model is not started, and steps S1 to S3 are repeated for a different patient.

[0039] Then, when the number of collected training samples reaches the required number and it is determined that training of the reliability judgment model should be started (step S4: YES), the processing circuit 21 trains the reliability judgment model based on the situation data and the correct label by implementing the first learning function 214 (step S5).

[0040] FIG. 7 illustrates the input / output relationship of the reliability determination model 71 during the learning phase. As shown in FIG. 7, the processing circuitry 21 trains the learning parameters of the reliability determination model 71, which is a multi-class classification model, based on supervised learning using situation data 72 as input and correct labels 73 as supervised learning. Classes may be established according to the candidate examination protocols. For example, if there are three candidates, "Protocol A," "Protocol B," and "Protocol C," three classes are established. In FIG. 7, the situation data 72 includes a judgment time of "40 seconds," reference items "patient name," "age," "contrast agent presence / absence," and "diagnosed disease," and the correct label 73 includes the examination protocol "Protocol A." During the training process, the processing circuitry 21 inputs the situation data 72 into the reliability determination model 71 and performs a forward propagation operation to calculate a predicted label. It then calculates the error between the predicted label and the correct label 73 and updates the learning parameters using an optimization method such as stochastic gradient descent. The predicted label is the output of the reliability determination model 71 and is a vector quantity representing the probability of each class. The learning parameters are optimized to minimize the above error by repeating update calculations using multiple training samples. A trained reliability determination model is generated by assigning the optimized learning parameters to the machine learning model. The learning parameters refer to the weight coefficients and biases of the functions representing the transformation between layers included in the reliability determination model 71.

[0041] As described above, by training the reliability determination model 71 based on supervised learning in which the situation data 72 is input and the correct label 73 is used as a teacher, the reliability determination model 71 learns the correlation between the situation data 72 and the correct label 73. Here, the correlation between the situation data 72 and the correct label 73 will be described.

[0042] FIG. 8 is a diagram illustrating an example of training data. As shown in FIG. 8, the training data includes situation data (judgment time and reference items) and correct answer labels (examination protocols). From the training data, it can be seen that protocol A takes a relatively long time to make a judgment and tends to refer to a large amount of information, such as the presence or absence of contrast and the imaging range. In contrast, protocol B tends to take a short time to make a judgment and refer to less information. The reliability assessment model learns these tendencies. For example, from the training data, it can be seen that protocol B was selected by an operator in a relatively short time of 20 seconds, after checking only the patient's name and diagnosed disease. When the situation data includes a short judgment time and a small number of reference items, the reliability assessment model outputs a higher probability of protocol B being selected than the probability of protocol A being selected. The probability of each protocol being selected represents the degree of appropriateness of the protocol being selected based on the input situation data; in other words, it represents the degree of reliability of the protocol. In this embodiment, this probability is used as the reliability of labeling.

[0043] After step S5 is performed, the processing circuit 21 realizes the reliability calculation function 215 to calculate the reliability from the situation data using the trained reliability determination model (step S6).

[0044] FIG. 9 is a diagram showing the input / output relationship of the trained reliability determination model 91 in the operational phase. As shown in FIG. 9, the reliability determination model 91 receives situation data 92 as input and outputs reliability 93. As described above, the reliability determination model 91 is a class classification model, and there are two classes: protocol A and protocol B. The reliability determination model 91 outputs the probability (likelihood) of each protocol being correct. The probability of correctness is used as the reliability. In FIG. 9, the situation data 92, like the situation data 72 in FIG. 7, includes a judgment time of "40 seconds," reference items "patient name," "age," "presence or absence of contrast agent," and "diagnosed disease name," and the reliability 93 includes protocol A "20%" and protocol B "80%."

[0045] In step S6, the processing circuitry 21 calculates the reliability of the condition data of each target patient. The reliability is registered in the reliability database in association with the condition data.

[0046] Fig. 10 is a diagram showing an example of a reliability database. As shown in Fig. 10, the reliability database associates situation data, correct labels, and reliability. Reliabilities are registered for each of protocol A and protocol B. For example, for ID "1," the reliability is registered as "0.95" for protocol A and "0.05" for protocol B.

[0047] When step S6 is performed, the processing circuit 21 trains the examination protocol classification model based on the input data, the correct label, and the reliability by implementing the second learning function 216 (step S7). The input data is input data for the examination protocol classification model and is part of the examination data. That is, the input data is data for items in the examination data that are recognized to be correlated with the examination protocol. The items of the input data are determined in advance. The input data is assumed to be associated with the situation data, the correct label, and the reliability via an ID or the like.

[0048] Fig. 11 is a diagram showing an example of input data. As shown in Fig. 11, the input data includes data on the patient's age (e.g., "70" or "75"), the patient's gender (e.g., "male" or "female"), the diagnosis (e.g., "hepatocellular carcinoma" or "renal cell carcinoma"), and whether or not a contrast agent is used (e.g., "yes" or "no").

[0049] FIG. 12 is a diagram showing the input / output relationship of the examination protocol classification model 121 in the learning phase. The processing circuit 21 identifies input data 122 to be processed and extracts a correct label 123 and a reliability 124 associated with the input data 122 from a reliability database. As shown in FIG. 12, the processing circuit 21 trains learning parameters of the examination protocol classification model 121, which is a multi-class classification model, based on supervised learning using the input data 122 as an input and the correct label 123 weighted by the reliability 124 as a teacher. In FIG. 12, the input data 122 includes a patient age of "70," a patient sex of "male," a diagnosed disease name of "hepatocellular carcinoma," and the presence or absence of a contrast agent of "absence," and the correct label 73 is "protocol A," and the reliability 124 is "0.95."

[0050] In the training process, the processing circuit 21 inputs the input data 122 into the examination protocol classification model 121, performs a forward propagation operation to calculate the output (hereinafter referred to as the model output), calculates the error between the model output and the correct label 73, and updates the learning parameters using the error according to an optimization method such as the stochastic gradient descent method. The learning parameters are optimized so as to minimize the error by repeating the update operation using multiple training samples.

[0051] Here, the error is expressed by the loss function L(T, p, w) based on the correct label T, model output p, and the reliability w of the correct label T, as shown in the following equation (1). Note that the subscript k represents the number of the training sample.

[0052]

number

[0053] As shown in equation (1), the loss function L(T, p, w) is defined as the sum of cross-entropy, which represents the error between the correct label T weighted by the confidence level w and the model output p, across multiple training samples. As an example, in the case of the training sample shown in FIG. 12, the confidence level 124 of the correct label 123 is "0.95," so when calculating the cross-entropy, "0.95" is multiplied by the value "1.00" of the correct label 123 "Protocol A." The learning parameters of the testing protocol determination model are trained to minimize the loss function L(T, p, w) across multiple training samples k. A trained testing protocol classification model is generated by assigning the optimized learning parameters that minimize the loss function L(T, p, w) to the machine learning model.

[0054] Fig. 13 is a diagram showing the input / output relationship of the trained inspection protocol classification model 131 in an operational phase. As shown in Fig. 13, the inspection protocol classification model 131 receives input data 132 and outputs an inspection protocol 133. More specifically, it calculates the probability of each of protocol A and protocol B being applicable, and outputs the name of the protocol with the highest probability of application as the inspection protocol 133. Note that the inspection protocol classification model 131 may be designed to output the probability of each of protocol A and protocol B being applicable.

[0055] When step S7 is performed, the medical information processing according to this embodiment ends.

[0056] The above-described flow of medical information processing is merely an example, and the present embodiment is not limited thereto. For example, the processing circuit 21 is configured to successively train the reliability determination model (S5) and the examination protocol classification model (S7). However, the training of the examination protocol classification model (S7) may be performed at any time after the training of the reliability determination model (S5), such as several days, several weeks, or several months. Furthermore, the training of the reliability determination model (S5) and the training of the examination protocol classification model (S7) do not have to be performed by the same medical information processing device 2, but may be performed by different medical information processing devices. Furthermore, multiple training samples (correct labels and situation data) do not have to be collected by the same medical information processing device 2, but may be collected by different medical information processing devices.

[0057] According to the above embodiment, the medical information processing device 2 has an assignment function 212, a collection function 213, a first learning function 214, an acquisition function 211, and a second learning function 216. The assignment function 212 assigns correct labels used to train a decision-making model, which is a machine learning model used for medical decision-making, in accordance with input instructions from an operator. The collection function 213 collects situation data, which is data representing the situation of the operator when assigning the correct labels. The first learning function 214 trains a reliability determination model, which is a machine learning model that inputs situation data and outputs the reliability of the correct labels, based on the situation data and the correct labels. The acquisition function 211 acquires input data for the decision-making model. The second learning function 216 trains the decision-making model that inputs input data and outputs output data, which is data representing the results of decision-making, based on the input data, the correct labels, and the reliability.

[0058] The above configuration makes it possible to evaluate the reliability of each correct label based on situation data that represents the situation in which the worker assigns the correct label. Since the decision-making model is trained based on correct labels evaluated with reasonable reliability, correct labels with low reliability relatively do not contribute to learning, making it possible to improve the prediction accuracy of the decision-making model.

[0059] The above medical information processing is an example, and the present embodiment is not limited to this, and various modifications are possible.

[0060] (Variation 1) In the above embodiment, the reliability determination model is trained based on the situation data and the correct answer label. However, this embodiment is not limited to this. Hereinafter, training of the reliability determination model according to the first modification will be described.

[0061] FIG. 14 is a diagram showing the input / output relationship of the reliability determination model 141 according to Modification 1 in the learning phase. As shown in FIG. 14, the processing circuit 21 trains the learning parameters of the reliability determination model 141, which is a multi-class classification model, based on supervised learning in which situation data 142 and skill data 143 are input and a correct label 144 is used as a teacher. The situation data 142, as in the above embodiment, is data representing the situation of the worker at the time the correct label is assigned. The correct label 144, as in the above embodiment, is a label assigned by the worker. The skill data 143 is data representing the skill of the worker. Specifically, as shown in FIG. 14, the name of the worker is used as the skill data 143. The skill data 143 is not limited to the name of the worker, and any data correlated with skill, such as years of service, years of experience, or job title, may be used.

[0062] During the training process, the processing circuit 21 inputs the situation data 142 and skill data 143 into the reliability judgment model 141 and performs a forward propagation operation to calculate a predicted label, calculates the error between the predicted label and the correct label 144, and updates the learning parameters using the error according to an optimization method such as stochastic gradient descent. The learning parameters are optimized so as to minimize the error by repeating the update operation using multiple training samples. A trained reliability judgment model is generated by assigning the optimized learning parameters to a machine learning model. The above training method makes it possible to generate a reliability judgment model that inputs situation data and skill data and outputs a reliability. As in the above embodiment, the reliability is used as a weight in training the examination protocol classification model.

[0063] FIG. 15 is a diagram showing an example of training data according to Modification 1. As shown in FIG. 15, the training data includes skill data (operator name), situation data (judgment time and reference items), and correct answer label (examination protocol). From the training data, it can be seen that Doctor A tends to select Protocol A when referring to many reference items and taking a long judgment time, and Doctor B tends to select Protocol B when referring to many reference items and taking a long judgment time. As such, since skill data may also have a correlation with the reliability of the correct answer label, it is possible to further improve the accuracy of the reliability by training a reliability determination model using skill data in addition to situation data.

[0064] (Variation 2) The training of the reliability determination model according to the second modification will be described below.

[0065] FIG. 16 is a diagram showing the input / output relationship of the reliability determination model 161 according to Modification 2 in the learning phase. As shown in FIG. 16, the processing circuit 21 trains learning parameters of the reliability determination model 161, which is a multi-class classification model, based on supervised learning using situation data 162, skill data 163, and additional data 164 as inputs and a correct label 165 as a teacher. The situation data 162, as in the above embodiment, is data representing the situation of the worker at the time of assigning the correct label. The skill data 163, as in Modification 1, is data representing the skill of the worker. The correct label 165, as in the above embodiment, is a label assigned by the worker. The additional data 164 is data acquired before and after the correct label assignment task and is expected to correlate with reliability. Specifically, the additional data 164 is data related to freshness, confidence, quality, and / or required time. FIG. 16 illustrates freshness and confidence as examples of the additional data 164.

[0066] Freshness is the time when the worker assigned the correct label. The newer the time, the fresher the freshness. The higher the freshness, the higher the reliability of the correct label. Confidence represents the subjective degree of confidence the worker felt in the assigned correct label or the labeling process. The confidence is entered by the worker after assigning the correct label. For example, if the worker recognizes that it took a long time to assign the correct label, a lower confidence level is assigned compared to a worker who recognizes that it took less time. A higher confidence level indicates a higher reliability of the correct label. Quality represents the quality of the information used as a reference when assigning the correct label. For example, when annotating a medical image as a correct label, information about the quality of the medical image is used. Specifically, the quality includes the image format of the medical image, the presence or absence of artifacts, differences in SNR, and shooting conditions. Required time is the difference between a preset optimal value for required time (judgment time) and the actual required time (judgment time), or a score based on this difference.

[0067] During the training process, the processing circuit 21 inputs the situation data 162, skill data 163, and additional data 164 into the reliability judgment model 161, performs a forward propagation operation to calculate a predicted label, calculates the error between the predicted label and the correct label 165, and updates the learning parameters using the error according to an optimization method such as stochastic gradient descent. The learning parameters are optimized so as to minimize the error by repeating the update operation using multiple training samples. A trained reliability judgment model is generated by assigning the optimized learning parameters to a machine learning model. The above training method makes it possible to generate a reliability judgment model that inputs the situation data 162, skill data 163, and additional data 164 and outputs a reliability. As in the above embodiment, the reliability is used as a weight in training the examination protocol classification model.

[0068] By training the reliability determination model using additional data in addition to the situation data and skill data, it is expected that the accuracy of the reliability will be improved more than in the second modification.

[0069] (Variation 3) In the various embodiments described above, the reliability output from the reliability determination model is used as a weight in training a decision-making model such as an examination protocol classification model. However, this embodiment is not limited to this. The processing circuit 21 according to the third modification displays the reliability. The display of the reliability will be described below.

[0070] The decision-making model according to the third modification is an image diagnosis model that receives a medical image as input and outputs annotations representing disease candidate regions. The annotations are included in the correct answer labels. More specifically, the image diagnosis model is a multi-class classification model that outputs the probability of each disease candidate for each unit region, such as a pixel. The image diagnosis model outputs the probability of each disease candidate for each unit region, and outputs the disease candidate with the highest probability of being identified. The processing circuitry 21 holds a color table that defines the correspondence between disease candidates and color values, and determines the disease candidate with the highest probability of being identified for each unit region and displays the unit region with a color value corresponding to the disease candidate. A collection of unit regions displayed with a color value corresponding to the disease candidate forms an annotation.

[0071] Annotations are used as correct labels to train the image diagnostic model, and as in the above embodiment, the annotations are added by an operator. The reliability determination model according to the third modification is trained based on the annotations and situation data related to the time of annotation addition, and inputs the situation data to output a reliability for each unit region. The processing circuitry 21 displays the reliability on the display device 25 by superimposing it on the medical image.

[0072] FIG. 17 is a diagram showing an example of a reliability display screen 170. As shown in FIG. 17, a medical image 171 and a text information display field 172 are displayed on the display screen 170. The medical image 171 is a medical image to which an annotation, which is a correct label, is to be added. An annotation 173 added by an operator is superimposed on the medical image 171. Each pixel of the annotation 173 is assigned a color value (color code) according to the reliability determined for each pixel by a reliability determination model, and the annotation 173 is displayed with a color value according to the reliability. The correspondence between the reliability and the color value is defined by a color table 174, which may be displayed on the display screen 170 so that an operator or the like can understand the relationship between the reliability and the color value. The text information display field 172 displays test data, etc. of the patient who is the subject of the medical image 171, which was used as a reference when the annotation 173 was added.

[0073] Typically, the display screen 170 is viewed by the worker who added the annotation 173. By displaying the annotation 173 in different colors according to the reliability, the reliability of each region of the annotation 173 can be easily confirmed. For example, it becomes possible to prompt the worker to reconsider the annotation for a region with a low reliability.

[0074] The correspondence between reliability and color value can be designed arbitrarily according to requirements. As an example, reliability is divided into an arbitrary number of color values, such as four, as shown in Fig. 17. In this case, the reliability range for each color value can be set by equally dividing the possible range of reliability (for example, from "0" to "1").

[0075] FIG. 18 is a diagram showing an example of the correspondence between the frequency distribution of reliability and color values. FIG. 19 is a diagram showing another example of the correspondence between the frequency distribution of reliability and color values. As shown in FIGS. 18 and 19, the correspondence between reliability and color values ​​may be set according to the frequency distribution of reliability. As an example, color values ​​are set based on reliability and frequency. When there are four color value categories, the color value categories are divided into four groups, from lowest to highest reliability: lowest frequency to medium frequency, medium frequency to highest frequency, highest frequency to medium frequency, and medium frequency to lowest frequency. As shown in FIG. 18, when the frequency of reliability is evenly distributed around the intermediate value "0.5," the boundary, color value categories are set according to the range of reliability (e.g., "0" to "1") divided equally.

[0076] 19, for example, when the difficulty level of annotation is relatively high, the frequency is biased toward lower reliability, and the lower the reliability, the more detailed the color value categories are set. In other words, the processing circuit 21 sets the correspondence between reliability and color value according to the difficulty level of annotation. When there are a finite number of color value categories, it is possible to set the color value categories in detail within the reliability range to be focused on according to the difficulty level.

[0077] The display of the reliability is not limited to when the decision-making model is an image diagnosis model, but can be applied to any type of model, such as an examination protocol classification model.

[0078] According to at least one of the embodiments described above, it is possible to improve the prediction accuracy of a machine learning model.

[0079] The term "processor" used in the above description refers to a circuit such as a CPU, a GPU, an application specific integrated circuit (ASIC), a programmable logic device (e.g., a simple programmable logic device (SPLD), a complex programmable logic device (CPLD), and a field programmable gate array (FPGA)). A processor realizes its functions by reading and executing a program stored in a memory circuit. Note that instead of storing a program in a memory circuit, a program may be directly embedded in the processor circuit. In this case, the processor realizes its functions by reading and executing the program embedded in the circuit. Furthermore, instead of executing a program, a function corresponding to the program may be realized by combining logic circuits. Note that each processor in this embodiment is not limited to being configured as a single circuit, but may be configured as a single processor by combining multiple independent circuits to realize its function. Furthermore, multiple components in FIGS. 1 and 2 may be integrated into a single processor to realize its function.

[0080] Although several embodiments have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, and combinations of embodiments can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, as well as within the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0081] 1 Medical information processing system 2 Medical information processing equipment 3 Medical device terminals 21 Processing circuit 22 Storage device 23 Input Devices 24. Communications equipment 25 Display equipment 211 Acquisition Function 212 Grant Function 213 Collection Function 214 First Learning Function 215 Reliability calculation function 216 Second Learning Function 217 Display Control Function

Claims

1. an assigning unit that assigns correct labels to be used in training a decision-making model, which is a machine learning model used for medical decision-making, in accordance with input instructions from an operator; a collection unit that collects situation data that represents the situation of the worker when the correct label assignment task is performed; a first learning unit that trains a reliability determination model, which is a machine learning model that inputs the situation data and outputs the reliability of the correct label, based on the situation data and the correct label; an acquisition unit that acquires input data for the decision-making model; a second learning unit that trains the decision-making model based on the input data, the correct label, and the reliability, and that inputs the input data and outputs output data that represents the result of the decision-making; A medical information processing device comprising:

2. The medical information processing apparatus according to claim 1 , wherein the collection unit collects, as the situation data, data relating to the operator's operation, line of sight, speech, and / or facial expression, which reflects the decision-making process of the operator during the implementation.

3. The medical information processing device according to claim 2, wherein the collection unit collects reference item data, which is data relating to an item on which the worker's gaze is focused among various items displayed on the display screen of the assignment work, as the data relating to the gaze.

4. The medical information processing apparatus according to claim 3 , wherein the collection unit collects, as the reference item data, an identifier of a reference item that is an item on which the operator's gaze is focused.

5. The collection unit further collects skill data which is data related to the skill of the worker, the first learning unit trains the reliability determination model, which receives the situation data and the skill data and outputs the reliability, based on the situation data, the skill data, and the correct label; The medical information processing device according to claim 1.

6. The collecting unit further collects additional data, which is data relating to freshness, reliability, quality, and / or required time, the first learning unit trains the reliability determination model, which inputs the situation data, the skill data, and the additional data and outputs the reliability, based on the situation data, the skill data, the additional data, and the correct label; The medical information processing device according to claim 5.

7. The medical information processing apparatus according to claim 1 , wherein the reliability determination model is a multi-class classification model that outputs the probability of each of a plurality of classes relating to the decision-making result as the reliability.

8. the second learning unit trains the decision-making model to minimize a loss function; The loss function includes an error between the output of the decision-making model and the correct label weighted by the confidence level. The medical information processing device according to claim 1.

9. The medical information processing apparatus according to claim 1 , further comprising a display control unit that displays the reliability via a display device.

10. The decision-making is annotating a disease candidate region in a medical image; the display control unit displays the annotation with a color value according to the reliability. The medical information processing device according to claim 9.

11. The medical information processing apparatus according to claim 10 , wherein the correspondence between the reliability and the color value is set according to a degree of difficulty in adding the annotation.

12. On the computer, An assignment function that assigns correct labels to be used in training a decision-making model, which is a machine learning model used for medical decision-making, according to input instructions from the operator; and a collection function for collecting situation data that represents the situation of the worker when assigning the correct label; a first learning function for training a reliability determination model, which is a machine learning model that inputs the situation data and outputs the reliability of the correct label, based on the situation data and the correct label; an acquisition function for acquiring input data for the decision-making model; a second learning function for training the decision-making model based on the input data, the correct label, and the confidence level, which inputs the input data and outputs output data that represents the result of the decision-making; A medical information processing program that makes this possible.

Images