Information processing device, information processing method, program, and information processing system

Abstract

To provide an information processing system configured to properly notify a user who has made an inference with a learned model before modification, when the learned model is modified by update or the like.SOLUTION: An information processing apparatus including a model information storage unit which stores a learned model and configured to make an inference on medical image data by using the learned model stored in the model information storage unit includes: an inference information storage unit which stores an inference object subjected to first inference by using a first learned model; an inference unit which performs second inference on the inference object by using a second learned model when the first learned model is changed to the second learned model; a comparison unit which compares a first inference result obtained by the first inference with a second inference result obtained by the second inference; and a notification unit which notifies a user on the basis of a comparison result of the comparison unit.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, a program, and an information processing system that provide a notification when a trained model is changed. [Background technology]

[0002] There is known a Computer Aided Diagnosis (CAD) system that analyzes medical data acquired by a medical imaging device and presents diagnostic support information to a doctor. The CAD system applies machine learning technology to medical data, such as medical image data, and outputs diagnostic support information.

[0003] Patent Document 1 discloses that in order to improve the reliability of analysis based on machine learning, multiple processed medical signals related to a subject are acquired by performing at least one of different imaging methods and different signal processing, and then inference is performed using multiple trained models for each of the multiple acquired processed medical signals. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-42810 A Summary of the Invention [Problem to be solved by the invention]

[0005] Patent Document 1 does not disclose how to notify users who performed inference using the trained model before the change when the trained model is changed due to an update or the like.

[0006] Therefore, the present invention aims to provide an information processing device that can appropriately notify users who performed inference using a trained model before the model was changed when the trained model is changed due to an update or the like. [Means for solving the problem]

[0007] In order to achieve the object of the present invention, an information processing device according to the present invention is an information processing device having a model information memory unit that stores a learned model, and performs inference on medical image data using the learned model stored in the model information memory unit, and is equipped with an inference information memory unit that stores an inference target for which a first inference was performed using a first learned model, an inference unit that performs a second inference on the inference target using the second learned model when the first learned model is changed to a second learned model, a comparison unit that compares the first inference result based on the first inference with the second inference result based on the second inference, and a notification unit that notifies a user based on the comparison result by the comparison unit. Effect of the Invention

[0008] According to the present invention, when a trained model is changed due to an update or the like, it is possible to appropriately notify users who performed inference using the trained model before the change. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing a configuration of an information processing system according to the present invention. [Diagram 2] FIG. 1 is a diagram showing a configuration of an information processing device according to a first embodiment of the present invention. [Diagram 3] FIG. 2 is a diagram showing the configuration of a storage unit in the information processing device of the present invention. [Figure 4] FIG. 2 is a diagram showing a configuration of a model information storage unit in the storage unit in the information processing device of the present invention. [Diagram 5] FIG. 2 is a diagram showing a configuration of an inference information storage unit of the storage unit in the information processing device of the present invention. [Figure 6] FIG. 2 is a flowchart illustrating a processing procedure of the information processing device according to the first embodiment of the present invention. [Figure 7] FIG. 11 is a diagram showing the configuration of an inference information storage unit of a storage unit according to a second embodiment of the present invention. [Figure 8]FIG. 11 is a flowchart for explaining a processing procedure of an information processing device according to a second embodiment of the present invention. [Figure 9] FIG. 11 is a diagram showing the configuration of an information processing device according to a third embodiment of the present invention. [Figure 10] FIG. 11 is a diagram showing the configuration of an information processing device according to a fourth embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings.

[0011] [First embodiment] FIG. 1 shows the configuration of an information processing system according to the present invention.

[0012] The information processing system of the present invention comprises a medical imaging device 100 that acquires medical data related to a subject, an information terminal 110, a network 120, and an information processing device 130. The information terminal 110 may be a single information terminal or may be a plurality of information terminals.

[0013] An operation unit 112 and a display unit 114 are connected to the information terminal 110. The operation unit 112 accepts various instructions from an operator and transmits the various instructions to the information terminal 110 and the medical imaging apparatus 100. The operation unit 112 is composed of, for example, a mouse, a keyboard, a button, a panel switch, a foot switch, a trackball, a joystick, etc. The display unit 114 displays a GUI for inputting various instructions in the operation unit 112, and displays image data based on medical data acquired by the medical imaging apparatus 100. The information terminal 110 is capable of communicating with the information processing apparatus 130 via a network.

[0014] In FIG. 1, operation unit 112 and display unit 114 are provided separately from information terminal 110, but information terminal 110 may have the functions of operation unit 112 and display unit 114 internally.

[0015] The medical imaging apparatus 100 is an apparatus that acquires medical data of a subject, such as an X-ray CT (Computed Tomography) apparatus, an MRI (Magnetic Resonance Imaging) apparatus, or an ultrasonic diagnostic apparatus.

[0016] X-ray CT equipment is equipped with an X-ray source and an X-ray detector. While rotating the X-ray source and the X-ray detector around the subject, the subject is irradiated with X-rays from the X-ray source and the data detected by the X-ray detector is projected to generate CT data.

[0017] The MRI apparatus generates a predetermined magnetic field for a subject placed within a static magnetic field, and generates MRI data by performing a Fourier transform on the acquired data.

[0018] An ultrasound diagnostic device transmits ultrasound waves to a subject, receives ultrasound waves reflected from the subject, and generates ultrasound data.

[0019] The medical data (CT data, MRI data, ultrasound data, etc.) generated by the medical imaging device 100 is three-dimensional data (volume data) or two-dimensional data. The medical data is, for example, image data relating to a subject. The image data also includes raw data. The medical image data may be moving image data consisting of multiple frame data. The medical data also includes measurement data obtained by performing various measurements using the medical image data.

[0020] The medical imaging apparatus 100 is connected to an information terminal 110. The information terminal 110 is a PC terminal, a mobile phone such as a smartphone, a notebook terminal, a tablet terminal, or the like. The information terminal 110 can also set subject information and can associate medical data acquired from the medical imaging apparatus 100 with the subject information. The information terminal 110 can also display various types of data, such as medical data and measurement data acquired from the medical imaging apparatus 100.

[0021] The information terminal 110 and the information processing device 130 are connected to a network 120. The network 120 includes a communication network outside the hospital, such as wireless communication (Wi-Fi), the Internet, a wireless base station, a provider, a communication line, etc. The network 120 may also include an intranet, which is a communication network within the hospital. The information terminal 110 is capable of communicating with the information processing device 130 via the network 120. The information terminal 110 can transmit medical data (including medical image data) to the information processing device 130. The information processing device 130 can transmit an inference result obtained by performing inference using the medical data (including medical image data) to the information terminal 110.

[0022] 2 shows the configuration of the information processing device 130 of the present invention. The information processing device 130 has a teacher data generation unit 200 that generates teacher data using medical image data, and a learning unit 202 that performs learning on the medical image data using the teacher data generated in the teacher data generation unit 200. The information processing device 130 further has a storage unit 204 that stores model information composed of a trained model generated in the learning unit 202 and an inference target, and an inference unit 206 that performs inference using the trained model. The information processing device 130 also has a comparison unit 208 that compares an inference result for an inference target inferred using the trained model before the update with an inference result for an inference target inferred using the updated trained model when the trained model is changed by an update or the like, and a notification unit 210 that notifies a user based on the comparison result by the comparison unit 208.

[0023] The teacher data generating unit 200 is connected to the network 120 and can acquire medical data including medical image data, measurement data, etc. The teacher data generating unit 200 generates teacher data using the medical image data. The teacher data generated by the teacher data generating unit 200 is determined according to the inference task and classification target performed by the model based on the neural network.

[0024] Inference tasks performed by neural networks include classification tasks to classify the classes of medical image data, detection tasks to detect what is in which position in the medical image data, and segmentation tasks to extract target regions from medical image data.

[0025] When learning a model based on a neural network to perform a classification task, the teacher data generation unit 200 generates teacher data that pairs medical image data with a correct answer label, which is a label indicating an object that appears in the medical image data.

[0026] On the other hand, a case where a detection task is performed using a model based on a neural network will be described. The training data generating unit 200 generates training data that pairs medical image data with a correct answer image to which a ROI (Region Of Interest) indicating the position of an object reflected in the medical image data and a correct answer label indicating the object are assigned to the medical image data.

[0027] The case where the task performed by the neural network-based model is segmentation will be described below. The training data generating unit 200 generates training data that pairs medical image data with a correct answer image to which position information of pixels of an object reflected in the medical image data and a correct answer label, which is a label indicating the object, are assigned.

[0028] The following describes a case where a model based on a neural network that performs the task of segmenting (extracting) a lesion area is trained. The training data generating unit 200 generates training data that pairs medical image data having a lesion area with information on a correct answer label indicating the type of lesion in the medical image data and a correct answer image with position information of the pixels of the lesion.

[0029] The teacher data generating unit 200 may also perform preprocessing of medical data according to the model based on a neural network that is learned by the learning unit 202. For example, when the target of inference by the model based on a neural network is medical image data, the acquired medical image data is subjected to processes such as noise removal, filtering, image cropping, and resolution change. When the target of inference is a natural language such as a sentence, the teacher data generating unit 200 performs preprocessing of the data to be processed, such as performing morphological analysis and applying vector conversion technology, according to the target of inference in the model based on a neural network and the task.

[0030] 2 shows a form in which the information processing device 130 has the functions of the teacher data generation unit 200 inside, the information terminal 110 or another information processing device may have the functions of the teacher data generation unit 200 inside. For example, after the teacher data as described above is generated by the information terminal 110 or an external information processing device, the model may be learned by the learning unit 202 of the information processing device 130 via the network 120.

[0031] The learning unit 202 is connected to the teacher data generating unit 200. The learning unit 202 generates a trained model by making the model based on the neural network learn by associating medical image data with teacher data. Here, the trained model indicates parameters determined by performing the learning process up to a predetermined standard and information on the model corresponding to the parameters. The trained model may be used to learn other models by transfer learning or fine tuning, or further learning processing (additional learning) may be performed on the trained model. The learning process by the learning unit 202 may be performed by an information processing device or information terminal 110 outside the information processing device 130. For example, the external information processing device or information terminal 110 may have the teacher data generating unit 200 and the learning unit 202 as functions, perform the learning process on the model based on the neural network, and store the generated trained model in the storage unit 204 of the information processing device 130.

[0032] A model based on a neural network includes multiple layers. Among neural networks, CNN (Convolutional Neural Network), which is a type of deep learning technology, has multiple intermediate layers between the input layer and the output layer, although not shown. The multiple intermediate layers include a convolution layer, a pooling layer, an upsampling layer, and a synthesis layer. The convolution layer is a layer that performs convolution processing on a group of input values. In the convolution layer, the input medical image data is convolved to extract the features of the medical image data.

[0033] The pooling layer is a layer that performs processing to make the number of output value groups smaller than the number of input value groups by thinning out or synthesizing the input value groups. The upsampling layer is a layer that performs processing to make the number of output value groups larger than the number of input value groups by duplicating the input value groups or adding values ​​interpolated from the input value groups. The synthesis layer is a layer that inputs value groups such as the output value groups of a certain layer or pixel values ​​constituting medical image data from multiple sources and synthesizes them by concatenating or adding them. The number of intermediate layers can be changed at any time depending on the learning content.

[0034] The storage unit 204 is connected to the learning unit 202. The storage unit 204 stores a trained model that has been trained to extract the type and area of ​​a lesion in medical image data, for example. The trained model may store, for example, a model of CNN or RNN (Recurrent Neural Network), which is one of deep learning technologies in neural network technology. In addition to models derived from CNN or RNN, models based on other machine learning technologies such as support vector machines, logistic regression, and random forests may be stored, or a method based on a rule base may be stored. The storage unit 204 further stores inference information such as an inference target for which inference is performed using the trained model stored in the storage unit 204. The storage unit 204 stores the trained model and the information of the inference target in association with each other as inference information. Furthermore, the storage unit 204 stores account information in association with the inference information, and when a user connects to the information processing device 130, for example, via the network 120, the user can refer to past inference information associated with the account information of the user. Furthermore, the account information in the storage unit 204 may be stored as a group account for multiple users.

[0035] The inference unit 206 is connected to the network 120 and can acquire medical data including medical image data, measurement data, etc. The inference unit 206 is connected to the storage unit 204 and can perform inference using a trained model stored in the storage unit 204.

[0036] The inference unit 206 performs inference on the medical image data to be inferred, for example, by using a trained model that has been trained to extract the type and area of ​​a lesion in the medical image data. When the medical image data to be inferred contains a lesion, the inference unit 206 can output the type and area of ​​the lesion.

[0037] When the trained model stored in the storage unit 204 is changed, such as updated, the comparison unit 208 compares a first inference result obtained by the trained model before the change inferring the inference target with a second inference result obtained by the inference unit 206 using the trained model after the change in the inference target. When the difference between the first inference result and the second inference result is equal to or greater than a predetermined value, the comparison unit 208 notifies at least one of the user or the manager of the model of the inference result via the notification unit 210. When the difference between the first inference result and the second inference result is less than a predetermined value, the comparison unit 208 does not notify. The comparison of the inference results by the comparison unit 208 will be described in detail below.

[0038] The notification unit 210 notifies the notification destination determined according to the comparison result by the comparison unit 208. Here, when the notification unit 210 notifies the user of the comparison result, it refers to the user's account information stored in the storage unit 204 and notifies the user. If the result of the comparison by the comparison unit 208 is less than a predetermined value, the notification unit 210 does not notify the user. With this configuration, the user can reduce the trouble of receiving a notification every time the trained model is changed, and can notify the user when the difference between the past inference result and the inference result by the updated model is large. If the account is a group account, the notification unit 210 determines multiple users constituting the group account as notification destinations. Alternatively, it may notify a specific user among the multiple users constituting the group account. Note that the notification by the notification unit 210 only needs to be understood by the user, and may be confirmed by the user accessing, for example, a display unit (not shown) provided in the information processing device 130 or the information processing device 130 via the information terminal 110 or the like. Alternatively, the information processing device 130 may notify the information terminal 110 via the network 120 .

[0039] The configuration of the information processing device 130 of the present invention makes it possible, when a trained model is changed, to appropriately notify a user who performed inference using the trained model before the update.

[0040] 3 shows the configuration of the storage unit 204 of the present invention. The storage unit 204 has a model information storage unit 300 that stores the trained model trained in the learning unit 202 and associated information, and an inference information storage unit 302 that stores information related to inference such as an inference target and account information. Note that each storage unit in the storage unit 204 may function as a single storage unit 204.

[0041] The model information storage unit 300 will be described with reference to Fig. 4. The model information storage unit 300 stores a trained model for each inference target and task. For example, the model information storage unit 300 stores a plurality of trained models, such as a trained model for detecting abnormal shadows from lung field regions and a trained model for performing differential diagnosis of lung cancer. The inference unit 206 performs inference processing using a model selected by a user via the network 120.

[0042] The model information storage unit 300 assigns the first auxiliary information 410 to the first trained model 400 that extracts abnormal shadows trained in the training unit 202 as a model for extracting abnormal shadows from the lung field region, and stores the first trained model 400. The model information storage unit 300 further stores a second trained model 402 as a model for extracting abnormal shadows from the lung field region, and the second trained model is assigned with the second auxiliary information. The model information storage unit 300 further stores a first trained model 420 as a model for performing lung cancer discrimination, and the first trained model is assigned with the first auxiliary information. The auxiliary information is information for identifying the model. The auxiliary information includes at least one of model version information, update information, inference target, and classification target information. The trained model can be identified by the auxiliary information.

[0043] Here, the trained models stored in the model information storage unit 300 are updated or changed as appropriate by the model administrator, and the larger the number, the later the trained model that was stored chronologically. Also, the trained model used by the inference unit 206 when performing inference is a newer trained model.

[0044] For example, when a model for extracting abnormal shadows from lung field regions is selected by the user, the second trained model, which is the latest model, is selected from among the models for extracting abnormal shadows from lung field regions in the model information storage unit 300. Then, the inference unit 206 performs inference processing using the selected model.

[0045] Furthermore, each of a plurality of trained models performing the same task, such as extraction of abnormal shadows in lung field regions, has at least a part of the task to be inferred, the class indicating the classification target, the model structure, and the teacher data in common. For example, a second trained model performing extraction of abnormal shadows in lung field regions may be generated by transfer learning or fine tuning the first trained model. The second trained model may also be a model generated by performing additional learning processing (additional learning) on ​​the first trained model. In other words, the second trained model is a trained model generated based on the first trained model.

[0046] The model information selected from the model information storage unit 300, the inference target, and the inference result inferred by the selected model are stored together with the user's account information in the inference information storage unit 302. Note that, although Fig. 4 shows a form in which three trained models are stored together with accompanying information in the storage unit 204, the model information storage unit 300 can also store four or more trained models together with accompanying information.

[0047] The inference information storage unit 302 will be described with reference to FIG. 5. The inference information storage unit 302 stores first account information 500 of a first user in association with inference information 510. The inference information 510 associated with the first account information 500 stores an inference target, model information used for inference performed on the inference target, and an inference result as first inference information. The inference information 510 also stores an inference target, model information, and an inference result as second inference information. The inference information storage unit 302 also stores second account information 502 in association with inference information 512. The inference information stored in the inference information storage unit 302 is managed for each account, and the inference information for each account is stored as different inference information for each inference target and model information used for inference. The information stored in the inference information storage unit 302 may be stored in its entirety or in part in the user's information terminal 110, or the storage medium may be determined depending on the capacity of the information to be stored and the line of the network 120 connecting the information terminal 110 and the information processing device 130. By operating the information terminal 110, the user can specify the model to be inferred, and refer to the inference process and inference results for the information processing device 130. Furthermore, by accessing the information stored in the inference information storage unit 302, the user can refer to past inference results for the inference target. Moreover, the account information may be stored as a group account for multiple users.

[0048] Hereinafter, a processing flow up to a notification when a trained model of the information processing device 130 in this embodiment is changed will be described with reference to FIG.

[0049] In step S600, the comparison unit 208 determines whether the trained model used for inference has been changed. If the trained model used for inference has not been updated, the process ends. If the trained model used for inference has been updated, the process proceeds to step S602.

[0050] In step S602, the comparison unit 208 determines whether an inference result by the first trained model before the update for the inference target is stored in the inference information storage unit 302 in the storage unit 204. The inference information storage unit 302 stores inference information in association with account information, and stores, for example, the inference target, the model information used for inference, and the inference result as the inference information. However, when the data storage capacity of the inference result is large, a part of the inference information may be stored in the user's information terminal 110 or an external information processing device. Therefore, if the first inference result by the first trained model is not stored in the inference information stored in the inference information storage unit 302, proceed to step S604. If the first inference result by the first trained model is stored, proceed to step S606.

[0051] In step S604, the comparison unit 208 acquires the inference target and model information used for inference from the inference information storage unit 302, acquires the model used for the inference from the model information storage unit 300, and transmits information on the model and the inference target to the inference unit 206. The inference unit 206 acquires a first inference result for the inference target transmitted from the comparison unit 208, using the model (first trained model) transmitted from the comparison unit 208. When the comparison unit 208 acquires the first inference result, the process proceeds to step S606.

[0052] In step S606, the inference unit 206 performs a second inference on the inference target using the second trained model, which is the changed trained model, and obtains a second inference result. After obtaining the second inference result, the comparison unit 208 proceeds to step S608.

[0053] In step S608, the comparison unit 208 compares the first inference result by the acquired first trained model with the second inference result by the second trained model. The comparison unit 208 determines whether the difference between the first inference result and the second inference result is equal to or greater than a predetermined value, and ends the process if it is less than the predetermined value. If the difference between the first inference result and the second inference result is equal to or greater than a predetermined value, the process proceeds to step S610. Here, the method of comparing both inference results differs depending on the output of the model. For example, if the trained model that performs the inference is a trained model that outputs a classification result, the comparison unit 208 compares the classification result of the class with the likelihood for the classification to calculate the difference. Specifically, if the trained model that performed the inference is a model that classifies into three classes, class A, class B, and class C, the comparison unit 208 calculates the difference in the likelihood assigned to each class. If the first inference result by the first trained model is (0.8, 0.1, 0.1) and the second inference result by the second trained model is (0.5, 0.3, 0.2), the difference between the two inference results is |0.8-0.5|+|0.1-0.3|+|0.1-0.2|=0.6. For example, if 0.5 is set as the predetermined value, the comparison unit 208 determines that the difference is equal to or greater than the predetermined value, and the process proceeds to step S610.

[0054] If the trained model that performs inference is a trained model that outputs an extraction result, the comparison unit 208 compares the class classification result with the corresponding pixel to calculate the difference. For example, the comparison unit 208 calculates the difference in the likelihood of the inference result for the same pixel position as described above, and if the sum of the differences is equal to or greater than a predetermined value, the process proceeds to step S610.

[0055] Here, the predetermined value is, for example, a threshold value set based on a value derived from the user or past statistics.

[0056] In step S610, the notification unit 210 determines the notification destination based on the comparison result transmitted from the comparison unit 208, and notifies the determined notification destination. When the user is determined as the notification destination, the notification unit 210 refers to the account information in the inference information storage unit 302 and notifies the inference result, etc. Here, the notification unit 210 determines the user and the manager of the model as notification destinations when, for example, the classes with the highest likelihood in the classification result are different. When the classes with the highest likelihood in the classification result are the same, notification based on the difference in likelihood is made to the user. That is, the notification unit 201 changes the notification destination depending on the comparison result by the comparison unit 208. When the notification by the notification unit 210 is made, the process ends.

[0057] That is, the information processing device 130 has a model information storage unit 300 that stores a trained model, and performs inference on medical image data using the trained model stored in the model information storage unit. The information processing device 130 further has an inference information storage unit 302 that stores an inference target for which a first inference has been performed using a first trained model. The information processing device 130 further has an inference unit 206 that performs a second inference on the inference target using the second trained model when the first trained model is changed to a second trained model, a comparison unit 208 that compares the first inference result by the first inference with the second inference result by the second inference, and a notification unit 210 that notifies the user based on the comparison result by the comparison unit 208.

[0058] In the above embodiment, the comparison unit 208 compares the first inference result by the first trained model before the update with the second trained model after the update, and the notification unit 210 notifies the user when the output of the first trained model and the output of the second trained model are significantly different. On the other hand, when the output is not significantly different, the notification unit 210 does not notify the user, and the user is notified every time a change is made, thereby reducing the effort of the user to check. That is, the notification unit 210 in the information processing device 130 is characterized in that it determines whether or not to notify the user depending on the result of the comparison by the comparison unit 208.

[0059] Therefore, the configuration and flow of the information processing device 130 of the present invention make it possible, when a trained model is changed, to appropriately notify users who performed inference using the trained model before the update.

[0060] Second Embodiment In the above-described embodiment, the first inference result of a first trained model for the inference target and the second inference result of a second trained model are compared to determine whether or not to notify and to whom to notify.

[0061] In this embodiment, the inference information storage unit 702 further stores diagnostic information determined by a doctor or the like upon receiving the first inference result by the first trained model. The comparison unit 208 then compares the first inference result by the first trained model and the second inference result by the second trained model with the diagnostic information, and the notification unit 210 determines whether or not to notify and the notification destination. Below, differences from the above-described embodiments will be described, and overlapping parts will be omitted as appropriate.

[0062] FIG. 7 shows information stored in the inference information storage unit 702 in this embodiment. The inference information storage unit 702 stores account information and inference information in association with each other. In addition, the inference information storage unit 702 further stores diagnostic information determined by a doctor or the like in response to an inference result in the first inference information among the inference information 710 corresponding to the first account information. In addition, in the inference information 712 corresponding to the second account information 502, the first diagnostic information stores diagnostic information determined by a doctor or the like in response to an inference result. The diagnostic information can be input or written by a doctor regardless of the inference result of the trained model that performed the inference. In this embodiment, the comparison unit 208 compares the diagnostic information with the inference result of the trained model, and the notification unit 210 determines whether or not to notify and the notification destination. The diagnostic information is information determined by a doctor or the like in response to an inference result, and is, for example, information indicating a disease name or a location of a disease name.

[0063] In this embodiment, when the trained model for performing inference is changed, the comparison unit 208 compares the first inference result by the first trained model, which is the trained model before the change, the second inference result by the second trained model, which is the trained model after the change, and the diagnostic information. Specifically, the comparison unit 208 compares the first difference between the diagnostic information and the first inference result by the first trained model with the second difference between the diagnostic information and the second inference result by the second trained model, and when the first difference is large, the notification unit 210 sets the notification destination of the inference result, etc. to a notification destination including the user. When the second difference is large, the notification destination of the inference result, etc. by the notification unit 210 is set to a notification destination including the manager of the model.

[0064] As a comparison method in the comparison unit 208, when the doctor's diagnosis information is a disease name, the value of the class corresponding to the disease name is set to 1, and the difference between the first inference result and the second inference result is calculated.

[0065] In the case where the trained model used for inference is a model that classifies into three classes, class A, class B, and class C, and the doctor inputs the name of a disease corresponding to class A, the value for class A is set to 1, and the difference between the first inference result and the second inference result is calculated as (1.0, 0.0, 0.0). As in the first embodiment, a case will be described where the first inference result by the first trained model is (0.5, 0.3, 0.2), and the second inference result by the second trained model is (0.8, 0.1, 0.1). In this case, the first difference between the diagnosis information and the first inference result calculated by the comparison unit 208 is |1.0-0.5|+|0.0-0.3|+|0.0-0.2|=1.0. Furthermore, the second difference between the diagnostic information and the second inference result calculated by the comparison unit 208 is |1.0-0.8|+|0.0-0.1|+|0.0-0.1|=0.4. As a result of the comparison by the comparison unit 208, the first difference is greater than the second difference, and therefore the notification unit 210 determines the notification destinations of the inference result, etc. to include the user.

[0066] As a comparison method in comparison unit 208, when the doctor's diagnosis information is the name of a disease and the location of the disease, the value of the class corresponding to the disease name of the pixel corresponding to the location of the disease name is set to 1, and the difference between the first inference result and the second inference result is calculated.

[0067] If the first difference is large, it means that the change to the second trained model has improved the inference accuracy of the inference target in the inference unit 202, and if the second difference is large, it means that the change to the second trained model has deteriorated the inference accuracy of the inference target in the inference unit 202. If the inference accuracy has improved, the administrator does not need to understand the inference result, but if the inference accuracy has deteriorated, it may be necessary to have the administrator of the model understand the inference result. By understanding the inference target and the inference results by both trained models, the administrator of the model can determine whether to update, stop, or continue the second trained model.

[0068] Hereinafter, the flow in this embodiment will be described with reference to Fig. 8. In step S800 in Fig. 8, the comparison unit 208 determines whether or not there is diagnostic information for the inference result of the first trained model, which is the trained model before the change in the inference information storage unit 702 in the storage unit 204. If there is no diagnostic information, the process proceeds to step S608 in the above embodiment. If there is diagnostic information, the process proceeds to step S802.

[0069] In step S802, the comparison unit 208 determines whether the difference between each inference result and the diagnostic information is equal to or greater than a predetermined value. Specifically, it determines whether at least one of the first difference between the inference result by the first trained model and the diagnostic information and the second difference between the inference result by the second trained model and the diagnostic information is equal to or greater than a predetermined value, and if it is equal to or greater than the predetermined value, proceeds to step S804, and if it is less than the predetermined value, ends the process.

[0070] In step S804, when the first difference is large, the notification unit 210 sets notification destinations including the user as notification destinations. On the other hand, when the second difference is large, the notification unit 210 sets notification destinations including the manager of the model as notification destinations. The notification destinations can be changed or set arbitrarily. According to this embodiment, when the inference accuracy in the inference unit 202 improves by a predetermined value or more, the user is notified, and when the inference accuracy in the inference unit 202 decreases by a predetermined value or more, the manager of the model is notified. In this way, when the trained model is updated, it is possible to appropriately notify the user who performed inference using the trained model before the update and the manager of the model.

[0071] (Variation 1) The comparison unit 208 compares a first difference between the first inference result and the diagnostic information using the first trained model with a second difference using the second trained model, and if the second difference is large, the administrator of the model notified by the notification unit 210 or the information processing device 130 can stop the use of the second trained model.

[0072] When the second trained model whose use has been stopped or a third trained model that updates the first trained model or the second trained model becomes available, the notification unit 210 notifies the user that use of the model has been resumed.

[0073] Third Embodiment In this embodiment, the information processing device 130 further includes an update unit 900 that adds and updates the diagnostic information and the inference target determined by a doctor or the like to the teacher data. The update unit 900 will be described with reference to FIG. 9. When the diagnostic information is stored in the inference information storage unit 702, the update unit 900 acquires the inference target and the diagnostic information from the storage unit 204 and transmits them to the teacher data generation unit 200. The teacher data generation unit 200 adds teacher data that pairs the transmitted inference target and the diagnostic information to the pooled teacher data, and the learning unit 202 performs learning of a model based on a neural network. Note that, when learning a model based on a neural network in the learning unit 202, the penalty of the calculation of the loss function for the teacher data generated from the diagnostic information is reduced. Alternatively, the influence on the model may be reduced compared to the teacher data held in advance by processing such as reducing the update rate of the parameters. By increasing the teacher data by the update unit 900, the accuracy of the generated trained model is improved, and the accuracy of the notification by the notification unit 210 is improved.

[0074] [Fourth embodiment] In the above-described embodiment, the inference unit 206 performs inference using a trained model selected by the user, and the inference information storage unit 302 in the storage unit 204 stores the inference target, the model information used for the inference, the inference result, and the like. In this embodiment, the information processing device 130 further includes a determination unit 1010 that determines whether the trained model that performed the inference is appropriate as a model for performing inference on the inference target. In addition, when there is a trained model that is determined to be more appropriate by the determination unit 1010, the notification unit 210 notifies the user that the trained model is recommended to be selected. According to this embodiment, when a different trained model added after the time of inference by the user is more suitable for inference than the second trained model obtained by changing the first trained model (i.e., when it is determined that the model is not appropriate as a model for inferring the inference target), the user can be notified of the recommendation of inference using a different trained model.

[0075] The information processing device 130 in this embodiment will be described with reference to FIG.

[0076] The judgment unit 1010 acquires the inference target and the model information used for inference in the inference information storage unit 302 of the storage unit 204. The judgment unit 1010 further acquires the teacher data used when learning each model based on the acquired model information, and learns a judgment model with the model as a class for the teacher data. The judgment unit 1010 performs inference on the inference target acquired from the inference information storage 302 of the storage unit 204 using the learned judgment model. If the result of the inference matches the model indicated by the acquired model information, the judgment unit 1010 ends the process, and if the result of the inference differs from the model indicated by the acquired model information, the judgment unit 1010 transmits the inference result to the notification unit 210. The notification unit 210 notifies the user that the model indicated by the inference result transmitted from the judgment unit 1010 is recommended. In addition, when a new inference target for inference is transmitted to the inference unit 206, the judgment unit 1010 may determine a learned model suitable for inference from among the models stored in the model information storage unit 300 of the storage unit 204, and the notification unit 210 may notify the recommended model.

[0077] According to this embodiment, when the judgment unit 1010 determines that a different trained model added after the time of inference by the user is more suitable for inference, it is possible to notify the recommendation of inference using the different trained model.

[0078] (Variation 2) Although the above has been described in terms of each embodiment, the above-described embodiments and modified examples may be appropriately combined and implemented.

[0079] (Other variations) A computer program for realizing the functions of the above-described embodiments can be supplied to a computer via a network or a storage medium (not shown), and the computer program can be executed. This is a computer program for causing a computer to execute the above-described information processing method. In other words, the computer program is a program for causing a computer to realize the functions of an information processing device. The storage medium stores the computer program. [Explanation of symbols]

[0080] 130 Information processing equipment 200 Teacher Data Generation Unit 202 Learning Department 204 Storage section 206 Reasoning section 208 Comparison Section 210 Notification Department

Claims

1. An information processing device having a model information storage unit that stores a trained model, and performing inference on medical image data using the trained model stored in the model information storage unit, an inference information storage unit that stores an inference target for which a first inference has been performed using a first trained model; An inference unit that performs a second inference on the inference target using the second trained model when the first trained model is changed to a second trained model; a comparison unit that compares a first inference result based on the first inference with a second inference result based on the second inference; a notification unit that notifies a user based on a comparison result by the comparison unit; having The information processing apparatus according to claim 1, wherein the notification section does not issue the notification when the comparison result is within a predetermined range, and issues the notification when the comparison result is outside the predetermined range.

2. 2 . The information processing apparatus according to claim 1 , wherein the inference information storage unit stores the inference target in association with account information of a user who has instructed the first inference.

3. 3. The information processing apparatus according to claim 2, wherein the notification unit issues a notification to the user based on the account information stored in the inference information storage unit.

4. 4. The information processing apparatus according to claim 1, wherein the notification unit determines whether or not to notify a user depending on a result of the comparison made by the comparison unit.

5. The information processing device according to any one of claims 1 to 4, characterized in that the first trained model and the second trained model have at least a portion in common, including a model structure, a class to be classified, model parameters, and teacher data used for learning.

6. The information processing device according to any one of claims 1 to 5, wherein the second trained model is a trained model generated based on the first trained model.

7. The information processing device according to claim 6, wherein the second trained model is a trained model generated from the first trained model by at least one of transfer learning, fine tuning, and additional learning.

8. 8. The information processing apparatus according to claim 1, wherein the model information storage unit further stores at least one of version information and update information of the stored model.

9. The information processing device according to any one of claims 1 to 8, further comprising a judgment unit that judges whether the trained model that performed the inference is appropriate as a model for inferring the inference target.

10. 10. The information processing device according to claim 9, wherein, when the determination unit determines that the model is not appropriate for inferring the inference target, the notification unit notifies a user of a model different from the inferred model.

11. The comparison unit further acquires diagnostic information determined by a user in response to the first inference result; 11. The information processing apparatus according to claim 1, wherein each of the first inference result and the second inference result is compared with the diagnostic information.

12. the comparison unit compares a first difference between the diagnostic information and the first inference result with a second difference between the diagnostic information and the second inference result; 12. The information processing apparatus according to claim 11, wherein the notification unit notifies the user when the first difference is greater than the second difference.

13. the comparison unit compares a first difference between the diagnostic information and the first inference result with a second difference between the diagnostic information and the second inference result; 13. The information processing apparatus according to claim 11, wherein the notification unit notifies an administrator of the model when the second difference is greater than the first difference.

14. The information processing device according to claim 13 , wherein the comparison unit stops use of the second trained model when the second difference is greater than the first difference.

15. The information processing device according to claim 14, characterized in that when the second trained model that has become unavailable, or the first trained model or a third trained model that has updated the second trained model, becomes available, the notification unit notifies the user.

16. An information processing device having a model information storage unit that stores a trained model, and performing inference on medical image data using the trained model stored in the model information storage unit, an inference information storage unit that stores an inference target for which a first inference has been performed using a first trained model; An inference unit that performs a second inference on the inference target using the second trained model when the first trained model is changed to a second trained model; a comparison unit that compares a first inference result based on the first inference and a second inference result based on the second inference with diagnostic information determined by a user in response to the first inference result; a notification unit that notifies a user based on a comparison result by the comparison unit; 13. An information processing device comprising:

17. the comparison unit compares a first difference between the diagnostic information and the first inference result with a second difference between the diagnostic information and the second inference result; 17. The information processing apparatus according to claim 16, wherein the notification unit notifies the user when the first difference is greater than the second difference.

18. the comparison unit compares a first difference between the diagnostic information and the first inference result with a second difference between the diagnostic information and the second inference result; 18. The information processing apparatus according to claim 16, wherein the notification unit notifies an administrator of the model when the second difference is greater than the first difference.

19. The information processing device according to claim 18 , wherein the comparison unit stops use of the second trained model when the second difference is greater than the first difference.

20. The information processing device according to claim 19, characterized in that when the second trained model that has become unavailable, or the first trained model or a third trained model that updates the second trained model, becomes available, the notification unit notifies the user.

21. An information processing method executed by a computer that performs inference on medical image data using a stored trained model, comprising: When a first trained model that has performed a first inference on an inference target is changed to a second trained model, an inference step of performing a second inference on the inference target stored in an inference target storage unit using the second trained model; a comparison step of comparing a first inference result based on the first inference with a second inference result based on the second inference; a notification step of notifying a user based on a comparison result by the comparison step, The information processing method according to the present invention, wherein the notification step does not notify when the comparison result is within a predetermined range, and does not notify when the comparison result is outside the predetermined range.

22. An information processing method for performing inference on medical image data using a stored trained model, comprising: When a first trained model that performed a first inference on an inference target is changed to a second trained model, an inference step of performing a second inference on the inference target using the second trained model; a comparison step of comparing a first inference result based on the first inference and a second inference result based on the second inference with diagnostic information determined by a user in response to the first inference result; a notification step of notifying a user based on a comparison result by the comparison step; 13. An information processing method comprising:

23. 23. A program for causing a computer to execute the information processing method according to claim 21 or 22.

24. An information processing system including: an information processing device having a model information storage unit that stores a trained model, and performing inference on medical image data using the trained model stored in the model information storage unit; and an information terminal capable of communicating with the information processing device via a network, The information processing device includes: an inference information storage unit that stores an inference target for which a first inference has been performed using a first trained model; An inference unit that performs a second inference on the inference target using the second trained model when the first trained model is changed to a second trained model; a comparison unit that compares a first inference result based on the first inference with a second inference result based on the second inference; a notification unit that notifies the information terminal based on a comparison result by the comparison unit, The information processing system, wherein the notification unit does not issue the notification when the comparison result is within a predetermined range, and issues the notification when the comparison result is outside the predetermined range.

25. An information processing system including: an information processing device having a model information storage unit that stores a trained model, and performing inference on medical image data using the trained model stored in the model information storage unit; and an information terminal capable of communicating with the information processing device via a network, The information processing device includes: an inference information storage unit that stores an inference target for which a first inference has been performed using a first trained model; An inference unit that performs a second inference on the inference target using the second trained model when the first trained model is changed to a second trained model; a comparison unit that compares a first inference result based on the first inference and a second inference result based on the second inference with diagnostic information determined by a user in response to the first inference result; a notification unit that notifies the information terminal based on a comparison result by the comparison unit; An information processing system comprising:

Images