Medical image analysis system and medical image analysis method

Abstract

To obtain high classification accuracy even with a small number of learning data.SOLUTION: A medical image analysis system includes a learning unit and an inference unit. The learning unit divides a plurality of pieces of learning data including information regarding a lesion region extracted from a medical image and a label assigned to the lesion region into a plurality of groups on the basis of the label included in each piece of learning data, assigns the divided pieces of learning data to a plurality of logical units, and individually learns each logical unit, thereby generating a plurality of learned logical units that receive an input of the information regarding the lesion region and output an inference result regarding a disease type of the lesion region. The inference unit acquires a plurality of inference results regarding a disease type of a lesion region by inputting examination data including information regarding the lesion region extracted from a medical image of a subject to be examined to each of the plurality of learned logic devices, and displays a classification result of the disease type on the basis of the plurality of acquired inference results.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The embodiments disclosed in this specification and the drawings relate to a medical image analysis system and a medical image analysis method. [Background technology]

[0002] Various methods have been proposed for classifying the disease types of lesion regions contained in medical images, and there is a need to improve the accuracy of disease type classification in such methods. However, the known methods have a problem in that they require a large amount of training data to improve the accuracy of disease type classification. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2001-511372 [Patent Document 2] Japanese Patent Application Publication No. 2018-061771 [Non-patent literature]

[0004] [Non-Patent Document 1] Wikipedia, the free encyclopedia, "bagging", [online], [searched June 27, 2024], Internet<URL:https: / / ja.wikipedia.org / wiki / %E3%83%90%E3%82%AE%E3%83%B3%E3%82%B0> Summary of the Invention [Problem to be solved by the invention]

[0005] One of the problems to be solved by the embodiments disclosed in this specification and the drawings is to achieve high classification accuracy even with a small amount of training data. However, the problems to be solved by the embodiments disclosed in this specification and the drawings are not limited to the above problem. Problems corresponding to the effects of each configuration shown in the embodiments described below can also be positioned as other problems. [Means for solving the problem]

[0006] A medical image analysis system according to an embodiment is a medical image analysis system including a learning unit and an inference unit. The learning unit divides a plurality of pieces of training data, each containing information about a lesion area extracted from a medical image and a label assigned to the lesion area, into a plurality of groups based on the label included in each piece of training data, assigns the divided training data to a plurality of logic units, and trains each logic unit individually to generate a plurality of trained logic units that accept input of information about a lesion area and output an inference result regarding the disease type of the lesion area. The inference unit inputs test data containing information about a lesion area extracted from a medical image of a subject to be examined into each of the plurality of trained logic units, thereby obtaining a plurality of inference results regarding the disease type of the lesion area, and displays a classification result of the disease type based on the obtained plurality of inference results. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a medical image analysis system according to the first embodiment. [Figure 2] FIG. 2 is a diagram showing the flow of processing performed by the learning function and the inference function according to the first embodiment. [Figure 3] FIG. 3 is a diagram showing an example of allocation information used by the learning function according to the first embodiment. [Figure 4] FIG. 4 is a diagram showing an example of allocation of learning data to inference devices during learning performed by the learning function according to the first embodiment. [Figure 5]FIG. 5 is a diagram showing an example of histogram estimation performed by the learning function according to the first embodiment. [Figure 6] FIG. 6 is a diagram showing an example of a classification result displayed by the inference function according to the first embodiment. [Figure 7] FIG. 7 is a flowchart showing the processing procedure of the learning process performed by the learning function according to the first embodiment. [Figure 8] FIG. 8 is a flowchart showing the processing procedure of the inference process performed by the inference function according to the first embodiment. [Figure 9] FIG. 9 is a diagram illustrating an example of the configuration of a medical image analysis system according to the second embodiment. [Figure 10] FIG. 10 is a diagram showing the flow of processing performed by the learning function and the inference function according to the second embodiment. [Figure 11] FIG. 11 is a diagram showing an example of displaying a classification result according to a modified example. [Figure 12] FIG. 12 is a diagram showing an example of displaying a classification result according to a modified example. [Figure 13] FIG. 13 is a diagram showing an example of displaying a classification result according to a modified example. [Figure 14] FIG. 14 is a diagram showing an example of displaying a classification result according to a modified example. [Figure 15] FIG. 15 is a diagram showing an example of displaying a classification result according to a modified example. [Figure 16] FIG. 16 is a diagram showing an example of displaying a classification result according to a modified example. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments of a medical image analysis system and a medical image analysis method will be described in detail with reference to the drawings.

[0009] (First embodiment) FIG. 1 is a diagram showing an example of the configuration of a medical image analysis system according to the first embodiment.

[0010] For example, as shown in Fig. 1, a medical image analysis system 100 according to this embodiment includes a medical image capturing device 110, a medical image storage device 120, and a learning and inference device 130. Here, each device is connected to each other so as to be able to communicate with each other via a network 10 such as the Internet or a LAN (Local Area Network). Note that the medical image analysis system 100 may further include various other devices not shown.

[0011] The medical imaging device 110 captures an image of a subject to generate a medical image. For example, the medical imaging device 110 is a medical imaging diagnostic device that generates various images of the inside of a subject's body, such as an X-ray CT (Computed Tomography) device, an MRI (Magnetic Resonance Imaging) device, an ultrasound diagnostic device, an X-ray diagnostic device, a PET (Positron Emission Tomography) device, a SPECT (Single Photon Emission Computed Tomography) device, a PET-CT device that combines a PET device and an X-ray CT device, or a SPECT-CT device that combines a SPECT device and an X-ray CT device. The medical imaging device 110 may also be a device that generates a pathological image of tissue extracted from the subject by endoscopy or surgery, or a device that generates a camera image of the subject's skin surface.

[0012] The medical image storage device 120 stores medical images generated by the medical imaging device 110. For example, the medical image storage device 120 is realized by a computer device that implements a PACS (Picture Archiving and Communication System) or an electronic medical record system, and stores medical images in a format that complies with DICOM (Digital Imaging and Communications in Medicine).

[0013] The learning and inference device 130 performs various processes using medical images. Specifically, the learning and inference device 130 acquires medical images from the medical image acquisition device 110 or the medical image storage device 120 via the network 10, and processes the acquired medical images. For example, the learning and inference device 130 is realized by computer equipment such as a server, a workstation, or a personal computer.

[0014] For example, the learning and inference device 130 includes a network (NW) interface 131, a memory circuit 132, an input interface 133, a display 134, and a processing circuit 135.

[0015] The NW interface 131 controls the transmission and communication of various data transmitted and received between other devices connected via the network 10. Specifically, the NW interface 131 is connected to the processing circuit 135, and controls the transmission and communication of various data under the control of the processing circuit 135. For example, the NW interface 131 is realized by a network card, a network adapter, a NIC (Network Interface Controller), or the like.

[0016] The memory circuitry 132 stores various data, various programs, etc. Specifically, the memory circuitry 132 is connected to the processing circuitry 135, and stores and reads out various data, various programs, etc. under the control of the processing circuitry 135. For example, the memory circuitry 132 is realized by a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, a hard disk, an optical disk, etc.

[0017] The input interface 133 accepts input operations of various instructions and various information from an operator. Specifically, the input interface 133 is connected to the processing circuitry 135, converts the input operations received from the operator into electrical signals, and outputs the electrical signals to the processing circuitry 135. For example, the input interface 133 may be realized by a mouse, a keyboard, a touchpad that performs input operations by touching the operation surface, a touchscreen that integrates a display screen and a touchpad, a non-contact input circuit using an optical sensor, a voice input circuit, etc.

[0018] The display 134 displays various types of information and various types of data. Specifically, the display 134 is connected to the processing circuit 135 and displays various types of information and various types of data output from the processing circuit 135. For example, the display 134 is realized by a liquid crystal monitor, a CRT (Cathode Ray Tube) monitor, a touch panel, or the like.

[0019] The processing circuitry 135 controls the components of the learning and inference device 130 in response to input operations received from an operator via the input interface 133. For example, the processing circuitry 135 stores data received by the NW interface 131 in the storage circuitry 132. Furthermore, for example, the processing circuitry 135 reads data stored in the storage circuitry 132 in response to a request from the operator and displays the read data on the display 134.

[0020] The above describes an example of the configuration of the medical image analysis system 100 according to this embodiment. For example, the medical image analysis system 100 according to this embodiment is installed in a medical facility such as a hospital or clinic, and supports various diagnoses and the formulation of treatment plans performed by users such as doctors.

[0021] Specifically, the medical image analysis system 100 presents information that serves as a reference for the user when classifying a lesion area contained in a medical image into a clinically useful disease type.

[0022] Various methods have been known to date for classifying the disease type of a lesion region contained in a medical image, and there is a need to improve the accuracy of disease type classification in such methods. However, the known methods have a problem in that a large amount of training data is required to improve the accuracy of disease type classification.

[0023] For example, a method for classifying disease types using a large number of image features is known. This method calculates a large number of image features to classify lesions as benign or malignant based on images of the lesion, and then uses a classification method such as an artificial neural network based on the values ​​of the large number of image features to obtain a classification result of the lesion (probability of malignancy).

[0024] This method requires training data from a large number of cases in order to classify lesions into several disease types (benign, malignant, etc.). Generally, when using a large number of features, several to several tens of times as many data samples are required as training data. For example, when classifying using 100 image features, several hundred to several thousand cases are required.

[0025] Let us now assume that there is a mathematical model that expresses the relationship between feature quantities and disease types. For example, this may be a model that estimates for the entire population. If this model expresses the relationship between feature quantities and lesion types in a simple manner, i.e., if the number of model parameters is small, good results can be obtained with a smaller amount of training data. Furthermore, if the model has good lesion type prediction accuracy, good results can also be obtained with a smaller amount of training data. However, in general, the relationship between feature quantities and lesion types is not simple, and prediction accuracy is often not very high. In such cases, there is the problem of having to prepare a large amount of training data.

[0026] Another known method is to use a two-stage inference model that utilizes visual image feature labels. This method uses a two-stage configuration in which a doctor labels multiple visual image features when interpreting an image to classify the lesion as benign or malignant based on a large number of feature quantities in the image, and these labels are used as training data to learn intermediate image features, and then a further learning process is performed to estimate benign or malignant based on the intermediate image features.

[0027] This method still has the problem of requiring a large amount of training data. If the relationship between image features and intermediate image features is simple and well-matched, and if the relationship between intermediate image features and disease types is also simple and well-matched, the amount of image data required will be small. However, the labels that doctors visually classify often require advanced judgment, and the correspondence is not necessarily simple. Furthermore, there are many exceptions to the correspondence between visual image features and disease classifications, and they may not always be good. In such cases, the amount of training data required will be large.

[0028] Furthermore, this method suffers from a deterioration in accuracy and an increase in the amount of training data due to inconsistencies in visual classification. When visual image features are trained, inconsistencies in visual classification limit the accuracy and reproducibility of image feature labels. If the label accuracy is low, a large amount of training data is required for label training, and the estimation accuracy of the obtained intermediate image features also deteriorates, resulting in the problem that a large amount of training data is required for disease type estimation.

[0029] Furthermore, this method cannot obtain new, effective features beyond those known clinically. While visual image features correspond to a certain degree with diseases, disease types are not necessarily determined by these image features alone, and unknown factors may exist. In such cases, the correspondence between intermediate image features and disease types is poor, which results in an increase in the amount of training data required.

[0030] Also, for example, a method using bootstrap sampling in ensemble learning is known. This method divides the data to create multiple training subsets from the entire training set, constructs a classifier for each training subset, and synthesizes the results of each classifier to obtain a final inference result. Here, random data selection is used when creating (resampling) the training subsets. This method can improve stability by synthesizing the inference results of multiple classifiers.

[0031] However, this method still requires a large amount of training data. The original problem is that when the relationship between image features and lesion types is not simple and prediction accuracy is limited, each classifier needs to be somewhat complex, and a small number of training subsets is not sufficient. Since the total training data needs to be larger than the number of training subsets, a large amount of training data is still required.

[0032] For this reason, the medical image analysis system 100 according to this embodiment is configured to be able to obtain high classification accuracy even with a small amount of training data.

[0033] Specifically, the processing circuit 135 of the learning and inference device 130 has a learning function 135a and an inference function 135b. Here, the learning function 135a is an example of a learning unit, and the inference function 135b is an example of an inference unit.

[0034] The learning function 135a divides a plurality of learning data sets, each set including information about a lesion area extracted from a medical image and a label assigned to the lesion area, into a plurality of groups based on the label included in each learning data set. The learning function 135a also assigns the divided learning data sets to a plurality of logic units and trains each logic unit individually, thereby generating a plurality of trained logic units that accept input of information about a lesion area and output an inference result about the disease type of the lesion area.

[0035] The inference function 135b inputs test data containing information about a lesion area extracted from a medical image of an examination target into a plurality of trained logic units, thereby acquiring a plurality of inference results regarding the disease type of the lesion area. The inference function 135b then displays a classification result of the disease type of the lesion area based on the acquired plurality of inference results. For example, the inference function 135b displays the plurality of inference results together with information about the allocation of training data to logic units during training. Furthermore, for example, the inference function 135b displays a result obtained by combining the plurality of inference results.

[0036] According to this configuration, by dividing multiple training data into multiple groups based on the labels, and assigning the divided training data to multiple logic units and training each logic unit individually, the model of the logic unit can be simplified compared to when multiple training data are assigned to a single logic unit without being divided. This makes it possible to obtain high classification accuracy even with a small amount of training data. Furthermore, when compared with the same number of training data, classification accuracy can be improved.

[0037] The learning function 135a and the inference function 135b of the processing circuit 135 of the learning and inference device 130 will be described in detail below.

[0038] FIG. 2 is a diagram showing the flow of processing performed by the learning function 135a and the inference function 135b according to the first embodiment.

[0039] In this embodiment, during learning, the processing shown in the upper part of Figure 2 is performed by learning function 135a, and after the processing during learning is completed, during inference, the processing shown in the lower part of Figure 2 is performed by inference function 135b.

[0040] First, the learning process performed by the learning function 135a will be described.

[0041] For example, as shown in the upper part of FIG. 2, the learning function 135a acquires a plurality of learning data including information about a lesion area extracted from a medical image and a label assigned to the lesion area (see "Learning Data" shown in the upper part of FIG. 2).

[0042] Specifically, the learning function 135a acquires a plurality of medical images including a lesion area of ​​the subject of examination from the medical imaging device 110 or the medical image storage device 120 via the network 10. For example, the learning function 135a acquires, as medical images, two-dimensional images such as tomographic images, or three-dimensional images including a plurality of slices. Here, for example, the medical images are dynamic images obtained by dynamic imaging using a contrast agent. Furthermore, for example, when the subject of examination is lung cancer, the medical images may be CT images taken by chest thin slice CT (a method of taking CT images of the lung area without contrast and with thin slices about 1 mm thick).

[0043] The learning function 135a then extracts a lesion region from each of the acquired medical images. For example, the learning function 135a may extract a lesion region (or a target tissue region) from the medical image using manual extraction or lesion region segmentation techniques. For example, the learning function 135a may three-dimensionally extract a nodule region from a three-dimensional medical image.

[0044] Thereafter, the learning function 135a calculates various image features for the lesion region extracted from the medical image. For example, the learning function 135a calculates morphological features such as circularity and flatness, and various texture features for the lesion region.

[0045] Then, the learning function 135a generates learning data including various image feature amounts calculated for the lesion area, background information about the subject and diagnostic information about the lesion area, and a label assigned to the lesion area.

[0046] Here, the subject's background information includes, for example, the subject's (patient's) age, sex, smoking history, etc. Furthermore, the diagnostic information regarding the lesion area includes, for example, other test results such as blood tests, image features of shadows determined by a doctor through image interpretation, image diagnosis results, tissue classification obtained through pathological testing, and the subject's subsequent progress, etc. Before learning is performed, these background information and diagnostic information are acquired from the medical imaging device 110, the medical image storage device 120, or other devices such as an electronic medical record system, a hospital information system (HIS), or a radiology information system (RIS) via the NW interface 131, and stored in the memory circuitry 132. Of these pieces of information, only the information used for labels, which will be described below, is used as learning data.

[0047] The labels are used to divide the training data into a relatively small number of groups based on the background information of the subject, diagnostic information related to the lesion area, or image features. These labels are set by the operator via the input interface 133, for example, when training is performed, based on the background information of the subject, diagnostic information related to the lesion area, or image features.

[0048] In the following, as an example, a case will be described in which tissue classification obtained by a pathological examination is used as the viewpoint and granularity of division, and the tissue classification is used as a label.

[0049] Here, tissue classification is the result of testing using tissue biopsied from the target lesion, and is broadly divided into malignant and benign lesions (borderline lesions may also be included). Furthermore, malignant and benign lesions are each further divided into hierarchical classifications. Clinically assigned tissue classifications may include not only classification terms but also natural language. In such cases, classification is performed after determining which classification terms to treat as natural sources. Furthermore, tissue classifications are divided into multiple classifications after determining the appropriate granularity for use in learning.

[0050] In one example, based on the tissue classification obtained from the pathological examination, malignant lesions are divided into two categories, hepatocellular carcinoma and cholangiocarcinoma, and benign lesions are divided into three categories, cyst, hemangioma, and others, and these five categories are used as labels. In addition, in one example, for all of these categories, only lesions with a diameter of 5 mm or more are used as training data.

[0051] That is, in this embodiment, based on a predetermined division perspective and granularity, if a portion of the total lesions is designated as A and another portion is designated as B, the labels A1 and A2 for A and the labels B1 and B2 for B are expressed as follows:

[0052] - Division perspective and granularity e.g. organizational classification - When the area containing all lesions is A, A1 ⊂ A. Example: Hepatocellular carcinoma ⊂ malignant lesion - When the area of ​​all lesions is A, A2 ⊂ A. Example: Cholangiocarcinoma ⊂ Malignant lesion - When the area containing all lesions is B, B1 ⊂ B. Example: Hemangioma ⊂ Benign lesion - If the area containing all lesions is B, then B2 ⊂ B. Example: cyst ⊂ benign lesion

[0053] Returning to FIG. 2, the learning function 135a then divides the acquired plurality of learning data into a plurality of groups based on the labels included in each learning data (see "Division" shown at the top of FIG. 2).

[0054] Specifically, the learning function 135a divides multiple pieces of learning data into multiple groups of first classifications based on the labels, and further divides the learning data divided into each of the multiple groups of first classifications into multiple groups of second classifications included in each of the multiple first classifications.

[0055] For example, the learning function 135a divides multiple pieces of learning data into a group belonging to all malignant lesions, which is a major classification of malignant lesions, and a group belonging to all benign lesions, which is a major classification of benign lesions. Furthermore, the learning function 135a further divides the learning data divided into the all malignant lesions group into a malignant lesion 1 group, which is a minor classification of malignant lesions, and a malignant lesion 2 group, which is a minor classification of malignant lesions. Furthermore, the learning function 135a further divides the learning data divided into the all benign lesions group into a benign lesion 1 group, which is a minor classification of benign lesions, and a benign lesion 2 group, which is a minor classification of benign lesions.

[0056] In one example, the learning function 135a divides multiple pieces of training data into a group whose tissue classification belongs to all malignant lesions and a group whose tissue classification belongs to all benign lesions. The learning function 135a further divides the training data divided into the all malignant lesion group into a group whose tissue classification belongs to hepatocellular carcinoma and a group whose tissue classification belongs to cholangiocarcinoma. The learning function 135a further divides the training data divided into the all benign lesion group into a group whose tissue classification belongs to cysts and a group whose tissue classification belongs to hemangioma or other.

[0057] In this case, the learning function 135a may divide the multiple learning data into multiple groups based on information contained in each learning data that is different from the label, and then divide the multiple learning data contained in one group into multiple groups based on the label.

[0058] In one example, the learning function 135a divides multiple training data into a group of lesions with a diameter of 5 mm or more and a group of lesions with a diameter of less than 5 mm based on the image features contained in each image data, and then divides multiple training data included in the group of lesions with a diameter of 5 mm or more into multiple groups based on the labels.

[0059] Then, the learning function 135a records division information including the labels used to divide the learning data (e.g., hepatocellular carcinoma, cholangiocarcinoma, cyst, hemangioma, etc.), classifications corresponding to positive and negative (e.g., malignant lesion, benign lesion), and the viewpoint and granularity of division (e.g., tissue classification) in the storage circuitry 132. This division information is used by the inference function 135b, which will be described later, when displaying the classification result of the disease type of the lesion area during inference.

[0060] In the above example, the learning function 135a divides the plurality of learning data into two levels of groups by dividing them into large groups and then into small groups, but the embodiment is not limited to this. For example, the learning function 135a may further divide the learning data divided into two levels into three or more levels of groups.

[0061] Thus, this embodiment differs from bootstrap sampling, which uses random data selection when creating (resampling) training subsets, in that it divides the training data based on the labels assigned to the lesion areas.

[0062] Returning to FIG. 2, the learning function 135a then assigns the divided learning data to multiple logic units and trains each logic unit individually, thereby generating multiple trained logic units that accept input of information about the lesion area and output inference results about the disease type of the lesion area (see "Assignment" and "Inference Unit" shown at the top of FIG. 2).

[0063] Specifically, the learning function 135a generates multiple trained logic units by assigning learning data of one first classification group among multiple first classification groups and learning data of a second classification group included in another first classification to one logic unit and performing learning.

[0064] Here, when the learning function 135a assigns the learning data of the first classification group and the learning data of the second classification group to one logic unit for learning, it performs learning by weighting the learning data of one group greater than the weighting the learning data of the other group.

[0065] More specifically, the learning function 135a learns each logic unit based on assignment information that defines which group of learning data is assigned to each of the multiple logic units. This assignment information is, for example, input by an operator via the input interface 133 before learning is performed, and is recorded in the memory circuit 132 by the learning function 135a.

[0066] FIG. 3 is a diagram showing an example of allocation information used by the learning function 135a according to the first embodiment.

[0067] For example, as shown in FIG. 3, the allocation information is information that allocates, for each of a plurality of logic units, a group of learning data to be used for learning the logic unit and learning conditions for learning the logic unit.

[0068] For example, each of the multiple reasoners is assigned a combination of a group of minor classifications of malignant lesions and a group of the first classification of benign lesions, or a combination of a group of minor classifications of benign lesions and a group of the first classification of malignant lesions.

[0069] In one example, Reasoner 1 is assigned a group of hepatocellular carcinomas with a diameter of 5 mm or greater and a group of all benign lesions with a diameter of 5 mm or greater. Reasoner 2 is assigned a group of cholangiocarcinomas with a diameter of 5 mm or greater and a group of all benign lesions with a diameter of 5 mm or greater. Reasoner 3 is assigned a group of cysts with a diameter of 5 mm or greater and a group of all malignant lesions with a diameter of 5 mm or greater. Reasoner 3 is assigned a group of hemangiomas or other lesions with a diameter of 5 mm or greater and a group of all malignant lesions with a diameter of 5 mm or greater.

[0070] Furthermore, for each inference unit, learning conditions including a target specificity or target sensitivity for training the logic unit are set. Here, the target specificity is the specificity that should be achieved by the inference unit alone, and the target sensitivity is the sensitivity that should be achieved by the inference unit alone.

[0071] For example, the target specificity is set to 0.98 for Inferer 1 and Inferer 2, specifying that the probability of false positives, in which a benign lesion is mistakenly classified as malignant (positive), is low. This suggests that if the inference result of either Inferer 1 or Inferer 2 is positive, there is a high probability that it is malignant. In other words, Inferer 1 and Inferer 2 are defined as inferers for malignant lesions.

[0072] Furthermore, for example, the target sensitivity for inferencing devices 3 and 4 is set to 0.98, which specifies a low probability of false negatives, where a malignant lesion is mistakenly classified as benign (negative). This suggests that if the inference result of either inferencing device 3 or inferencing device 4 is negative, there is a high probability that the lesion is benign. In other words, inferencing devices 3 and 4 are defined as inferencing devices for benign lesions.

[0073] In this manner, in this embodiment, the inference units for malignant lesions and those for benign lesions are trained using different accuracy targets, thereby reducing either false positives or false negatives. This makes it less likely that accuracy will deteriorate even when the inference results of multiple inference units are combined, as in this embodiment.

[0074] Then, the learning function 135a generates a plurality of trained logic elements by learning each logic element by machine learning using a model such as a neural network or a decision tree, using the learning data of the group assigned to each logic element in accordance with the above-mentioned assignment information and based on the learning conditions assigned to each logic element.

[0075] As an example, the learning function 135a generates a learned logic unit that receives input of information about a lesion area and outputs an output value indicating whether the disease type of the lesion area is a malignant lesion (positive) or a benign lesion (negative) as an estimation result. Here, the output value of each logic unit is a continuous value that becomes high for a malignant lesion (positive).

[0076] FIG. 4 is a diagram showing an example of allocation of learning data to inference devices during learning performed by the learning function 135a according to the first embodiment.

[0077] 4, in one example, the learning function 135a performs learning using the learning data of the group of malignant lesions 1 (e.g., hepatocellular carcinoma) and the learning data of the group of all benign lesions to generate logic unit 1. Also, the learning function 135a performs learning using the learning data of the group of malignant lesions 2 (e.g., cholangiocarcinoma) and the learning data of the group of all benign lesions to generate logic unit 2.

[0078] Furthermore, the learning function 135a performs learning using the learning data of the group of benign lesions 1 (e.g., cysts) and the learning data of the group of all malignant lesions, thereby generating a logic unit 3. Furthermore, the learning function 135a performs learning using the learning data of the group of benign lesions 2 (e.g., hemangioma or other), which are a part of benign lesions, and the learning data of the group of all malignant lesions, thereby generating a logic unit 4.

[0079] The learning function 135a then records the model parameters determined during learning for the inference model corresponding to each logic unit in the memory circuitry 132. For example, the model parameters are the number of positive determinations or the number of decision trees in a model such as a neural network or decision tree. These model parameters are used by the inference function 135b (described later) to obtain an inference result regarding the disease type of a lesion area included in test data using each inference unit during inference.

[0080] Here, various known methods can be used as a method for training each logic unit. In this embodiment, however, a plurality of pieces of training data are divided into a plurality of groups based on labels, and the divided pieces of training data are assigned to a plurality of logic units, and each logic unit is trained individually. Therefore, a simpler logic model can be used compared to the case where a plurality of pieces of training data are assigned to a single logic unit without being divided.

[0081] Returning to FIG. 2, the learning function 135a then estimates a histogram for calculating the probability of a disease type for each inference unit based on the estimation results of each inference unit (see "Probability Calculation" shown at the top of FIG. 2).

[0082] Specifically, the learning function 135a performs histogram estimation for each inference device using the learning data of the malignant lesion group and the learning data of the benign lesion group assigned during learning, thereby estimating the histogram of the learning data of the malignant lesions and the histogram of the learning data of the benign lesions, respectively.

[0083] Although an example in which the learning function 135a performs histogram estimation using training data will be described here, the embodiment is not limited to this. For example, the learning function 135a may perform histogram estimation using verification data separate from the training data.

[0084] FIG. 5 is a diagram showing an example of histogram estimation performed by the learning function 135a according to the first embodiment.

[0085] For example, as shown in FIG. 5, the learning function 135a may plot, for each inference unit, the output value y of the inference unit on the horizontal axis and the frequency of malignant lesions (F 悪性 (y)) and frequency of benign lesions (F 良性 (y)), the histogram of the training data of malignant lesions and the histogram of the training data of benign lesions are estimated, respectively.

[0086] In one example, the learning function 135a estimates a histogram of the training data of hepatocellular carcinoma and a histogram of the training data of all benign lesions for the inferencing device 1. The learning function 135a also estimates a histogram of the training data of cholangiocarcinoma and a histogram of the training data of all benign lesions for the inferencing device 2. The learning function 135a also estimates a histogram of the training data of cysts and a histogram of the training data of all benign lesions for the inferencing device 3. The learning function 135a also estimates a histogram of the training data of hemangiomas or other lesions and a histogram of the training data of all benign lesions for the inferencing device 4.

[0087] Then, the learning function 135a records inference parameters including the histogram obtained by histogram estimation and information indicating the probability of the calculated disease type (e.g., information (label name) indicating malignant probability or benign probability) in the storage circuit 132. These inference parameters are used by the inference function 135b, which will be described later, when calculating the probability of the disease type for each inference unit based on the inference results of each inference unit during inference.

[0088] In this embodiment, the inference function 135b uses a histogram of inference parameters for each inference unit to calculate the probability of a disease type based on the estimation results of the inference unit.

[0089] For example, the inference function 135b calculates, for each inference unit, a malignancy probability P from the output value y of the inference unit using the histogram of the inference parameters and the following equations (1) and (2): 悪性 (y) or benign probability P 良性 Calculate (y).

[0090]

number

number

[0091] In one example, inference function 135b calculates benign probability 1 from the output value of inference device 1, benign probability 2 from the output value of inference device 2, malignant probability 1 from the output value of inference device 3, and malignant probability 2 from the output value of inference device 4.

[0092] Returning to FIG. 2, the learning function 135a then determines a synthesis formula for synthesizing the probabilities of the disease types calculated for each reasoner (see "Synthesis" shown at the top of FIG. 2).

[0093] Specifically, the learning function 135a determines a synthesis formula for calculating the overall probability by synthesizing the probabilities of disease types calculated based on the estimation results of each inference module.

[0094] In one example, the learning function 135a combines the benign probability 1 calculated from the output value of inference device 1, the benign probability 2 calculated from the output value of inference device 2, the malignant probability 1 calculated from the output value of inference device 3, and the malignant probability 2 calculated from the output value of inference device 4 to determine a combination formula for calculating the overall malignant probability or benign probability.

[0095] For example, the learning function 135a determines an equation designated by the operator from among the following equations (3) to (5) as the composite equation for calculating the overall malignancy probability or benign probability.

[0096]

number

number

number

[0097] Then, the learning function 135a records synthesis information including the determined synthesis formula in the storage circuitry 132. This synthesis information is used by the inference function 135b (described later) when synthesizing the probabilities of disease types calculated for each inference unit during inference.

[0098] Returning to FIG. 2, the learning function 135a then determines a threshold for classifying the disease type based on the combined probability (see "Classification" at the top of FIG. 2).

[0099] Specifically, the learning function 135a performs histogram estimation for each inference device in the same way as when estimating a histogram for calculating the probability of a disease type, and then calculates a cumulative histogram by accumulating the estimated multiple histograms, thereby obtaining a sensitivity curve and a specificity curve (the horizontal axis is a threshold value for the composite output).The learning function 135a then obtains, based on the obtained sensitivity curve and specificity curve, a horizontal axis value that satisfies the target sensitivity specified in the allocation information and provides the highest specificity, or a horizontal axis value that satisfies the target specificity specified in the allocation information and provides the highest sensitivity, and determines the obtained horizontal axis value as the threshold.

[0100] Then, the learning function 135a records classification information including a threshold determined based on the cumulative histogram and information (label name) indicating whether a positive result suggests malignancy or benignity in the storage circuitry 132. This classification information is used by the inference function 135b, which will be described later, when classifying disease types based on the combined probability during inference.

[0101] Next, the inference process performed by the inference function 135b will be described.

[0102] For example, as shown in the lower part of Figure 2, after the above-mentioned learning processing is performed, the inference function 135b acquires, during inference, multiple test data including information about the lesion area extracted from the medical image of the subject being examined (see "Test Data" shown in the lower part of Figure 2).

[0103] Specifically, the inference function 135b acquires medical images of the subject to be examined from the medical imaging device 110 or the medical image storage device 120 via the network 10. For example, like the learning function 135a, the inference function 135b acquires, as medical images, two-dimensional images such as tomographic images or three-dimensional images including multiple slices.

[0104] Then, the inference function 135b extracts a lesion area from the acquired medical image. For example, like the learning function 135a, the inference function 135b extracts a lesion area (or a target tissue area) from the medical image using manual extraction or lesion area segmentation techniques. Also, for example, like the learning function 135a, the inference function 135b three-dimensionally extracts a nodule area from a three-dimensional medical image.

[0105] Thereafter, the inference function 135b calculates various image features for the lesion area extracted from the medical image, similar to the learning function 135a. For example, the inference function 135b calculates morphological features such as circularity and flatness, and various texture features for the lesion area, similar to the learning function 135a.

[0106] Then, the inference function 135b generates examination data including various image feature amounts calculated for the lesion area.

[0107] Returning to Figure 2, the inference function 135b then inputs the test data into each of multiple learned logic units to obtain multiple inference results regarding the disease type of the lesion area (see ``Inference Unit'' shown at the bottom of Figure 2).

[0108] Specifically, the inference function 135b reads out the model parameters recorded in the memory circuitry 132 by the learning function 135a, and applies the read-out model parameters to the inference models corresponding to the inference units to acquire multiple logic units (trained logic units).The inference function 135b then inputs the same test data to all of the acquired multiple logic units and acquires the inference results output from each logic unit, thereby acquiring multiple inference results related to the disease type of the lesion area included in the test data.

[0109] In one example, the inference function 135b inputs the same test data into each of logic unit 1 trained using the learning data of the group of malignant lesion 1 (e.g., hepatocellular carcinoma) and the learning data of the group of all benign lesions, logic unit 2 trained using the learning data of the group of malignant lesion 2 (e.g., cholangiocarcinoma) and the learning data of the group of all benign lesions, logic unit 3 trained using the learning data of the group of benign lesion 1 (e.g., cyst) and the learning data of the group of all malignant lesions, and logic unit 4 trained using the learning data of the group of benign lesion 2 (e.g., hemangioma or other) and the learning data of the group of all malignant lesions, thereby obtaining multiple inference results regarding the disease type of the lesion area included in the test data.

[0110] In this case, the inference function 135b may divide the multiple test data into multiple groups based on information contained in each test data that is different from the label, and then obtain multiple inference results by inputting one test data contained in one group into each of the multiple trained logic units.

[0111] In one example, the inference function 135b divides multiple test data into a group of lesions with a diameter of 5 mm or more and a group of lesions with a diameter of less than 5 mm based on the image features contained in each test data, and then inputs the test data contained in the group of lesions with a diameter of 5 mm or more into each of multiple trained logic units to obtain multiple inference results.

[0112] Here, the test data may include, in addition to image feature quantities calculated from the image, background information of the subject known before the image test, information on blood tests, etc. However, the test data does not include information that is not available at the time the image test is performed, such as tissue classification obtained by a pathological test. In this embodiment, the test data is not segmented using labels as in the training data, so it is possible to infer disease types even if such information is not included.

[0113] Returning to FIG. 2, the inference function 135b then calculates the probability of the disease type for each inference unit based on the inference results of each inference unit (see "Probability Calculation" shown at the bottom of FIG. 2).

[0114] Specifically, the inference function 135b reads out the inference parameters recorded in the memory circuit 132 by the learning function 135a, and uses a histogram of the read out inference parameters to calculate the probability of the disease type based on the estimation result of the inference unit.

[0115] For example, as described above, the inference function 135b calculates the probability of malignancy P from the output value y of each inference unit using the histogram of the inference parameters and equations (1) and (2). 悪性 (y) or benign probability P 良性 Calculate (y).

[0116] In one example, as described above, inference function 135b calculates benign probability 1 from the output value of inference device 1, benign probability 2 from the output value of inference device 2, malignant probability 1 from the output value of inference device 3, and malignant probability 2 from the output value of inference device 4.

[0117] Returning to FIG. 2, the inference function 135b then combines the probabilities of the disease types calculated for each inference unit (see "Combination" shown at the bottom of FIG. 2).

[0118] Specifically, the inference function 135b reads out the synthetic information recorded in the memory circuit 132 by the learning function 135a, and uses the synthesis formula of the read out synthetic information to synthesize the probabilities of the disease types calculated for each inference unit to calculate the overall probability.

[0119] In one example, the inference function 135b combines the benign probability 1 calculated from the output value of inference device 1, the benign probability 2 calculated from the output value of inference device 2, the malignant probability 1 calculated from the output value of inference device 3, and the malignant probability 2 calculated from the output value of inference device 4 to calculate the overall malignant probability or benign probability.

[0120] For example, the inference function 135b calculates the overall probability of malignancy or benignity using a composite formula determined from the above-mentioned formulas (3), (4), and (5).

[0121] Returning to FIG. 2, the inference function 135b then classifies the disease type based on the combined probability (see "Classification" at the bottom of FIG. 2).

[0122] Specifically, the inference function 135b reads out the classification information recorded in the memory circuitry 132 by the learning function 135a, and classifies the disease type based on the combined probability using the threshold value of the read classification information.

[0123] For example, inference function 135b applies a threshold to the calculated overall benign or malignant probability to classify the disease type as positive (suggesting malignancy) or negative (suggesting benign).

[0124] Returning to FIG. 2, the inference function 135b then displays the disease type classification results on the display 134 based on the multiple inference results obtained (see "Display" shown at the bottom of FIG. 2). For example, the inference function 135b displays the multiple inference results together with information regarding the allocation of learning data to logic units during learning. Also, for example, the inference function 135b displays the result of combining the multiple inference results.

[0125] Specifically, the inference function 135b reads out the division information and allocation information recorded in the memory circuit 132 by the learning function 135a, and based on the read out division information and allocation information, displays multiple inference results regarding the disease type of the lesion, a synthesis result of the multiple inference results, and a classification result of the disease type.

[0126] FIG. 6 is a diagram showing an example of a classification result displayed by the inference function 135b according to the first embodiment.

[0127] For example, as shown in FIG. 6, the inference function 135b displays information indicating the viewpoint and granularity of division, and information indicating the labels used to divide the training data, based on the division information.

[0128] For example, the inference function 135b displays tissue classification as information indicating the viewpoint and granularity of division, and displays malignant lesion, malignant lesion 1, malignant lesion 2, benign lesion, benign lesion 1, and benign lesion 2 as information indicating labels. In one example, "hepatocellular carcinoma" is displayed for malignant lesion 1, "cholangiocarcinoma" is displayed for malignant lesion 2, "cyst" is displayed for benign lesion 1, and "hemangioma or other" is displayed for benign lesion 2.

[0129] In addition, based on the assignment information, the inference function 135b displays, for each piece of information indicating a label, a probability calculated from the output value of a logic unit to which learning data of each label group is assigned, as multiple inference results regarding the disease type of the lesion.

[0130] For example, for malignant lesion 1, the inference function 135b uses benign probability 1 (e.g., 98%) calculated from the output value of logic unit 1 to which learning data for malignant lesion 1 is assigned to display 1 - benign probability 1 (e.g., 2%) as the probability of occurrence and displays benign probability 1 (e.g., 98%) as the probability of non-occurrence. Furthermore, for malignant lesion 2, the inference function 135b uses benign probability 2 (98%) calculated from the output value of logic unit 2 to which learning data for malignant lesion 2 is assigned to display 1 - benign probability 2 (e.g., 2%) as the probability of occurrence and displays benign probability 2 (98%) as the probability of non-occurrence. Furthermore, for benign lesion 1, the inference function 135b uses malignant probability 1 (e.g., 70%) calculated from the output value of logic unit 3 to which learning data for benign lesion 1 is assigned to display 1 - malignant probability 1 (e.g., 30%) as the probability of occurrence and displays malignant probability 1 (e.g., 70%) as the probability of non-occurrence. Furthermore, for benign lesion 2, the inference function 135b uses the malignancy probability 2 (e.g., 90%) calculated from the output value of logic unit 4 to which learning data for benign lesion 2 is assigned, to display 1 - malignancy probability 2 (e.g., 10%) as the relevant probability, and displays malignancy probability 2 (e.g., 90%) as the non-relevant probability.

[0131] Furthermore, the inference function 135b displays the overall malignancy probability and benign probability calculated based on the estimation results of each inference unit as a composite result of the multiple inference results. At this time, if the overall malignancy probability is calculated using the above-mentioned composite formula, the inference function 135b displays 1 - overall malignancy probability as the malignancy probability, and if the overall benign probability is calculated, the inference function 135b displays 1 - overall benign probability as the benign probability.

[0132] For example, the inference function 135b displays the overall probability of malignancy (e.g., 2%) as the probability of occurrence for malignant lesions, and displays the overall probability of benignity (e.g., 30%) as the probability of occurrence for benign lesions.

[0133] The inference function 135b also displays the classification results of the disease type based on the overall probability of benignity or malignancy.

[0134] For example, the inference function 135b displays a classification result in which the disease type is classified as positive (suggesting malignancy) or negative (suggesting benign). For example, the inference function 135b displays "negative" and "benign lesion (NPV 99%)" as the classification result. Here, NPV (Negative Predictive Value) represents the probability of benignity when a negative determination is made. Note that, when the classification result is positive, the inference function 135b may display PPV (Positive Predictive Value), which represents the probability of malignancy when a positive determination is made, as the classification result.

[0135] The above has described the learning function 135a and the inference function 135b possessed by the processing circuitry 135 of the learning and inference device 130. In this embodiment, the processing circuitry 135 is realized by, for example, a processor. In this case, the above-described learning function 135a and inference function 135b are stored in the storage circuitry 132 in the form of programs executable by a computer. The processing circuitry 135 then reads out and executes each program stored in the storage circuitry 132, thereby realizing the function corresponding to each program. In other words, when each program has been read out, the processing circuitry 135 has the learning function 135a and inference function 135b shown in FIG. 1.

[0136] The following describes the processing procedures performed by the learning function 135a and the inference function 135b of the processing circuitry 135 according to the first embodiment.

[0137] FIG. 7 is a flowchart showing the processing procedure of the learning process performed by the learning function 135a according to the first embodiment.

[0138] For example, as shown in FIG. 7, the learning function 135a acquires a plurality of learning data sets including information on a lesion area extracted from a medical image and a label assigned to the lesion area (step S101).

[0139] Next, the learning function 135a divides the acquired plurality of learning data into a plurality of groups based on the labels included in each learning data (step S102).

[0140] Next, the learning function 135a assigns the divided learning data to multiple logic units and learns each logic unit individually, thereby generating multiple logic units that accept input of information about the lesion area and output inference results about the disease type of the lesion area (step S103).

[0141] Next, the learning function 135a estimates a histogram for calculating the probability of a disease type for each inference unit based on the estimation result of each inference unit (step S104).

[0142] Next, the learning function 135a determines a synthesis formula for synthesizing the probabilities of the disease types calculated for each inference unit (step S105).

[0143] Next, the learning function 135a determines a threshold value for classifying the disease type based on the combined probability (step S106).

[0144] For example, when the processing circuitry 135 is realized by a processor, the processing of steps S101 to S106 described above is realized by the processing circuitry 135 reading out a program corresponding to the learning function 135a from the storage circuitry 132 and executing it.

[0145] FIG. 8 is a flowchart showing the processing procedure of the inference process performed by the inference function 135b according to the first embodiment.

[0146] For example, as shown in FIG. 8, the inference function 135b acquires a plurality of examination data including information on a lesion area extracted from a medical image of a subject to be examined (step S201).

[0147] Next, the inference function 135b inputs the test data into each of a plurality of logic units to obtain a plurality of inference results regarding the disease type of the lesion area (step S202).

[0148] Next, the inference function 135b calculates the probability of the disease type for each inference unit based on the inference result of each inference unit (step S203).

[0149] Next, the inference function 135b combines the probabilities of the disease types calculated for each inference unit (step S204).

[0150] Next, the inference function 135b classifies the disease type based on the combined probability (step S205).

[0151] Next, the inference function 135b displays the disease type classification result on the display 134 based on the acquired multiple inference results (step S206).

[0152] For example, when the processing circuitry 135 is realized by a processor, the processing of steps S201 to S206 described above is realized by the processing circuitry 135 reading out a program corresponding to the inference function 135b from the storage circuitry 132 and executing it.

[0153] As described above, in the first embodiment, the learning function 135a divides multiple pieces of learning data, each containing information about a lesion area extracted from a medical image and a label assigned to the lesion area, into multiple groups based on the label included in each piece of learning data. The learning function 135a then assigns the divided learning data to multiple logic units and trains each logic unit individually, thereby generating multiple trained logic units that accept input of information about a lesion area and output inference results about the disease type of the lesion area. The inference function 135b then inputs test data containing information about a lesion area extracted from a medical image of an examination target into each of the multiple trained logic units, thereby obtaining multiple inference results about the disease type of the lesion area. The inference function 135b then displays a classification result of the disease type of the lesion area based on the obtained inference results.

[0154] According to the above configuration, by dividing multiple training data into multiple groups based on labels, and allocating the divided training data to multiple logic units and training each logic unit individually, the model of the logic unit can be simplified compared to when multiple training data are assigned to a single logic unit without being divided. This makes it possible to obtain high classification accuracy even with a small amount of training data. Furthermore, when compared with the same number of training data, classification accuracy can be improved.

[0155] (Second embodiment) In the first embodiment described above, an example was described in which one learning and inference device 130 included in the medical image analysis system 100 has both a learning function 135a and an inference function 135b, but the embodiment is not limited to this.

[0156] For example, a medical image analysis system may be configured to include a learning device and an inference device, with the learning device having a learning function and the inference device having an inference function. Such an example will be described below as a second embodiment. Note that the second embodiment will be described focusing on differences from the first embodiment, and detailed descriptions of content that overlaps with the first embodiment will be omitted.

[0157] FIG. 9 is a diagram illustrating an example of the configuration of a medical image analysis system according to the second embodiment.

[0158] 9, a medical image analysis system 200 according to this embodiment includes a medical image capturing device 210, a medical image storage device 220, a learning device 240, and an inference device 250. Here, each device is connected to each other so as to be able to communicate with each other via a network 20 such as the Internet or a LAN (Local Area Network). Note that the medical image analysis system 200 may further include various other devices not shown.

[0159] Here, the medical imaging device 210 and the medical image storage device 220 are the same devices as the medical imaging device 110 and the medical image storage device 120 described in the first embodiment.

[0160] The learning device 240 and the inference device 250 each perform various processes using medical images. Specifically, the learning device 240 and the inference device 250 acquire medical images from the medical image acquisition device 210 or the medical image storage device 220 via the network 20, and process the acquired medical images. For example, the learning device 240 and the inference device 250 are realized by computer equipment such as a server, a workstation, or a personal computer.

[0161] For example, the learning device 240 includes a network interface 241, a memory circuit 242, an input interface 243, a display 244, and a processing circuit 245.

[0162] Here, the NW interface 241, memory circuit 242, input interface 243, display 244 and processing circuit 245 are configured in the same manner as the NW interface 131, memory circuit 132, input interface 133, display 134 and processing circuit 135 of the learning inference device 130 described in the first embodiment.

[0163] However, while in the first embodiment the processing circuit 135 of the learning and inference device 130 has both the learning function 135a and the inference function 135b, in this embodiment the processing circuit 245 of the learning device 240 has only the learning function 245a. Here, the learning function 245a is an example of a learning unit.

[0164] The inference device 250 also has a NW interface 251 , a memory circuit 252 , an input interface 253 , a display 254 , and a processing circuit 255 .

[0165] Here, the NW interface 251, memory circuit 252, input interface 253, display 254 and processing circuit 255 are configured in the same manner as the NW interface 131, memory circuit 132, input interface 133, display 134 and processing circuit 135 of the learning inference device 130 described in the first embodiment.

[0166] However, while in the first embodiment the processing circuit 135 of the learning and inference device 130 has both the learning function 135a and the inference function 135b, in this embodiment the processing circuit 255 of the inference device 250 has only the inference function 255b. Here, the inference function 255b is an example of an inference unit.

[0167] The learning function 245a of the processing circuit 245 of the learning device 240 and the inference function 255b of the processing circuit 255 of the inference device 250 will be described in detail below.

[0168] FIG. 10 is a diagram showing the flow of processing performed by the learning function 245a and the inference function 255b according to the second embodiment.

[0169] In this embodiment, during learning, the processing shown in the upper part of Figure 2 is performed by the learning function 245a of the learning device 240, and after the processing during learning is completed, during inference, the processing shown in the lower part of Figure 2 is performed by the inference function 255b of the inference device 250.

[0170] First, the learning process performed by the learning function 245a of the learning device 240 will be described.

[0171] For example, as shown in the upper part of FIG. 10, the learning function 245a, similar to the learning function 135a described in the first embodiment, acquires multiple pieces of learning data including information about the lesion area extracted from the medical image and the label assigned to the lesion area (see "Learning Data" shown in the upper part of FIG. 10).

[0172] Next, similar to the learning function 135a described in the first embodiment, the learning function 245a divides the acquired multiple learning data into multiple groups based on the labels included in each learning data (see "Division" shown at the top of Figure 10).

[0173] Next, similar to the learning function 135a described in the first embodiment, the learning function 245a assigns the divided learning data to multiple logic units and trains each logic unit individually, thereby generating multiple logic units that accept input of information about the lesion area and output inference results about the disease type of the lesion area (see "Assignment" and "Inference Unit" shown at the top of Figure 10).

[0174] Next, the learning function 245a, similar to the learning function 135a described in the first embodiment, estimates a histogram for calculating the probability of a disease type for each inference device based on the estimation results of each inference device (see "Probability Calculation" shown at the top of Figure 10).

[0175] Next, the learning function 245a, similar to the learning function 135a described in the first embodiment, determines a synthesis formula for synthesizing the probabilities of the disease types calculated for each inference unit (see "Synthesis" shown at the top of Figure 10).

[0176] Next, the learning function 245a determines a threshold for classifying the disease type based on the combined probability, similar to the learning function 135a described in the first embodiment (see "Classification" shown at the top of Figure 10).

[0177] Then, in this embodiment, after performing the above-mentioned processing, the learning function 245a outputs recorded information of the learning results obtained during learning to the inference device 250 (see "Learning result output" shown at the top of Figure 10).

[0178] Specifically, the learning function 245a outputs the information recorded in the memory circuitry 242 during the above-mentioned processing as recording information of the learning result to the inference device 250 via the network 20. Here, the recording information of the learning result includes the division information, allocation information, model parameters, inference parameters (learning model parameters, etc.), synthesis information, and classification information (learning method and model identification information) described in the first embodiment.

[0179] Next, the processing during inference performed by the inference function 255b of the inference device 250 will be described.

[0180] In this embodiment, the inference function 255b acquires the record information of the learning results output by the learning function 245a of the learning device 240, and uses the record information to acquire multiple inference results regarding the disease type of the lesion area.

[0181] For example, as shown in the lower part of FIG. 10, in this embodiment, the inference function 255b acquires, via the network 20, the record information of the learning result output by the learning function 245a of the learning device 240.

[0182] Next, similar to the inference function 135b described in the first embodiment, after the above-mentioned learning processing is performed, the inference function 255b acquires, during inference, multiple test data including information about the lesion area extracted from the medical image of the subject being examined (see ``Test Data'' shown at the bottom of Figure 10).

[0183] Next, in the same manner as the inference function 135b described in the first embodiment, the inference function 255b inputs the test data into each of multiple logic units to obtain multiple inference results regarding the disease type of the lesion area (see the "inference unit" shown at the bottom of Figure 10).

[0184] At this time, in this embodiment, the inference function 255b uses model parameters included in the record information of the learning results acquired from the learning device 240 to acquire multiple inference results regarding the disease type of the lesion area.

[0185] Next, the inference function 255b, similar to the inference function 135b described in the first embodiment, calculates the probability of the disease type for each inference device based on the estimation results of each inference device (see "Probability Calculation" shown at the bottom of Figure 10).

[0186] At this time, in this embodiment, the inference function 255b uses a histogram of inference parameters included in the recording information of the learning results acquired from the learning device 240 to calculate the probability of the disease type based on the estimation results of the inference device.

[0187] Next, the inference function 255b combines the probabilities of the disease types calculated for each inference unit, similar to the inference function 135b described in the first embodiment (see "Combination" shown at the bottom of FIG. 10).

[0188] At this time, specifically, in this embodiment, the inference function 255b uses a synthesis formula of the synthesis information contained in the recording information of the learning results acquired from the learning device 240 to synthesize the probabilities of the disease types calculated for each inference device to calculate the overall probability.

[0189] Next, the inference function 255b classifies the disease type based on the combined probability, similar to the inference function 135b described in the first embodiment (see "Classification" shown at the bottom of FIG. 10).

[0190] At this time, in this embodiment, the inference function 255b uses the threshold value of the classification information included in the record information of the learning result acquired from the learning device 240 to classify the disease type based on the combined probability.

[0191] Next, the inference function 255b, similar to the inference function 135b described in the first embodiment, displays the disease type classification results on the display 254 based on the multiple inference results obtained (see "Display" shown at the bottom of Figure 10).

[0192] At this time, in this embodiment, the inference function 255b uses the division information and allocation information included in the record information of the learning result acquired from the learning device 240 to display the classification result of the disease type.

[0193] Here, the above-mentioned inference function 255b is realized by, for example, incorporating the record information of the learning result into software corresponding to the inference function 255b.

[0194] In this case, for example, the inference function 255b generates an installer or an executable file for incorporating the stored information of the learning result acquired from the learning device 240 into the software corresponding to the inference function 255b. Then, at the time of inference, the installer or executable file is executed by an operator or the like, and the software into which the recorded information is incorporated executes the above-mentioned processing.

[0195] Alternatively, the inference function 255b may be realized by software corresponding to the inference function 255b reading the record information of the learning results.

[0196] In this case, for example, when the inference function 255b acquires stored information of the learning result from the learning device 240, the inference function 255b stores the recorded information in the memory circuitry 252. Then, at the time of inference, software corresponding to the inference function 255b reads the recorded information from the memory circuitry 252 and executes the above-described processing.

[0197] Furthermore, in this embodiment, the learning function 245a may output the counting results, algorithm information, rights information, license management information, and the like in addition to the recording information of the learning results.

[0198] Here, the aggregated results are, for example, the sensitivity, specificity, NPV, PPV, etc. of the training data or other validation data. The algorithm information is, for example, the name and version number of the algorithm used for training. The rights information is the rights information of the developer of the training data or algorithm, the developer of the inference application (the person who carried out the training), etc. The license management information recorded includes the license name and period of the algorithm or training result, the number of licenses, and an administration password.

[0199] For example, the inference function 255b may acquire this information along with records of learning results via the network 20 and control the inference process based on the acquired information (see "Inference Control" shown at the bottom of Figure 10).

[0200] For example, the inference function 255b may display the sensitivity, specificity, NPV, PPV, etc. of the training data or other validation data together with the disease type classification result on the display 254 based on the aggregation result. Also, for example, the inference function 255b may display the name of the algorithm used for training, the version number, the developer of the training data or algorithm, the developer of the inference application (the person who performed the training), etc., together with the disease type classification result on the display 254 based on algorithm information and rights information. Also, for example, the inference function 255b may manage the license of software corresponding to the inference function 255b based on license management information.

[0201] In addition, in this embodiment, during learning, the learning function 245a may use the learning data or other verification data to calculate the overall malignant probability or benign probability using a determined synthesis formula, similar to the inference function 255b, apply the determined threshold to the malignant probability or benign probability to classify the disease type, and evaluate the classification result, thereby optimizing the threshold for classifying the disease type (see ``Optimization'' shown at the top of Figure 10).

[0202] As described above, in the second embodiment, the learning function 245a of the learning device 240 divides multiple pieces of learning data, including information about lesion areas extracted from medical images and labels assigned to the lesion areas, into multiple groups based on the labels included in each piece of learning data. The learning function 245a of the learning device 240 then assigns the divided learning data to multiple logic units and trains each logic unit individually, thereby generating multiple logic units that accept input of information about lesion areas and output inference results regarding the disease type of the lesion area. The inference function 255b of the inference device 250 then inputs test data, including information about lesion areas extracted from medical images of the test target, into multiple trained logic units, respectively, to obtain multiple inference results regarding the disease type of the lesion area. The inference function 255b of the inference device 250 then displays a classification result of the disease type of the lesion area based on the obtained multiple inference results.

[0203] According to the above configuration, similar to the first embodiment, by dividing multiple training data into multiple groups based on labels, assigning the divided training data to multiple logic units, and training each logic unit individually, the model of the logic unit can be simplified compared to when multiple training data are assigned to a single logic unit without being divided. This makes it possible to obtain high classification accuracy even with a small amount of training data. Furthermore, when compared with the same number of training data, classification accuracy can be improved.

[0204] (Variation) In the above-described embodiment, an example was described in which malignant lesions were classified into hepatocellular carcinoma and cholangiocarcinoma, and benign lesions were classified into cysts, hemangiomas, and others, using tissue classification obtained from a pathological examination as the division viewpoint and granularity, but the method of classifying lesions is not limited to this. Therefore, hereinafter, an example in which another classification method is used will be described as a modified example.

[0205] 11 to 16 are diagrams showing examples of displaying classification results according to modified examples.

[0206] In the examples shown in Figures 11 to 16, only information indicating the division viewpoint and granularity, and information indicating the label are shown, and multiple inference results regarding the disease type of the lesion, the synthesis results of multiple inference results, and the classification results of the disease type are not shown.

[0207] For example, Figure 11 shows an example in which the tissue type obtained from a pathological examination is used as the division perspective and granularity, and malignant lesions are classified into cancer tissue type A and cancer tissue type B, and benign lesions are classified into benign tumor tissue type A and benign tumor tissue type B.

[0208] In this example, the training data is divided into a group whose tissue type belongs to malignant lesions and a group whose tissue type belongs to benign lesions. The training data divided into the malignant lesion group is further divided into a group whose tissue type belongs to cancer tissue type A and a group whose tissue type belongs to cancer tissue type B. The training data divided into the benign lesion group is further divided into a group whose tissue type belongs to benign tumor tissue type A and a group whose tissue type belongs to benign tumor tissue type B.

[0209] Figure 12 shows an example in which malignant lesions are classified into cancer tissue type A and cancer tissue type B, and benign lesions are classified into benign tumor tissue type A and benign tumor tissue type B, based on the viewpoint and granularity of the division based on the imaging findings of a doctor in a pathological examination.

[0210] In this example, the training data is divided into a group whose imaging findings correspond to malignant lesions and a group whose imaging findings correspond to benign lesions. The training data divided into the malignant lesion group is further divided into a group whose imaging findings correspond to cancer histology type A and a group whose imaging findings correspond to cancer histology type B. The training data divided into the benign lesion group is further divided into a group whose imaging findings correspond to benign tumor histology type A and a group whose imaging findings correspond to benign tumor histology type B.

[0211] Figure 13 also shows an example in which the blood test results and the gender included in the patient (subject) background information are used as the division perspective and granularity, and malignant lesions and benign lesions are classified into cases where the albumin value is above the standard value (+) and cases where it is below the standard value (-), and further, cases where the albumin value is above the standard value (+) and cases where it is below the standard value (-) are classified into cases where the subject's gender is male (M) and cases where it is female (F), respectively.

[0212] In this example, the training data is divided into a group whose albumin values ​​correspond to malignant lesions and a group whose albumin values ​​correspond to benign lesions. Furthermore, the training data divided into the malignant lesion group and the training data divided into the benign lesion group are each further divided into a group whose albumin values ​​are positive and a group whose albumin values ​​are negative. Furthermore, the training data divided into the albumin value positive group and the training data divided into the albumin value negative group are each further divided into a group whose gender is M and a group whose gender is F. That is, the example in FIG. 13 is an example of dividing the training data into three stages.

[0213] Figure 14 also shows an example in which malignant lesions are classified into those with gene mutations and those without gene mutations, using the genes or tissue type present and the lesion size as the division criteria and granularity, and those without gene mutations are further classified into those where the lesion size is less than a predetermined standard value (small size) and those where the lesion size is equal to or greater than the standard value (large size). Benign lesions are classified into benign tumor tissue type A and benign tumor tissue type B, and benign tumor tissue type A and benign tumor tissue type B are further classified into those where the lesion size is less than a predetermined standard value (small size) and those where the lesion size is equal to or greater than the standard value (large size).

[0214] In this example, the training data is divided into a group whose histology belongs to malignant lesions and a group whose histology belongs to benign lesions. The training data divided into the malignant lesion group is further divided into a group whose genes have gene mutations and a group whose genes do not have gene mutations. The training data divided into the group whose genes have gene mutations is not divided, while the training data divided into the group whose genes do not have gene mutations is further divided into a group whose lesion size is small and a group whose lesion size is large. The training data divided into the benign lesion group is further divided into a group whose histology is benign tumor histology type A and a group whose histology is benign tumor histology type B. The training data divided into the benign tumor histology type A group and the training data divided into the benign tumor histology type B group are each further divided into a group whose lesion size is small and a group whose lesion size is large. That is, the example in FIG. 14 is an example in which division into two stages and division into three stages are combined.

[0215] Figure 15 shows an example of an image segmentation where the region label (e.g., intraperitoneal fluid accumulation) is used as the segmentation perspective and granularity, and intraperitoneal fluid accumulation is classified into intraperitoneal hematoma and ascites, and cases that are not intraperitoneal fluid accumulation are classified into tissue (organ) A and tissue (organ) B.

[0216] In this example, the training data is divided into a group with a region label of intraperitoneal fluid accumulation and a group with a region label other than intraperitoneal fluid accumulation. The training data divided into the group with a region label of intraperitoneal fluid accumulation is further divided into a group with intraperitoneal hematoma and a group with a region label other than intraperitoneal fluid accumulation. The training data divided into the group with a region label other than intraperitoneal fluid accumulation is further divided into a group with a region label of tissue (organ) A and a group with a region label of tissue (organ) B.

[0217] Figure 16 shows an example in which the region labels used for lesion detection are used as the division viewpoint and granularity, and lesions are classified into lesion A and lesion B, and non-lesions are classified into normal tissue (organ) A and normal tissue (organ) B.

[0218] In this example, the training data is divided into a group whose region label corresponds to a lesion and a group whose region label corresponds to a non-lesion. The training data divided into the lesion group is further divided into a group whose region label is lesion A and a group whose region label is lesion B. The training data divided into the non-lesion group is further divided into a group whose region label is normal tissue (organ) A and a group whose region label is normal tissue (organ) B.

[0219] As shown in FIGS. 11 to 16, according to the above-described embodiment, it is possible to display the results of disease type classification from various perspectives and with various granularities.

[0220] (Other embodiments) The medical image analysis system according to the above-described embodiment can also be implemented as a cloud system that provides cloud services. In this case, for example, the learning function and inference function described above are implemented in one or more server devices included in the cloud system. Then, the inference results and classification results obtained by the learning function and inference function of the server device are transmitted via a network to a client device used by a service user and displayed on a display or the like of the client device.

[0221] In addition, in the above-described embodiment, an example has been described in which the learning unit and the inference unit in this specification are realized by the learning function and the inference function of a processing circuit, respectively, but the embodiment is not limited to this. For example, in addition to being realized by the learning function and the inference function described in the above-described embodiment, the learning unit and the inference unit in this specification may also be realized by hardware only, software only, or a combination of hardware and software.

[0222] Furthermore, in the above-described embodiment, an example in which the processing circuit is realized by a single processor has been described, but the embodiment is not limited thereto. For example, the processing circuit may be configured by combining multiple independent processors, and each processor may execute a program to realize each processing function. Furthermore, each processing function of the processing circuit may be realized by being appropriately distributed or integrated among a single or multiple processing circuits. Furthermore, each processing function of the processing circuit may be realized by a combination of hardware and software, such as a circuit. Furthermore, while the above-described embodiment describes an example in which programs corresponding to each processing function are stored in a single storage circuit, the embodiment is not limited thereto. For example, a configuration in which programs corresponding to each processing function are distributed and stored among multiple storage circuits, and the processing circuit reads and executes each program from each storage circuit, may also be used.

[0223] Furthermore, the term "processor" used in the description of the above-mentioned embodiments refers to circuits such as a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a programmable logic device (e.g., a simple programmable logic device (SPLD), a complex programmable logic device (CPLD), and a field programmable gate array (FPGA)). Here, instead of storing a program in a memory circuit, the processor may be configured so that the program is directly embedded in the circuit. In this case, the processor realizes its function by reading and executing the program embedded in the circuit. Furthermore, each processor in the present embodiment is not limited to being configured as a single circuit for each processor, but may also be configured as a single processor by combining multiple independent circuits to realize its function.

[0224] Here, the program executed by the processor is provided in advance in a read-only memory (ROM) or a storage circuit. The program may be provided by being recorded on a computer-readable, non-transitory storage medium such as a compact disk (CD)-ROM, a flexible disk (FD), a recordable CD-R (CD-R), or a digital versatile disk (DVD) in a format that can be installed or executed on these devices. The program may also be provided or distributed by being stored on a computer connected to a network such as the Internet and downloaded via the network. For example, the program may be composed of modules including the above-described processing functions. In actual hardware, a CPU reads and executes the program from a storage medium such as a ROM, whereby each module is loaded into a main memory device and generated on the main memory device.

[0225] In the above-described embodiments, the components of each device shown in the drawings are conceptual functional units and do not necessarily have to be physically configured as shown. In other words, the specific form of distribution or integration of each device is not limited to that shown in the drawings, and all or part of the devices can be functionally or physically distributed or integrated in any unit depending on various loads, usage conditions, etc. Furthermore, all or any part of the processing functions performed by each device can be realized by a CPU and a program analyzed and executed by the CPU, or can be realized as hardware using wired logic.

[0226] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using a known method.In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified.

[0227] The various data handled in this specification are typically digital data.

[0228] According to at least one of the embodiments described above, high classification accuracy can be obtained even with a small amount of training data.

[0229] Although several embodiments have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, and combinations of embodiments can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, as well as within the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0230] 100,200 Medical image analysis system 110,210 Medical imaging equipment 120,220 Medical image archiving device 130 Learning and Inference Device 240 Learning Device 250 Reasoning device 135,245,255 processing circuit 135a,245a Learning Function 135b,255b inference function

Claims

1. A medical image analysis system comprising a learning unit and an inference unit, The learning unit Dividing a plurality of learning data sets, each set including information on a lesion area extracted from a medical image and a label assigned to the lesion area, into a plurality of groups based on the label included in each learning data set; the divided learning data is assigned to a plurality of logic units, and each logic unit is trained individually, thereby generating a plurality of trained logic units that accept input of information about a lesion area and output an inference result about the disease type of the lesion area; The inference unit inputting test data including information about a lesion area extracted from a medical image of a subject to be examined into each of the plurality of trained logic units, thereby obtaining a plurality of inference results regarding the disease type of the lesion area; displaying a classification result of the disease type based on the plurality of inference results obtained; Medical image analysis system.

2. the learning unit divides the plurality of training data into a plurality of first classification groups based on the labels, and further divides the training data divided into each of the plurality of first classification groups into a plurality of second classification groups included in each of the plurality of first classifications; The medical image analysis system according to claim 1 .

3. the learning unit generates the plurality of trained logic units by assigning learning data of one first classification group among the plurality of first classification groups and learning data of a second classification group included in another first classification to one logic unit and performing learning. The medical image analysis system according to claim 2 .

4. when the learning unit performs learning by assigning the learning data of the first classification group and the learning data of the second classification group to the one logic unit, the learning unit performs learning by assigning a weight to the learning data of one group greater than a weight to the learning data of the other group. The medical image analysis system according to claim 3 .

5. the learning unit divides the plurality of training data into a plurality of groups based on information included in each training data and different from the label, and then divides the plurality of training data included in one group into a plurality of groups based on the label; the inference unit divides a plurality of test data into a plurality of groups based on information contained in each test data that is different from the label, and then inputs one test data item contained in one group to each of the plurality of trained logic units, thereby acquiring the plurality of inference results. The medical image analysis system according to claim 1 .

6. the inference unit displays a result obtained by combining the plurality of inference results. The medical image analysis system according to claim 1 .

7. the inference unit displays the plurality of inference results together with information regarding allocation of the learning data to the logic units during learning. The medical image analysis system according to claim 1 .

8. A learning and inference device is included, the learning and inference device includes the learning unit and the inference unit; The medical image analysis system according to any one of claims 1 to 7.

9. a learning device and a reasoning device, the learning device includes the learning unit, The inference device includes the inference unit. The medical image analysis system according to any one of claims 1 to 7.

10. the learning unit outputs record information of the learning results obtained during learning; the inference unit acquires record information of the learning results output by the learning unit, and acquires a plurality of inference results regarding disease types of the lesion area using the record information; The medical image analysis system according to claim 9 .

11. the inference unit is realized by incorporating the record information of the learning result into software corresponding to the inference unit; The medical image analysis system according to claim 10.

12. the inference unit is realized by software corresponding to the inference unit reading the record information of the learning result; The medical image analysis system according to claim 10.

13. A medical image analysis method applied to a medical image analysis system including a learning unit and an inference unit, The learning unit: a step of dividing a plurality of learning data sets, each set including information on a lesion area extracted from a medical image and a label assigned to the lesion area, into a plurality of groups based on the label included in each learning data set; a step of allocating the divided learning data to a plurality of logic units and learning each logic unit individually to generate a plurality of trained logic units that accept input of information about a lesion area and output an inference result about the disease type of the lesion area; The inference unit: inputting test data including information on a lesion area extracted from a medical image of a subject to be examined into each of the plurality of trained logic units, thereby obtaining a plurality of inference results regarding disease types of the lesion area; displaying a classification result of the disease type based on the plurality of inference results obtained; A medical image analysis method comprising:

Images