Medical diagnostic support device, identifier generation device, identifier generation method and program

Abstract

Even with biased training data, stable classification results can be obtained. [Solution] In the identification device 100, the identification processing unit 115 uses a classifier 30 that has undergone multitask learning to output first information regarding the quality of the identification target in the medical image or whether it should be referred, and second information regarding at least one of the size, age, body part, and race of the identification target, to identify the identification target in the medical image.

Description

Technical Field

[0001] The present invention relates to a medical diagnosis support device, an identifier generation device, an identifier generation method, and a program.

Background Art

[0002] A technique for identifying an object to be identified in an image using an identifier learned by machine learning is known. For example, Patent Document 1 discloses a medical image classification device that classifies medical images into a plurality of types of case regions using a discriminator learned by a deep learning method.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, depending on the object to be identified, it may be difficult to collect training data for machine learning in a balanced manner. In such a case, since the training data is biased, there is a risk of bias in the identification result.

[0005] The present invention is for solving the above problems, and an object thereof is to obtain a stable identification result even when there is bias in the training data.

Means for Solving the Problems

[0006] To achieve the above objective, the medical diagnostic support device according to the present invention is characterized by comprising an identification processing unit that identifies the object to be identified in a medical image using a classifier that has undergone multitask learning to output first information relating to the quality of the object to be identified in the medical image or a recommendation for referral, and second information relating to at least one of the size, age, body part, and race of the object to be identified. [Effects of the Invention]

[0007] According to the present invention, stable classification results can be obtained even if the training data is biased. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing the configuration of the identification device according to Embodiment 1. [Figure 2] This figure shows the input and output of the classifier during the learning phase according to Embodiment 1. [Figure 3] This is a diagram showing the configuration of the identifier according to Embodiment 1. [Figure 4] This flowchart shows the flow of the classifier generation process performed by the identification device according to Embodiment 1. [Figure 5] This figure shows the input and output of the discriminator in the inference phase according to Embodiment 1. [Figure 6] This flowchart shows the flow of the identification process performed by the identification device according to Embodiment 1. [Figure 7] Figures (a) to (c) show the results of evaluating the classifier according to Embodiment 1. [Figure 8] This is a block diagram showing the configuration of the identifier generation device according to Embodiment 2. [Figure 9] This is a block diagram showing the configuration of the identification device according to Embodiment 2. [Modes for carrying out the invention]

[0009] Embodiments of the present invention will be described below with reference to the drawings. In the drawings, the same or corresponding parts are denoted by the same reference numerals. The identification device 100 according to Embodiment 1 is a device that uses a classifier 30 generated by machine learning to identify whether an object to be identified in an input image is benign or malignant. In particular, the identification device 100 according to Embodiment 1 functions as a medical diagnostic support device that identifies whether a lesion in a medical image in which a lesion has been photographed is benign or malignant. Here, a medical image is an image taken for medical diagnostic purposes, in which a part of the living body suspected of having a disease has been photographed. An example of a medical image is a dermatological image, which is an image of a lesion on the skin. Alternatively, the medical image is not limited to this, and may be any other image capable of photographing a lesion, such as an endoscopic image, X-ray image, CT (Computed Tomography) image, or ultrasound image.

[0010] As shown in Figure 1, the identification device 100 comprises a control unit 11, a storage unit 12, an operation unit 13, a display unit 14, and a communication unit 15. The control unit 11 includes a CPU (Central Processing Unit), ROM (Read Only Memory), and RAM (Random Access Memory). The CPU is a central processing unit that includes a microprocessor and performs various processes and calculations. In the control unit 11, the CPU reads the control program stored in the ROM and controls the operation of the entire identification device 100 while using the RAM as work memory. The processing of the control unit 11 may be executed by a single CPU or by multiple CPUs.

[0011] The memory unit 12 is a non-volatile memory such as flash memory or a hard disk. The memory unit 12 stores programs and data executed by the control unit 11, and data generated by the control unit 51. Specifically, the memory unit 12 stores the classifier 30, training data 121, and evaluation data 122. Details of these will be described later.

[0012] The operation unit 13 is equipped with input devices such as a keyboard, mouse, and touch panel, and accepts user input. The display unit 14 is equipped with a display device such as a liquid crystal display or an organic EL (Electro-Luminescence) display, and displays various images under the control of the control unit 11. The communication unit 15 is equipped with a communication interface for communicating with external devices of the identification device 100. For example, the communication unit 15 communicates with external devices in accordance with well-known communication standards such as LAN (Local Area Network) and USB (Universal Serial Bus).

[0013] The control unit 11 performs processing in two phases: a first phase, which is the learning phase of the classifier 30, and a second phase, which is the inference phase of the classifier 30. The control unit 11 includes a learning unit 111, an accuracy calculation unit 112, and a classifier determination unit 113 as functions in the learning phase. The control unit 11 also includes an image receiving unit 114, a classification processing unit 115, and a result output unit 116 as functions in the inference phase. In the control unit 11, the CPU functions by reading a program stored in ROM into RAM and executing that program to control each of these units. Note that in the control unit 11, one CPU may function as the learning phase and the inference phase units, or multiple CPUs may jointly function as the learning phase and the inference phase units.

[0014] First, let's explain the learning phase. The learning phase is the phase in which a classifier 30 capable of accurately identifying input data is generated using machine learning techniques. Here, the classifier 30 is a computer program for identifying whether an object to be identified in an input image is good or bad, and is a trained model that has been trained by machine learning using training data 121.

[0015] Specifically, as shown in FIG. 2, the discriminator 30 receives an input of a medical image in which a lesion to be discriminated is photographed. Then, as output information for the input of the medical image, the discriminator 30 outputs first information regarding the quality of the lesion (region of interest) photographed in the medical image and second information regarding matters other than the quality of the lesion. Specifically, the first information is information indicating a discrimination result as to whether the lesion photographed in the medical image is benign or malignant. The second information is information indicating an estimated value of the size of the lesion in the medical image. Thus, for the input of one medical image, the discriminator 30 outputs output information indicating two discrimination results, namely, whether the lesion photographed in the medical image is benign or malignant and the size of the lesion.

[0016] More specifically, as shown in FIG. 3, the discriminator 30 includes an NN (Neural Network) unit 31 and a benign / malignant discrimination unit 32. The NN unit 31 is a unit that executes main operations in the discriminator 30. Specifically described, the NN unit 31 outputs a malignancy degree M and a size S for the input medical image by a method such as logistic regression or DNN (Deep Neural Network). Here, the malignancy degree M is a value indicating the probability that the lesion is malignant. The malignancy degree M takes a value between 0 and 1, meaning that the closer the malignancy degree M is to 0, the higher the probability that the lesion is benign, and the closer the malignancy degree M is to 1, the higher the probability that the lesion is malignant. The size S is the size of the lesion. For example, the major axis of the lesion is used as the size S. [[ID=⑤]] [[ID=⑥]]

[0017] [[ID=⑦]] [[ID=⑧]]As an example, the NN unit 31 is constituted by a neural network having a multi-layer structure, and includes an input layer to which input data is input, an intermediate layer (hidden layer) that performs operations such as convolution and pooling on the input data, and an output layer (fully connected layer) that outputs the operation result. The NN unit 31 calculates the malignancy degree M and the size S of the lesion photographed in the medical image input to the input layer in the intermediate layer, and outputs the calculated malignancy degree M and size S from the output layer. [[ID=⑨]] [[ID=⑩]]

[0018] [[ID=⑪]] Based on the malignancy degree M output from the NN unit 31, the benign / malignant discrimination unit 32 outputs first information indicating whether the lesion captured in the input medical image is benign or malignant. Specifically, the benign / malignant discrimination unit 32 compares the malignancy degree M output from the NN unit 31 with a preset cut-off value. Then, when the malignancy degree M is greater than the cut-off value, the benign / malignant discrimination unit 32 determines that the lesion is malignant, and when the malignancy degree M is less than the cut-off value, the benign / malignant discrimination unit 32 determines that the lesion is benign. The cut-off value is preset to an appropriate value between 0 and 1 so that the benign / malignant discrimination unit 32 can appropriately discriminate between benign and malignant.

[0019] The benign / malignant discrimination unit 32 outputs such a discrimination result as the first information of the discriminator 30. For example, when the benign / malignant discrimination unit 32 determines that the lesion is malignant, it outputs the value "1" as the first information, and when it determines that the lesion is benign, it outputs the value "0" as the first information. Note that the size S output from the NN unit 31 is directly output from the discriminator 30 as the second information.

[0020] Returning to FIG. 1, in the learning phase, the learning unit 111 performs machine learning using the training data 121. Here, the training data 121 is a data set (learning data set) used by the learning unit 111 to perform machine learning. The training data 121 includes a plurality of training input images (hereinafter referred to as "training images") used as teacher data. Each of the plurality of training images is an image in which a lesion is captured, and the correct answer of whether the captured lesion is benign or malignant and its size is known in advance. In the training data 121, for each training image, a correct malignancy degree LM and a correct size LS, which are the correct answers of the captured lesion, are pre-assigned as teacher labels (correct labels). The correct malignancy degree LM is represented by two values, 1 or 0, which becomes 1 when the lesion is malignant and 0 when the lesion is benign. As the correct malignancy degree LM, the result of pathological diagnosis (biopsy) can be used.

[0021] The learning unit 111 performs multitask learning using multiple training images included in the training data 121 as training data, and learns the computational parameters of the classifier 30. Here, multitask learning refers to a machine learning method in which a model is trained by having it learn multiple tasks simultaneously. In Embodiment 1, the multiple tasks specifically correspond to a benign or malignant discrimination process that identifies whether a lesion captured in a medical image is benign or malignant, and a size estimation process that estimates the size of the lesion. The learning unit 111 trains a single classifier 30 on these two tasks.

[0022] Specifically, the learning unit 111 inputs each of the multiple training images included in the training data 121 into the classifier 30. In the classifier 30, the NN unit 31 calculates and outputs estimated values ​​of the malignancy degree M and size S of the lesion captured in the input training images. The learning unit 111 adjusts the calculation parameters of the classifier 30 using methods such as backpropagation so that the malignancy degree M and size S output from the NN unit 31 approach the correct malignancy degree LM and correct size LS, respectively, attached to the input training images. The calculation parameters of the classifier 30 are, for example, the connection weights of each layer of the neural network in the NN unit 31, that is, weights indicating the connection strength between multiple neurons (nodes). By adjusting the calculation parameters, the estimated values ​​of malignancy degree M and size S output by the NN unit 31 for the input training images change. The learning unit 111 adjusts the computation parameters by varying them in various ways so that the malignancy score M and size S output from the NN unit 31 are as close as possible to the correct malignancy score LM and correct size LS, respectively. The learning unit 111 performs this computation parameter adjustment process for each of the multiple training images included in the training data 121, thereby optimizing the computation parameters of the NN unit 31 and constructing the neural network in the NN unit 31.

[0023] More specifically, the learning unit 111 performs multitask learning using the loss function E expressed by equation (1) below. The loss function E is a function for evaluating the estimation error by the classifier 30. The loss function E is expressed by the estimation error of the first piece of information, the malignancy degree M, and the estimation error of the second piece of information, the size S. The estimation error of the malignancy degree M is calculated by squaring the difference (M-LM) between the malignancy degree M output from the NN unit 31 for the input of a training image and the correct malignancy degree LM attached to that training image. Similarly, the estimation error of the size S is calculated by squaring the difference (S-LS) between the size S output from the NN unit 31 for the input of a training image and the correct size LS attached to that training image. The learning unit 111 calculates the loss function E by adding these two estimation errors weighted by the hyperparameter α. E = α×(M-LM)^2+(1-α)×(S-LS)^2 …(1)

[0024] In equation (1) above, the hyperparameter α is a parameter that indicates the weight of the estimation error of malignancy M and the estimation error of size S in the loss function E. The hyperparameter α is an external variable that can be freely set by the user within the range of 0 to 1. By changing the hyperparameter α, the weights of the two estimation errors can be adjusted. Specifically, increasing the hyperparameter α, i.e., approaching 1, increases the weight of the estimation error of malignancy M in equation (1) above, thus improving the estimation accuracy of malignancy M by the NN unit 31. Conversely, decreasing the hyperparameter α, i.e., approaching 0, increases the weight of the estimation error of size S in equation (1) above, thus improving the estimation accuracy of size S by the NN unit 31.

[0025] The hyperparameter α is set to one of several different values ​​within the range of 0 to 1. For each of the multiple variations in hyperparameter α, the learning unit 111 calculates a loss function E from the malignancy M and size S, which are the outputs of the NN unit 31. The learning unit 111 then updates the calculation parameters of the NN unit 31 so that the loss function E approaches 0 as close to 0 as possible, and adopts the calculation parameters that bring the loss function E closest to 0 as the calculation parameters of the NN unit 31 for the set hyperparameter α. In this way, the learning unit 111 generates multiple candidate classifiers 30 by performing multitask learning for each of the multiple variations in hyperparameter α.

[0026] Returning to Figure 1, the accuracy calculation unit 112 calculates the discrimination accuracy of the classifier 30, whose calculation parameters for the NN unit 31 have been learned by the learning unit 111. Here, the discrimination accuracy of the classifier 30 is a value that represents the degree to which the classifier 30 can correctly distinguish whether a lesion captured in a medical image is benign or malignant. As an example, the accuracy calculation unit 112 calculates the diagnostic accuracy P(α), which is expressed by the following equation (2), as the discrimination accuracy. In the following equation (2), the number of correct benign predictions is the number of benign cases, i.e., lesions that are benign, that were correctly identified as benign, and the number of correct malignant predictions is the number of malignant cases, i.e., lesions that are malignant, that were correctly identified as malignant. Accuracy P(α) = (Estimated number of correct benign cases + Estimated number of correct malignant cases) / (Number of data points) …(2)

[0027] The accuracy calculation unit 112 calculates the discrimination accuracy of the classifier 30 using the evaluation data 122. Here, the evaluation data 122 is a dataset used by the accuracy calculation unit 112 to evaluate the discrimination accuracy of the classifier 30. The evaluation data 122 includes multiple evaluation input images (hereinafter referred to as "evaluation images"). Each of the multiple evaluation images is pre-labeled with the correct malignancy grade LM and correct size LS of the captured lesion as training labels, similar to the training images. Note that all or part of the multiple evaluation images in the evaluation data 122 may be the same as the training images. Alternatively, machine learning by the learning unit 111 and the calculation of discrimination accuracy by the accuracy calculation unit 112 may be performed using only the training data 121 by cross-validation.

[0028] The accuracy calculation unit 112 inputs each evaluation image included in the evaluation data 122 to the classifier 30. The accuracy calculation unit 112 then compares the first information indicating benign or malignant, output from the classifier 30 for each input evaluation image, with the correct malignancy score LM for each evaluation image. For evaluation images with a correct malignancy score LM of malignant (1), the accuracy calculation unit 112 counts the number of evaluation images with a first information of malignant (1), i.e., the number of images correctly identified as malignant, as the "estimated number of correct malignancies". Similarly, for evaluation images with a correct malignancy score LM of benign (0), the accuracy calculation unit 112 counts the number of evaluation images with a first information of benign (0), i.e., the number of images correctly identified as benign, as the "estimated number of correct benignancies". The accuracy calculation unit 112 then calculates the accuracy rate P(α) of equation (2) above by dividing the sum of the estimated number of malignant and benign cases by the number of evaluation images (number of data) input to the classifier 30. The accuracy calculation unit 112 calculates this accuracy rate P(α) for each of the multiple candidates of the classifier 30 that are generated by the learning unit 111 performing multitask learning by changing the hyperparameter α used in multitask learning in multiple ways.

[0029] Returning to Figure 1, the classifier determination unit 113 selects a candidate from among the multiple candidates for classifier 30 generated by the learning unit 111 whose accuracy P(α) calculated by the accuracy calculation unit 112 satisfies a predetermined standard as classifier 30. Specifically, for each of the accuracy P(α) calculated with multiple hyperparameters α, the classifier determination unit 113 calculates the difference D between it and the accuracy P(1) of classifier 30 when hyperparameter α is 1, using equation (3) below. The classifier determination unit 113 then determines that a candidate whose calculated difference D is less than or equal to the standard value DS is a candidate whose accuracy P(α) satisfies a predetermined standard, and selects that candidate as classifier 30. D = P(1) - P(α) …(3)

[0030] Here, when the hyperparameter α is 1, it corresponds to the case where the weight of the estimation error of size S in the loss function E is 0, that is, the case where the weight of the estimation error of size S is smallest. Therefore, the classifier determination unit 113 calculates the difference D between each of the accuracy rates P(α) of multiple candidate classifiers 30 generated by varying the hyperparameter α in multiple ways and the accuracy rate P(1) of the candidate classifier 30 generated by multitask learning where the weight of the estimation error of size S in the loss function E is 0. The classifier determination unit 113 then determines that candidates whose difference D is less than or equal to the reference value DS are candidates that satisfy a predetermined criterion. The reference value DS is set to a very small value because the purpose is to detect cases that are not equivalent to the case where the hyperparameter α=1.

[0031] Here, the hyperparameter α corresponds to the weight of the estimation error of the malignancy grade M in the loss function E. Typically, the accuracy P(1) is highest when the hyperparameter α is 1, and the accuracy P(α) decreases as the hyperparameter α decreases from 1. Therefore, the criterion that the difference D is less than or equal to the reference value DS corresponds to the degree of decrease in discrimination accuracy from when the hyperparameter α is 1 being small and remaining within a predetermined range. On the other hand, decreasing the hyperparameter α from 1 improves the estimation accuracy of size S in multi-task learning, and it is thought that features related to size S are being extracted and utilized more effectively.

[0032] The classifier determination unit 113 determines the final classifier 30 in such a way that the estimation accuracy of size S is as high as possible while minimizing the decrease in the accuracy of the classifier 30 in distinguishing between benign and malignant lesions. To this end, the classifier determination unit 113 determines the candidate generated with the hyperparameter α that has the largest weight of the estimation error of size S in the loss function E, i.e., the hyperparameter α from 1 to the smallest, among the candidates whose accuracy rate P(α) satisfies a predetermined criterion, as the classifier 30.

[0033] The reason for increasing the accuracy of size S estimation during the learning phase of the classifier 30 is to stabilize (regularize) the classifier 30. Here, stabilization (regularization) means that the learning result does not fall into a local optimum but is optimized as a global optimum. For example, consider a case where there are extremely few small-sized malignant cases as training data 121. When training the classifier 30, the computational parameters in the NN unit 31 are learned so that the loss function E is as small as possible. In other words, since the classifier 30 is learned to improve its overall performance, data of small-sized malignant cases with a small sample size are more likely to be excluded. If the two features, malignancy degree M and size S, are independent, it is desirable to divide the training data 121 and generate multiple classifiers 30 separately. However, if there is some correlation between malignancy degree M and size S, it is preferable to train a single classifier 30 within a large dataset. In particular, deep learning can acquire features that humans cannot recognize. Therefore, by incorporating size S information into the learning process, the classifier 30 is expected to actively utilize features related to size S. This is expected to ensure that data from small-sized malignant cases with a small sample size are not excluded but actively utilized. By making full use of size information in this way, the classifier 30 can be stabilized (regularized). Taking the above into consideration, the classifier determination unit 113 determines the classifier 30 as the final classifier 30, which has been multi-task trained to maximize the estimation accuracy of size S while ensuring that the accuracy of distinguishing between benign and malignant lesions does not decrease significantly.

[0034] Next, with reference to Figure 4, the flow of the classifier generation process performed by the classifier device 100 during the learning phase will be explained. The classifier generation process shown in Figure 4 starts when the operation unit 13 receives a start command from the user. The classifier generation process shown in Figure 4 is an example of a classifier generation method.

[0035] When the classifier generation process is started, the control unit 11 prepares the training data 121 and evaluation data 122, and initializes the hyperparameter α of the loss function E to 1 (step S101). Next, the control unit 11 selects one of the multiple training images included in the training data 121. Then, the control unit 11 uses the classifier 30 to infer the malignancy grade M and size S of the lesion captured in the selected training image (step S102). Specifically, the control unit 11 inputs the training image into the classifier 30 and obtains the malignancy grade M and size S output from the classifier 30. Having obtained the malignancy grade M and size S, the control unit 11 calculates the loss function E according to equation (1) above using the obtained malignancy grade M and size S, and the correct malignancy grade LM and correct size LS attached to the training image (step S103). Then, the control unit 11 updates the calculation parameters of the classifier 30 so that the loss function E approaches 0 (step S104).

[0036] When the computation parameters are updated, the control unit 11 determines whether or not the processing in steps S102 to S104 has been performed using a predetermined number of training images (step S105). Here, the predetermined number of training images may be all the training images included in the training data 121, or it may be only a portion of them, as long as there are enough to perform machine learning. If the processing has not been completed using the predetermined number of training images (step S105; NO), the control unit 11 returns to step S102. The control unit 11 then selects one unselected training image from among the multiple training images included in the training data 121, and repeats the processing in steps S102 to S105 for the newly selected training image. As a result, the control unit 11 performs multi-task learning using each of the multiple training images included in the training data 121, when the hyperparameter α is set to an initial value of 1. The control unit 11 then updates the computation parameters of the classifier 30 so that the loss function E approaches 0, and generates candidate classifiers for the classifier 30.

[0037] After that, once processing is completed using a predetermined number of training images (step S105; YES), the control unit 11 functions as an accuracy calculation unit 112 and calculates the recognition accuracy of the classifier 30 (step S106). Specifically, the control unit 11 calculates the accuracy rate P(α) using equation (2) above. After calculating the recognition accuracy, the control unit 11 determines whether the difference D between the calculated recognition accuracy and the recognition accuracy of the classifier 30 when α=1 is greater than the reference value DS (step S107). Note that if the hyperparameter α is initially set to 1, the difference D is 0, so step S107 is determined to be NO.

[0038] If the difference D is less than or equal to the reference value DS (step S107; NO), the control unit 11 sets a new hyperparameter α by reducing the current hyperparameter α by a certain percentage X (step S108). The percentage X is pre-set to a value such as 0.1 or 0.05. Once the new hyperparameter α is set, the control unit 11 returns to step S102 and executes the processes in steps S102 to S107 with the new hyperparameter α. As a result, the control unit 11 generates candidates for the classifier 30 by performing multitask learning with the new hyperparameter α and calculates the accuracy P(α) of the generated candidates.

[0039] The control unit 11 repeats steps S102 to S107, gradually decreasing the hyperparameter α until the difference D between the newly calculated accuracy P(α) and the accuracy P(1) at α=1 becomes greater than the reference value DS. Finally, when the difference D becomes greater than the reference value DS (step S107; YES), the control unit 11 terminates the update of the hyperparameter α. Then, the control unit 11 functions as a classifier determination unit 113 and determines the candidate with the smallest hyperparameter α among multiple candidates whose difference D is less than or equal to the reference value DS as the classifier 30 (step S109). With this, the classifier generation process shown in Figure 4 is completed.

[0040] Returning to Figure 1, the second phase will be described. The inference phase is a phase in which the classifier 30 generated in the learning phase is used to identify whether an unknown lesion in an unknown medical image is benign or malignant. In the inference phase, the image receiving unit 114 receives input of an unknown medical image to be identified. Here, an unknown medical image is an image in which a lesion of unknown benign or malignant nature has been captured. The image receiving unit 114 accepts the designation of an unknown medical image to be identified from among a plurality of medical images pre-stored in the storage unit 12, according to the user's instructions from the operation unit 13. Alternatively, the image receiving unit 114 may receive the unknown medical image to be identified from an external source via the communication unit 15.

[0041] The identification processing unit 115 uses the classifier 30 determined by the classifier determination unit 113 to identify whether the lesion in the unknown medical image received by the image receiving unit 114 is good or bad. Specifically, as shown in Figure 5, the identification processing unit 115 inputs the unknown medical image received by the image receiving unit 114 into the classifier 30.

[0042] In the classifier 30, the NN unit 31 calculates the malignancy degree M and size S of the lesion captured in the input unknown medical image using computation parameters for which multitasking learning has been performed by the learning unit 111. The benign / malignant discrimination unit 32 compares the malignancy degree M calculated by the NN unit 31 with a cutoff value to determine whether the lesion is benign or malignant. The classifier 30 outputs first information indicating whether it is benign or malignant as determined by the benign / malignant discrimination unit 32, and second information indicating the size S calculated by the NN unit 31. The identification processing unit 115 then identifies whether the lesion is benign or malignant based on the first information from the first and second information output from the classifier 30 for the unknown medical image. In this case, the identification processing unit 115 does not use the second information, i.e., the size S information, output from the classifier 30 during the inference phase.

[0043] The result output unit 116 outputs the identification result from the identification processing unit 115. Specifically, the result output unit 116 displays on the display unit 14 output information indicating whether the lesion in the unknown medical image is benign or malignant, which is the result identified by the identification processing unit 115. Alternatively, the result output unit 116 may output the output information as audio, or output it to an external device via the communication unit 15. This allows the user to confirm the identification result from the identification device 100.

[0044] Next, with reference to Figure 6, the flow of the identification process performed by the identification device 100 in the inference phase will be explained. The identification process shown in Figure 6 starts when the operation unit 13 receives a start command from the user, while the identification device 30 generated by the identification device generation process shown in Figure 4 is stored in the storage unit 12.

[0045] When the identification process is started, the control unit 11 functions as an image receiving unit 114 and receives input of an unknown medical image in which an unknown lesion to be identified is captured (step S301). Next, the control unit 11 functions as an identification processing unit 115 and inputs the unknown medical image to the classifier 30, and obtains the identification result of benign or malignant, indicated by the first information, from the first information and second information output from the classifier 30 (step S302). Then, the control unit 11 functions as a result output unit 116 and outputs output information indicating the acquired identification result (step S303). With this, the identification process shown in Figure 6 is completed. Because the identification process described above uses a classifier 30 generated by multitask learning regarding the benign / malignant nature and size of the lesion in the classifier generation process, it can accurately identify whether the lesion is benign or malignant even if there is a bias in the training data 121 regarding the size of the lesion.

[0046] As described above, the identification device 100 according to Embodiment 1 generates a classifier 30 through multitask learning that, in response to an input medical image, outputs first information indicating whether the lesion captured in the medical image is benign or malignant, and second information indicating the size of the lesion. Using the generated classifier 30, the device identifies whether an unknown lesion captured in an unknown medical image is benign or malignant. Thus, in order to identify whether a lesion is benign or malignant, the identification device 100 according to Embodiment 1 uses a classifier 30 that has undergone multitask learning to output not only first information regarding whether the lesion is benign or malignant, but also second information regarding the size of the lesion, which is other than whether it is benign or malignant. As a result, even if the training images are not collected in a balanced manner regarding the size of the lesion, and there is a bias in the training data 121 regarding the size of the lesion, a stable identification result can be obtained.

[0047] In particular, in clinical practice, many small lesions are benign. Therefore, it is difficult to collect enough training images of small malignant lesions. When there is a bias in the disease distribution within the training data 121, the identification results are more likely to be biased. For example, if the number of malignant cases is smaller than the number of benign cases, the identification results are more likely to be biased towards benign lesions. In contrast, the identification device 100 according to Embodiment 1 uses lesion size information, which is not used in the inference phase, in the learning phase to generate the classifier 30. Therefore, even if there is a large bias in the case distribution of the training data 121, the learning results do not fall into a local optimum but are optimized as a global optimum, thus stabilizing (regularizing) the classifier 30. As a result, stable identification results can be obtained.

[0048] Figures 7(a) to 7(c) show the results of experimental evaluation of the classifier 30 according to Embodiment 1 described above. In Figures 7(a) to 7(c), the horizontal axis represents the size of the lesion captured in the medical image. In Figures 7(a) and 7(b), the vertical axes represent the sensitivity and specificity of the classifier 30 evaluated without limiting the type of lesion, respectively. In Figure 7(c), the vertical axis represents the sensitivity of the classifier 30 evaluated only when the lesion is melanoma. Here, sensitivity is the percentage of malignant cases that were correctly identified as malignant, and specificity is the percentage of benign cases that were correctly identified as benign. As an evaluation method, using multiple evaluation images divided by lesion size, the sensitivity or specificity of the classifier 30 generated by multi-task learning as described in Embodiment 1 and a conventional classifier generated without performing multi-task learning were evaluated for each size. In Figures 7(a) to 7(c), the solid lines show the results obtained by the classifier 30 in Embodiment 1, while the dashed lines show the results obtained by the conventional (original) classifier.

[0049] As a result, as shown in Figures 7(a) and (b), when the type of lesion was not limited, there was no significant change in sensitivity and specificity between the classifier 30 of Embodiment 1 and the conventional classifier, regardless of the size of the lesion. On the other hand, for melanoma shown in Figure 7(c), the sensitivity for small size lesions (less than 7 mm) was 0.667 for the conventional classifier, while it was confirmed that the classifier 30 of Embodiment 1 improved to 0.767. Since melanoma is a dangerous malignant tumor, it is especially important to identify it with high accuracy. However, melanoma is rare, and there are few cases regardless of its size, but there are especially few cases of small-sized melanoma. Therefore, it is difficult to collect small-sized melanoma as training data 121. In contrast, it was confirmed that the classifier 30 of Embodiment 1 can identify small-sized melanoma, which have few cases, with higher sensitivity than conventional methods without reducing the overall identification accuracy. Based on the experimental results described above, it can be seen that the classifier 30 generated by multi-task learning is effective in obtaining stable classification results even when there is a bias in the case distribution in the training data 121.

[0050] Next, Embodiment 2 will be described. The same configuration and functions as in Embodiment 1 will not be described. As shown in Figure 1, the identification device 100 according to Embodiment 1 had both a learning phase function and an inference phase function. In contrast, the identification device 100 according to Embodiment 2 does not have a learning phase function, and a separate device, the classifier generation device 200, has the learning phase function.

[0051] Specifically, as shown in Figure 8, the classifier generation device 200 according to Embodiment 2 comprises a control unit 21, a storage unit 22, an operation unit 23, a display unit 24, and a communication unit 25. The hardware configuration of each of these units is the same as that of the control unit 11, storage unit 12, operation unit 13, display unit 14, and communication unit 15 in the classifier device 100, so a detailed explanation is omitted. The control unit 21 includes a learning unit 111, an accuracy calculation unit 112, and a classifier determination unit 113 as functions for the learning phase. In the control unit 21, the CPU functions by reading the program stored in ROM into RAM and executing the program to control these units. The storage unit 22 stores training data 121 and evaluation data 122 as data used in the learning phase.

[0052] Each function of the control unit 21 is the same as that of the learning unit 111, accuracy calculation unit 112, and classifier determination unit 113 provided in the identification device 100 in Embodiment 1. Specifically, the control unit 21 executes the classifier generation process shown in Figure 4 using the functions of the learning unit 111, accuracy calculation unit 112, and classifier determination unit 113. As a result, the control unit 21 generates a classifier 30 on which multitask learning has been performed.

[0053] On the other hand, the identification device 100 according to Embodiment 2 is configured as shown in Figure 9. In the identification device 100 according to Embodiment 2, the control unit 11 does not have a learning phase function, but as an inference phase function, it includes an image receiving unit 114, an identification processing unit 115, and a result output unit 116. The functions of each of these units are the same as in Embodiment 1. The identification device 100 acquires the classifier 30 generated by the classifier generation device 200 from the classifier generation device 200, for example, through communication via the communication unit 15, and stores it in the storage unit 12. The control unit 11, using the functions of the image receiving unit 114, the identification processing unit 115, and the result output unit 116, uses the classifier 30 acquired from the classifier generation device 200 to execute the identification process shown in Figure 6. As a result, the control unit 11 identifies whether the lesion captured in the medical image is benign or malignant. Thus, in Embodiment 2, since the functions of the learning phase and the inference phase are separated into different devices, more flexible operation becomes possible, such as utilizing the classifier 30 generated by the classifier generation device 200 in multiple classifier devices 100.

[0054] Although embodiments of the present invention have been described above, these embodiments are merely examples, and the scope of application of the present invention is not limited thereto. In other words, Embodiment 1 of the present invention can be applied in various ways, and all embodiments fall within the scope of the present invention.

[0055] For example, in the above embodiment, the accuracy calculation unit 112 calculated the diagnostic accuracy P(α) expressed by equation (2) as the discrimination accuracy. However, the accuracy calculation unit 112 may calculate other index values ​​as discrimination accuracy, not just the diagnostic accuracy. For example, the accuracy calculation unit 112 may calculate the F1 score, which is the harmonic mean of sensitivity and precision, as the discrimination accuracy. Here, sensitivity is the proportion of malignant cases that were correctly identified as malignant. Sensitivity is obtained by calculating the proportion of the first information of the classifier 30 that is malignant (1) out of the number of input evaluation images in which the correct malignancy grade LM is malignant (1). Precision is the proportion of data identified as malignant that are actually malignant. Precision is obtained by calculating the proportion of evaluation images in which the correct malignancy grade LM is malignant (1) out of the number of evaluation images in which the first information of the classifier 30 is malignant (1). In addition, in the above embodiment, the learning unit 111 evaluated the estimation error between the first and second information using the loss function E expressed by equation (1). However, the learning unit 111 is not limited to this; it may also evaluate the estimation error between the first and second information using the error due to cross-entropy.

[0056] In the above embodiment, the second information used in multi-task learning, other than whether the object to be identified is good or bad, was information about the size of the lesion. However, the second information may be information other than size, for example, information indicating the age, body part, race, etc., of the object to be identified. Here, the age of the object to be identified means the age of the patient who has the lesion that is the object to be identified. Generally, just as there are few small malignant cases, there are few malignant cases at a young age. Therefore, by performing multi-task learning using age information as the second information, stable identification results can be obtained even if there is a bias in the training data 121 with respect to age, as in the above embodiment. Furthermore, the body part of the object to be identified means the body part where the lesion that is the object to be identified is located (e.g., face, palms and soles, mucous membranes, etc.). The race of the object to be identified means the race of the patient who has the lesion that is the object to be identified (e.g., Caucasian, Black, Asian, etc.). Since it can be difficult to collect training images in a balanced manner depending on the body part and race, by performing multi-task learning using this information as the second information, stable identification results can be obtained even if there is a bias in the training data 121. Thus, by using information that is prone to bias in the training data 121 as secondary information, it is possible to obtain stable classification results.

[0057] In the above embodiment, the identification device 100 identified whether the lesion was benign or malignant. However, the identification device 100 is not limited to this and may also be a device that identifies a referral recommendation for the identified subject. Here, a referral recommendation means advising a patient with the identified lesion to be referred to another hospital. For example, it is conceivable that a small clinic would refer a patient to a larger hospital capable of more specialized examinations. In the above embodiment, each training image was labeled with a correct malignancy score LM and a correct size LS indicating malignancy (1) or benignity (0) as a teacher label. However, when the identification device 100 identifies a referral recommendation, instead of the correct malignancy score LM, information indicating whether a referral recommendation is necessary (1) or unnecessary (0) is attached to each training image as a teacher label. The learning unit 111 performs multitask learning using such training data 121. As a result, the learning unit 111 generates a classifier 30 that, in response to an input medical image, outputs first information indicating whether or not a referral recommendation is necessary for the lesion captured in the medical image (presence or absence of referral recommendation) and second information regarding matters other than whether the lesion is benign or malignant. Alternatively, the training label remains the correct malignancy grade LM, but the cutoff value used by the benign / malignant discrimination unit 32 can be set higher than when identifying whether the lesion site is benign or malignant, thereby allowing the classifier 100 to be used as a referral recommendation classifier.

[0058] In the above embodiment, the identification device 100 was a medical diagnostic support device that identified whether a lesion captured in a medical image was benign or malignant. However, the identification device 100 is not limited to being a medical diagnostic support device. For example, the identification device 100 may be an inspection device that receives input of inspection images of structures such as buildings, roads, and bridges, and identifies whether or not there is an abnormality in the structure based on cracks, surface shape, etc., of the structure captured in the inspection image. In this case, the object of identification corresponds to the structure captured in the inspection image, not the lesion captured in the medical image. Furthermore, the goodness or badness of the object of identification corresponds to the presence or absence of an abnormality in the structure, not whether the lesion is benign or malignant.

[0059] Furthermore, the identification device 100 is not limited to outputting binary information as an identification result, such as whether the lesion is benign or malignant, whether a referral is necessary, or whether the building is in good condition (whether there is an abnormality). It may also output information that is more than binary as an identification result. For example, the identification device 100 may use an identifier 30 capable of identifying eight diseases to output an identification result indicating which of the eight diseases the lesion belongs to. Here, the eight diseases are, for example, eight major diseases: melanoma, basal cell carcinoma, other malignant diseases, pigmented nevus, seborrheic keratosis, dermatofibroma, hemangioma, and other benign diseases. In this case, the NN unit 31 outputs a probability value indicating the probability that the lesion captured in the input medical image corresponds to each of the eight diseases, instead of the malignancy grade M in the above embodiment. For example, if the input medical image is a melanoma image, the learning unit 111 trains the NN unit 31 so that the probability value for melanoma approaches 1, and the probability values ​​for the other seven diseases approach 0. The benign / malignant discrimination unit 32 then determines that the disease with the largest probability value among the eight probability values ​​output from the NN unit 31 is the disease corresponding to the lesion captured in the input medical image. For example, if the probability value for melanoma is the largest, the benign / malignant discrimination unit 32 determines that the lesion is melanoma. The classifier 30 outputs this discrimination result from the benign / malignant discrimination unit 32 as first information. In the identification device 100, the identification processing unit 115 uses the first information output from the classifier 30 to identify which of the eight diseases the lesion captured in the input medical image corresponds to, and the result output unit 116 outputs the identification result.

[0060] In the above embodiment, the control units 11 and 21 functioned as the respective parts shown in Figures 1, 8, or 9 by having the CPU execute a program stored in the ROM or storage unit 12. However, the control units 11 and 21 may be dedicated hardware. Dedicated hardware includes, for example, a single circuit, a composite circuit, a programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof. If the control units 11 and 21 are dedicated hardware, each function of each part may be realized by separate hardware, or the functions of each part may be realized together by a single piece of hardware. Furthermore, some of the functions of each part may be realized by dedicated hardware, and other parts by software or firmware. In this way, the control unit 110 can realize the above-mentioned functions by hardware, software, firmware, or a combination thereof.

[0061] It is also possible to make an existing computer, such as a personal computer or cloud server, function as the identification device 100 or identification device 200 by applying the program that defines the operation of the identification device 100 or identification device 200 described above to the computer. Furthermore, the distribution method of such a program is arbitrary; for example, it may be distributed by storing it on a computer-readable recording medium such as a CD-ROM (Compact Disk ROM), DVD (Digital Versatile Disk), MO (Magneto Optical Disk), or memory card, or it may be distributed via a communication network such as the Internet.

[0062] Although preferred embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above, and various modifications and substitutions can be made to the embodiments described above without departing from the scope of the claims. [Explanation of symbols]

[0063] 11...Control unit, 30...Identifier, 115...Identification processing unit, 100...Identification device

Claims

1. An identification processing unit that identifies the object to be identified in a medical image using a classifier that has undergone multitask learning to output first information regarding the quality of the object to be identified in the medical image or whether it should be referred, and second information regarding at least one of the size, age, body part, and race of the object to be identified. A medical diagnostic support device characterized by being equipped with the following.

2. The identification processing unit identifies the quality of the object to be identified in the medical image using the classifier which has undergone multitask learning to output the first information and the second information regarding the quality of the object to be identified in the medical image. The medical diagnostic support device according to feature 1.

3. The identification processing unit identifies the referral recommendation for the identified target in the medical image using the classifier on which the multitask learning has been performed to output the first information and the second information regarding the referral recommendation for the identified target in the medical image. The medical diagnostic support device according to feature 1.

4. A learning unit that generates multiple candidates for the classifier by performing the multitask learning by changing the hyperparameters used in the multitask learning in multiple ways, A precision calculation unit calculates the identification accuracy of each of the multiple candidates generated by the learning unit, The system further comprises a classifier determination unit that, among the plurality of candidates, determines a candidate whose classification accuracy calculated by the accuracy calculation unit satisfies a predetermined standard as the classifier, The identification processing unit uses the identifier determined by the identifier determination unit to identify the object to be identified in the medical image. A medical diagnostic support device according to any one of claims 1 to 3.

5. The learning unit performs the multi-task learning using a loss function represented by the estimation error of the first information and the estimation error of the second information. The hyperparameters are parameters that indicate the weights of the estimation error of the first information and the estimation error of the second information in the loss function. The medical diagnostic support device according to feature 4.

6. The predetermined criterion is satisfied when the difference between the identification accuracy calculated by the accuracy calculation unit and the identification accuracy of the candidate generated with the hyperparameter that has the smallest weight of the estimation error of the second information in the loss function among the plurality of candidates is less than or equal to the criterion value. The classifier determination unit determines, as the classifier, the candidate generated with the hyperparameter that has the largest weight of the estimation error of the second information in the loss function among the candidates whose classification accuracy satisfies the predetermined criteria. The medical diagnostic support device according to feature 5.

7. When the identification processing unit identifies the object to be identified in the medical image, it does not use the second information output from the classifier in response to the input of the medical image. A medical diagnostic support device according to any one of claims 1 to 3.

8. A device for generating an identifier that identifies an object to be identified in a medical image, A learning unit generates a classifier, which, in response to the input of the medical image, outputs first information regarding the quality of the object to be identified in the medical image or whether a referral should be recommended, and second information regarding at least one of the size, age, body part, and race of the object to be identified, through multitask learning. A discriminator generating device characterized by comprising the following features.

9. A method for generating a classifier that identifies an object to be identified in a medical image, The classifier is generated by multitask learning, which, in response to the input of the medical image, outputs first information regarding the quality of the object to be identified in the medical image or whether a referral should be recommended, and second information regarding at least one of the size, age, body part, and race of the object to be identified. A method for generating a classifier characterized by the following features.

10. A program that generates a classifier to identify objects in medical images, Computers, A learning unit generates a classifier, which, in response to the input of the medical image, outputs first information regarding the quality of the object to be identified in the medical image or whether a referral should be recommended, and second information regarding at least one of the size, age, body part, and race of the object to be identified, through multitask learning. A program designed to function as such.

Images