Machine learning device, machine learning method, machine learning program, and inference device

Abstract

To provide a machine learning device that improves accuracy of machine learning using a label.SOLUTION: A machine learning device according to an embodiment includes a first learning unit. The first learning unit accepts input of first processing data and a first processing label assigned to the first processing data by a first user to use a first calibration model that outputs calibration data for calibrating individual characteristics of label assigned by the first user, and learns a target model on the basis of at least processing data and calibration data, or a calibration label with individual characteristics calibrated by using the calibration data.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The embodiments disclosed in this specification and the drawings relate to a machine learning device, a machine learning method, a machine learning program, and an inference device. [Background technology]

[0002] Machine learning requires a large number of labels. For example, in machine learning using medical data such as medical images, labels are manually assigned to the medical data. The large number of labels may be prepared by one or a small number of people, or by many people. If there are individual differences in the assignment of labels, the reliability of the labels will be low, and the accuracy of machine learning will be poor. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] US Patent Application Publication No. 2020 / 0160509 Summary of the Invention [Problem to be solved by the invention]

[0004] One of the problems to be solved by the embodiments disclosed in this specification and the drawings is to improve the accuracy of machine learning using labels. However, the problems to be solved by the embodiments disclosed in this specification and the drawings are not limited to the above problems. Problems corresponding to the effects of each configuration shown in the embodiments described later can also be positioned as other problems. [Means for solving the problem]

[0005] A machine learning device according to an embodiment includes a first learning unit that uses a first calibration model that inputs first processed data and a first processed label assigned to the first processed data by a first user and outputs calibration data related to calibration of personal characteristics of the label assigned by the first user, and learns a target model based on at least the first processed data and the calibration data or a calibrated label whose personal characteristics have been calibrated using the calibration data. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a machine learning device according to this embodiment. [Diagram 2] FIG. 2 is a diagram illustrating individual differences in labeling. [Diagram 3] FIG. 3 is a diagram illustrating a flow of an example of a learning process of the calibration model according to the first embodiment. [Figure 4] FIG. 4 is a diagram illustrating a flow of an example of a learning process of the calibration model according to the first embodiment. [Diagram 5] FIG. 5 is a diagram illustrating a flow of an example of a learning process of a target model according to the first embodiment. [Figure 6] FIG. 6 is a diagram illustrating a flow of an example of the learning process of the target model according to the first embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of the positional relationship between the application label and the calibration label. [Figure 8] FIG. 8 is a diagram illustrating a flow of an example of a learning process of a calibration model according to the second embodiment. [Figure 9] FIG. 9 is a diagram illustrating a flow of an example of a learning process of a calibration model according to the second embodiment. [Figure 10] FIG. 10 is a diagram illustrating a flow of an example of a learning process of a target model according to the second embodiment. [Figure 11] FIG. 11 is a diagram illustrating a flow of an example of the learning process of the target model according to the second embodiment. [Figure 12] FIG. 12 is a diagram illustrating a flow of an example of a learning process of a calibration model according to the third embodiment. [Figure 13]FIG. 13 is a diagram illustrating a flow of an example of a learning process of a calibration model according to the third embodiment. [Figure 14] FIG. 14 is a diagram illustrating a flow of an example of a learning process of a target model according to the third embodiment. [Figure 15] FIG. 15 is a diagram illustrating a flow of an example of the learning process of the target model according to the third embodiment. [Figure 16] FIG. 16 is a diagram illustrating an example of processing of the machine learning device according to the application example 1. [Figure 17] FIG. 17 is a diagram illustrating an example of calibration of personal characteristics of labeling of a first user and a second user. [Figure 18] FIG. 18 is a diagram illustrating a configuration example of a blend network according to application example 3. [Figure 19] FIG. 19 is a diagram showing an example of personal characteristics of labeling of a first user and a second user in relation to an imaging section detection task. [Figure 20] FIG. 20 is a diagram showing an example of personal characteristics of the labels of the first user and the second user in relation to the abnormal / normal judgment task. [Figure 21] FIG. 21 is a diagram illustrating an example of the configuration of an inference device. [Figure 22] FIG. 22 is a diagram showing the input / output relationship of the target model. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, embodiments of a machine learning device, a machine learning method, a machine learning program, and an inference device will be described in detail with reference to the drawings.

[0008] The machine learning device according to the present embodiment is a computer that uses labels to learn a machine learning model. The label means information indicating a correct answer that is assigned to input data related to machine learning. The label may take various forms depending on the task of the machine learning model to be learned. The task of the machine learning model may be any task such as a classification problem, a regression problem, an object detection problem, an image generation problem, etc. The machine learning model according to the present embodiment can be any of a neural network, a support vector machine, a random forest, etc. The machine learning model to be learned according to the present embodiment is referred to as a target model. The inference device according to the present embodiment is a computer that performs inference using a target model learned by the machine learning device.

[0009] The machine learning device according to the present embodiment uses a machine learning model for calibrating personal characteristics of label assignment for each user who assigns a label. Hereinafter, this machine learning model will be referred to as a calibration model. Personal characteristics of label assignment refer to personal habits of users in assigning labels or individual differences in label assignment between users. The machine learning device according to the present embodiment uses the calibration model to perform machine learning of a target model while calibrating personal characteristics of the labels assigned by users.

[0010] Fig. 1 is a diagram showing an example of the configuration of a machine learning device 100 according to this embodiment. As shown in Fig. 1, the machine learning device 100 has a processing circuit 1, a storage device 3, a display device 5, an input interface 7, and a communication interface 9. Data communication between the processing circuit 1, the storage device 3, the display device 5, the input interface 7, and the communication interface 9 is performed via a bus.

[0011] The processing circuit 1 has a processor such as a CPU (Central Processing Unit). The processor starts various programs installed in a storage device 3 or the like, thereby realizing an acquisition function 11, a labeling function 12, a calibration model learning function 13, an object model learning function 14, a display control function 15, and the like. Each of the functions 11 to 15 is not limited to being realized by a single processing circuit. A processing circuit may be configured by combining a plurality of independent processors, and each processor may execute a program to realize each of the functions 11 to 15.

[0012] By implementing the acquisition function 11, the processing circuit 1 acquires data to be labeled. The acquired data is divided into data for training a target model and data for training a calibration model. Data used to train a target model is collectively called target training data, and data used to train a calibration model is collectively called calibration training data. The type of data to be labeled is not particularly limited. The data to be labeled may be any data with one or more dimensions. In other words, the data to be labeled may be any of image data, character data, and waveform data, or a combination of these.

[0013] By implementing the label assignment function 12, the processing circuit 1 assigns a label to the data acquired by the acquisition function 11. "Assignment" means superimposing a label on data or associating a label with data. A label assigned to target learning data is called a target learning label, and a label assigned to calibrated learning data is called a calibrated learning label. The label assignment task is performed by a user assigning a label to data via the input interface 7. The processing circuit 1 assigns a label to data according to instructions from the user via the input interface 7.

[0014] By implementing the calibration model learning function 13, the processing circuit 1 learns a calibration model for calibrating personal characteristics of labeling by a user based on the calibration learning data and the calibrated learning labels assigned by the user to the calibration learning data. The calibration model is trained to input the calibration learning data and the calibrated learning labels and output the calibration data. The calibration data is data related to the calibration of personal characteristics of labeling by a user. The input and output formats of the calibration model, or the calibration data, may vary. The user means a person who assigns labels.

[0015] By implementing the target model learning function 14, the processing circuit 1 learns a target model based on the target learning data, a target learning label assigned to the target learning data by a user, and a calibration model for the user. More specifically, the processing circuit 1 learns a target model based on the target learning data and the calibration data or a calibration label in which personal characteristics are calibrated using the calibration data. The target model is a machine learning model trained to input input data and output inference data corresponding to the input data. The inference data is data obtained by subjecting the input data to a conversion according to the task of the target model. There are various methods for learning the target model depending on the input / output format of the calibration model, i.e., the format of the calibration data.

[0016] By implementing the display control function 15, the processing circuit 1 displays various information on the display device 5. For example, the processing circuit 1 displays target learning data, calibrated learning data, target learning labels, calibrated learning labels, and the like.

[0017] The storage device 3 is a storage device such as a ROM (Read Only Memory), a RAM (Random Access Memory), an HDD (Hard Disk Drive), an SSD (Solid State Drive), an integrated circuit storage device, etc. In addition to the above storage devices, the storage device 3 may be a portable storage medium such as a CD (Compact Disc), a DVD (Digital Versatile Disc), or a flash memory, or a drive device that reads and writes various information between the storage device and a semiconductor memory element, etc.

[0018] The display device 5 displays various data according to the display control function 15 of the processing circuit 1. A liquid crystal display (LCD), a cathode ray tube (CRT) display, an organic electroluminescence display (OELD), a plasma display, or any other display may be appropriately used as the display device 5. The display device 5 may also be a projector.

[0019] The input interface 7 accepts various input operations from a user, converts the accepted input operations into electrical signals, and outputs the electrical signals to the processing circuit 1. Specifically, the input interface 7 is connected to input devices such as a mouse, a keyboard, a trackball, a switch, a button, a joystick, a touchpad, and a touch panel display. The input interface 7 outputs electrical signals corresponding to the input operations to the input devices to the processing circuit 1. The input devices connected to the input interface 7 may be input devices provided in other computers connected via a network or the like. The input interface 7 may be a voice recognition device that converts voice signals collected by a microphone into instruction signals.

[0020] The communication interface 9 is an interface that connects to various computers via a LAN (Local Area Network) or the like. A LAN card, a network adapter, a network interface card, or the like is used as the communication interface 9. For example, the communication interface 9 performs data communication with a generating device and a storing device for the target learning data and the calibration learning data.

[0021] Next, the machine learning device 100 according to this embodiment will be described in detail.

[0022] First, the personal characteristics of labeling by a user will be described. In the following description, as an example, it is assumed that data used for labeling is medical data generated by a medical device. The medical device may be a single modality device such as an X-ray computed tomography device (X-ray CT device), a magnetic resonance imaging device (MRI device), an X-ray diagnostic device, a PET (Positron Emission Tomography) device, a SPECT (Single Photon Emission CT) device, an ultrasound diagnostic device, an optical coherence tomography device (fundus camera), and an optical ultrasound diagnostic device, or a multi-modality device such as a PET / CT device, a SPECT / CT device, a PET / MRI device, and a SPECT / MRI device.

[0023] As described above, the target model according to the present embodiment may execute any task as long as it is a machine learning model that is trained using labels. As an example, in the following description, the task of the target model is a measurement voxel position detection problem in which a medical image is input and the position of a measurement voxel, which is a data collection region of MR spectroscopy by a magnetic resonance imaging apparatus, is output. The medical image is an example of input data to the target model, and the position of the measurement voxel is an example of predicted data output from the target model. The data used for labeling is an MR image in which the measurement voxel of MR spectroscopy is set. The label is a mark indicating the position of the measurement voxel.

[0024] MR spectroscopy includes a single voxel method in which data is collected from one measurement voxel, and a multi-voxel method in which data is collected from multiple measurement voxels. This embodiment can be applied to either method, but hereinafter, MR spectroscopy is assumed to be the single voxel method.

[0025] MR spectroscopy using the single voxel method is performed according to the following procedure. First, a pre-scan called local shimming is performed on a measurement area of ​​about 10 cm square. The magnetic field distribution in the measurement area is obtained by local shimming. The data collection time for local shimming is often less than one minute. After local shimming is performed, the measurement voxels for MR spectroscopy are set. The measurement voxels are set in a relatively small area of ​​about 1 to 2 cm square. A user such as a medical professional observes the MR image and sets a lesion such as a tumor as the measurement target for MR spectroscopy. For example, the user operates an input device such as a mouse or stylus to set an ROI mark at the position of the measurement target included in the head MR image. The voxel indicated by the ROI mark is set as the measurement voxel. MR spectroscopy is performed on the measurement voxel. The data collection time for MR spectroscopy is often relatively long, about 4 to 5 minutes.

[0026] Setting the measurement voxels is a relatively difficult task due to factors such as the relatively small size of the measurement target. The target model is generated to automate this task. The target model is a machine learning model that inputs an MR image and outputs the position of the measurement voxel. A large number of training samples must be prepared for machine learning of the target model. The training sample is a combination of an input training sample and an output training sample. The input training sample is the target training data input to the target model. The output training sample is a label. In this embodiment, the input training sample is an MR image, and the output training sample is an ROI mark indicating the position of the measurement voxel. The label is manually assigned to the MR image.

[0027] FIG. 2 is a diagram illustrating individual differences in labeling. As shown in FIG. 2, user 1 and user 2 observe head MR images I1 and I2 on which lesions such as tumors are drawn, and manually add ROI marks M11 and M12 to surround the lesions as labels. It is assumed that the head MR images I1 and I2 shown in FIG. 2 are the same image. However, the head MR images I1 and I2 do not need to be the same image, and may be different images. The ROI marks M11 and M12 are set inside the ROI marks M21 and M22 indicating the measurement region of local shimming. The left diagram in FIG. 2 illustrates the ROI mark M11 added by user 1, and the right diagram illustrates the label M12 added by user 2. It is assumed that the positions of the ROI marks M21 and M22 are the same between the left diagram and the right diagram.

[0028] As shown in FIG. 2, even when ROI marks M11 and M12 are added to the same head MR images I1 and I2 by user 1 and user 2, it is assumed that the shapes of the ROI marks M11 and M12, such as the position and size, are different. That is, individual differences in labeling may occur. This individual difference can be said to be due to the individual characteristics of labeling that exist for each user. For example, if the centers of the ROI marks M21 and M22 are the true correct label positions, it can be said that user 1 tends to add the ROI mark M11 to the left, and user 2 tends to add the ROI mark M12 downward. Such a difference in the rough tendency of labeling between users is classified as "bias". In addition, for example, even for the same user 1 or user 2, the shape of the ROI mark M11 or M12 may differ depending on the position or type of the lesion. Such a difference in individual labeling between the same users is classified as "variation". It is assumed that the individual characteristics are a concept including bias and variation. If personal characteristics exist, the ROI marks M11 and M12 will deviate from the expected values, increasing the possibility that the accuracy of machine learning using the ROI marks M11 and M12 will decrease.

[0029] The machine learning device 100 according to the present embodiment calibrates the personal characteristics of each user with a calibration model to perform machine learning of a target model. The processing of the machine learning device 100 is divided into a learning phase of the calibration model and a learning phase of the target model. Combinations of the learning phase of the calibration model and the learning phase of the target model can be divided into various examples according to the input and output of the calibration model. Below, examples of the operation of the machine learning device 100 will be described by dividing them into various examples.

[0030] In the following various embodiments, the calibration learning data used for the machine learning of the calibration model is test data. The test data is the calibration learning data to which a label used for the machine learning of the calibration model is attached. The test data is an MR image, but may be an MR image generated by actually imaging a patient or a phantom with a magnetic resonance imaging device, or may be a pseudo MR image artificially generated by image processing or predictive calculation. In the following embodiments, the target learning data used for learning the target model is real data. The real data is the target learning data to which a label used for the machine learning of the target model is attached. The real data is an MR image generated by actually imaging a patient with a magnetic resonance imaging device. The task of the target model is, as an example, the attachment of a label indicating a measurement voxel of MR spectroscopy. In addition, the user to be calibrated is a first user.

[0031] Example 1 The calibration model according to the first embodiment is a machine learning model that receives input data and an assigned label assigned to the input data and outputs a calibration parameter. The calibration parameter is an example of calibration data output from the calibration model. By implementing the calibration model learning function 13, the processing circuit 1 calculates a calibration parameter for calibrating the personal characteristics of the user's label assignment from the calibrated learning label assigned by the user to the calibrated learning data, and learns a calibration model that receives input data and outputs a calibration parameter corresponding to the input data based on the calibrated learning data, the calibrated learning label, and the calibration parameter. The target model according to the first embodiment is learned based on the input data, the assigned label, and a calibrated label obtained by calibrating the assigned label with the calibration parameter. By implementing the target model learning function 14, the processing circuit 1 applies the calibration model to the target learning data and the target learning label to generate a calibration parameter for the target learning label, applies the calibration parameter to the target learning label to generate a calibrated label, and learns the target model based on the target learning data and the calibrated label. First, a learning method of the calibration model according to the first embodiment will be described.

[0032] Fig. 3 is a diagram illustrating a flow of an example of a learning process of a calibration model according to the first embodiment. Fig. 4 is a diagram illustrating a schematic flow of an example of a learning process of a calibration model according to the first embodiment.

[0033] As shown in FIG. 3 and FIG. 4, the processing circuit 1 acquires test data with a correct answer by implementing the acquisition function 11 (step SA1). The test data with a correct answer is test data in which a correct answer regarding labeling is recognized or can be recognized. For example, a head MR image in which a tumor, which is a measurement target of MR spectroscopy, is drawn is used as the test data with a correct answer. In this case, a label is given to indicate the tumor, so that the drawn tumor means the correct answer. The test data with a correct answer may be an MR image in which an actual tumor obtained by MR imaging a patient presenting with a tumor in the head is drawn, or an MR image in which a pseudo tumor is drawn may be used. The pseudo tumor is given at an arbitrary position in the head region, for example, by performing image processing on an MR image in which a tumor is not drawn. As the image processing, a GAN (generative adversarial network) may be used, or a technique for simply replacing a pseudo tumor region may be used, or other image processing may be used. Hereinafter, when a real tumor and a pseudo tumor are not particularly distinguished from each other, they will be collectively referred to simply as tumors.

[0034] When step SA1 is performed, processing circuit 1, by implementing label assignment function 12, assigns a label (assigned label) to the test data acquired in step SA1 (step SA2). Specifically, in step SA2, processing circuit 1 first displays test data (head MR image) on display device 5. A tumor is depicted in the displayed head MR image. At this time, processing circuit 1 may image-recognize the tumor and visually highlight the image-recognized tumor to display it. The first user observes the displayed test data and assigns an ROI mark to surround the tumor via input interface 7. The ROI mark is assigned to the test data as an assigned label.

[0035] When step SA2 is performed, the processing circuit 1 generates a calibration parameter based on a comparison between the correct answer and the assigned label by implementing the calibration model learning function 13 (step SA3). For example, in step SA3, the processing circuit 1 calculates the distance between the correct answer and the position indicated by the assigned label as the calibration parameter. Specifically, the processing circuit 1 calculates the distance between the position of the tumor, which is the correct answer, and the position indicated by the ROI mark, which is the assigned label, as the calibration parameter. The processing circuit 1 may also generate an ROI mark indicating the tumor as the correct answer label by image processing, and calculate the distance between the correct answer label and the assigned label as the calibration parameter. The distance may be defined as a scalar quantity or a higher-order vector quantity.

[0036] Steps SA1 to SA3 are repeated for different test data to collect a plurality of training samples including test data, assigned labels, and calibration parameters for the first user. The test data and assigned labels are collected as input training samples, and the calibration parameters are collected as output training samples.

[0037] When step SA3 is performed, the processing circuit 1 generates a calibration model based on the test data, the assigned label, and the calibration parameters by implementing the calibration model learning function 13 (step SA4). Specifically, the processing circuit 1 trains the learning parameters of the calibration model based on the input learning sample including the test data and the assigned label, and the output learning sample including the calibration parameters. The assigned label may be input to the machine learning model as image data representing the ROI mark, or as numerical data representing the coordinates at which the ROI mark is drawn or the coordinates indicated by the ROI mark. The learning parameters are, for example, network parameters to be learned, such as the weights and biases of the calibration model. The learning method is not particularly limited, but it is sufficient to use supervised learning in which the test data and the assigned label are the input learning samples and the calibration parameters are the output learning samples. In this case, the processing circuit 1 may iteratively update the learning parameters so as to minimize the error between the predicted calibration parameters obtained by forward propagating the test data and the assigned label to the calibration model, and the calibration parameters that are the output learning samples. The optimization method is not particularly limited, and may be any method, such as stochastic gradient descent. In addition, the learning parameters may be updated for each learning sample, or for each set of multiple learning samples. Of course, as long as the above test data, assigned labels, and calibration parameters are used, the learning method is not limited to supervised learning, and other learning methods such as weakly supervised learning, anti-supervised learning, and contrastive learning may be used.

[0038] According to step SA4, it is possible to generate a calibration model that inputs test data and a label (assigned label) assigned to the test data by the first user and outputs a calibration parameter for correcting the assigned label to a correct label. That is, it is possible to generate a calibration model for calibrating personal characteristics related to label assignment of the first user. The calibration model is stored in the storage device 3 in association with the user identification information of the first user.

[0039] Next, a learning process of a target model using a calibration model according to the embodiment 1 will be described. Fig. 5 is a diagram illustrating a flow of an example of the learning process of a target model according to the embodiment 1. Fig. 6 is a diagram illustrating a schematic flow of an example of the learning process of a target model according to the embodiment 1.

[0040] 5 and 6, the processing circuit 1 acquires real data by implementing the acquisition function 11 (step SB1). It is assumed that the real data includes a tumor to be labeled. The tumor depicted in the real data does not need to be recognized.

[0041] When step SB1 is performed, the processing circuit 1 assigns a label to the real data acquired in step SB1 by implementing the label assignment function 12 (step SB2). Specifically, in step SB2, the processing circuit 1 first displays the real data on the display device 5. The first user observes the displayed real data and performs an operation to assign an ROI mark indicating the position of the tumor to the real data via the input interface 7. The processing circuit 1 assigns the ROI mark to the real data as an assigned label in accordance with the operation. It is assumed that the personal characteristics of the first user are reflected in the position of the assigned label.

[0042] When step SB2 is performed, the processing circuit 1, by implementing the object model learning function 14, applies the actual data acquired in step SB1 and the attached label attached in step SB2 to a calibration model to generate calibration parameters (step SB3). Specifically, first, the processing circuit 1 searches the storage device 3 using the identification information of the first user as a search key to select a calibration model related to the first user. Next, the processing circuit 1 applies the actual data and the attached label to the selected calibration model to generate calibration parameters for calibrating the personal characteristics of the first user related to the attached label.

[0043] When step SB3 is performed, the processing circuit 1, by implementing the target model learning function 14, applies the assigned label assigned in step SB2 to the calibration parameters generated in step SB3 to generate a calibrated label (step SB4). By applying the assigned label to the calibration parameters, a label (calibrated label) in which the personal characteristics of the first user are calibrated is generated from the assigned label. The calibrated label indicates the exact position of the tumor contained in the real data. In other words, the calibrated label is expected to have the same accuracy as the true answer label, with the personal characteristics of the first user suppressed. It can be said that such a calibrated label is more reliable than the assigned label.

[0044] FIG. 7 is a diagram illustrating the positional relationship between the assigned label and the calibrated label. As shown in FIG. 7, a tumor T1 is drawn on a head MR image I3, which is real data. In step SB2, a first user assigns an ROI mark (assigned label) M31. The first user assigns the label M31 to the left side of the image of the tumor T1. In step SB4, a calibrated label M41 in which the personal characteristics of the first user appearing in the assigned label M31 are calibrated is generated by a calibration model for the first user. The calibrated label M41 is assigned to the approximate center of the tumor T1. It can be said that the personal characteristic of being assigned to the left side of the image is suppressed by the calibration model for the first user. The calibrated label 41 is assigned to an accurate position, and is therefore more accurate and reliable than the assigned label M31.

[0045] Steps SB1 to SB4 are repeated for different real data to collect a plurality of training samples including real data and calibrated labels for the first user. The real data is collected as input training samples, and the calibrated labels are collected as output training samples.

[0046] When step SB4 is performed, the processing circuit 1 learns the target model based on the real data acquired in step SB1 and the calibrated label generated in step SB4 by implementing the target model learning function 14 (step SB5). Specifically, the processing circuit 1 trains the learning parameters of the target model based on the input learning sample, which is the real data, and the output learning sample, which is the calibrated label. The learning method is not particularly limited, but supervised learning in which the real data is the input learning sample and the calibration is the output learning sample may be used. In this case, the processing circuit 1 may iteratively update the learning parameters so as to minimize the error between the predicted label obtained by forward propagating the real data to the target model and the calibrated label, which is the output learning sample. The optimization method is not particularly limited, and may be any method such as stochastic gradient descent. In addition, the learning parameters may be updated for each learning sample or for each of a plurality of learning samples. Of course, as long as the above-mentioned real data and calibrated labels are used, the method is not limited to supervised learning, and other learning methods such as weakly supervised learning, anti-supervised learning, and contrastive learning may be used.

[0047] According to step SB5, it is possible to generate a target model that inputs real data and outputs a label indicating a tumor drawn on the real data. Since the target model is trained using calibrated labels in which the personal characteristics of the first user regarding labeling are calibrated, it is possible to perform machine learning of the target model while suppressing the influence of the personal characteristics of the label. This also makes it possible to improve the accuracy of the target model.

[0048] Example 2 The calibration model according to the second embodiment is a machine learning model that receives input data and an assigned label assigned to the input data and outputs a calibrated label. The calibrated label is an example of calibrated data output from the calibration model. By implementing the calibration model learning function 13, the processing circuit 1 learns the calibration model based on the calibrated learning data, the calibrated learning label assigned by the user to the calibrated learning data, and the correct label for the calibrated learning data. The target model according to the second embodiment is learned based on the input data, the assigned label, and the calibrated label. By implementing the target model learning function 14, the processing circuit 1 applies the calibration model to the target learning data and the target learning label to generate a calibrated label, and learns the target model based on the target learning data and the calibrated label. First, a learning method of the calibration model according to the second embodiment will be described.

[0049] Fig. 8 is a diagram illustrating a flow of an example of a learning process of a calibration model according to the second embodiment. Fig. 9 is a diagram illustrating a schematic flow of an example of a learning process of a calibration model according to the second embodiment.

[0050] 8 and 9, the processing circuit 1 acquires test data with correct answers (step SC1) by implementing the acquisition function 11. Step SC1 is similar to step SA1.

[0051] After step SC1 is performed, the processing circuit 1 assigns a label (assigned label) to the test data acquired in step SC1 by implementing the label assignment function 12 (step SC2). Step SC2 is similar to step SA2.

[0052] Steps SC1 to SC2 are repeated for different test data, thereby collecting a plurality of learning samples including test data, annotated labels, and correct labels for the first user. The correct labels are generated based on the correct answers included in the test data with correct answers. Specifically, the correct labels are generated as ROI marks indicating the tumors drawn on the MR images, which are the test data. The test data and the annotated labels are collected as input learning samples, and the correct answers are collected as output learning samples.

[0053] When step SC2 is performed, the processing circuit 1 generates a calibration model based on the test data, the assigned label, and the correct answer (correct answer label) by implementing the calibration model learning function 13 (step SC3). Specifically, the processing circuit 1 trains learning parameters of the calibration model based on an input learning sample including the test data and the assigned label, and an output learning sample including the correct answer label. The learning method is not particularly limited, but it is sufficient to use supervised learning in which the test data and the assigned label are used as input learning samples and the correct answer label is used as an output learning sample. In this case, the processing circuit 1 iteratively updates the learning parameters so as to minimize the error between the predicted label obtained by forward propagating the test data and the assigned label to the calibration model, and the correct answer label which is the output learning sample.

[0054] According to step SC3, it is possible to generate a calibration model that inputs test data and a label (assigned label) assigned to the test data by the first user and outputs a correct label for the test data. That is, it is possible to generate a calibration model for calibrating personal characteristics related to label assignment by the first user. The calibration model is stored in the storage device 3 in association with the user identification information of the first user.

[0055] Next, a learning process of a target model using a calibration model according to the embodiment 2 will be described. Fig. 10 is a diagram illustrating a flow of an example of the learning process of a target model according to the embodiment 2. Fig. 11 is a diagram illustrating a schematic flow of an example of the learning process of a target model according to the embodiment 2.

[0056] 10 and 11, the processing circuit 1 acquires real data by implementing the acquisition function 11 (step SD1). Step SD1 is similar to step SB1.

[0057] After step SD1 is performed, the processing circuit 1 assigns a label to the actual data acquired in step SD1 by implementing the label assignment function 12 (step SD2). Step SD2 is similar to step SB2.

[0058] When step SD2 is performed, the processing circuit 1, by implementing the object model learning function 14, applies the actual data acquired in step SD1 and the assigned label assigned in step SD2 to a calibration model to generate a calibrated label (step SD3). Specifically, first, the processing circuit 1 searches the storage device 3 using the identification information of the first user as a search key to select a calibration model related to the first user. Next, the processing circuit 1 applies the actual data and the assigned label to the selected calibration model to generate a label (calibrated label) in which the personal characteristics of the first user related to the assigned label are calibrated.

[0059] Steps SD1 to SD3 are repeated for different real data to collect a plurality of training samples including real data and calibrated labels for the first user. The real data is collected as input training samples, and the calibrated labels are collected as output training samples.

[0060] After step SD3, the processing circuit 1, by implementing the object model learning function 14, learns an object model based on the real data acquired in step SD1 and the calibrated labels generated in step SD3 (step SD4). Step SD4 is similar to step SB5.

[0061] According to step SD4, it is possible to generate a target model that inputs real data and outputs a label indicating a tumor drawn on the real data. Since the target model is trained using calibrated labels in which the personal characteristics of the first user regarding labeling are calibrated, it is possible to perform machine learning of the target model while suppressing the influence of the personal characteristics of the label. This also makes it possible to improve the accuracy of the target model.

[0062] Example 3 The calibration model according to the third embodiment is a machine learning model that inputs input data and an assigned label assigned to the input data and outputs the reliability of the assigned label. The reliability is an example of calibration data output from the calibration model. By implementing the calibration model learning function 13, the processing circuit 1 determines the reliability of the calibrated learning label for the calibrated learning data, and learns the calibration model according to the third embodiment based on the calibrated learning data, the calibrated learning label, and the reliability. The target model according to the third embodiment is learned based on the input data, the assigned label, and the reliability. By implementing the target model learning function 14, the processing circuit 1 applies the calibration model according to the third embodiment to the target learning data and the target learning label to output the reliability of the target learning label, and learns the target model based on the target learning data and the target learning label using the reliability as a parameter of an error function. First, a learning method of the calibration model according to the third embodiment will be described.

[0063] Fig. 12 is a diagram illustrating a flow of an example of a learning process of a calibration model according to the embodiment 3. Fig. 13 is a diagram illustrating a schematic flow of an example of a learning process of a calibration model according to the embodiment 3.

[0064] 12 and 13, the processing circuit 1 acquires test data with correct answers (step SE1) by implementing the acquisition function 11. Step SE1 is similar to step SA1.

[0065] After step SE1 is performed, the processing circuit 1 assigns a label (assigned label) to the test data acquired in step SE1 by implementing the label assignment function 12 (step SE2). Step SE2 is similar to step SA2.

[0066] When step SE2 is performed, the processing circuit 1 determines the reliability of the assigned label based on a comparison between the correct answer included in the test data and the assigned label assigned in step SE2 by implementing the calibration model learning function 13 (step SE3). The reliability is an index that quantifies the likelihood of the assigned label for the input data. Various methods are possible for determining the reliability. For example, the processing circuit 1 calculates the distance between the assigned label and the correct label as the reliability. The correct label is a label generated based on the correct answer included in the test data. Specifically, the processing circuit 1 calculates the distance between the position indicated by the ROI mark, which is the assigned label, and the position indicated by the ROI mark, which is the correct label, as the reliability. Note that the reliability may be the numerical value of the distance itself, may be the numerical value of the distance scaled to a predetermined value range, or may be a category classified according to the numerical value of the distance. Note that the reliability may be determined to a numerical value specified by the first user or the like via the input interface 7. Hereinafter, the reliability is assumed to be a numerical value scaled to a range of 0 to 1, as an example.

[0067] Steps SE1 to SE3 are repeated for different test data to collect a plurality of training samples including test data, attached labels, and confidence levels for the first user. The test data and attached labels are collected as input training samples, and the confidence levels are collected as output training samples.

[0068] When step SE3 is performed, the processing circuit 1 generates a calibration model based on the test data, the attached label, and the reliability by implementing the calibration model learning function 13 (step SE4). Specifically, the processing circuit 1 trains the learning parameters of the calibration model based on the input learning sample including the test data and the attached label, and the output learning sample including the reliability. The learning method is not particularly limited, but supervised learning may be used in which the test data and the attached label are the input learning sample and the reliability is the output learning sample. In this case, the processing circuit 1 may iteratively update the learning parameters so as to minimize the error between the predicted reliability obtained by forward propagating the test data and the attached label to the calibration model, and the reliability which is the output learning sample. The optimization method is not particularly limited, and may be any method such as stochastic gradient descent. In addition, the learning parameters may be updated for each learning sample, or for each of a plurality of learning samples. Of course, as long as the above test data, the attached label, and the reliability are used, the method is not limited to supervised learning, and other learning methods such as weakly supervised learning and anti-supervised learning may be used.

[0069] According to step SE4, it is possible to generate a calibration model that inputs test data and a label (assigned label) assigned to the test data by the first user, and outputs the reliability of the assigned label. That is, it is possible to generate a calibration model for calibrating personal characteristics related to label assignment by the first user. The calibration model is stored in the storage device 3 in association with the user identification information of the first user.

[0070] Next, a learning process of a target model using a calibration model according to the embodiment 3 will be described. Fig. 14 is a diagram showing a flow of an example of the learning process of a target model according to the embodiment 3. Fig. 15 is a diagram showing a schematic flow of an example of the learning process of a target model according to the embodiment 3.

[0071] 14 and 15, the processing circuit 1 acquires real data (step SF1) by implementing the acquisition function 11. Step SF1 is similar to step SB1.

[0072] After step SF1 is performed, the processing circuit 1 assigns a label to the actual data acquired in step SF1 by implementing the label assignment function 12 (step SF2). Step SF2 is similar to step SB2.

[0073] When step SF2 is performed, the processing circuit 1, by implementing the object model learning function 14, applies the actual data acquired in step SF1 and the attached label assigned in step SF2 to a calibration model to determine the reliability (step SF3). Specifically, first, the processing circuit 1 searches the storage device 3 using the identification information of the first user as a search key to select a calibration model related to the first user. Next, the processing circuit 1 applies the actual data and the attached label to the selected calibration model to determine the reliability of the attached label.

[0074] By repeating steps SF1 to SF3 for different real data, a plurality of learning samples including real data, attached labels, and confidences for the first user are collected. The real data is collected as an input learning sample, the attached labels are collected as an output learning sample, and the confidences are collected as parameters of an error function.

[0075] When step SF3 is performed, the processing circuit 1, by implementing the target model learning function 14, learns a target model based on the real data acquired in step SF1, the calibrated labels assigned in step SF2, and the reliability determined in step SF3 (step SF4). In step SF4, the processing circuit 1 trains learning parameters of the target model based on the real data, which is an input learning sample, and the assigned labels, which are output learning samples. The learning method is not particularly limited as long as real data and calibrated labels are used, and supervised learning, weakly supervised learning, anti-supervised learning, contrastive learning, etc. are possible. In the following, as an example, the learning method is assumed to be supervised learning.

[0076] In this case, the processing circuit 1 performs supervised learning using real data as input learning samples and the assigned labels as output learning samples. In supervised learning, the processing circuit 1 iteratively updates the learning parameters so as to minimize the error between the predicted labels obtained by forward propagating the real data to the target model and the assigned labels, which are the output learning samples. The optimization method is not particularly limited, and may be any method such as stochastic gradient descent.

[0077] Here, the predicted label y i and the assigned label t i Error L i is expressed, for example, by the following formula (1). "i" is the number of the learning sample. In formula (1), since the task of the target model according to this embodiment is a regression problem, the error L i is expressed as a least squares function, but any function such as cross entropy may be used depending on the task of the target model. i is a function f(W, x i ) where W is the set of learning parameters for the target model, and x i represents real data (input learning sample). The processing circuit 1 updates the learning parameter W so as to minimize the error function L shown in the following equation (3). The error function L is the error L for the input learning sample i=1 to N (integer), i and confidence level c i The confidence level c is expressed as the sum of the products of i is the input training sample x i is a parameter of the error function L that represents the reliability corresponding to

[0078] L i =||t i -y i || 2 (1) y i =f(W,x i ) (2) L=Σ i=1 N (c i L i ) (3)

[0079] As shown in the above equations (1) to (3), the reliability of the learning sample “i” c i is the error L i It acts as a weight for the confidence level c i If is relatively small, the learning sample “i” will not contribute to machine learning, and the confidence level c i When is relatively large, the learning sample "i" contributes relatively to machine learning. i The error L i By applying this method, it is possible to machine-learn a target model that reduces the influence of personal characteristics of the labels. This also makes it possible to improve the accuracy of the target model.

[0080] This concludes the description of the first to third embodiments. According to the first to third embodiments, it is possible to learn a target model while calibrating the personal characteristics appearing in the labels given by the user. As described above, bias and variability are considered as types of personal characteristics. Whether the calibration model can calibrate the bias or the variability depends on the type of personal characteristics appearing in the labels used to learn the calibration model. That is, if the first user gives a label with a constant bias without variability in step SA2, the calibration model can calibrate the bias specific to the first user. On the other hand, if the first user gives a label with a constant variability without bias in step SA2, the calibration model can calibrate the variability specific to the first user. Of course, if the first user gives a label with a constant bias and variability in step SA2, the calibration model can calibrate the bias and variability specific to the first user.

[0081] Note that the above calibration model learning method is an example, and other learning methods may be used. As an example, the calibration model may be generated by transfer learning. In this case, the processing circuit 1 first learns a calibration model for another user (e.g., a second user) different from the first user according to any of the above embodiments by implementing the calibration model learning function 13. The processing circuit 1 copies the learning parameters of the learned calibration model corresponding to the second user to an unlearned calibration model. The processing circuit 1 trains the learning parameters of the unlearned calibration model so as to input the calibration learning data and the calibration learning label and output the calibration data, thereby learning a calibration model for the first user. According to this transfer learning, the learning parameters of the calibration model of another person that has already been learned are initially set to the unlearned calibration model corresponding to the first user, so that it is possible to easily learn the calibration model corresponding to the first user.

[0082] (Application example 1) The above Examples 1 to 3 focus on the calibration of personal characteristics of labeling of one user. The machine learning device 100 according to this embodiment may calibrate personal characteristics of labeling of multiple users. Below, an example of the operation of the machine learning device 100 according to an application example will be described. In the following description, components having substantially the same functions as those in the above embodiment will be given the same reference numerals and will be described repeatedly only when necessary.

[0083] The processing circuit 1 according to the application example 1 learns a target model based on a combination of target learning data, target learning labels, and a calibration model related to a first user, and a combination of target learning data, target learning labels, and a calibration model related to a second user, by implementing the target model learning function 14. The second user is a different person having different personal characteristics from the first user. Note that the second user is a collective term for a person different from the first user, and does not necessarily mean one person, but can also mean multiple people with different personal characteristics.

[0084] FIG. 16 is a diagram illustrating a processing example of the machine learning device 100 according to the application example 1. As shown in FIG. 16, a calibration model of a first user and a calibration model of a second user are generated by implementing the calibration model learning function 13. The calibration model of the first user and the calibration model of the second user may be any of the types in the first to third embodiments as long as they are the same type. The processing circuit 1 uses the calibration model of the first user to generate calibration data 23 related to the calibration of the label assignment of the first user from the real data and the assigned label 21 assigned by the first user. Similarly, the processing circuit 1 uses the calibration model of the second user to generate calibration data 23 from the real data and the assigned label 21 assigned by the second user.

[0085] The format of the calibration data 23 varies depending on the type of the calibration model. In the case of the calibration model of the first embodiment, the calibration data 23 includes an assigned label assigned by the first user to the real data, a calibration parameter output from the calibration model, and a calibration label obtained by calibrating the assigned label with the calibration parameter. In the case of the calibration model of the second embodiment, the calibration data 23 includes an assigned label assigned by the first user to the real data, and a calibration label output from the calibration model. In the case of the calibration model of the third embodiment, the calibration data 23 includes an assigned label assigned by the first user to the real data, and a reliability output from the calibration model.

[0086] The storage device 3 stores a plurality of calibration models corresponding to a plurality of users. Each of the plurality of calibration models is associated with corresponding user identification information. The plurality of calibration models may be systematically stored so as to be searchable by the user identification information. The storage device 3 may store a neural network to which learned learning parameters are assigned as a calibration model, or may store only learned parameters. When using a calibration model, the processing circuit 1 reads the user identification information associated with the assigned label to be processed, and selects and reads out the calibration model associated with the read user identification information from the plurality of calibration models stored in the storage device 3. For example, when the user identification information of a first user is associated with the assigned label to be processed, the calibration model of the first user is selected and read out.

[0087] A plurality of pieces of calibration data 23 are collected by the first user and the second user. When the calibration model is the type of Example 1 or Example 2, the first user calibration model obtains a calibration label in which the personal characteristics of the first user are calibrated as the calibration data 23, and the second user calibration model obtains a calibration label in which the personal characteristics of the second user are calibrated as the calibration data 23. That is, it is possible to obtain calibration data 23 in which individual differences are suppressed. When the calibration model is the type of Example 3, the first user calibration model obtains a label given by the first user and the reliability of the label as the calibration data 23, and the second user calibration model obtains a label given by the second user and the reliability of the label as the calibration data 23. These calibration data 23 are also stored in the storage device 3.

[0088] The processing circuit 1 learns a target model based on the actual data 21 and the calibration data 23 by implementing the target model learning function 14. The target model is learned according to the method described in the first to third embodiments depending on the type of the calibration model. At this time, the personal characteristics of the first user are suppressed from the calibration data 23 regarding the first user obtained using the first user calibration model, and the personal characteristics of the second user are suppressed from the calibration data 23 regarding the second user obtained using the second user calibration model. Therefore, by using these calibration data 23, it becomes possible to perform highly accurate machine learning on the target model in which the personal characteristics of each user are calibrated.

[0089] FIG. 17 is a diagram showing an example of calibration of the personal characteristics of labeling of the first user and the second user. As shown in FIG. 17, the first user assigns an ROI mark M151 indicating the position of a voxel of interest as an assigned label in the ROI mark M25 of the local shimming of the head MR image I5. The calibration model of the first user eliminates the labeling habit of the first user, and the ROI mark M151 is moved to the ROI mark M152. Similarly, the second user assigns an ROI mark M261 indicating the position of a voxel of interest as an assigned label to the ROI mark M26 of the local shimming of the head MR image I65. The calibration model of the second user eliminates the labeling habit of the second user, and the ROI mark M261 is moved to the ROI mark M262. It is expected that the ROI mark M152 and the ROI mark M262 suppress individual differences in labeling.

[0090] (Application example 2) In Application Example 1, we focused on the individual characteristics of label assignment for each user. However, even for the same user, if the label assignment conditions are different, the accuracy of the assigned labels may vary. Assumed label assignment conditions include the type of application used for label assignment and the assignment process. In Application Example 2, we focus on the label assignment conditions.

[0091] The processing circuit 1 according to the application example 2 learns the target model based on a combination of the target learning data, the target learning label, and the calibration model related to the first assignment condition and a combination of the target learning data, the target learning label, and the calibration model related to the second assignment condition by implementing the target model learning function 14. It is assumed that the target learning label related to the first assignment condition and the target learning label related to the second assignment condition are both assigned by the first user. The calibration model related to the first assignment condition is a calibration model for calibrating the personal characteristics of the label assignment by the first assignment condition of the first user. The second calibration model is a calibration model for calibrating the personal characteristics of the label assignment by the second assignment condition of the first user. The first assignment condition and the second assignment condition mean assignment conditions with different condition values. The condition value means the type of application used for label assignment and the specific contents of the assignment process. The type of application means the manufacturer, version, etc. of a specific application. For example, it is possible to use software called an MRI simulator as a label assignment application, which predicts MR imaging by a magnetic resonance imaging device and the behavior of various devices. The software for predicting MR imaging is also called a Bloch simulator. MRI simulators are sold by multiple companies. The types of MRI simulator products by such different sales companies correspond to the types of applications. The labeling process refers to the processing order of multiple processes leading to labeling when a specific application is used, the setting values ​​of the application, etc. The personal characteristics of Application Example 2 also include fluctuations in labeling between different labeling conditions for the same user.

[0092] The processing circuit 1 according to the application example 2 learns a calibration model for the first labeling condition by using the assigned labels assigned by the first user according to the first labeling condition through the implementation of the calibration model learning function 13 in any one of the methods of the first to third embodiments. The processing circuit 1 also learns a calibration model for the second labeling condition by using the assigned labels assigned by the first user according to the second labeling condition through the implementation of the calibration model learning function 13 in any one of the methods of the first to third embodiments. The calibration model for the first labeling condition and the calibration model for the second labeling condition are stored in the storage device 3. The calibration model is stored in association with identification information of the labeling condition (hereinafter referred to as labeling condition identification information).

[0093] The processing circuit 1 according to the application example 2 assigns a target learning label to target learning data according to a specific label assignment condition by implementing the target model learning function 14. The processing circuit 1 reads the assignment condition identification information related to the label assignment condition, and selects and reads out a calibration model associated with the read assignment condition identification information from among a plurality of calibration models stored in the storage device 3. For example, when the target learning label is assigned according to the second label assignment condition, the processing circuit 1 selects and reads out a calibration model associated with the assignment condition identification information of the second label assignment condition. Next, the processing circuit 1 learns the target model while calibrating the target learning label using the read calibration model. As described above, the target model is learned according to the type of calibration model.

[0094] In this way, according to Application Example 2, since the target model can be trained using labels in which personal characteristics resulting from differences in labeling conditions have been calibrated, the reliability of the labels is improved, and as a result, the reliability of the target model is also improved. In addition, since the target model is trained using labels in which personal characteristics resulting from differences in labeling conditions have been calibrated, it is expected that the accuracy of the target model will improve.

[0095] In the application example 2, it is preferable that the labeling conditions when labels are assigned to the calibration learning data and the labeling conditions when labels are assigned to the target learning data are unified. Specifically, the types of applications and the labeling process are unified between the labeling of the test data and the labeling of the actual data. By unifying the labeling conditions, it is possible to further improve the accuracy of the machine learning of the target model.

[0096] Application example 2 and application example 1 may be combined. That is, a calibration model may be generated for each user and for each labeling condition. In this case, the calibration model is stored in the storage device 3 in association with the user identification information and the labeling condition identification information of the labeling condition. When using the calibration model, the processing circuit 1 reads the user identification information and the labeling condition identification information associated with the label to be processed, and selects and reads out a calibration model associated with the combination of the read user identification information and the labeling condition identification information from among a plurality of calibration models stored in the storage device 3. For example, when the user identification information of the first user and the labeling condition identification information of the first labeling condition are associated with the label to be processed, the calibration model corresponding to the combination of the first user and the first labeling condition is selected and read out.

[0097] (Application example 3) A neural network that integrates multiple calibration models for multiple users obtained by the above various embodiments may be used as a calibration model. Hereinafter, the neural network that integrates multiple calibration models for multiple users will be referred to as a blend network.

[0098] Fig. 18 is a diagram showing a configuration example of a blend network NB according to Application Example 3. The calibration target of the blend network NB shown in Fig. 18 is a first user. The blend network NB has a plurality of calibration models respectively corresponding to a plurality of users different from the first user, and an addition layer 181 that performs weighted addition of a plurality of calibration data from the plurality of calibration models according to weights trained for calibration of labeling of the first user, and outputs calibration data related to calibration of labeling of the first user. The blend network NB functions as a calibration model that calibrates labeling of the first user.

[0099] The number of other users may be any number greater than or equal to two. In the following description, the other users are assumed to be three users, namely, the second user, the third user, and the fourth user. The second user calibration model is a calibration model for the second user according to the various embodiments described above, which receives actual data and an assigned label and outputs calibration data related to the calibration of the personal characteristics of the second user's label assignment. The third user calibration model is a calibration model for the third user according to the various embodiments described above, which receives actual data and an assigned label and outputs calibration data related to the calibration of the personal characteristics of the third user's label assignment. The fourth user calibration model is a calibration model for the fourth user according to the various embodiments described above, which receives actual data and an assigned label and outputs calibration data related to the calibration of the personal characteristics of the fourth user's label assignment.

[0100] The summation layer 181 performs weighted addition of the calibration data from the second user calibration model, the calibration data from the third user calibration model, and the calibration data from the fourth user calibration model, and outputs the final calibration data for the first user. Specifically, the summation layer 181 multiplies the calibration data from the second user calibration model by a weight w2 to generate weighted calibration data, multiplies the calibration data from the third user calibration model by a weight w3 to generate weighted calibration data, multiplies the calibration data from the fourth user calibration model by a weight w4 to generate weighted calibration data, and adds the weighted calibration data for the second user, the weighted calibration data for the third user, and the weighted calibration data for the fourth user to generate the final calibration data. Hereinafter, the weights will be referred to as output weights.

[0101] The output weights w2, w3, and w4 are determined by learning the blend network NB. The output weights w2, w3, and w4 are learned to calibrate the personal characteristics of the labeling of a first user different from the user with which each calibration model included in the blend network NB was trained. By inputting real data and the assigned labels assigned to the real data by a fourth user to a calibration model for the first user, a calibration model for the second user, and a calibration model for the third user included in the blend network NB, calibration data of the personal characteristics of the labeling of the fourth user is output from the blend network NB.

[0102] Here, the learning of the blend network NB will be specifically described. First, the second user calibration model, the third user calibration model, and the fourth user calibration model are individually learned according to any of the above-mentioned embodiments. Next, an unlearned blend network NB is designed in which the addition layer 181 is provided at the output destinations of the second user calibration model, the third user calibration model, and the fourth user calibration model. The initial values ​​of the output weights w2, w3, and w4 may be set arbitrarily.

[0103] For an untrained blend network NB, the output weights w2, w3, and w4 are trained based on supervised learning in which test data and the labels assigned to the test data by the first user are input, and calibration data related to the calibration of personal characteristics of the labels assigned by the first user is output. In the training of the blend network NB, the learning parameters of the second user calibration model, the third user calibration model, and the fourth user calibration model included in the blend network NB may be fixed. Note that, together with the output weights w2, w3, and w4, some or all of the learning parameters of the second user calibration model, the third user calibration model, and the fourth user calibration model may be trained.

[0104] By training the output weights of the blend network NB that integrates multiple calibration models in this way, it becomes possible to generate a calibration model for calibrating the personal characteristics of other users' labeling. Since the number of output weights is small compared to the learning parameters, it is possible to simply learn the calibration model.

[0105] For users other than the first user (e.g., the fifth user, etc.), it is possible to train the output weights w2, w3, and w4 of the blend network NB that calibrates the personal characteristics of the labeling of the user, in the same manner as in the case of the first user. The blend network NB to which the output weights w2, w3, and w4 after training are assigned is stored in the storage device 23 in association with the user identification information. Note that the combination of the output weights w2, w3, and w4 may be stored in the storage device 23 in association with the user identification information.

[0106] (Other task examples) In the above embodiment, as an example, the task of the target model is the setting of measurement voxels of MR spectroscopy. However, as described above, the task of the target model according to the present embodiment is not limited to this. Below, some other specific examples of the task of the target model will be described.

[0107] The task of the target model according to the specific example 1 is to detect an imaging section. The target model according to the specific example 1 receives a medical image in which an imaging target region is drawn, and detects an imaging section from the imaging target region. In machine learning, a learning parameter is trained to receive a medical image and output an imaging section position. When collecting learning data, a label corresponding to the imaging section position is assigned by various users.

[0108] FIG. 19 is a diagram showing an example of personal characteristics of labeling of a first user and a second user in relation to an imaging section detection task. As shown in FIG. 19, each user refers to medical images I6 and I7 displayed on a display device 5 or the like, and assigns an imaging section mark CS indicating an imaging section to an imaging target area drawn on the medical images I6 and I7 as an assignment label. The imaging target area is not particularly limited, and may be any part of the human body such as the heart, head, or abdomen, but in the example shown in FIG. 19, it is assumed to be the heart. In addition, the type of medical image diagnostic device that performs imaging is not particularly limited, and any type such as a magnetic resonance imaging device or an X-ray computed tomography device is possible.

[0109] Each user observes the medical image in which the imaging target region is drawn, and identifies the imaging target such as a lesion. Then, each user sets a mark corresponding to the imaging section (hereinafter, imaging section mark) on the medical image via the input interface 7 so as to include the imaging target. The imaging section mark is set as an attached label. The medical image is an example of the test data in the above-mentioned Examples 1 to 3. That is, the test data may be an image obtained by performing medical imaging on an actual patient, or an image obtained by performing medical imaging on a phantom. The imaging target such as a lesion may be actually included inside the patient or phantom, or may be pseudo-generated by image processing such as GAN.

[0110] Specifically, as shown in Fig. 19, a first user observes a medical image I6 to identify an imaging target in the cardiac region, and assigns an imaging section mark CS1 as an assigned label to the imaging target. Similarly, a second user observes a medical image I7 to determine an imaging target in the cardiac region, and assigns an imaging section mark CS2 as an assigned label to the imaging target. The determination of the imaging target and the assignment of the imaging section marks CS1 and CS2 depend on the skill of the individual. Therefore, even if the cardiac region is anatomically the same, the positions of the imaging section marks CS1 and CS2 reflect the personal characteristics of the user.

[0111] As described above, in the imaging section detection task, there may be individual differences in the accuracy of the assigned labels (imaging section marks CS1 and CS2), so it is useful in machine learning of the target model to calibrate the individual characteristics of the assigned labels using the calibration models exemplified in Examples 1 to 3 above.

[0112] In the first specific example, the task of the target model is an imaging slice detection task, but in this embodiment, the task may be a slice detection task, and the use of the slice is not particularly limited. Examples of uses other than the imaging slice include display slices for post-processing of medical images. In this case, the type of medical image is not particularly limited, and may be generated by any medical image diagnostic device such as a magnetic resonance imaging device or an X-ray computed tomography device.

[0113] The task of the target model according to the specific example 2 is to judge whether the measurement target is abnormal or normal. The target model according to the specific example 2 receives a spectrum of a measurement target and judges whether the measurement target is abnormal or normal from the spectrum. In machine learning, learning parameters are trained to receive a spectrum and output a classification of abnormal or normal. The spectrum is typically a series of data of any component of a measurement value obtained by various measurements on a measurement target sample. The measurement method may be an optical, magnetic, chemical, or immunological method for a biological sample, or may be a method using various image analyses on pixels or voxels of a medical image or optical image. When collecting learning data, labels corresponding to the judgment results of abnormality or normality are assigned by various users.

[0114] Fig. 20 is a diagram showing an example of personal characteristics of labeling of a first user and a second user in relation to an abnormal / normal judgment task. As shown in Fig. 20, each user refers to a spectrum and assigns a label of abnormality or normality as an assigned label. In the example shown in Fig. 20, the spectrum is a spectrum that is a frequency distribution of MR signal intensity measured by MR spectroscopy.

[0115] Each user observes the spectrum and judges whether it is abnormal or normal. If each user judges that there is an abnormality, he / she gives a mark indicating that it is abnormal (hereinafter referred to as an abnormal mark) to the spectrum via the input interface 7. If each user judges that there is no abnormality, he / she gives a mark indicating that it is normal (hereinafter referred to as a normal mark) to the spectrum via the input interface 7. The spectrum is an example of the test data in the above-mentioned Examples 1 to 3. That is, the test data may be a spectrum obtained by performing MR spectroscopy on an actual patient, or may be a spectrum obtained by performing MR spectroscopy on a phantom. The imaging target such as a lesion may be actually included inside the patient or the phantom, or may be pseudo-generated by image processing such as GAN.

[0116] Specifically, as shown in FIG. 20, the first user observes spectrum I8 and determines that there is no abnormality, and therefore adds a normal mark M8 to spectrum I8 as a stamp of the characters "normal". The second user observes spectrum I9 and determines that there is an abnormality, and therefore adds an abnormal mark M9 to spectrum I9. Specifically, the abnormal mark M9 may be an abnormal mark M91 that is a stamp of the characters "abnormal", or a mark M92 that indicates the location of the abnormality. The determination of whether something is abnormal or normal and the determination of the location of the abnormality depend on the skill of the individual. Therefore, even if the spectrum is the same, the presence and / or position of the normal mark M8 and the abnormal mark M9 reflect the individual characteristics of the user.

[0117] As such, in the abnormal / normal judgment task, there may be individual differences in the accuracy of the assigned labels (normal mark M8 and abnormal mark M9), so it is useful in the machine learning of the target model to calibrate the individual characteristics of the assigned labels using the calibration models exemplified in Examples 1 to 3 above.

[0118] (Inference device) Fig. 21 is a diagram showing an example of the configuration of an inference device 200 according to this embodiment. As shown in Fig. 21, the inference device 200 has a processing circuit 21, a storage device 23, a display device 25, an input interface 27, and a communication interface 29. Data communication between the processing circuit 21, the storage device 23, the display device 25, the input interface 27, and the communication interface 29 is performed via a bus.

[0119] The processing circuit 21 has a processor such as a CPU. The processor starts various programs installed in the storage device 23 or the like, thereby realizing an acquisition function 211, an inference function 212, a display control function 213, and the like. Each of the functions 211 to 213 is not limited to being realized by a single processing circuit. A processing circuit may be configured by combining a plurality of independent processors, and each processor may execute a program to realize each of the functions 211 to 213.

[0120] By implementing the acquisition function 211, the processing circuit 21 acquires target data and a target model. The target data is actual data to be processed by the target model. The target data is the same type of data as the calibration learning data and the target learning data. The target model is a machine learning model trained by the machine learning device 100, and is a machine learning model trained to input input data such as the target data and output inference data corresponding to the target data. The target data and the target model are stored in the storage device 23.

[0121] By implementing the inference function 212, the processing circuit 21 applies the target data to the target model and outputs inferred data corresponding to the target data.

[0122] By implementing the display control function 213, the processing circuitry 21 displays various information on the display device 25. For example, the processing circuitry 21 displays processing target data, inference data, and the like.

[0123] The storage device 23 is a storage device such as a ROM, RAM, HDD, SSD, integrated circuit storage device, etc., that stores various data. In addition to the above storage devices, the storage device 23 may be a portable storage medium such as a CD, DVD, or flash memory, or a drive device that reads and writes various information between the storage device and a semiconductor memory element, etc.

[0124] The display device 25 displays various data in accordance with the display control function 213 of the processing circuit 21. A liquid crystal display, a CRT display, an organic EL display, a plasma display, or any other display may be appropriately used as the display device 25. The display device 25 may also be a projector.

[0125] The input interface 27 accepts various input operations from a user, converts the accepted input operations into electrical signals, and outputs the electrical signals to the processing circuit 21. Specifically, the input interface 27 is connected to input devices such as a mouse, a keyboard, a trackball, a switch, a button, a joystick, a touchpad, and a touch panel display. The input interface 27 outputs electrical signals corresponding to the input operations to the input devices to the processing circuit 21. The input devices connected to the input interface 27 may be input devices provided in other computers connected via a network or the like. The input interface 27 may be a voice recognition device that converts a voice signal collected by a microphone into an instruction signal.

[0126] The communication interface 29 is an interface that connects to various computers via a LAN or the like. A LAN card, a network adapter, a network interface card, or the like is used as the communication interface 29. For example, the communication interface 29 performs data communication with a generation device or a storage device for processing learning data. The communication interface 29 also performs data communication with the machine learning device 100 to receive a target model.

[0127] FIG. 22 is a diagram showing an input / output relationship of a target model. As shown in FIG. 22, the target model is a machine learning model that has been trained to input real data to be processed and output inference data corresponding to the real data. For example, the real data to be processed is acquired from a generating device or a storage device for the real data via a communication interface 29. Alternatively, if the real data to be processed is stored in the storage device 23 in advance, it may be acquired from the storage device 23. The target model is acquired from the machine learning device 100 via the communication interface 29. Alternatively, if the target model is stored in the storage device 23 in advance, it may be acquired from the storage device 23.

[0128] For example, when the task of the target model is a problem of detecting the position of a measurement voxel in MR spectroscopy, an MR image is input as actual data, and an ROI mark indicating the position of the measurement voxel or an MR image with an ROI mark superimposed thereon is output as inference data.

[0129] In the target model according to this embodiment, the personal characteristics of the label assigner are suppressed from the correct labels used in machine learning, so that the accuracy of the inference data is expected to be high.

[0130] According to at least one of the embodiments described above, it is possible to improve the accuracy of machine learning using labels.

[0131] The term "processor" used in the above description means a circuit such as a CPU, a GPU, or an application specific integrated circuit (ASIC), a programmable logic device (e.g., a simple programmable logic device (SPLD), a complex programmable logic device (CPLD), and a field programmable gate array (FPGA)). The processor realizes its function by reading and executing a program stored in a memory circuit. Instead of storing a program in a memory circuit, the processor may be configured to directly incorporate the program in the circuit. In this case, the processor realizes its function by reading and executing a program incorporated in the circuit. Instead of executing a program, a function corresponding to the program may be realized by a combination of logic circuits. Each processor in this embodiment is not limited to being configured as a single circuit for each processor, but may be configured as one processor by combining multiple independent circuits to realize its function. Furthermore, the multiple components in FIG. 1 and FIG. 21 may be integrated into one processor to realize its function.

[0132] Although some embodiments have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, and combinations of embodiments can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope of the invention and its equivalents as described in the claims, as well as in the scope and spirit of the invention. [Explanation of symbols]

[0133] 1 Processing circuit 3 Storage device 5 Display equipment 7 Input Interface 9 Communication Interface 11 Acquisition Function 12 Labeling function 13 Calibration model learning function 14. Target model learning function 15 Display control function

Claims

1. a first learning unit that uses a first calibration model that inputs first processed data and a first processed label assigned to the first processed data by a first user and outputs calibration data related to calibration of personal characteristics of the label assigned by the first user, and learns a target model based on at least the first processed data and the calibration data or a calibrated label whose personal characteristics have been calibrated using the calibration data; A machine learning device comprising:

2. The first learning unit is applying the first calibration model to the first processed data and the first processed label to generate calibration parameters for the first processed label as the calibration data; applying the calibration parameters to the first processed label to generate the calibrated label; training the object model based on the first processed data and the calibration labels; The machine learning device according to claim 1 .

3. The first learning unit is applying the first calibration model to the first processed data and the first processed label to generate the calibrated label as the calibrated data; training the object model based on the first processed data and the calibration labels; The machine learning device according to claim 1 .

4. The first learning unit is applying the first calibration model to the first processed data and the first processed label to output a confidence level for the first processed label as the calibration data; training the object model based on the first processed data and the first processed label using the confidence measures as parameters of an error function; The machine learning device according to claim 1 .

5. The machine learning device according to claim 1 , further comprising a second learning unit that generates the first calibration model based on first input learning data, a first learning label assigned to the first input learning data by the first user, and the calibration data.

6. The machine learning device according to claim 5 , further comprising an assignment unit that assigns the first learning label to the first input learning data in accordance with an instruction from the first user.

7. The machine learning device according to claim 5 , wherein the first input learning data is medical data generated by a medical device.

8. the first input learning data is an MR image in which measurement voxels of MR spectroscopy by a magnetic resonance imaging apparatus are set, The first learning label is a mark indicating a position of the measurement voxel. The machine learning device according to claim 5.

9. The second learning unit is calculating calibration parameters from the first learned labels to calibrate a personal characteristic of the labeling of the first user; learning the first calibration model based on the first input learning data, the first learning label, and the calibration parameters, which inputs input data and outputs calibration parameters corresponding to the input data as the calibration data; The machine learning device according to claim 5.

10. 6. The machine learning device of claim 5, wherein the second learning unit learns the first calibration model that inputs input data and a label assigned to the input data by the first user, and outputs a correct label for the input data as the calibration data, based on the first input learning data, the first learning label, and a correct label for the first input learning data.

11. The second learning unit is determining a confidence level of the first training label for the first input training data; learning the first calibration model based on the first input learning data, the first learning label, and the reliability, which inputs input data and a label assigned to the input data by the first user, and outputs the reliability of the label for the input data as the calibration data; The machine learning device according to claim 5.

12. 6. The machine learning device according to claim 5, wherein the first learning unit learns the target model based on a combination of the first processed data, the first processed label, and the first calibration model, and a combination of second processed data, a second processed label assigned to the second processed data by a second user, and a second calibration model for calibrating personal characteristics of the label assigned by the second user.

13. the first learning unit learns the target model based on a combination of the first processed data, the first processing label assigned to the first processed data by the first user under a first assignment condition, and the first calibration model, and a combination of the second processed data, the second processing label assigned to the second processed data by the first user under a second assignment condition, and a second calibration model; The first calibration model is a calibration model for calibrating a personal characteristic of the first user's label assignment under a first assignment condition, The second calibration model is a calibration model for calibrating a personal characteristic of the first user's label assignment under a second assignment condition, The machine learning device according to claim 5.

14. The machine learning device according to claim 1 , wherein the target model is a machine learning model that is trained to input input process data and output prediction data corresponding to the input process data.

15. 6. The machine learning device of claim 5, wherein the second learning unit copies learning parameters of a learned calibration model corresponding to another user different from the first user to an unlearned first calibration model, and trains learning parameters of the unlearned first calibration model based on the first input learning data, the first learning label, and the calibration data to learn the first calibration model.

16. 2. The machine learning device according to claim 1, wherein the first calibration model includes a plurality of calibration models each corresponding to a plurality of users different from the first user, and an addition layer that weights and adds a plurality of calibration data from the plurality of calibration models according to weights trained for calibrating the labeling of the first user, and outputs the calibration data.

17. using a calibration model that inputs processed data and a processed label assigned by a user to the processed data and outputs calibration data regarding calibration of personal characteristics of the label assigned by the user, and learning a target model based on at least the processed data and the calibration data or the calibration label whose personal characteristics have been calibrated using the calibration data; A machine learning method comprising:

18. On the computer, a function of using a calibration model that inputs processed data and a processed label assigned by a user to the processed data and outputs calibration data regarding calibration of personal characteristics of the label assigned by the user, and training a target model based on at least the processed data and the calibration data or the calibration label whose personal characteristics have been calibrated using the calibration data; A machine learning program to achieve this.

19. an inference unit that uses a calibration model that inputs processed data and a processing label assigned by a user to the processed data and outputs calibration data regarding calibration of personal characteristics of the label assigned by the user, and performs inference using a model trained based on at least the processed data and the calibration data or a calibration label whose personal characteristics have been calibrated using the calibration data; An inference device comprising:

Images