Information processing device and information processing method

Abstract

To improve the visibility of the sight line direction in a medical image.SOLUTION: An information processing device according to an embodiment comprises: an acquisition unit; and a generation unit. The acquisition unit acquires a first model and a second model being any of: the first model executing a first task and the second model executing a second task different from the first task; the first model trained on the basis of first training data about a first domain and the second model trained on the basis of second training data about a second domain different from the first domain; and the first model executing the first task and trained on the basis of the first training data and the second model executing the second task and trained on the basis of the second training data. The generation unit generates a third model corresponding to both of the first model and the second model on the basis of difference information about the first model and the second model.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] FIELD Embodiments described herein relate generally to information processing devices and methods, such as devices for training models for successive learning of different tasks and / or domains. [Background technology]

[0002] It is known to train neural networks or other machine learning models on multiple different tasks sequentially, and such sequential learning may be considered a type of continuous learning.

[0003] In some situations, a model needs to learn a series of tasks sequentially, but the model is not allowed to store the raw data: for example, the model may be trained on medical data to which access is restricted.

[0004] In one continuous learning scenario, the model is trained using incremental class learning, which comprises a protocol in which the model incrementally learns to recognize new classes from a data distribution that is still similar to the original data distribution.

[0005] Since the task identity is always given, it is possible to train a model with task-specific components. A typical neural network architecture used in this scenario has a multi-head output layer, which means that each task has its own output, but the rest of the network will be shared between tasks.

[0006] In one example of incremental class learning, a model is trained to perform multiple classification tasks, each with a separate segmentation, and the model sequentially learns to recognize new classes.

[0007] A first task of the plurality of classification tasks comprises calcium segmentation. A second task comprises nodule detection. When a model is trained to perform the nodule detection task, the model must remember the calcium segmentation task. A third task comprises lung segmentation. When a model is trained to perform the lung segmentation task, the model must remember the calcium segmentation task and the nodule detection task. A fourth task comprises landmark detection. When a model is trained to perform the landmark detection task, the model must remember the calcium segmentation task, the nodule detection task, and the lung segmentation task.

[0008] The model needs to be able to both solve each previous task and reason about which task is being presented.

[0009] Another type of continuous learning is incremental domain learning. An incremental domain learning protocol may involve protocols in which the configuration of the task to be performed is always the same, but the input distribution changes. For example, the input distribution will change if a different scanner is used or if the task is performed on a different population. In domain incremental learning, subsequent domains may be related to but different from the first domain, for example, the same anatomical structure but a different source.

[0010] A model is trained to perform a task with a first input distribution, and then the model is trained to perform the task with a further, different input distribution. Models trained with different input distributions are sometimes referred to as being trained in different domains.

[0011] Typically, task identity is not available at test time: the model only needs to be able to solve the task at hand, and is not required to infer what task it is.

[0012] Figure 1 shows an example of training with different input distributions. A model is trained to perform stroke detection using a set of training data that includes images 12 acquired at a first institution in Glasgow. The images 12 acquired at the Glasgow institution have a first input distribution, shown in Figure 1 as a histogram 11. The histogram may, for example, represent intensity values ​​measured across all of the images 12 acquired at the Glasgow institution. In one example, the measured intensity values ​​may represent CT numbers in Hounsfield units.

[0013] The model is then trained to perform stroke detection using a set of training data that includes images 14 obtained at a second institution in Japan. The training data obtained at the Japanese institution has a different input distribution than the training data obtained at the Glasgow institution. This different input distribution is shown in FIG. 1 as histogram 13. Histogram 13 might, for example, represent intensity values ​​measured across all of the images 14 obtained at the Japanese institution.

[0014] The model is further trained to perform stroke detection using a set of training data that includes images 16 obtained at a third institution in the United States. The training data obtained at the United States institution has a different input distribution than the training data obtained at the Glasgow institution and the training data obtained at the Japanese institution. This different input distribution is shown in FIG. 1 as histogram 15. Histogram 15 may, for example, represent intensity values ​​measured across all of the images 16 obtained at the United States institution. Histogram 15 differs from histogram 13, which represents the input distribution of the training data from Japan, and from histogram 11, which represents the input distribution of the training data from Glasgow.

[0015] It has been found that neural network models trained sequentially on different tasks or domains may perform worse in the first task or domain when the model is later trained on a second, different task or domain. The models may suffer from catastrophic forgetting. Catastrophic forgetting may include forgetting how to perform a first task when the model is later trained to perform a second, different task. Catastrophic forgetting may include forgetting how to perform a task in a first domain (e.g., with a first input distribution) when the model is later trained on data from the second domain (e.g., with a second input distribution).

[0016] Figure 2 shows an example of catastrophic forgetting. Figure 2 plots the Dice score for the task. The Dice score is a measure of similarity, measuring how well the model's results match the ground truth results. The Dice score is used as a metric of model performance.

[0017] In a continuous incremental class learning setup, tasks are incrementally added to the model. In the example shown in Figure 2, a global distillation continuous learning approach is used. Global distillation differs from previous distillation methods in that they use knowledge from a trained model in a task-aware manner rather than a global manner, which makes it impossible to distinguish classes across different tasks.

[0018] In the first training period, denoted by 1 on the horizontal axis in Figure 2, the model is trained to segment the left lung. The Dice score is used as a measure of segmentation performance. After the first training period, the model achieves a Dice score of 0.924 for segmenting the left lung.

[0019] In the second training period, denoted by 2 on the horizontal axis of Figure 2, the same model is trained to segment the right lung. After the second training period, the model achieves a Dice score of 0.933 for segmenting the left lung and a Dice score of 0.892 for segmenting the right lung.

[0020] In the third training period, denoted by 3 on the horizontal axis of Figure 2, the same model is trained to segment the spinal cord. After the third training period, the model achieves a Dice score of 0.932 for the left lung segmentation, a Dice score of 0.889 for the right lung segmentation, and a Dice score of 0.835 for the spinal cord segmentation.

[0021] In the fourth training period, denoted by 4 on the horizontal axis of Figure 2, the same model is trained to segment the trachea. After the fourth training period, the model achieves a Dice score of 0.932 for the left lung segmentation, a Dice score of 0.896 for the right lung segmentation, a Dice score of 0.835 for the spinal cord segmentation, and a Dice score of 0.863 for the trachea segmentation.

[0022] In the fifth training period, denoted by 5 on the horizontal axis of Figure 2, the same model is trained to segment the esophagus. After the fifth training period, the model achieves a Dice score of 0.932 for the left lung segmentation, a Dice score of 0.892 for the right lung segmentation, a Dice score of 0.827 for the spinal cord segmentation, a Dice score of 0.841 for the trachea segmentation, and a Dice score of 0.463 for the esophagus segmentation.

[0023] For example, when training the right lung after the left lung, assume that the new class has overlapping features with the previous class. When learning to segment the esophagus, there will be a large gap in the representation that the model needs to learn to perform the new segmentation task.

[0024] In the sixth training period, denoted by 6 on the horizontal axis in Figure 2, the model is trained to segment the lesion class, ground glass opacity. After the sixth training period, the model achieves a Dice score of 0.912 for left lung segmentation, a Dice score of 0.87 for right lung segmentation, and a Dice score of 0.791 for trachea segmentation. The model achieves a Dice score of 0.8 for segmenting the new ground glass class. However, the Dice scores drop to zero for both the esophagus segmentation task and the spinal cord segmentation task. The model has forgotten how to segment the esophagus and spinal cord.

[0025] The drop to zero is highlighted in FIG.

[0026] In the seventh training period, indicated by 7 on the horizontal axis in Figure 2, the model is trained to segment consolidations that refer to fluid within the lungs. After the seventh training period, the model achieves a Dice score of 0.907 for the left lung segmentation, a Dice score of 0.856 for the right lung segmentation, a Dice score of 0.758 for the trachea segmentation, and a Dice score of 0.776 for the ground-glass segmentation. The model achieves a Dice score of 0.293 for the consolidation segmentation. The Dice scores remain 0 for the spinal cord segmentation and the esophagus segmentation.

[0027] In Figure 2, we can see that the global distillation continuous learning approach worked well until task 6, the ground glass class, was added. As a lesion class, the ground glass class would be considered very different from the previous anatomy classes. This would be hypothesized to cause catastrophic forgetting of the spinal cord and esophagus classes.

[0028] Some continuous learning approaches may mitigate the effects of catastrophic forgetting. However, as more classes are learned, the degree of similarity between tasks or domains and the relative difficulty of the trained tasks will affect the model's performance. Existing continuous learning approaches may not be robust for learning new classes with overlapping features with previously learned classes. Existing continuous learning approaches may not be robust for learning new domains where the data distribution in the new domain is very different from the previous domain. [Prior art documents] [Patent documents]

[0029] [Patent Document 1] U.S. Patent Application Publication No. 2020 / 050943 Summary of the Invention [Problem to be solved by the invention]

[0030] One of the problems that the embodiments disclosed in this specification and the drawings aim to solve is to suppress a decrease in the accuracy of the distillation model during continuous learning. However, the problems that the embodiments disclosed in this specification and the drawings aim to solve are not limited to the above problem. Problems corresponding to each configuration shown in the embodiments described below can also be positioned as other problems. [Means for solving the problem]

[0031] An information processing apparatus according to an embodiment includes an acquisition unit and a generation unit. The acquisition unit acquires a first model for executing a first task, a second model for executing a second task different from the first task, a first model trained based on first training data related to a first domain, a second model trained based on second training data related to a second domain different from the first domain, and a first model for executing the first task and trained based on the first training data, and a second model for executing the second task and trained based on the second training data. The generation unit generates a third model corresponding to both the first model and the second model based on difference information related to the first model and the second model. [Brief explanation of the drawings]

[0032] Embodiments will now be described by way of example and not limitation and are illustrated in the following figures: [Figure 1] Figure 1 is a schematic diagram of different input distributions. [Figure 2] Figure 2 is a plot of the performance of the global distillation continuous learning model. [Figure 3] FIG. 3 is a schematic diagram of an apparatus according to an embodiment. [Figure 4A] FIG. 4A is a schematic diagram of the training of the first model. [Figure 4B] FIG. 4B is a schematic diagram of the training of the second model. [Figure 4C] FIG. 4C is a schematic diagram of training a third model according to an embodiment. [Figure 5] FIG. 5 is a flow chart that schematically illustrates a method according to an embodiment. [Figure 6] FIG. 6 is a schematic diagram of a method for incremental class learning according to an embodiment. [Figure 7]FIG. 7 is a schematic diagram of a method for incremental domain learning according to an embodiment. [Figure 8] FIG. 8 is a diagram of a user interface according to an embodiment. [Figure 9A] FIG. 9A is an intensity frequency histogram of the left lung. [Figure 9B] FIG. 9B is an intensity frequency histogram of the trachea. [Figure 9C] FIG. 9C is an intensity frequency histogram of the esophagus. [Figure 9D] FIG. 9D is an intensity frequency histogram of the right lung. [Figure 9E] FIG. 9E is an intensity frequency histogram of the spinal cord. [Figure 9F] FIG. 9F is an intensity frequency histogram of ground glass. [Figure 9G] FIG. 9G is an intensity frequency histogram of consolidations. [Figure 10] Figure 10 is a plot showing the distance between model weights for models trained on various classes of chest CT. DETAILED DESCRIPTION OF THE INVENTION

[0033] One embodiment provides an apparatus (information processing apparatus) including a processing circuit. The processing circuit is configured to receive a first model and a second model. The first model is trained to perform a first task, the first model being trained with first training data for a first domain, and the second model being trained with different second training data, and at least one of a) and b) is configured: a) the second model is trained to perform a second task different from the first task, and b) the second training data is data for a second domain different from the first domain. The processing circuit is configured to determine difference information representing differences between the first model and the second model, and / or between the first task and the second task, and / or between the first domain and the second domain, and to create a third model using the first model, the second model, and the difference information. The creation of the third model includes training the third model to perform both the first task and the second task and / or to function in both the first domain and the second domain.

[0034] One embodiment provides a method including: receiving a first model and a second model; the first model is trained to perform a first task, the first model being trained with first training data for a first domain, and the second model being trained with different second training data, and at least one of a) and b) where a) the second model is trained to perform a second task different from the first task, and b) the second training data is data for a second domain different from the first domain; determining difference information representing differences between the first model and the second model, and / or between the first task and the second task, and / or between the first domain and the second domain; and creating a third model using the first model, the second model, and the difference information, wherein the creating of the third model includes training the third model to perform both the first task and the second task and / or to function in both the first domain and the second domain.

[0035] An apparatus 20 according to an embodiment is shown schematically in FIG. 3. The apparatus 20 is configured to obtain a trained first model and a trained second model, and to train a third model using the trained first model and the trained second model. In this embodiment, the apparatus 20 is also configured to apply the third model once it has been trained. In other embodiments, the third model may be trained by a first apparatus and used by a different second apparatus. Here, the apparatus 20 is an example of an information processing apparatus.

[0036] The device 20 comprises a computing device 22, which in this example is a personal computer (PC) or workstation. The device 20 is connected to at least one scanner 24 via a data store 30. The device 20 is also connected to one or more display screens 26 and one or more input devices 28, such as a computer keyboard, mouse, trackball, etc.

[0037] At least one scanner 24 may comprise any scanner configured to perform medical imaging. The or each scanner 24 is configured to generate medical image data, which may include two-dimensional, three-dimensional, or four-dimensional data for any imaging modality. For example, scanner 24 may comprise a magnetic resonance (MR or magnetic resonance imaging) scanner, a computed tomography (CT) scanner, a cone-beam CT scanner, an x-ray scanner, an ultrasound scanner, a positron emission tomography (PET) scanner, or a single photon emission computed tomography (SPECT) scanner.

[0038] Image data sets obtained by the at least one scanner 24 are stored in a data store 30 and then provided to the computing device 22 .

[0039] In other embodiments, the device 20 and data store 30 are not connected to any scanner 24 .

[0040] In this embodiment, data store 30 stores multiple training image datasets and associated ground truth data. The ground truth data may be obtained by manual annotation of the training image datasets by an expert, or by any suitable automated or semi-automated method. Data store 30 also stores at least one unannotated image dataset.

[0041] All of the training image dataset may not be available at the same time, and access to the training image dataset may be limited.

[0042] In alternative embodiments, the training image data set and / or other image data sets are sourced from one or more further data stores (not shown), which may form a Picture Archiving and Communication System (PACS). Data store 30 or the further data stores may comprise any suitable form of memory storage.

[0043] In further embodiments, the training dataset and / or the dataset to which the trained model is applied may not comprise an image dataset, for example, the dataset may comprise clinical data such as vital sign data, lab value data, or text data.

[0044] The computing device 22 includes a processing unit 32 for processing data, including image data. The processing unit includes a central processing unit (CPU) and a graphical processing unit (GPU). In other embodiments, the processing unit may include a CPU without a GPU.

[0045] The processing unit 32 comprises a differencing circuit 34 configured to obtain differences between models, tasks, and / or domains, a model training circuit 36 ​​configured to train a third model based on the trained first and second models, and an inference circuit 38 configured to apply the trained model to unseen data to obtain model outputs. The processing unit 32 may further comprise a user interface circuit 39 configured to accept input via a user interface and / or display output via the user interface.

[0046] In this embodiment, the circuits 34, 36, 38, and 39 are each implemented in a CPU and / or GPU by a computer program having computer-readable instructions that are executable to perform the method of the embodiment. In other embodiments, the various circuits may be implemented as one or more Application Specific Integrated Circuits (ASICs) or Field Programmable Gate Arrays (FPGAs).

[0047] Computing device 22 also has a hard drive and other components of a PC, including RAM, ROM, a data bus, an operating system including various device drivers, and hardware devices including a graphics card, although such components are not shown in FIG.

[0048] The apparatus of Figure 3 is configured to carry out the method illustrated generally in Figures 4A-4C.

[0049] 4A illustrates training a first model 42 using a first set of training data 40 that includes a plurality of first training image datasets, the first set of training data relating to a first task and including training image datasets labeled with ground truth data relating to the first task.

[0050] In this embodiment, the model training circuitry 36 is configured to train the first model 42 based on the first set of training data 40. In other embodiments, the first model 42 is trained by a different device and received by the model training circuitry 36 as a trained model. In such embodiments, the model training circuitry 36 would not have access to the first set of training data 40. In other embodiments, the model training circuitry 36 may have time-limited access to the first set of training data 40.

[0051] 4B illustrates training a second model 52 using a second set of training data 50 that includes a second plurality of training image datasets. In this embodiment, the trained first model 42 is used as a starting point for training the second model 52. The second model 52 is trained to perform a second task that is different from the first task for which the first model 42 was trained. The second set of training data relates to the second task and includes a training image dataset labeled with ground truth data for the second task.

[0052] The first model 42 is trained only to perform the first task, not the second task. The second model is not specifically trained to perform the first task. The trained first model 42 is used as a starting point to train the second model 52, so that the second model 52 can perform both the first task and the second task.

[0053] In other embodiments, the first model 42 and the second model 52 are each trained from the same starting point, e.g., from the same untrained model. The first model 42 is trained only to perform the first task and not to perform the second task. The second model 52 is trained only to perform the second task and not to perform the first task.

[0054] In a further embodiment, both the first model 42 and the second model 52 are trained to perform a first task, but the second model 52 is trained in a different domain, e.g., using a different input distribution, than the first model. The first model is trained on a first set of training data, the first set of training data relating to the first task and including a training image dataset labeled with ground truth data relating to the first task. The second model is trained on a second set of training data, the second set of training data relating to the first task and including a training image dataset labeled with ground truth data relating to the first task. However, the second set of training data relates to a different domain than the first set of training data. For example, the second set of training data may be obtained from a different scanner than the scanner from which the first set of training data was obtained. The second set of training data may represent a different population than the first set of training data.

[0055] In further embodiments, the second model may be trained on a different task and in a different domain than the first model.

[0056] In this embodiment, model training circuitry 36 is configured to train second model 52 based on a second set of training data 50. In other embodiments, second model 52 is trained by a different device and received as a trained model by model training circuitry 36. Model training circuitry 36 accesses the second set of training data 50.

[0057] FIG. 4C illustrates the training of a third model 60 by the model training circuitry 36. The third model 60 is a distillation model. A distillation model is a model that is trained to perform the same task as the initial model or models used when the distillation model is trained. A distillation model may also be referred to as a distillation model or a student model. Typically, a distillation model is smaller and / or less complex than the initial model or models. A distillation model may be trained using distillation loss, which uses a comparison of the output of an initial model with the output of the distillation model to train the distillation model to produce an output that is closer to the output of the initial model.

[0058] In this embodiment, the Distill model (third model 60) is trained to perform a first task and a second task. In other embodiments, the Distill model (third model 60) is trained to perform the first task in a first domain and a different second domain.

[0059] In this embodiment, the first model 42, the second model 52, and the third model 60 are all neural network models. In other embodiments, any suitable type of model may be used. The first model 42, the second model 52, and the third model 60 may be any type of model that learns features from data as part of model training in a supervised manner. For example, supervised dictionary learning may be used.

[0060] The model training circuit 36 ​​trains a distilled model (third model 60) with the second set of training data 50 and predictions from the first trained model 42 and the second trained model 52. The predictions from the first trained model 42 and the second trained model 52 may be considered to encapsulate information about both the first set of training data 40 and the second set of training data 50.

[0061] In this embodiment, the model training circuitry 36 only accesses the second set of training data 50 when training the Distill model (third model 60). It is known that accessing only the second set of training data 50 can bias the training of the Distill model (third model 60) toward the second task. If the first and second models are trained in different domains to perform the same task, accessing only the second set of training data 50 could bias the task toward the second domain.

[0062] Therefore, the model training circuit 36 ​​trains the distill model (third model 60) using a regularization term chosen to prevent the distill model (third model 60) from being strongly biased towards the second task (or second domain). The regularization term applies a loss to the features extracted by the model.

[0063] Using the regularization term, the model training circuit 36 ​​applies a regularization scheme to the training of the distilled model (third model 60). The regularization scheme incorporates the difference between the first and second tasks (or between the first and second domains). The determination and use of the difference between the first and second tasks (or domains) is described in more detail below.

[0064] The use of such a regularization scheme may improve performance preservation on previously learned tasks or domains compared to training without the regularization scheme. The use of a regularization scheme may maintain a similar level of performance on new tasks or domains compared to training without the regularization scheme.

[0065] For example, a distilled model (third model 60) may maintain similar performance to first model 42 for the first task and similar performance to second model 52 for the second task, despite being trained only on the second set of training data 50 and not on the first set of training data 40.

[0066] In this embodiment, model training circuitry 36 does not access the first set of training data 40 when training the distill model (third model 60). In other embodiments, model training circuitry 36 may access some or all of the first set of training data 40 when training the distill model (third model 60).

[0067] FIG. 5 is a flow chart that schematically illustrates a method of an embodiment, which may be the same as the embodiment described above with reference to FIG. 4C.

[0068] At stage 70, the device 20 obtains a trained first model 42 and a trained second model 52 that have been trained as described above with reference to FIGS. 4A and 4B.

[0069] At stage 72, the difference circuit 34 determines difference information for the trained first model 42 and the trained second model 52. The difference information represents the degree of similarity or difference between the trained first model 42 and the trained second model 52.

[0070] The differential information includes at least one differential measure, which may also be considered a similarity measure, and vice versa.

[0071] In this embodiment, the difference information is obtained by comparing the first trained model 42 and the second trained model 52. In another embodiment, the difference information is obtained by comparing information about a first task for training the first trained model 42 and a second task for training the second trained model 52. Significant differences in the tasks for training the models would be expected to result in differences in the models. In a further embodiment, the difference information is obtained by comparing information about a first domain for which the first trained model 42 is trained and a second domain for which the second trained model 52 is trained. Significant differences in the domains for which the models are trained would be expected to result in differences in the models.

[0072] In this embodiment, the first trained model 42 and the second trained model 52 are neural networks constructed in two parts. The first part of the network, called the body of the network, is constructed to perform feature extraction from the data. The first part comprises a first plurality of network layers. The second part of the network, called the head of the network, is constructed to combine the extracted features to obtain the final output. The second part comprises a second plurality of network layers. In other embodiments, feature extraction may be performed by any suitable supervised learning method, for example, by supervised learning, followed by feature combination.

[0073] In this embodiment, the difference measure calculated by differencing circuit 34 is the difference between the feature extraction layer parameters of trained first model 42 and the feature extraction layer parameters of trained second model 52. The feature extraction layer parameters include a subset of all model parameters. The feature extraction layer parameters include model parameters for a first portion of the network that performs feature extraction. In this embodiment, the difference determined is the L1 distance between the feature extraction layer parameters.

[0074] θ are all the model parameters,

number

number

number

number

[0075] In other embodiments, any suitable method may be used to calculate the distance between the models to be used as a measure of difference.

[0076] In some embodiments, the difference measure calculated by difference circuit 34 is a difference in the distribution of task intensities. For example, a first model may be configured to perform a first task that includes segmenting pixels or voxels having a first set of intensity values, such as Hounsfield values. A second model may be configured to perform a second task that includes segmenting pixels or voxels having a second set of intensity values. If the first set of intensity values ​​and the second set of intensity values ​​are similar, the tasks may be considered to have similar task intensities. Therefore, the first model and the second model may be considered to have similar task intensities.

[0077] In incremental class learning scenarios where the first and second tasks are different, a large overlap between task intensities is likely to require regularization. In incremental domain learning scenarios where the first and second domains are different, a large overlap between task intensities is likely to not require regularization.

[0078] In further embodiments, the difference measure calculated by the difference circuit 34 includes a learned distance metric. For example, the learned distance metric may be calculated using a siamese net or triplet net with a symmetric loss. A siamese net or triplet net is a neural network approach that learns to embed input data so as to minimize the difference between two data samples in the same category and maximize the difference between data in different categories. In some embodiments, the siamese net or triplet net is trained to generate embeddings that separate data from different patient populations. Given data from two different populations, the siamese net or triplet net is expected to produce outputs that are very different from each other, indicating a large distance between these data. This large distance may be used to indicate the need for regularization.

[0079] In a further embodiment, the differential measure may be provided by a user. In such an embodiment, the user provides user input using input device 28 of device 20. User interface circuitry 39 processes the user input to obtain the differential measure. Such an embodiment is described below with reference to FIG. 8.

[0080] At stage 74, differencing circuit 34 compares the determined difference measure to a threshold. In this embodiment, if the difference measure is greater than the threshold, differencing circuit 34 determines that there is a significant difference between the first and second models. If the difference measure is not greater than the threshold, differencing circuit 34 determines that there is not a significant difference between the first and second models.

[0081] At stage 76 , the model training circuit 36 ​​determines a loss function for training the third model 60 .

[0082] In this embodiment, if the difference measure is determined to be greater than the threshold at stage 74, the loss function includes a regularization term. If the difference measure is not determined to be greater than the threshold at stage 74, the loss function does not include a regularization term. The models are deemed sufficiently similar that a regularization term is not required.

[0083] In other embodiments, the regularization term may be included in all cases. In further embodiments, the regularization term may be used when there is a small difference measure. For example, in an embodiment where the difference measure is the difference between task intensities, the regularization term may be used when there is a small difference between the task intensity of a first task and the task intensity of a different second task.

[0084] The regularization term in the loss function is intended to force the output of the feature extraction layer of the third model 60, which is a distilled model, to approach the output of the feature extraction layer of the first model 42 or the output of the feature extraction layer of the second model 52.

[0085] A loss function with a regularization term may be written as:

number

[0086] L Distill is the distillation loss term. The distillation loss term compares the output of the distillation model (third model 60) to the first model 42 and the second model 52. The distillation loss is used to regress the third model output vs. the predictions of the first and second models. The distillation loss is applied to the soft output.

[0087] The soft output may be a probability output representing the individual probability of each label being classified by the model, and the hard output may be the final classification obtained by thresholding, normalizing, or other processing of the soft output.

[0088] The term logits is used to refer to the raw outputs of a network before normalization or processing to an appropriate range, for example, to obtain output values ​​between 0 and 1. Logits may be considered to be soft outputs. Hard outputs may typically be calculated by applying a softmax, sigmoid, or threshold function to the logits.

[0089] In this embodiment, the distillation loss is a combination of the difference between the logits output by the distillation model (third model 60) and the logits output by the first model 42 and the difference between the logits output by the distillation model (third model 60) and the logits output by the second model 52.

[0090] In other embodiments, any suitable distillation loss term may be used.

[0091] (L DistilVsFirst +L DistilVsSecond ) is the regularization term. L DistilVsFirst compares the output of the feature extraction layer of the distilled model (third model 60) with the output of the feature extraction layer of the first model 42. The outputs are compared by passing the input data through the feature extraction layer of the distilled model (third model 60) and the feature extraction layer of the first model 42. Comparing the outputs of the feature extraction layers rather than comparing model parameters may allow for more flexibility in training.

[0092] The outputs compared in the regularization term are not the final outputs of the distilled model (third model 60) and the first model 42. The outputs compared in the regularization term are not the same outputs used to calculate the distillation loss. The outputs compared in the regularization term are the outputs of the feature extraction layer that forms the first part of the neural network. The outputs compared in the distillation loss are the soft outputs from the second part of the neural network, where the features are combined.

[0093] Similarly, L DistilVsSecondcompares the output of the feature extraction layer of the distilled model (third model 60) with the output of the feature extraction layer of the second model 52.

[0094] w is a weighting factor. In this embodiment, L DistilVsFirst and L DistilVsSecond In another embodiment, the same weighting is used for L DistilVsFirst and L DistilVsSecond A different weighting may be used for each of

[0095] In this embodiment, the weighting is expressed as follows:

number

[0096] In Equation 4, θ is used to refer to a subset of the model parameters of the feature extraction layer. first is a set of model parameter weights for each node in the feature extraction layer of the first model 42, and φ second is a set of model parameter weights for each node in the feature extraction layer of the second model 52, and φ distill_model is the current set of model parameter weights for each node in the feature extraction layer of the Distill model (third model 60), and φ distill_model_start is the starting set of model parameter weights for each node in the feature extraction layer of the Distill model (third model 60). first , φ second , φ distill_model The value of φ changes during training as the model is updated. distill_model_start is a static term. The difference is calculated in the same way as the model difference described above, for example using the L1 distance.

[0097] The weighting used in this embodiment is decaying weighting, in which the weight of the loss regularization term decreases as the difference in parameters between the distilled model (third model 60) and the first and second models 42, 52 decreases.

[0098] In other embodiments, any suitable weighting may be used. In some embodiments, the regularization term weighting is learned. In further embodiments, the regularization term weighting is manually set by a user.

[0099] If the determined difference measure does not exceed the threshold at stage 74, no regularization term is used and the loss is simply the distilled loss L Distill is.

[0100] In stage 78, the model training circuit 36 ​​trains the distill model (third model 60) using the loss determined in stage 76. Any suitable model training method may be used, such as training using stochastic gradient descent. By using a regularization term in addition to the distillation loss in training the distill model, the distill model (third model 60) may achieve a better balance of similarity to the first model 42 and similarity to the second model 52, despite only having access to the second set of training data 50. The use of a regularization term may reduce bias toward the second model 52 and / or the second set of training data 50.

[0101] The trained Distill model (third model 60) provides a single model that performs both the tasks of the first and second models. All of the training data for training the first task need not be available at the same time as the data for training the second task. Instead, the second model and Distill model (third model 60) may be trained using the second set of training data without access to the first set of training data.

[0102] At stage 79, the inference circuitry 38 applies the trained distil model (third model 60) to new image data to perform one or both of a first task and a second task. For example, the input to the trained distil model (third model 60) may include image data obtained from a patient scan. The trained distil model (third model 60) may process the image data to obtain a first output for the first task and / or a second output for the second task. For example, the first task may be segmentation of a first anatomical feature within the image data, and the second task may be segmentation of a different second anatomical feature within the image data.

[0103] Figure 6 illustrates a method for incremental class learning according to an embodiment. In the embodiment of Figure 6, a set of training data 80 is available to model training circuitry 36. The set of training data 80 has been annotated with ground truth labels relevant to the current task. In the example of Figure 6, the current task is segmentation of the right lung.

[0104] The model training circuit 36 ​​inputs the training data 80 into a previous model 82. The previous model 82 has been previously trained to perform at least one different task. In the embodiment of Figure 6, the at least one different task includes segmentation of the left lung, where the previous model 82 is an example of a first model.

[0105] The model training circuitry 36 does not have access to the training data that trained the previous model 82 .

[0106] The model training circuit 36 ​​also inputs the training data 80 to a current model 84. The current model 84 may be referred to as a current task teacher model. The current model 84 was trained to perform the current task of segmenting the right lung. Here, the current model 84 is an example of a second model.

[0107] The model training circuit 36 ​​performs a global distillation process to obtain a new model 86, which is a distillation model. To perform the global distillation process, the model training circuit 36 ​​inputs training data 80 into an initial distillation model. The model training circuit 36 ​​adjusts the initial distillation model using a loss function corresponding to the loss function in Equation 3. Here, model 86 is an example of a third model.

number

[0108] L Distil is the distillation loss comparing the output of the Distiller Model 86 with the previous Model 82 and the current Model 84. DistilVsPast compares the output of the feature extraction layer of the distilled model 86 with the output of the feature extraction layer of the previous model 82. DistilVsCurrent compares the output of the feature extraction layer of the distilled model 86 with the output of the feature extraction layer of the current model 84.

[0109] In other embodiments, the previous model may be trained on any suitable number of previous tasks. The process described above with reference to FIG. 6 may be repeated. When a new task is added, the distill model 86 becomes the previous model. A new current model is trained on a new set of current data to perform the new task. The new current model may be trained from scratch, or using the previous model or any suitable model as a starting point. The distill model 86 and the new current model are then used as inputs to train a further distill model. The training of the new model may be repeated on any suitable number of tasks and any suitable number of training datasets.

[0110] In one example, a model was trained on a first set of training data for esophagus segmentation, and the model was separately trained on a second set of training data for consolidation segmentation without access to the first data set or the esophagus segmentation.

[0111] We found that the Distill model without regularization exhibited a degradation in task performance, even when only two tasks were included in the Distill model. Introducing regularization helped preserve performance on the esophagus segmentation task.

[0112] If both the first set of training data and the second set of training data were available to the model simultaneously, we could train a multi-task model for both tasks. This might be considered an ideal scenario. However, in reality, data is often available sequentially and access may be limited. Regularization can be used to improve the performance of models trained using global distillation.

[0113] Figure 7 illustrates a method for incremental domain learning according to an embodiment. In the embodiment of Figure 7, a set of training data 90 is available to model training circuitry 36. The set of training data 90 has been annotated with ground truth labels relevant to the task. In the example of Figure 7, the task is segmentation of the right lung.

[0114] The model training circuit 36 ​​inputs the training data 90 into a prior model 92, which was previously trained to perform the task in a prior domain, where the prior model 92 is an example of a first model.

[0115] The model training circuitry 36 does not have access to the training data that trained the previous model 92 .

[0116] The model training circuit 36 ​​also inputs the training data 90 to a current model 94. The current model 94 may be referred to as a current task teacher model. The current model 94 has been trained to perform the task in a current domain that is different from the previous domain. Here, the current model 94 is an example of a second model.

[0117] The model training circuit 36 ​​performs a global distillation process to obtain a new model 96, which is a distillation model. To perform the global distillation process, the model training circuit 36 ​​inputs the training data 90 into an initial distillation model. The model training circuit 36 ​​uses the loss function of Equation 5 to tune the initial distillation model. In the embodiment of FIG. 7, the weighting w may be calculated in a different manner than that shown in Equation 6. For example, a different distance metric may be used to calculate w. The distance metric may be selected to measure differences between domains, for example, using the distribution of data values ​​across the entire image instead of a task area. Here, model 96 is an example of a third model.

[0118] In the embodiment of Figure 7, two separate output channels are maintained in the distill model 96, one per domain. Using two separate output channels may facilitate learning weights for the domains. At prediction time, the inference circuitry 38 takes the maximum of the outputs of the two channels to derive the segmentation. The inference circuitry 38 takes the variance between the outputs to derive the uncertainty map.

[0119] In one example, a model was trained to perform spinal cord segmentation. The model was trained on a first set of training data. The model was then separately trained on a second set of training data with a different domain. Thus, there is a domain shift, where the model was trained on the same task but with a different input distribution. Two separate outputs were obtained, one for each domain. At prediction time, the maximum of the outputs is taken to obtain the segmentation. The variance between the outputs is taken to obtain the uncertainty map. Maintaining multiple outputs may make it easier to create the uncertainty map.

[0120] 8 is an example of a user interface for manually inputting difference information. The user is given the opportunity to provide user input to add information about a new task. User interface circuitry 39 processes the user input to obtain difference information. The difference information is used to determine whether to include a regularization term when training a distillation model to perform a new task in addition to one or more existing tasks.

[0121] 8, user interface circuitry 39 displays slice image 100 to the user on display screen 26. User interface circuitry 39 also displays a first set of checkboxes 110, 111, 112, and 113 and a second set of checkboxes 120, 121, 122, and 123.

[0122] A first set of checkboxes 110, 111, 112, 113 are shown associated below the text "The newly added 'Consolidation' class has an anatomically similar location to the one below."

[0123] A checkbox 110 is displayed next to the text "Lungs." In the illustrated example, the user has provided user input by checking the checkbox 110 to indicate that the user believes the new class, Consolidations, has an anatomical location similar to the lungs.

[0124] Checkbox 111 is displayed next to the text "Trachea." Checkbox 112 is displayed next to the text "Spinal Cord." Checkbox 113 is displayed next to the text "Esophagus." In the example shown in FIG. 8, the user has not checked any of checkboxes 111, 112, or 113.

[0125] The user interface circuitry 39 processes the user input and determines that the user believes that the new class of consolidations has an anatomical location similar to the lungs, and determines that the user does not believe that the new class of consolidations has an anatomical location similar to the trachea, spinal cord, or esophagus.

[0126] A second set of checkboxes 120, 121, 122, 123 are shown associated below the text "Has an HU intensity distribution that overlaps with that below."

[0127] Checkbox 120 is displayed next to the text "Lungs." Checkbox 121 is displayed next to the text "Trachea." In the example shown in Figure 8, the user has not checked checkboxes 120 and 121.

[0128] Checkbox 122 is displayed next to the text "Spinal Cord." Checkbox 123 is displayed next to the text "Esophagus." In the illustrated example, the user has provided user input by checking checkboxes 122 and 123 to indicate that the user believes the new class, Consolidation, has an intensity distribution similar to the spinal cord and esophagus.

[0129] The user interface circuitry 39 processes the user input and determines that the user believes the new class of consolidations has an intensity distribution similar to the spinal cord and esophagus, and determines that the user believes the new class of consolidations does not have an intensity distribution similar to the lungs and trachea.

[0130] The difference circuitry 34 uses decisions made by the user interface circuitry 39 based on user input at checkboxes 110, 111, 112, and 113 to determine a measure of difference between the task of segmenting the consolidation and one or more of the tasks of segmenting the lungs, segmenting the trachea, segmenting the spinal cord, and segmenting the esophagus. The difference circuitry 34 uses decisions made by the user interface circuitry 39 based on user input at checkboxes 120, 121, 122, and 123 to determine a measure of difference between the intensity of the consolidation and one or more of the intensity of the lungs, trachea, spinal cord, and esophagus.

[0131] The difference circuit 34 determines whether to use a regularization term in the loss function for training the distilled model (third model 60) based on the determined one or more difference measures. The one or more difference measures are based on user input. In other embodiments, any suitable user input may be used to determine one or more difference measures between tasks, and thereby determine one or more difference measures between models trained to perform those tasks. Any suitable user input may be used to determine one or more difference measures between domains, and thereby determine one or more difference measures between models trained to function in those domains.

[0132] The model training methods described above may be used to extend the performance of an existing model. In one embodiment, a user of scanner technology, such as an institution, adds a new data type. For example, the user may replace an old scanner with a new one. In the continuous learning setup described above, the user would utilize the trained model with data collected from both the new scanner and the trained model. Training the model on data from both the old and new scanner may require only adding a relatively small number of annotations to the new data.

[0133] The model training method described above may be used in federated learning. In a conventional federated learning algorithm, a model (e.g., a neural network) is trained simultaneously on multiple cohorts without data transmission between the cohorts. A collection of data-providing institutions is called a federation. A copy of the model is sent from a central server to each institution in the federation. The copy of the model sent to each institution is trained on the data cohort at that institution for a period of time. After the period, each institution returns the trained model parameters to the central server.

[0134] The continuous learning methods described above may be used to update model weights in-house at any institution in the federation.

[0135] Figures 9A-9G show a set of intensity distributions for inferring differences between models. The intensity distributions in Figures 9A-9G are for various anatomical and lesion classes in chest CT. Each intensity distribution is represented by a respective histogram in which frequency is plotted against intensity in Hounsfield units. Figure 9A shows an intensity frequency histogram for the left lung. Figure 9B shows an intensity frequency histogram for the trachea. Figure 9C shows an intensity frequency histogram for the esophagus. Figure 9D shows an intensity frequency histogram for the right lung. Figure 9E shows an intensity frequency histogram for the spinal cord. Figure 9F shows an intensity frequency histogram for ground glass. Figure 9G shows an intensity frequency histogram for consolidation.

[0136] It can be seen that the various anatomical structures and lesions in Figures 9A-9G each have a distinct intensity distribution. The intensity distribution of the consolidation may be compared to the intensity distribution of other anatomical structures or lesions. It can be seen that the esophagus has the highest HU intensity, which overlaps with the consolidation.

[0137] It may be possible to infer how similar a set of tasks is based on the overlap between the intensities associated with those tasks. For example, esophagus and consolidation are very similar. In some situations, it may be difficult to learn different tasks with similar ranges of intensity values.

[0138] Figure 10 is a plot showing the distance between model weights of SqueezUnet trained on various classes of chest CT images. The classes are consolidation, ground glass, left lung, esophagus, right lung, spine, and trachea. Larger model distances are indicated by darker colors in Figure 10. Consolidation and esophagus have the largest model weight distances. Therefore, there is a large distance between the model trained on consolidation and the model trained on esophagus.

[0139] If the consolidation and esophagus have the highest HU intensity overlap and the largest model weight distance, it may be inferred that regularization may be necessary in the sequential learning of these classes.

[0140] In some embodiments, information about the similarities between tasks is used to select the order in which models are trained on new tasks. In some embodiments, the network is first trained on the task and / or domain where the model parameters are most different. Training on the most different task or domain first will encourage the model to acquire a wide range of learned features early on. These wide range of learned features may then be used later to adapt to new tasks and / or domains.

[0141] In the above embodiment, a new distill model is initialized without attempting to adapt to past models. Training is performed on logits from past and current tasks without correct hard labels required after the previous and current models are trained. A regularization term is introduced to help balance past and current task performance. The method is adapted to incremental domain learning as well as incremental class learning.

[0142] The weight differences between the new distillation model and the past and current models are calculated and applied as weights for regularization based on global distillation loss, without requiring data from previous tasks to be used for training the distillation model or for the regularization.

[0143] The above embodiments have been described with reference to medical image data. In other embodiments, the model training methods described above may be used to train any suitable model. The data used to train the model need not be image data. The data used to train the model need not be medical data. For example, the data used to train the model may be language data for training a language model used in natural language processing. The language data may be image data or text data, similar to medical data.

[0144] In one embodiment, a method for incrementally adding knowledge to a medical decision support system is provided, comprising N machine learning models for N tasks / domains, sequentially available data for the tasks / domains, and a training process that encapsulates similarities / differences between the tasks.

[0145] The distance / similarity between the tasks / domains may be explicitly included in the model training loss function.

[0146] The distance / similarity between the tasks / domains may be assigned by the user.

[0147] The similarity between the domains or the difficulty of distinguishing between the tasks may be determined by the overlap of task intensity.

[0148] The distance / similarity between the tasks / domains may be calculated as the difference between the models.

[0149] The distance / similarity between the tasks / domains may be predicted by an external model.

[0150] The distance / similarity between the tasks / domains may be used to determine the best task / domain on which to train a model.

[0151] In one embodiment, an apparatus is provided that includes a processing circuit that receives a first model and a second model, the first model performing a first task and / or trained with first training data for a first domain, the second model performing a second task different from the first task and / or training with second training data for a second domain different from the first domain and different from the first training data, and the processing circuit generates a third model based on difference information regarding the first and second models, the third model performing both the first task and the second task and / or corresponding to both the first domain and the second domain.

[0152] The difference information may be determined based on a feature extraction layer between the first model and the second model.

[0153] In one embodiment, a method is provided for training a model, the model comprising a first model performing a first task and / or trained with first training data for a first domain, and a second model performing a second task and / or trained with training data for a second domain, the method determining differences between the first task and the second task and / or between the first domain and the second domain, and using the differences when training at least one of the first and second models.

[0154] The difference may be included in the model loss training function used for the training.

[0155] Using the difference in the training may include using the difference in regularization.

[0156] The training may include training the second model, the training including using the trained first model and predictions from the determined differences.

[0157] The method further includes generating a distilled model based on the training and the first and second models, the distilled model including information from training data in both the first and / or second domains and / or information related to the first and second tasks.

[0158] The first task may include segmenting a first anatomical feature or a first lesion, and the second task may include segmenting a second anatomical feature or a first lesion that is different from the first task.

[0159] The first domain and the second domain may relate to different locations and / or different scanners and / or different imaging modalities.

[0160] The difference may be determined based at least in part on input from a user.

[0161] The difference may be determined based on an overlap of task intensities.

[0162] The difference may be calculated as the difference between the models.

[0163] The difference may be determined based on predictions from further models.

[0164] Although particular circuits are described herein, in alternative embodiments, the functionality of one or more of these circuits may be provided by a single processing resource or other component, or the functionality provided by a single circuit may be provided by a combination of two or more processing resources or other components. A reference to a single circuit encompasses multiple components that provide the functionality of that circuit, whether or not such components are separate from one another. A reference to multiple circuits encompasses a single component that provides the functionality of those circuits.

[0165] According to at least one of the embodiments described above, it is possible to suppress a decrease in accuracy of the distilled model during continuous learning.

[0166] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are intended to be included within the scope and spirit of the invention, as well as within the scope of the inventions and their equivalents as defined in the claims. [Explanation of symbols]

[0167] 20 Device (information processing device) 22 Computing Devices 24 Scanner 26 display screens 28 Input Devices 30 Data storage unit 32 Processing equipment 34 Differential circuit 36 Model Training Circuits 38 Inference circuit 39 User Interface Circuit 40,50,80,90 training data 42, 82, 92 models (first model) 52, 84, 94 models (2nd model) 60, 86, 96 models (3rd model)

Claims

1. a first model that executes a first task, and a second model that executes a second task different from the first task; a first model trained on first training data relating to a first domain, and a second model trained on second training data relating to a second domain different from the first domain, the second training data being different from the first training data; a first model for performing the first task, the first model being trained based on the first training data; and a second model for performing the second task, the second model being trained based on the second training data. an acquisition unit that acquires the first model and the second model, which are any of the following: a generator that generates a third model that executes both the first task and the second task and / or functions in both the first domain and the second domain based on difference information between the first model and the second model; An information processing device comprising:

2. The information processing apparatus according to claim 1 , wherein the difference information is determined based on feature extraction layers of the first model and the second model.

3. The information processing apparatus according to claim 1 , wherein the difference information is determined based on a task or a domain processed by the first model or the second model.

4. The information processing apparatus of claim 1 , wherein the difference information is determined based on a learned distance metric.

5. The information processing apparatus according to claim 1 , wherein the difference information is determined based on a user input.

6. The information processing device according to claim 5 , wherein the user input includes information regarding anatomical similarity or intensity distribution.

7. The information processing device according to claim 1 , wherein the difference information is determined based on predictions by a further trained model.

8. The information processing device according to claim 1 , wherein the third model is a distilled model.

9. The information processing device according to claim 1 , wherein the third model is a model trained using a loss function that includes the difference information or is derived from the difference information.

10. 10. The information processing device according to claim 9, wherein the loss function includes a regularization term that balances a difference between the first model and the third model and a difference between the second model and the third model so as to suppress a bias toward either the first task or the second task and / or a bias toward either the first domain or the second domain in training of the third model.

11. 11. The information processing device according to claim 9, wherein the loss function further includes a distillation loss term based on a difference between an output of the first model and an output of the third model, and a difference between an output of the second model and an output of the third model.

12. The information processing device according to claim 10 , wherein the regularization term includes weighting based on model parameters of the first model, the second model, and the third model.

13. The information processing apparatus according to claim 12 , wherein the weighting is a decay weighting according to the similarity between the third model and the first model and between the third model and the second model.

14. The information processing device of claim 1 , wherein the third model is trained using the second training data.

15. The information processing device of claim 1 , wherein the third model is trained without access to the first training data.

16. the second model is trained to perform the second task, which is different from the first task; the second model is not trained to perform the first task; The information processing device according to any one of claims 1 to 15.

17. The information processing apparatus of claim 1 , wherein the first training data and the second training data each include medical imaging data.

18. the first task includes segmenting a first anatomical feature or a first lesion; the second task includes segmenting a second anatomical feature or a second lesion different from the first task; The information processing device according to any one of claims 1 to 17.

19. The information processing device according to claim 1 , wherein the first domain and the second domain differ in at least one of location, scanner, and imaging diagnostic method.

20. a first model that executes a first task, and a second model that executes a second task different from the first task; a first model trained on first training data relating to a first domain, and a second model trained on second training data relating to a second domain different from the first domain, the second training data being different from the first training data; a first model for performing the first task, the first model being trained based on the first training data; and a second model for performing the second task, the second model being trained based on the second training data. obtaining the first model and the second model, each of which is one of: generating a third model that performs both the first task and the second task and / or functions in both the first domain and the second domain based on difference information between the first model and the second model; An information processing method including:

Images