Medical information processing device and medical information processing program

Abstract

To provide a deep learning model, or the like, with higher accuracy, compared to the prior art.SOLUTION: A medical information processing apparatus includes a trained first model to input medical data to output first output data, and a trained second model to input medical data to output second output data. Training of the first model and training of the second model are executed by using a hierarchical relationship, which is a relationship between the first output data and the second output data, including at least one of a spatial hierarchy, a temporal hierarchy, an anatomical hierarchy, and a hierarchy of clinical conditions.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The embodiments disclosed in this specification relate to a medical information processing device and a medical information processing program. [Background technology]

[0002] It is known to train machine learning algorithms to process data, such as medical image data, and the machine learning algorithms may comprise deep learning models.

[0003] In general, a deep learning model can be trained to perform a specific task. For example, a deep learning model may be trained on training data comprising multiple training image datasets, some or all of which may be labeled with information relevant to the task being learned.

[0004] If there are multiple tasks to perform on an image dataset, a separate deep learning model may be trained to perform each task.

[0005] Effective knowledge transfer between related tasks remains a challenge in deep learning. In some situations, joint modeling of related tasks can result in increased algorithm performance.

[0006] The tasks of lung segmentation and lung nodule detection can be considered related tasks. The tasks of lung segmentation and lung nodule detection are related in that lung nodules always occur within the lungs.

[0007] Current deep learning systems may not be able to naturally capture high-level relationships, such as the relationship that lung nodules are always present in the lungs.

[0008] Transfer learning, multi-task learning, and alternating training are methods that attempt to use relationships between related tasks to improve algorithm performance. Transfer learning, multi-task learning, and alternating training each rely on deep learning algorithms to internally learn the relationships between tasks from training data. While these approaches can be beneficial in creating more robust feature representations, they have been shown to fall short in exploiting high-level task relationships, such as understanding that lung nodules can only reside in the lungs.

[0009] FIG. 1 schematically illustrates two tasks performed by one or more trained models. Input data 10 is represented in FIG. 1 as a cloud shape. Results 12 of a first task are shown as circles. Results 14 of a second task are shown as squares located inside the circles. Thus, FIG. 1 schematically illustrates a scenario in which results 14 of a second task are embedded in results 12 of the first task. For example, the first task may be lung segmentation and the second task may be lung nodule detection. The first result may be the lungs and the second result may be lung nodules embedded in the lungs.

[0010] Figure 2 shows a schematic of using a single multitask model 20 to perform the tasks shown in Figure 1. Input data 10 is input to the multitask model 20. The multitask model 20 outputs the results 12 of a first task and the results 14 of a second task. [Prior art documents] [Patent documents]

[0011] [Patent Document 1] U.S. Patent Application Publication No. 2016021736 [Non-patent literature]

[0012] [Non-Patent Document 1] Fu et al 'CNN with coarse-to-fine layer for hierarchical classification', 2017, IET Computer Vision, 2018, 12, (6), p. 892-899. Summary of the Invention [Problem to be solved by the invention]

[0013] One of the problems to be solved by the embodiments disclosed herein is to provide a deep learning model and its training process that are more accurate than conventional models, for example, when processing medical image data. However, the problems to be solved by the embodiments disclosed herein and in the drawings are not limited to the above problem. Problems corresponding to the effects of each configuration shown in the embodiments described below can also be considered as other problems. [Means for solving the problem]

[0014] The medical image processing device according to this embodiment includes a trained first model that receives medical data as input and outputs first output data, and a trained second model that receives the medical data and outputs second output data. The training of the first model and the training of the second model are performed using a hierarchical relationship between the first output data and the second output data, the hierarchical relationship including at least one of a spatial hierarchy, a temporal hierarchy, an anatomical hierarchy, and a clinical symptom hierarchy. [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 1 is a diagram showing a schematic diagram of a set of input data and the results of the first and second tasks. [Figure 2] FIG. 2 is a diagram illustrating an overview of using a multitasking model to perform a first task and a second task. [Figure 3] FIG. 3 is a schematic diagram of an apparatus according to an embodiment. [Figure 4]FIG. 4 is a flowchart outlining a method according to an embodiment. [Figure 5] FIG. 5 is a schematic diagram of an alternating training process according to an embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of a user interface according to an embodiment. [Figure 7] FIG. 7 illustrates an exemplary medical image. [Figure 8A] FIG. 8A shows the results of applying the nodule segmentation model to the image of FIG. [Figure 8B] FIG. 8B shows the results of applying the lung segmentation model to the image of FIG. [Figure 8C] FIG. 8C shows the results of combining the nodule segmentation model of FIG. 8A with the lung segmentation of FIG. 8B. [Figure 8D] FIG. 8D illustrates the results of applying a nodule segmentation model trained according to an embodiment to the image of FIG. [Figure 8E] FIG. 8E illustrates the results of applying a lung segmentation model trained according to an embodiment to the image of FIG. [Figure 8F] FIG. 8F shows the results of combining the nodule segmentation model of FIG. 8D with the lung segmentation of FIG. 8E. [Figure 9] FIG. 9 is a graph comparing the numerical performance of the models on the nodule segmentation task. [Figure 10] FIG. 10 is a schematic diagram of four cascaded models according to an embodiment. [Figure 11] FIG. 11 is a schematic diagram of an alternating training process for training four models according to an embodiment. [Figure 12] FIG. 12 is a schematic diagram of determining the duration of hypertension, hyperthermia, and heart failure from clinical data using a model trained according to an embodiment. [Figure 13]FIG. 13 is a diagram illustrating an example of a user interface according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0016] Hereinafter, a medical information processing apparatus, a medical information processing system, and a medical information processing program according to an embodiment will be described with reference to the accompanying drawings.

[0017] First, the overall configuration of the medical information processing system according to this embodiment will be described.

[0018] An apparatus 30 according to an embodiment is shown schematically in Fig. 3. The apparatus 30 is configured to train at least two models (e.g., at least two neural networks) to perform at least two tasks, sometimes referred to as image interpretation tasks, which are hierarchically related.

[0019] In this embodiment, the device is also configured to apply the trained model to perform the at least two tasks. In other embodiments, the trained model may be applied using one or more different devices.

[0020] The device 30 comprises a computing device 32, which in this example is a personal computer (PC) or workstation. The device 30 is connected to at least one scanner 34 via a data store 40. The device 30 is also connected to one or more display screens 36 and one or more input devices 38, such as a computer keyboard, mouse, or trackball.

[0021] At least one scanner 34 may comprise any scanner configured to perform medical imaging. The or each scanner 34 is configured to generate medical image data, which may comprise two-dimensional, three-dimensional, or four-dimensional data from any imaging modality. For example, scanner 34 may comprise a magnetic resonance (MR or MRI) scanner, a CT (computed tomography) scanner, a cone-beam CT scanner, an X-ray scanner, an ultrasound scanner, a PET (positron emission tomography) scanner, or a SPECT (single-photon emission computed tomography) scanner.

[0022] Image data sets acquired by the at least one scanner 34 are stored in a data store 40 and later provided to the computing device 32 .

[0023] In this embodiment, data storage 40 stores at least two sets of training data, each set comprising a plurality of training image datasets and associated ground truth data. The first set of training data comprises a training image dataset associated with a first task and labeled with ground truth data associated with the first task. The second set of training data comprises a training image dataset associated with a second task and labeled with ground truth data associated with the second task. The ground truth data may be obtained by manual annotation of the training image dataset by an expert or by any suitable automatic or semi-automatic method.

[0024] In this embodiment, the data store 40 also stores other image datasets that are not related to ground truth data, such as image datasets to which the trained model is applied to obtain a desired output, such as classification, segmentation, or region detection.

[0025] In alternative embodiments, the training image data set and / or other image data sets are provided from one or more additional data stores (not shown), which may form a picture archiving and communication system (PACS). Data store 40 or the additional data stores may comprise any suitable form of memory store.

[0026] In further embodiments, the training dataset and / or the dataset to which the trained model is applied may not comprise an image dataset. For example, the dataset may comprise clinical data such as vital sign data, lab value data, or text data. The model may be trained to provide any suitable output.

[0027] The computing device 32 includes a processing unit 42 for processing data, including image data. The processing unit includes a central processing unit (CPU) and a graphics processing unit (GPU). In other embodiments, the processing unit may include a CPU without a GPU.

[0028] The processing device 42 includes a training circuit 44 configured to train at least two models on data from a set of training data stored in the data storage unit 40, and an inference circuit 46 configured to apply the trained models to unseen data to obtain model outputs. The processing device 42 may further include a user interface circuit 48 configured to accept input via a user interface and / or display output via the user interface. The models trained by the training circuit 44 and used in the inference circuit 46 use medical data as input data. Here, medical data refers to data including at least one of medical image data captured by various modalities, vital sign data, clinical test value data, and text data such as electronic medical records and health records.

[0029] In this embodiment, the training circuitry 44, the inference circuitry 46, and the user interface circuitry 48 are each implemented in a CPU and / or GPU by a computer program having computer-readable instructions that are executable to perform the methods of the embodiments. In other embodiments, the various circuits may be implemented as one or more ASICs (application-specific integrated circuits) or FPGAs (field-programmable gate arrays).

[0030] Computing device 32 also has a hard drive and other components of a PC, including RAM, ROM, a data bus, an operating system including various device drivers, and hardware devices including a graphics card, although such components are not shown in FIG.

[0031] The apparatus of FIG. 3 is configured to carry out the method illustrated generally in FIG.

[0032] At stage 50, the training circuitry 44 receives a plurality of first training image data sets and associated ground truth data from the data store 40. The plurality of first training image data sets are derived from a first set of training data. Each of the training image data sets comprises a distinct set of volumetric medical image data, such as a set of voxel intensities in an array of voxel locations. Each training image data set is manually classified to obtain a distinct ground truth data set.

[0033] In this embodiment, the ground truth data comprises a separate lung segmentation for each training image dataset of the plurality of first training image datasets. The lung segmentation comprises a voxel-by-voxel classification of the volumetric medical image dataset indicating which voxels represent lungs. The ground truth data may be obtained in any suitable manner, such as, for example, manual classification of the training image dataset by an expert. The ground truth data for at least some of the plurality of first training datasets may not comprise data related to lung nodules, such as, for example, lung nodule segmentation data.

[0034] Training circuitry 44 accepts or generates a first neural network 72. First neural network 72 may be referred to as a lung segmentation model. In other embodiments, training circuitry 44 may accept or generate any suitable first model, such as, for example, any suitable deep learning model. In some embodiments, the first model does not comprise a neural network.

[0035] The first neural network 72 may be pre-trained before the start of stage 50. For example, the first neural network 72 may be pre-trained to recognize general image features. The first neural network 72 may be pre-trained to recognize features in medical images.

[0036] Pre-training may comprise training the model on a different task or on a different dataset and task. Pre-training may be performed in a supervised manner, using human-generated ground truth to inform updates to model parameters. Pre-training may also be performed in an unsupervised manner, where data used as ground truth may be automatically generated (e.g., image inpainting or encoder / decoder image reconstruction). Using model pre-training may mean that random network initialization is not used. In some embodiments, pre-training is used as part of the model. For example, pre-training may be used for a layer of the model. Pre-training may be used for the encoder portion of the model. Pre-training of part of the model may be used when the nature of the pre-training differs from the final task to be performed by the model, for example, when pre-training is image classification and the final task is image segmentation, or vice versa.

[0037] In this embodiment, the encoder of the first neural network 72 is pre-trained on an ImageNet classification task. In other embodiments, any suitable pre-training may be used.

[0038] The training circuitry 44 uses the training image data set and associated ground truth data to train the first neural network 72 to perform the task of lung segmentation.

[0039] The first neural network 72 is trained to receive an image dataset as input and output a lung segmentation comprising a voxel-by-voxel classification of the image dataset. During training of the first neural network 72, an error in the lung segmentation task is determined by comparison with ground truth data. The determined error is backpropagated to the first neural network 72 to improve the performance of the first neural network 72. Training of the first neural network 72 includes updating the weights of the first neural network 72. During training, the first neural network 72 may learn to perform lung segmentation based on any suitable features of the image dataset. Note that backpropagation is an example of a feedback process.

[0040] Any suitable training method may be used to train first neural network 72. The training of first neural network 72 in stage 50 may be considered to be performed separately and independently.

[0041] The output of stage 50 is an updated weighted version of first neural network 72. The weights of first neural network 72 are fixed at the end of stage 50.

[0042] At stage 52, the training circuitry 44 receives a plurality of second training image data sets and associated ground truth data from the data store 40. The plurality of second training image data sets are derived from the second set of training data, each of the plurality of second training image data sets comprising a distinct set of volumetric medical image data, such as a set of voxel intensities for an array of voxel locations.

[0043] In this embodiment, the ground truth data comprises a separate lung nodule segmentation for each training image dataset of the plurality of second training image datasets. The lung nodule segmentation comprises a voxel-by-voxel classification of the volumetric medical image dataset indicating which voxels represent lung nodules. The ground truth data may be obtained in any suitable manner, such as, for example, manual classification of the training image dataset by an expert. In one embodiment, the ground truth data for at least some of the plurality of second training datasets does not comprise lung segmentation data.

[0044] Training circuitry 44 accepts or generates a second neural network 82. Second neural network 82 may be referred to as a pulmonary nodule segmentation model. In other embodiments, training circuitry 44 may accept or generate any suitable second model, such as, for example, any suitable deep learning model. In some embodiments, the second model does not comprise a neural network.

[0045] The second neural network 82 may be pre-trained before the start of stage 52. For example, the second neural network 82 may be pre-trained to recognize general image features. The second neural network 82 may be pre-trained to recognize features in medical images. In this embodiment, the encoder of the second neural network is pre-trained on an ImageNet classification task. In other embodiments, any suitable pre-training may be used.

[0046] Training circuitry 44 uses the training image data set and associated ground truth data from the second set of training data to train a second neural network 82 to perform the task of pulmonary nodule segmentation. The second neural network 82 is trained within the context of the first neural network 72. During stage 52, the weights of the first neural network 72 are locked.

[0047] The second neural network 82 is trained to receive an image dataset as input and output a lung nodule segmentation with a voxel-by-voxel classification of the image dataset. The output of the second neural network 82 is multiplied by the output of the first neural network 72. During training of the second neural network 82, an error in the task of lung nodule segmentation is determined by comparison with ground truth data. The error in the task of lung nodule segmentation is backpropagated to the second neural network 82 to improve the performance of the second neural network 82. In stage 52, the lung nodule model is free to predict lung nodules outside the lung and is not penalized. Training of the second neural network 82 includes updating the weights of the second neural network 82. During training, the second neural network 82 may learn to perform lung nodule segmentation based on any suitable features of the image dataset.

[0048] Any suitable training method may be used to train the second neural network 82. The training of the second neural network 82 in stage 52 may be considered to be performed separately and independently, since only the second neural network 82 is updated. The first neural network 72 has locked parameters.

[0049] The output of stage 52 is an updated weighted version of second neural network 82. The weights of second neural network 82 are fixed at the end of stage 52.

[0050] Stage 50 is the initial training of first neural network 72 on the task of performing lung segmentation, and stage 52 is the initial training of second neural network 82 on the task of performing lung nodule segmentation. In some embodiments, the initial training of stage 50 and / or the initial training of stage 52 may be omitted from the method of FIG.

[0051] In stage 54, training circuit 44 performs an alternating training process, which comprises alternating between stages 56 and 58 of the flowchart of Figure 4. Figure 5 is a schematic diagram showing the alternating training process of stage 54 in more detail.

[0052] In stage 56, training circuitry 44 trains first neural network 72 independently (i.e., in isolation from the training of second neural network 82) using a batch of training image data sets from the first set of training data. The training image data set used in stage 56 may be different from the training image data set used in stage 50. A different training image data set may be used in each instance of stage 56.

[0053] The first neural network 72 is trained in isolation on the task of lung segmentation. At the start of the first instance of stage 56, the weights of the first neural network 72 at the start of stage 56 are the weights output at the end of stage 50.

[0054] 5 illustrates the training of a first neural network 72. For each training image dataset in the batch, the first neural network 72 receives the training image dataset 70 and outputs a dataset 74 comprising lung segmentation data, e.g., a voxel-by-voxel classification of the input image dataset. An error in the output of the first neural network 72 is determined by comparison with ground truth data. The error in the output of the first neural network 72 is backpropagated to the first neural network 72. The backpropagation of the error in the output of the first neural network 72 to the first neural network 72 is indicated in FIG. 5 by arrow 76.

[0055] The output of stage 56 is an updated set of weights for first neural network 72 .

[0056] At stage 58, training circuitry 44 jointly trains first neural network 72 and second neural network 82. First neural network 72 and second neural network 82 are combined by a multiplication operation and jointly trained on the task of lung nodule segmentation. The multiplication may be thought of, for example, as representing the calculation of the joint probability of nodules and lungs as follows:

[0057] P(lung∩nodule)=P(lung)P(nodule|lung) Here, given that it is in the lung, P(lung ∩ nodule) is the probability of being both lung and a lung nodule, P(lung) is the probability of being in the lung, and P(nodule | lung) is the probability of being in a nodule.

[0058] For example, for an individual voxel, P(lung ∩ nodule) is the probability that the voxel is part of a lung nodule in the lung, P(lung) is the probability that the voxel is in the lung, and P(nodule | lung) is the probability that the voxel is part of a lung nodule, given that the voxel is in the lung.

[0059] In the first instance of stage 58, the weights of first neural network 72 at the start of stage 56 are the weights output at the end of the first instance of stage 56, and the weights of second neural network 82 are the weights output from stage 52.

[0060] Training circuitry 44 trains a first neural network 72 to perform lung segmentation and a second neural network 82 to perform pulmonary nodule segmentation. The first neural network 72 and the second neural network 82 are trained together by multiplying the output of the first neural network 72 with the output of the second neural network 82. Stage 58 is performed using batches of training image data sets from the second set of training data. Training of the combined first neural network 72 and second neural network 82 may be performed using training data labeled only for pulmonary nodules.

[0061] The training of the first neural network 72 and the second neural network 82 is shown in FIG. 5. For each training data set 80 in the current batch, the first neural network 72 and the second neural network 82 each receive the same training data set. The first neural network 72 generates a first output 84 including lung segmentation. The first output 84 may be referred to as O1. The second neural network 82 generates a second output 86 including lung nodule segmentation. The second output 86 may be referred to as O2. The training circuit 44 multiplies the first output 84 and the second output 86 to obtain a third output 88. The third output 88 may be referred to as O1 × O2. Note that the first output 84 and the second output 86 are examples of first output data and second output data, respectively.

[0062] The training circuitry 44 determines an error in the third output 88 by comparison with ground truth data. The training circuitry 44 may determine whether the error in the third output 88 is due to an error in the first output 84 and / or an error in the second output 86. Different types of errors are described in more detail below. The error in the third output 88 is backpropagated to the first neural network 72 and / or the second neural network 82 depending on the type of error. The backpropagation of the error from the third output 88 to the first neural network 72 is represented by arrow 90. The backpropagation of the error from the third output 88 to the second neural network 82 is represented by arrow 92.

[0063] For the example tasks of lung segmentation (by first neural network 72) and lung nodule segmentation (by second neural network 82), there are four possible extremes of error. In the discussion below, a positive prediction (lung present, nodule present) is listed as 1. A negative prediction (lung not present, nodule not present) is listed as 0.

[0064] The first type of error is a false positive, predicting a lung and a nodule. That is, O1=1, O2=1, O1×O2=1. The prediction of a lung and a nodule is incorrect. The correct answer is neither a lung nor a nodule. If this is a first type of error, the error is backpropagated to both the first neural network 72 and the second neural network 82, as indicated by arrows 90 and 92. Both the first neural network 72 and the second neural network 82 are penalized.

[0065] The second type of error is a false negative, predicting lung and not a nodule. The correct answer is lung and nodule. The classification of lung, O1=1, is correct. The classification of not a nodule, O2=0, is incorrect. The multiplied output, O1×O2=0, is incorrect due to a misclassification of a nodule. If it is an error of the second type, the error is backpropagated only to the second neural network 82, as shown by arrow 92. The second neural network 82 is penalized. The first neural network 72 is not penalized.

[0066] The third type of error is a false negative, predicting a nodule and not a lung. The correct answer is lung and nodule. The classification O1=0, not lung, is incorrect. The classification O2=1, nodule, is correct. The multiplied output O1×O2=0 is incorrect due to a misclassification of lung. If it is a third type of error, the error is backpropagated only to the first neural network 72, as shown by arrow 90. The first neural network 72 is penalized. The second neural network 82 is not penalized.

[0067] The fourth type of error is a false negative, which predicts neither a nodule nor a lung. The correct answer is lung and nodule. The classification of O1=0, which is not lung, is incorrect. The classification of O2=0, which is not a nodule, is incorrect. The multiplied output O1×O2=0 is incorrect due to both a misclassification of lung and a misclassification of a nodule. If it is a fourth type of error, the error is backpropagated to the first neural network 72 and the second neural network 82, as shown by arrows 90 and 92. Both the first neural network 72 and the second neural network 82 are penalized.

[0068] The lung segmentation model 72 is penalized during joint training in stage 58 by excluding lung nodules from the interior of the lung. The nodule segmentation model 82 is free to predict high probabilities in regions outside the lung.

[0069] The weights of both the first neural network 72 and the second neural network 82 are updated during stage 58. The output of stage 58 is an updated set of weights for the first neural network 72 and an updated set of weights for the second neural network 82.

[0070] After stage 58, the alternating training returns to stage 56. In stage 56, the first neural network 72 is trained in isolation on further batches of image data sets from the first set of training data. After stage 56, the alternating training returns to stage 58. In stage 58, the first neural network 72 and the second neural network 82 are trained in conjunction with further batches of image data sets from the second set of training data.

[0071] Arrows 57 and 59 are used to indicate that during the alternating training process of stage 54, each instance of stage 56 is followed by an instance of stage 58, and each instance of stage 58 is followed by an instance of stage 56.

[0072] The alternation between stages 56 and 58 is repeated until convergence is reached, at which point stage 54 ends. The weights of first neural network 72 and second neural network 82 are fixed.

[0073] In stage 60, training circuit 44 outputs a trained first neural network with fixed weights at the end of stage 54 and outputs a trained second neural network with fixed weights at the end of stage 54.

[0074] At stage 62, inference circuitry 46 receives first trained neural network 72 and second trained neural network 82. Inference circuitry 46 receives a new image data set from data store 40. The new image data set may be obtained from a scan by scanner 34. The new image data set is an unseen data set that is not part of the first set of training data or the second set of training data.

[0075] The inference circuitry 46 applies the trained first neural network 72 to the new image data set to obtain a first output. In this embodiment, the first output is a lung segmentation. The inference circuitry 46 applies the trained second neural network 82 to the new image data set to obtain a second output. In this embodiment, the second output is a lung nodule segmentation. The inference circuitry 46 then multiplies the first output by the second output to obtain a third output. The third output may be referred to as a combined output. The multiplication of the first output by the second output provides a hierarchical relationship between the first neural network 72 and the second neural network 82. In this embodiment, the multiplication of the lung segmentation by the lung nodule segmentation provides a relationship that the lung nodule is within the lung boundary. Here, the hierarchical relationship is a relationship between the first output and the second output, including at least one of a spatial hierarchy, a temporal hierarchy, an anatomical hierarchy, and a clinical hierarchy. The hierarchical relationship is, for example, a correlation relationship in which the order of occurrence of one of the first output and the second output affects the other, and therefore does not necessarily have to be hierarchical.

[0076] In this embodiment, where the first output is a lung segmentation and the second output is a lung nodule segmentation, the third output is a lung nodule segmentation given the spatial hierarchical relationship between the lungs and the nodules. The third output may provide an improved lung nodule segmentation.

[0077] At stage 64, display circuitry 48 receives the first output and the third output from inference circuitry 46. Display circuitry 48 displays the first output and the third output to a user using display screen 36 or on any suitable display or displays. In other embodiments, the first output and the third output may not be displayed to a user. For example, the first output, the second output, and / or the third output may be used as inputs for further processing.

[0078] FIG. 6 illustrates an example of a user interface configured to display the results of applying the first trained model and the second trained model as described above to lung image data.

[0079] The user interface 300 includes a selection panel 302 and an imaging panel 304. The selection panel 300 displays multiple lung images 310, 312, and 314, which in this embodiment are single CT slices. The dates the images 310, 312, and 314 were acquired are also displayed.

[0080] In a first stage 1 of the user interface processing, a user selects at least one image from selection panel 302. In the embodiment shown in FIG. 6, the user selects lung images 310 and 312. Display circuitry 48 displays lung images 310 and 312 in imaging panel 304. In stage 2 of the user interface processing, a user views lung images 310 and 312. The user may view lung images 310 and 312 to manually assess whether lung images 310 and 312 contain pulmonary nodules.

[0081] For each lung image 310 , 312 , the display circuitry 48 displays a corresponding lung segmentation image and a nodule segmentation image on the imaging panel 304 .

[0082] For lung image 310, display circuitry 48 displays lung segmentation image 320 and nodule segmentation image 322. Lung segmentation image 320 comprises a first output of the first trained model when applied to the image data of lung image 310. Nodule segmentation image 322 comprises the combined output of the first trained model and the second trained model when applied to the image data of lung image 310.

[0083] For lung image 312, display circuitry 48 displays lung segmentation image 330 and nodule segmentation image 332. Lung segmentation image 330 comprises a first output of the first trained model when applied to image data for lung image 312. Nodule segmentation image 332 comprises a combined output of the first trained model and the second trained model when applied to image data for lung image 312.

[0084] In some embodiments, the trained model is applied to the lung images 310, 312 when the lung images 310, 312 are selected by a user. In other embodiments, the trained model is applied to the lung images 310, 312 in advance. For example, the trained model may be applied to all lung images stored in the data store 40.

[0085] In stage 3 of the user interface process, the user sees the results of the segmentation: the outputs 320, 330 of the trained first model, and the combined outputs 332, 334 of the trained first model and the trained second model.

[0086] In other embodiments, any suitable display method and user interface may be used, and the user may interact with the user interface in any suitable manner.

[0087] The method of FIG. 4 provides a method for training and deploying deep learning algorithms. A deep learning approach to hierarchical tasks is provided. The method of FIG. 4 may be described as hierarchical multi-task transfer with model symbiosis. Multiple models cooperate in structural relationships. Tasks are jointly modeled to facilitate knowledge transfer between tasks.

[0088] The model architecture and training strategy described above in connection with Figure 4 may allow for the integration of two related tasks, which may improve performance on both tasks.

[0089] The method of Figure 4 may result in improved sensitivity due to the combined model on lung nodule segmentation. The method of Figure 4 results in lung segmentation and lung nodule segmentation algorithms that can be considered coherent. In the training method, the two models are forced to remain together due to the hierarchical structure of the task.

[0090] The deep learning algorithm is given task relationships, where one object resides within the boundary of another. The individual models are combined at the output by a multiplication operation. At inference time (stage 62), this acts as the application of a mask. At training time, the method of Figure 4 allows errors from one task to be used to inform weight updates of models used in other tasks if the predictions by the models violate the hierarchical task relationships.

[0091] Model training alternates between tasks. Model training may also alternate between datasets. The method of FIG. 4 may be used to leverage multiple datasets with multiple associated labels for related tasks. For example, a first set of training data may comprise a lung segmentation dataset and a second set of training data may comprise a nodule segmentation dataset. Information captured in one dataset may be leveraged to improve performance of the other dataset and task. Alternating between datasets may allow the use of incompletely labeled data, for example, data that does not have all the labels for a hierarchical task.

[0092] In some situations, a training process that involves training each model individually before beginning interleaved training may produce better results than a training process that only involves interleaved training. The method in Figure 4 does not rely on deep learning algorithms to learn the internal relationships between tasks from the data. Instead, the relationships between tasks are provided as part of the training method.

[0093] While it is known that multitask models can be beneficial in generating more robust feature representations, they rarely exploit high-level task relations, such as understanding that lung nodules always exist in the lungs. This may be due to the simplistic nature of deep learning algorithms. The method in Figure 4 exploits high-level task relations that are strictly implied by model training and application of the trained model.

[0094] The method of Figure 4 is described with reference to the tasks of lung segmentation and lung nodule segmentation. In other embodiments, a hierarchical multi-task transfer method similar to that described with reference to Figure 4 may be used for any suitable task having outputs with hierarchical relationships, such as, for example, spatial hierarchical relationships.

[0095] Hierarchical multitask transfer can be used to improve algorithm performance. Figures 8A-8F and 9 show the algorithm performance for two example hierarchical tasks using independent datasets labeled separately for each task.

[0096] 7 shows an exemplary medical image 100, which is a single CT slice. The medical image includes a lung nodule 102.

[0097] Figures 8A-8C show the results of applying a lung segmentation model and a nodule segmentation model to the medical image 100 of Figure 7. In the method whose results are shown in Figures 8A-8C, the lung segmentation model and the nodule segmentation model were not trained alternately. The lung segmentation and nodule segmentation models were trained independently.

[0098] Figure 8A shows the output of the nodule segmentation model when applied to an image dataset corresponding to the image of Figure 7. Figure 8B shows the output of the lung segmentation model when applied to an image dataset corresponding to the image of Figure 7. The lung segmentation model omits the lung nodule in the lung segmentation. Figure 8C shows the result of multiplying the output of the nodule segmentation model and the output of the lung segmentation model. In Figure 8C, the nodule is correctly identified.

[0099] 8D-8F show the results of applying the lung and nodule segmentation models trained using the alternating training described above with reference to FIG.

[0100] Figure 8D shows the output of a nodule segmentation model trained using alternating training when applied to an image dataset corresponding to the images in Figure 7. Figure 8E shows the output of a lung segmentation model trained using alternating training when applied to an image dataset corresponding to the images in Figure 7. Figure 8F shows the result of multiplying the output of the nodule segmentation model and the output of the lung segmentation model. In Figure 8F, the nodule is correctly located inside the lung. As can be seen in Figure 8E, the lung segmentation model contains the lung nodule within the lung region.

[0101] FIG. 9 includes a graph 110 of the numerical performance of two different models on a lung nodule segmentation task. Graph 110 is the Free Receiver Operating Characteristic. The horizontal axis of graph 110 in FIG. 9 shows the mean false positives per volume (number). The vertical axis of graph 110 in FIG. 9 shows the sensitivity.

[0102] The first line 111 represents the performance of a combined model trained on both the lung and nodule segmentation tasks. The combined model comprises the lung and nodule segmentation models described above with reference to FIG. 4.

[0103] The second line 112 represents the performance of a model trained alone on the nodule segmentation task. A combined model comprising a lung segmentation model and a nodule segmentation model is not employed in this model, whose performance is shown in line 112. The alternating training method of FIG. 4 is not employed in this model, whose performance is shown in line 112.

[0104] It can be seen that the combined model (line 111) has a higher sensitivity than the model that does not employ the combined strategy (line 112). The combined model exhibits higher sensitivity at all false positive rates.

[0105] In the embodiment of Figure 4, the tasks performed by the model are lung segmentation and lung nodule segmentation. The task relationship is pathology within a visceral organ, in the case of Figure 4, a lung nodule located within the lung. In other embodiments, a hierarchical multi-task transfer method corresponding to that described above with reference to Figure 4 is generally applied to tasks where one object is within the boundary of another. The method used may comprise the method of Figure 4 with lung segmentation replaced by any suitable first task relating to a first object and lung nodule segmentation replaced by any suitable second task relating to a second object. Any suitable hierarchy of objects may be used.

[0106] A spatial hierarchical task relationship between two tasks may be used to leverage performance improvements in both tasks. Deep learning algorithms may be generated across tasks. Deep learning algorithms may be inherently constrained to provide results that are coherent with the hierarchical relationship. For example, a first task may relate to tumors and a second task may relate to regions of fluid or edema, with the tumor residing within the boundaries of the regions of fluid or edema. A first task may relate to the brain and a second task may relate to the hippocampus. For example, the hippocampus may be segmented using knowledge of its location in the brain.

[0107] In some embodiments, the first task relates to the eye and the second task relates to the eye's lens, for example, the first model may segment the entire eye and the second model may segment the eye's lens.

[0108] In a further embodiment, an alternating training method similar to that described with reference to FIG. 4 may be applied to more than two tasks where a cascading hierarchical task relationship exists.

[0109] Figure 10 is a schematic diagram of a set of four models trained to perform a four-hierarchical segmentation task. Figure 10 shows how the models are cascaded during inference to obtain four outputs. The model outputs are combined by successive hierarchical weighting.

[0110] An input image 120 is provided to the cascade of trained models. In Figure 10, the input image 120 is shown twice: first using a symbol for the image similar to that used in Figure 5, and secondly as a representation indicating that the image contains first through fourth objects 122, 124, 126, and 128. An arrow connects the two views of the input image 120.

[0111] Input image 120 includes a first object 122, shown as a square. Input image 120 also includes a second object 124, shown as a circle, that is contained within the boundary of first object 122. The input image further includes a third object 126, shown as a triangle, that is contained within the boundary of second object 124. The input image further includes a fourth object 128, shown as a line, that is contained within the boundary of third object 126. Thus, a set of spatial hierarchical relationships exists between first object 122, second object 124, third object 126, and fourth object 128.

[0112] The objects 122, 124, 126, and 128 are simplified objects used as examples. In an embodiment, the first through fourth objects 122, 124, 126, and 128 may be any objects having a spatial hierarchical relationship. For example, the objects 122, 124, 126, and 128 may be any suitable anatomical structures having a spatial hierarchical relationship.

[0113] Inference circuitry 46 applies trained first model 130 to input image 120 to generate first output 132 comprising a segmentation of first object 122. First output 132 is shown twice in Figure 10, first as a symbol and second as a representation indicating that first output 132 represents first object 122.

[0114] Inference circuitry 46 applies trained second model 140 to input image 120 to generate second output 142. Inference circuitry 46 multiplies second output 142 by first output 132 to obtain first combined output 144. First combined output 144 comprises a segmentation of second object 124. In Figure 10, first combined output 144 is shown twice: first as a symbol and second as a representation indicating that first combined output 144 represents second object 124.

[0115] Inference circuitry 46 applies trained third model 150 to input image 120 to generate third output 152. Inference circuitry 46 multiplies third output 152 by first combined output 144 to obtain second combined output 154. Second combined output 154 comprises a segmentation of third object 126. In Figure 10, second combined output 154 is shown twice: first as a symbol and second as a representation indicating that second combined output 154 represents third object 126.

[0116] Inference circuitry 46 applies trained fourth model 160 to input image 120 to generate fourth output 162. Inference circuitry 46 multiplies fourth output 162 by second combined output 154 to obtain third combined output 164. Third combined output 164 comprises a segmentation of fourth object 128. In Figure 10, third combined output 164 is shown twice: first as a symbol and second as a representation indicating that third combined output 164 represents fourth object 128.

[0117] The method of Figure 10 provides a hierarchical relationship between the outputs of multiple tasks. In other embodiments, the method of Figure 10 may be used for any suitable number of tasks, provided that the outputs of the tasks have a hierarchical relationship. Individual models are combined at their outputs by multiplication operations.

[0118] 10 illustrates how four trained models 130, 140, 150, and 160 are cascaded during inference to generate four outputs on a hierarchical segmentation task. The models 130, 140, 150, and 160 may be referred to as M1, M2, M3, and M4, respectively. That is, a first combined output 144 as first output data is obtained by combining the first model 130 with at least one additional trained model, the second model 140, combined with the first model 130. A third output 152 as second output data can also be obtained by combining the third model 150 with at least one additional trained model, the third model 150, combined with the third model 150.

[0119] Figure 11 shows a cyclical alternated training process for training the four models used in Figure 10. The cyclical alternated training process includes a first stage 170, a second stage 172, a third stage 174, and a fourth stage 176. Figure 11 shows an example order of stages 170, 172, 174, and 176. In other embodiments, training may not be limited to the order shown in Figure 11. Any suitable order may be used.

[0120] At stage 170, the training circuitry 44 trains the first model to perform a first task, which in this embodiment is segmentation of the first object 122. The training circuitry inputs first training data 180 to the first model 130 and obtains an output 182. The error in the output 182 is backpropagated to the first model 130. The first model 130 is trained using batches of first training data comprising an image dataset labeled with a ground truth segmentation of the first object 122.

[0121] In stage 172, the training circuit 44 jointly trains the first model 130 and the second model 140. The training circuit 44 inputs second training data 190 to the first model 130 to obtain a first output 192. The training circuit 44 inputs the same second training data 190 to the second model 140 to obtain a second output 194. The training circuit 44 multiplies the first output 192 and the second output 194 to obtain a third combined output 196. The error in the third output 196 is backpropagated to the first model 130 and / or the second model 140. The backpropagation of the error to the first model 130 and / or the second model 140 may depend on the type of error. The combination of the first model 130 and the second model 140 is trained using batches of second training data comprising an image dataset labeled with a ground truth segmentation of the second object 124.

[0122] In stage 174, the training circuit 44 jointly trains the first model 130, the second model 140, and the third model 160. The training circuit 44 inputs third training data 200 to the first model 130 to obtain a first output 202. The training circuit 44 inputs the same third training data 200 to the second model 140 to obtain a second output 204. The training circuit 44 inputs the same third training data 200 to the third model 150 to obtain a third output 206. The training circuit 44 multiplies the first output 202 and the second output 204 to obtain a first combined output 208. The training circuit 44 multiplies the first combined output 208 by the third output 206 to obtain a second combined output 210.

[0123] The error in the second combined output 210 is backpropagated to the first model 130 and / or the second model 140 and / or the third model 150. The backpropagation of the error to the first model 130 and / or the second model 140 and / or the third model 150 may depend on the type of error. The combination of the first model 130, the second model 140, and the third model 150 is trained using a third batch of training data comprising an image dataset labeled with a ground truth segmentation of the third object 126.

[0124] At stage 176, training circuitry 44 trains all four models 130, 140, 150, and 160 together. Training circuitry 44 inputs fourth training data 220 to first model 130, resulting in a first output 222. Training circuitry 44 inputs the same fourth training data 220 to second model 140, resulting in a second output 224. Training circuitry 44 inputs the same fourth training data 220 to third model 150, resulting in a second output 226. Training circuitry 44 inputs the same fourth training data 220 to fourth model 160, resulting in a fourth output 228.

[0125] The training circuit 44 multiplies the first output 222 and the second output 224 to obtain a first combined output 230. The training circuit 44 multiplies the first combined output 230 with the third output 226 to obtain a second combined output 232. The training circuit 44 multiplies the second combined output 232 with the fourth output 228 to obtain a third combined output 234.

[0126] The error in the third combination output 234 is backpropagated to the first model 130 and / or the second model 140 and / or the third model 150 and / or the fourth model 160. The backpropagation of the error to the first model 130 and / or the second model 140 and / or the third model 150 and / or the fourth model 160 may depend on the type of error. The combination of the first model 130, the second model 140, the third model 150 and the fourth model 160 is trained using a fourth batch of training data comprising an image dataset labeled with a ground truth segmentation of the fourth object 128.

[0127] After stage 176, the cyclic training process returns to stage 170. The cyclic training process cycles through stages 170, 172, 174, and 176 until convergence is reached. The output of the cyclic training process are four trained models 130, 140, 150, and 160.

[0128] In further embodiments, any suitable number of models may be trained to perform any suitable number of tasks. In an initial training, the highest model in the hierarchy is trained first, followed by the lower models. The lower models are further trained to develop model dependencies consistent with the hierarchy. For example, a first model at the highest level in the hierarchy may be trained first, followed by a second model below the first model, using the hierarchical relationship between the first output of the first model and the second output of the second model. A third model below the second model may be trained using the hierarchical relationship between the first output of the first model, the second output of the second model, and the third output of the third model. The same process is followed for the models below the third model.

[0129] In some embodiments, training circuitry 44 trains more than one model to perform a given task. For example, training circuitry 44 may train two or more models to perform lung segmentation. Training circuitry 44 may train two or more models to perform lung nodule segmentation. In some embodiments, the training process may be repeated to generate multiple pairs of lung segmentation and nodule segmentation models. In some embodiments, different combinations of lung segmentation and nodule segmentation models are trained together in succession.

[0130] During inference, inference circuitry 46 may combine the outputs of more than one model for each task. For example, inference circuitry 46 may combine the outputs from two lung segmentation models to obtain an overall lung segmentation output. Inference circuitry 46 may combine the outputs from two lung nodule segmentation models to obtain an overall lung nodule segmentation output. The overall lung segmentation output may then be multiplied by the overall lung nodule segmentation output to obtain a combined output.

[0131] In many of the embodiments described above, one model is trained to perform lung segmentation and another model is trained to perform lung nodule segmentation. In other embodiments, any two or more models may be trained to perform any suitable task. One or more tasks may comprise segmentation of any suitable object, such as, for example, any suitable anatomical structure. Segmentation may comprise pixel-level classification or voxel-level classification.

[0132] In some embodiments, one or more tasks comprise bounding box regression, where the bounding box may be placed around any suitable object, such as, for example, any suitable anatomical structure.

[0133] In some embodiments, one or more tasks comprise region detection. For example, a region of tissue may be detected. A lung nodule detection task may comprise generating coordinates at a lung nodule. In another example, a task may be performed to detect the center of mass of a tumor mass.

[0134] In some embodiments, one or more tasks comprise image-level classification. For example, a model may be trained to identify whether a given object (e.g., a given anatomical structure) occurs within an image dataset.

[0135] In some embodiments, different models are trained to perform different tasks. For example, one model in the hierarchy may be trained to perform segmentation, while another model in the hierarchy may be trained to perform bounding box regression. In some embodiments, a single model may be trained to perform multiple tasks related to a single object or multiple objects.

[0136] The task for which the model is trained may relate to any suitable object, such as, for example, any suitable anatomical structure. The image dataset may comprise a medical image dataset acquired using any suitable modality or modalities, such as, for example, CT data, MR data, X-ray data, PET data, or SPECT dataset. The image dataset may be two-dimensional, three-dimensional, or four-dimensional. The image dataset may be obtained by imaging any suitable anatomical region of any patient or other subject. The patient or other subject may be a human or an animal.

[0137] In further embodiments, the image dataset may not be a medical image dataset. The task for which the model is trained may relate to any suitable objects present in the image dataset. For example, the above method may be used for analyzing natural images. In an autonomous vehicle environment, the tasks of automatically segmenting roads and detecting road signs may be arranged as a hierarchical task.

[0138] In alternative embodiments, the hierarchical multi-task transfer method with model symbiosis may be performed on any suitable data. In some embodiments, the data for training the model does not comprise image data. In some embodiments, the data for training the model comprises a mixture of image data and non-image data.

[0139] Figure 12 shows an embodiment using three models trained on clinical data with time series data, which comprises data acquired at multiple time points.

[0140] The clinical data comprises medical image data. The clinical data may comprise medical image data obtained at multiple time points, for example on different dates.

[0141] The clinical data further comprises vital sign data. The vital sign data may comprise, for example, temperature data, blood pressure data, heart rate data, and / or respiratory rate data. The vital sign data is obtained at multiple time points. For example, the vital sign data may be obtained hourly or minutely.

[0142] Clinical data also includes laboratory data. The laboratory data may be obtained from any suitable laboratory test, such as, for example, blood tests or urine tests. The laboratory data may be obtained at multiple time points. For example, the laboratory data may be obtained daily or every few days.

[0143] Clinical data also includes text data, which may include, for example, patient medical data, and which may be obtained at multiple points in time, for example, daily.

[0144] The clinical data for training the model comprises data from a large number of patients, and ground truth information is provided regarding the duration of hypertension, hyperthermia, and heart failure in the patients who provided the clinical data.

[0145] The training circuit 44 trains a first model to determine duration of hypertension based on a time series of clinical data comprising vital sign data, lab value data, text data, and image data. The training circuit trains a second model to determine duration of hypertension based on the same time series of clinical data. The training circuit trains a third model to determine duration of heart failure based on the same time series of clinical data.

[0146] In the context of the embodiment of Figure 12, it is assumed that periods of high fever always occur as part of periods of high blood pressure, but high blood pressure can occur without high fever. It is assumed that periods of heart failure always occur within periods of high fever, but high fever can occur without heart failure. High blood pressure, high fever, and heart failure have a temporal hierarchy. Heart failure always occurs simultaneously with high fever and high blood pressure. High fever always occurs simultaneously with high blood pressure. In other embodiments, different relationships between clinical symptoms may apply.

[0147] The training circuit 44 trains the first, second, and third models using a temporal hierarchy between periods of hypertension, high fever, and heart failure. The first, second, and third models are trained using a cyclic training method similar to that described with reference to FIG. 11. In the first stage of the cycle, the first model is trained alone. In the second stage of the cycle, the first and second models are trained together. In the third stage of the cycle, all three models are trained together. The training process repeats the training cycle until convergence is reached.

[0148] Inference circuitry 46 applies the first trained model, the second trained model, and the third trained model to a data set associated with an individual patient, as shown in Figure 12. The data set includes vital sign time series data 240, laboratory test data 242, text data 244, and image data 246.

[0149] The trained first model, the trained second model, and the trained third model analyze the dataset to obtain a plurality of features. In Figure 12, the features are shown by elements 252 along a time axis 250. The features 252 may be arranged by feature type.

[0150] The first model outputs a period 260 of hypertension having a first length along the time axis 250, shown in a first box 260 in FIG.

[0151] The output of the second model is multiplied by the output of the first model to obtain a period of hyperthermia 262 having a second length along the time axis 250, shown as a second box 262 in FIG.

[0152] The output of the third model is multiplied by the outputs of the first and second models to obtain a period of heart failure 264 having a third length along the time axis 250, shown in a third box 264 in FIG.

[0153] The duration of the period of heart failure 264 is shorter than the duration of the period of high fever 262. The period of heart failure 264 is included within the period of high fever 262.

[0154] The length of the high fever period 262 is shorter than the length of the high blood pressure period 260. The high fever period 262 is included in the high blood pressure period 260.

[0155] The temporal hierarchy between periods of hypertension 260, periods of high fever 262, and periods of heart failure 264 is given by training the first, second, and third models.

[0156] In the embodiment of Figure 12, the symptoms manifest a particular hierarchical order. Heart failure occurs during periods of high fever, which in turn occurs during periods of high blood pressure. This hierarchical order may be specific to a particular disease, for which the approach of Figure 12 may be appropriate for use with a dataset acquired from a subject suffering from that disease. In other embodiments and other diseases, different hierarchies of symptoms may exist.

[0157] 13 shows an example of a user interface 270 that displays the results of an analysis of time series data performed using a model trained with a temporal hierarchy between outputs. The user interface 270 in FIG. 13 displays the output obtained in FIG. 12.

[0158] User interface 270 includes three selectable elements 271, 272, and 273, which represent high blood pressure, high fever, and heart failure, respectively. A user may select one or more of the selectable elements. User selection is indicated by arrow 274 in FIG. 13. In response to a user selection of selectable element 271, user interface circuitry 48 may highlight an aspect of the display related to high blood pressure. In response to a user selection of selectable element 272, user interface circuitry 48 may highlight an aspect of the display related to high fever. In response to a user selection of selectable element 273, user interface circuitry 48 may highlight an aspect of the display related to heart failure.

[0159] The user interface 270 includes a timeline display 275. The timeline display 275 represents time from left to right. The timeline display includes four rows 280, 282, 284, and 286. A set of blocks 281, 283, 285, and 287 represent information available at a particular time. Information such as text reports, images, and lab results may be available as chunks of data, each associated with a limited time, rather than a continuous timeline of data.

[0160] Rows 280 represent vital signs data. Each block 281A, 281B, 281C, 281D represents a distinct time at which a distinct chunk of vital signs data was collected.

[0161] Rows 282 represent lab data. Each block 283A, 283B represents a separate time at which a separate chunk of lab data was collected.

[0162] Rows 284 represent image data. Blocks 285 represent the time at which a chunk of image data was collected.

[0163] Rows 286 represent text data. Blocks 287 represent the time at which a chunk of text data was collected.

[0164] The duration of hypertension, hyperthermia, and heart failure is determined by inference circuit 46 using the first, second, and third trained models described above.

[0165] User interface circuitry 48 displays the determined periods of hypertension as first boxes 276 overlaying rows 280 , 282 , 284 , 286 of timeline display 275 .

[0166] User interface circuitry 48 displays the determined duration of high fever as a second box 277 overlaying rows 280 , 282 , 284 , 286 of timeline display 275 .

[0167] User interface circuitry 48 displays the determined duration of heart failure as a third box 278 overlaying rows 280 , 282 , 284 , and 286 of timeline display 275 .

[0168] By viewing the timeline display 275, a user may see what data was collected during the period of high blood pressure, high fever, and / or heart failure. The user may select to view any of the collected data. For example, in some embodiments, the user may select any of blocks 281A-281D, 283A, 283B, 285, or 287 to obtain more information about the data corresponding to the selected block.

[0169] 13, the vital sign data is further shown in a vital sign graph 290. The vital sign graph 290 includes a line 291 that indicates the trend of the vital sign measurement. The line 291 connects a number of data points 292A-292H.

[0170] Superimposed on the vital signs graph 290 are multiple colored regions 293, 294A, 294B, 295 representing high blood pressure, high fever, and / or heart failure.

[0171] Region 293 is colored green and is represented by a dotted outline in Figure 13. Region 293 indicates a period of hypertension in the absence of both high fever and heart failure.

[0172] Regions 294A and 294B are colored yellow and are represented by dashed outlines in FIG. 13. Regions 294A and 294B indicate periods of high fever in the absence of heart failure. According to the temporal hierarchy of symptoms, high blood pressure is present during high fever. Therefore, during the periods of regions 294A and 294B, high blood pressure is present.

[0173] Region 295 is colored red and is represented by a solid outline in FIG. 13. Region 295 indicates a period of heart failure. According to the temporal hierarchy of symptoms, high blood pressure and high fever are present during heart failure. Therefore, during the period of region 295, high blood pressure and high fever are present.

[0174] 13, the laboratory data is further shown in a laboratory data graph 296. Graph 296 includes a line 297 that indicates the trend of the laboratory measurement. Line 296 connects a number of data points 298A-298H.

[0175] Colored regions 293, 294A, 294B, and 295 overlay graph 296 in the same manner as they overlay graph 290.

[0176] By overlaying areas that indicate periods of high blood pressure, high fever, and / or heart failure, a user may more easily identify which measurements were taken when which symptoms were present. For example, a user may distinguish between measurements taken when the patient had heart failure and measurements taken when the patient did not have heart failure.

[0177] The timeline of Figure 13 and each of the graphs of Figure 13 assist a user, such as a clinician, in interpreting data obtained from a patient. The user may be assisted in correlating individual data points or data sets with periods of high blood pressure, high fever, and / or heart failure.

[0178] 13 to view the periods of high blood pressure, high fever, and heart failure determined by reasoning circuitry 46. For example, the user may use clinical judgment to verify that they agree with the periods determined by reasoning circuitry 46.

[0179] 12 and 13, durations of high blood pressure, high fever, and heart failure are determined. In other embodiments, durations associated with any suitable clinical condition may be determined. For example, the clinical condition may be any suitable symptom.

[0180] In some embodiments, the temporal hierarchy is used in the context of cardiac time-series imaging. For example, in some embodiments, a first model analyzes the input time series and distinguishes between ventricular diastole and ventricular systole in the cardiac cycle. A second model may be constrained to function within the diastole. A second model may detect when arterial contraction occurs. A second model may be constrained to function within the systole. A third model may identify when ventricular ejection occurs.

[0181] 12 and 13, the model learns to determine time periods associated with clinical symptoms based on input data comprising a combination of vital sign data, lab data, image data, and text data. Using a combination of different types of data may provide a more complete picture of a patient's condition. In some situations, certain types of data may be available for periods when other types of data are not available.

[0182] In other embodiments, a model may be trained to determine time periods based on any suitable type or types of data. For example, in some embodiments, a model may be trained to determine time periods based solely on vital sign data. In some embodiments, a model may be trained to determine time periods based on a combination of vital sign data and image data.

[0183] In further embodiments, a model may be trained to perform a task on an object, such as an anatomical structure, based on a combination of image data and at least one other type of data, such as text data.

[0184] In some embodiments, a model is trained to perform object-related tasks and time-related tasks, and the output of the model may have both spatial and temporal hierarchies.

[0185] In the above embodiment, the outputs of the models are multiplied to obtain a combined output. In other embodiments, any method of combining model outputs may be used. The method of combining models may be weighted. The combination of models does not have to be multiplicative. For example, soft weighting may be used. Soft weighting may comprise the application of percentages or probabilities rather than the hard constraints of multiplication.

[0186] In some embodiments, levels of the hierarchy may not always be completely contained within other levels of the hierarchy, for example, for a given patient, symptoms of fever, high blood pressure, and hypertension may not be completely simultaneous in time.

[0187] A spatial hierarchy can exist when one object is mostly, but not completely, contained within another object. A temporal hierarchy can exist when one time period is mostly, but not completely, contained within another time period. A spatial hierarchy can exist when one object is usually, but not always, contained within another object. A temporal hierarchy can exist when one time period is usually, but not always, contained within another time period.

[0188] Although particular circuits are described herein, in alternative embodiments, the functionality of one or more of these circuits may be provided by a single processing resource or other component, or the functionality provided by a single circuit may be provided by a combination of two or more processing resources or other components. A reference to a single circuit encompasses multiple components that provide the functionality of that circuit, whether or not such components are separate from one another. A reference to multiple circuits encompasses a single component that provides the functionality of those circuits.

[0189] According to at least one of the embodiments described above, when determining whether the current handling of a drug being administered to a target patient is appropriate, it is possible to reduce the burden on doctors and assist in speeding up the response.

[0190] With respect to the above embodiment, the following supplementary notes are disclosed as one aspect and optional features of the invention. (Appendix 1) In a first aspect, a processing device is provided, comprising a processing circuit configured to apply a first trained model to input data comprising clinical data to obtain a first output based on the input data and to apply a second trained model to input data to obtain a second output based on the input image data, the first trained model and the second trained model being trained depending on a hierarchical relationship between the first output and the second output. The hierarchical relationship may comprise a spatial hierarchy. The hierarchical relationship may comprise a temporal hierarchy. The hierarchical relationship may comprise an anatomical hierarchy. The hierarchical relationship may comprise a clinical symptom hierarchy.

[0191] (Appendix 2) The input data may comprise medical image data.

[0192] (Appendix 3) The first output may relate to a first object, the second output may relate to a second object, and the hierarchical relationship between the first output and the second output may comprise a spatial hierarchy in which the second object is contained within a boundary of the first object.

[0193] (Appendix 4) The first output may comprise a first anatomical structure, and the second output may comprise a second anatomical structure.

[0194] (Appendix 5) The first output may comprise a first time period associated with a first type of clinical condition, the second output may comprise a second time period associated with a second type of clinical condition, and the hierarchical relationship between the first output and the second output may comprise a temporal hierarchy in which the second time period is contained within the first time period.

[0195] (Appendix 6) Training the first model and the second model may comprise providing feedback to the first model if the output of the second model is erroneous.

[0196] (Appendix 7) The processing circuitry may be configured to multiply the first output and the second output to obtain a combined output.

[0197] (Appendix 8) The processing circuitry may be further configured to apply at least one first additional trained model to the input data, and obtaining the first output may comprise combining an output of the first trained model and an output of the first additional trained model.

[0198] (Appendix 9) The processing circuitry may be further configured to apply at least one second additional trained model to the input data, and obtaining the second output may comprise combining an output of the second additional trained model with an output of the second additional trained model.

[0199] (Appendix 10) The processing circuitry may further display the first output and / or the second output and / or the combined input on a display screen.

[0200] (Appendix 11) In a further aspect that may be provided independently, there is provided a method of processing comprising applying a first trained model to input data comprising clinical data to obtain a first output based on the input data, and applying a second trained model to the input data to obtain a second output based on the input data, wherein the first trained model and the second trained model have been trained depending on a hierarchical relationship between the first output and the second output. The hierarchical relationship may comprise a spatial hierarchy. The hierarchical relationship may comprise a temporal hierarchy. The hierarchical relationship may comprise an anatomical hierarchy. The hierarchical relationship may comprise a clinical symptom hierarchy.

[0201] (Appendix 12) The input data may comprise medical image data.

[0202] (Appendix 13) In a further aspect that may be provided independently, a processing device is provided that includes processing circuitry configured to perform a training process, the training process comprising training a first model to obtain a first output from input data comprising clinical data and training a second model to obtain a second output from the input data, wherein the training of the first model and the second model is dependent on a hierarchical relationship between the first output and the second output. The hierarchical relationship may comprise a spatial hierarchy. The hierarchical relationship may comprise a temporal hierarchy. The hierarchical relationship may comprise an anatomical hierarchy. The hierarchical relationship may comprise a clinical symptom hierarchy.

[0203] (Appendix 14) The input data may comprise medical image data.

[0204] (Appendix 15) The first output may relate to a first anatomical structure, the second output may relate to a second anatomical structure, and the hierarchical relationship between the first output and the second output may comprise a spatial hierarchy in which the second anatomical structure is contained within a boundary of the first anatomical structure.

[0205] (Appendix 16) The first output may comprise a first time period associated with a first type of clinical condition, the second output may comprise a second time period associated with a second type of clinical condition, and the hierarchical relationship between the first output and the second output may comprise a temporal hierarchy in which the second time period is contained within the first time period.

[0206] (Appendix 17) The training process may comprise alternating between a) and b). a) training the first model in isolation; b) Jointly training the first model and the second model.

[0207] (Appendix 18) The joint training of the first model and the second model may comprise combining an output of the first model and an output of the second model via a multiplication operation to obtain a combined input.

[0208] (Appendix 19) The joint training of the first model and the second model may comprise providing feedback to the first model if the output of the second model is erroneous.

[0209] (Appendix 20) The training of the first model to obtain a first output from input data may comprise training the first model to perform a first task, which may comprise at least one of pixel or voxel-level classification, segmentation, bounding box regression, region detection, and image-level classification.

[0210] (Appendix 21) The training of the second model to obtain a second output from the input data may comprise training the second model to perform a second task, which may comprise at least one of pixel or voxel-level classification, segmentation, bounding box regression, region detection, and image-level classification.

[0211] (Appendix 22) The training of the first model may be performed using a first training data set. The training of the second model may be performed using a different second training data set. The first training data set may comprise ground truth labeled data of the first output. The second training data set may comprise ground truth labeled data of the second output.

[0212] (Appendix 23) The processing circuitry may be further configured to pre-train the first model and / or the second model.

[0213] (Appendix 24) The processing circuitry may be further configured to train at least one further model to obtain at least one further output from the input data, wherein the training of the at least one further model may be performed depending on a hierarchical relationship between the first output, the second output, and the at least one further output.

[0214] (Appendix 25) The processing circuitry may further train at least one first additional model to obtain the first output, and at least one second additional model to obtain the second output.

[0215] (Appendix 26) In a further aspect, there is provided a method of training a first model and a second model using a training process, the training process comprising training a first model to obtain a first output from input data comprising clinical data and training a second model to obtain a second output from the input data, wherein the training of the first model and the second model relies on a hierarchical relationship between the first output and the second output. The hierarchical relationship may comprise a spatial hierarchy. The hierarchical relationship may comprise a temporal hierarchy. The hierarchical relationship may comprise an anatomical hierarchy. The hierarchical relationship may comprise a clinical symptom hierarchy.

[0216] (Appendix 27) The input data may comprise medical image data.

[0217] (Appendix 28) In a further aspect, which may be provided independently, there is provided a method for training a deep learning model architecture using backpropagation, the method comprising: 1. Multiple image interpretation tasks where the relationships between tasks are hierarchical in nature; 2. A plurality of deep learning neural network models for performing the image interpretation task; 3. Model outputs are combined through successive hierarchical weighting; 4. During training, the training alternates between models after each training batch; 5. During inference, model outputs are simply combined through successive hierarchical weighting; Equipped with.

[0218] (Appendix 29) The tasks can be voxel-level classification (segmentation), bounding box regression, region detection, image-level classification, or a mixture of the above. One or more of the models can be pre-trained before hierarchical multi-task transfer via model symbiosis. More than one model can be applied to each image interpretation task.

[0219] (Appendix 30) In a further aspect that may be provided independently, a processing device is provided that includes processing circuitry configured to apply a neural network to input data, the neural network including a first model and a second model, the first model outputting a first output based on the input data, and the second model outputting a second output based on the input data, the second output having a hierarchical relationship to the first output.

[0220] (Appendix 31) The first and second models may be configured to provide feedback to a complementary model if the output of the first or second model is erroneous.

[0221] (Appendix 32) The hierarchical relationship may be one in which a second object related to the second result is contained within a boundary of a first object related to the first result.

[0222] (Appendix 33) In a further aspect, which may be provided independently, there is provided a model training method for training a first model to perform a first image analysis task with respect to a first anatomical structure and training a second model to perform a second image analysis task with respect to a second anatomical structure, wherein the first anatomical structure and the second anatomical structure have a spatial hierarchical relationship such that the second anatomical structure, if present, occurs anatomically within the first anatomical structure, and the training of the first and second models comprises alternating between a) and b). a) training the first model in isolation; b) Jointly training the first model and the second model. Thereby, the first model and the second model are trained to match the spatial hierarchical relationship.

[0223] (Appendix 34) The first model and the second model may each comprise a respective deep learning model, and the deep learning models may comprise a deep learning neural network.

[0224] (Appendix 35) The spatial hierarchical relationship may be explicitly imparted by hierarchical training. The outputs of the first and second models may be combined by successive hierarchical weighting. The joint training of the first and second models may comprise combining the first and second models by a multiplication operation.

[0225] (Appendix 36) The training of the first model and the second model may comprise training the first model on a first data set labeled for the first image analysis task, and training the second model on a different second data set labeled for the second image analysis task.

[0226] (Appendix 37) The first image analysis task may comprise at least one of voxel-level classification, segmentation, bounding box regression, region detection, and image-level classification. The second image analysis task may comprise at least one of voxel-level classification, segmentation, bounding box regression, region detection, and image-level classification. The first image analysis task may comprise lung segmentation, and the second image analysis task may comprise lung nodule detection.

[0227] (Appendix 38) At least one of the first model and the second model may be pre-trained.

[0228] (Appendix 39) The joint training of the first model and the second model may include at least one of a) to d). a) If it is a false positive that results in an incorrect prediction for both the first image analysis task and the second image analysis task, backpropagate the error to both the first model and the second model. b) penalizing the second model for false negatives that result in correct predictions for the first image analysis task and incorrect predictions for the second image analysis task; c) penalizing the first model for false negatives that result in an incorrect prediction in the first image analysis task and a correct prediction in the second image analysis task; d) penalizing both the first model and the second model for false negatives that result in incorrect predictions for both the first image analysis task and the second image analysis task;

[0229] (Appendix 40) The method may further comprise training at least one further model to perform at least one further image analysis task relating to at least one further anatomical structure having a spatial hierarchical relationship to the first anatomical structure and / or the second anatomical structure.

[0230] (Appendix 41) The method may further comprise training at least one further model to perform the first image analysis task such that a plurality of models are trained to perform the first image analysis task, and / or training at least one additional model to perform the second image analysis task such that a plurality of models are trained to perform the second image analysis task.

[0231] (Appendix 42) In a further aspect, which may be provided independently, there is provided a method of analysis comprising receiving data representing a subject, and applying a first model and a second model trained according to the methods claimed or described herein to the data to perform the first analysis task with respect to the first anatomical structure and the second analysis task with respect to the second anatomical structure, wherein the data may comprise image data, such as medical image data.

[0232] (Appendix 43) The applying of the first model and the second model may comprise obtaining an output of the first model, obtaining an output of the second model, processing the output of the first model to determine information related to the first anatomical structure, multiplying the output of the first model by the output of the second model, and processing a result of the multiplication to determine information related to the second anatomical structure.

[0233] (Appendix 44) Features of one embodiment may be provided as features of any other embodiment, as appropriate. For example, method features may be provided as apparatus features, and vice versa. Any one or more features of one embodiment may be provided in combination with any suitable one or more features of the other embodiments.

[0234] Although several embodiments have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, and combinations of embodiments can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, as well as within the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0235] 30 Data processing device 32 Computing equipment (medical information processing equipment) 34 Scanner 36 display screens 38 Input Devices 40 Data storage unit 42 Processing equipment 44 Training Circuit 46 Inference circuit 48 Display circuit

Claims

1. a processing circuit for training a first model that receives medical data and outputs first output data, and a processing circuit for training a second model that receives medical data and outputs second output data; the first output data includes a first object; the second output data includes a second object; performing an alternating training process that alternates between combined training, in which backpropagation to at least one of the first model and the second model is performed based on an error of the third output data obtained by comparing the third output data with ground truth data, and independent training, in which backpropagation to the first model is performed based on an error of the first output data obtained by comparing the first output data with ground truth data, using a relationship based on the spatial hierarchy among the first output data, the second output data, and third output data obtained by combining the first output data and the second output data by a multiplication process, when there is a spatial hierarchy in which the second object is contained within the boundary of the first object; Medical information processing equipment.

2. The medical image processing apparatus of claim 1 , wherein the medical data comprises medical image data.

3. combining the training of the first model with the training of the second model, and providing feedback to the training of the first model when the second output data of the second model is erroneous; The medical information processing device according to claim 1 or 2.

4. the first output data is obtained by combining the first model with at least one first additional trained model combined with the first model; and the second output data is obtained by a combination of the second model and at least one second additional trained model combined with the second model; at least one of The medical information processing device according to claim 1 or 2.

5. displaying at least one of the first output data, the second output data, and the medical data on a display circuit; The medical information processing device according to claim 1 or 2.

6. the first model is trained by alternating between training in isolation from the second model and training together with the second model; The medical information processing device according to claim 1 or 2, comprising:

7. training the first model includes training the first model to perform a first task; training the second model includes training the second model to perform a second task; At least one of the first task and the second task includes at least one of pixel or voxel level classification, segmentation, bounding box regression, region detection, and image level classification. The medical information processing device according to claim 1 or 2.

8. the training of the first model is performed using a first training data set labeled with ground truth of the first output data; the training of the second model is performed using a second training dataset that is labeled with ground truth for the second output data and that is different from the first training dataset. The medical information processing device according to claim 1 or 2.

9. At least one of the first model and the second model is pre-trained. The medical information processing device according to claim 1 or 2.

10. On the computer, A medical information processing program for executing training of a first model that receives medical data and outputs first output data, and training of a second model that receives medical data and outputs second output data, the first output data includes a first object; the second output data includes a second object; and performing an alternating training process that alternates between a combined training process in which backpropagation is performed to at least one of the first model and the second model based on an error of the third output data obtained by comparing the third output data with ground truth data, and an independent training process in which backpropagation is performed to the first model based on an error of the first output data obtained by comparing the first output data with ground truth data, using a relationship based on the spatial hierarchy among the first output data, the second output data, and third output data obtained by combining the first output data and the second output data by a multiplication process, when there is a spatial hierarchy in which the second object is contained within the boundary of the first object. Medical information processing program.

Images