Improving the performance of machine learning models for automatically quantifying coronary artery disease
Patent Information
- Application Number
- CN202010885335.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-08-30
- Filing Date
- 2020-08-28
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2040-08-28
AI Technical Summary
Existing machine learning models have problems such as insufficient generalization ability, difficulty in verification and difficulty in feedback in the diagnosis of coronary artery disease, making it difficult for users to directly verify or improve model performance.
Improve the model's predictive ability for primary and secondary tasks by receiving medical images and predicting metrics of interest using trained machine learning models, and receiving user feedback for online retraining.
The accuracy of machine learning models and the controllability of users' model performance are improved, and the model is continuously optimized through user feedback to enhance the generalization ability of unseen data.
Smart Images

Figure CN112446499B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates generally to improving the performance of machine learning models, and more particularly to improving the performance of machine learning models for automatically quantifying coronary artery disease based on user feedback. Background Art
[0002] Machine learning models have been applied to perform a variety of tasks for an increasing number of applications. In one example, a machine learning model has been applied to characterize coronary artery disease by predicting the functional severity of coronary artery lesions. Such a machine learning model can be trained to perform a task using labeled training images to map a set of input features to predicted output values. In supervised learning, during the training phase, the machine learning model is optimized to maximize prediction accuracy by evaluating the output predicted from the training images relative to their labels. Machine learning models are typically trained for a specific task.
[0003] The challenges associated with such machine learning models are numerous. One challenge is generalization - a trained machine learning model may not generalize correctly to unseen data. This is also true in cases where a trained machine learning model is too specific to the training images or to the task for which it was trained. Another challenge is control - a user may not have a direct way to verify the correct behavior of a trained machine learning model, such as in the case where a trained machine learning model predicts a fractional flow reserve (FFR) value. Another challenge is feedback - a user may not be able to easily provide feedback to improve the performance of a trained machine learning model because the user may not be able to easily identify or easily evaluate input features. Summary of the invention
[0004] According to one or more embodiments, a system and method for online retraining of a trained machine learning model is provided. One or more input medical images are received. Using the trained machine learning model, a metric of interest for a primary task and a secondary task is predicted based on the one or more input medical images. The predicted metric of interest for the primary task and the secondary task is output. User feedback on the predicted metric of interest for the secondary task is received. Based on the user feedback on the output for the secondary task, the trained machine learning model is retrained for predicting the metric of interest for the primary task and the secondary task.
[0005] In one embodiment, the user cannot directly verify the metrics of interest for the primary task based on the one or more input medical images, and the user can directly verify the metrics of interest for the secondary task based on the one or more input medical images. The metrics of interest for the primary task may include a hemodynamic indicator, such as, for example, a virtual fractional flow reserve. The metrics of interest for the secondary task may include at least one of the following: the location of a measurement point, the location of a common image point, the location of a stenosis, and the segmentation of a vessel.
[0006] In one embodiment, the user feedback may include acceptance or rejection of the predicted metric of interest for the secondary task, or modification of the predicted metric of interest for the secondary task.
[0007] In one embodiment, a measure of confidence may also be determined. The measure of confidence may be determined by predicting additional measures of interest for the primary task and the secondary task based on the one or more input medical images using the retrained machine learning model, and determining a measure of confidence in the predicted additional measures of interest for the primary task and the secondary task based on a difference between the measures of interest predicted using the trained machine learning model and the additional measures of interest predicted using the retrained machine learning model. A measure of confidence may also be determined by predicting, using a trained machine learning model, measures of interest for a primary task and a secondary task based on a single input medical image from a first image sequence and a single input medical image from a second image sequence; predicting, using a trained machine learning model, additional measures of interest for the primary task and the secondary task based on two input medical images from the first image sequence and two input medical images from the second image sequence; and determining a measure of confidence in the predicted measures of interest and / or the predicted additional measures of interest for the primary task and the secondary task based on the difference between: 1) the measures of interest predicted using the trained machine learning model based on the single input medical image from the first image sequence and the single input medical image from the second image sequence, and 2) additional measures of interest predicted using the trained machine learning model based on the two input medical images from the first image sequence and the two input medical images from the second image sequence.
[0008] In one embodiment, the one or more input medical images are selected by: receiving corresponding electrocardiogram signals acquired during acquisition of a first image sequence and a second image sequence; associating indicators with each image in the first image sequence and the second image sequence based on the corresponding electrocardiogram signals of the first image sequence and the second image sequence; matching images of the first image sequence with images of the second image sequence based on the associated indicators of the first image sequence and the second image sequence; and selecting the matched images of the first sequence and the second sequence as the one or more input medical images.
[0009] These and other advantages of the present invention will become apparent to those of ordinary skill in the art by reference to the following detailed description and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] Figure 1 A method for online retraining of a trained machine learning model based on user feedback is shown;
[0011] Figure 2 A method for training a machine learning model for predicting a metric of interest for a primary task and one or more secondary tasks is shown;
[0012] Figure 3 Exemplary training data is shown;
[0013] Figure 4 A workflow for synchronizing a first training image time series and a second training image time series is shown;
[0014] Figure 5 shows the network architecture of the machine learning model; and
[0015] Figure 6 A high-level block diagram of a computer is shown. DETAILED DESCRIPTION
[0016] The present invention generally relates to methods and systems for improving the performance of machine learning models based on user feedback. Embodiments of the present invention are described herein to give an intuitive understanding of methods for improving the performance of machine learning models based on user feedback. A digital image is typically composed of digital representations of one or more objects (or shapes). Digital representations of objects are generally described herein in terms of identifying and manipulating objects. Such manipulations are virtual manipulations performed in the memory or other circuitry / hardware of a computer system. Therefore, it should be understood that embodiments of the present invention can be performed within a computer system using data stored within the computer system.
[0017] Furthermore, it should be understood that while the embodiments discussed herein may be discussed with respect to improving the performance of a machine learning model trained to predict a metric of interest based on one or more input medical images, the invention is not limited thereto. Embodiments of the invention may be applied to improve the performance of a machine learning model trained to predict any metric of interest for performing any type of task using any type of input.
[0018] According to one or more embodiments, a system and method for improving the performance of a trained machine learning model based on user feedback is provided. The trained machine learning model is trained to predict metrics of interest for multiple tasks (a primary task and one or more secondary tasks). The user cannot directly verify the metric of interest for the primary task, but the user can directly verify the metric of interest for one or more secondary tasks. By training the trained machine learning model to predict the metric of interest for the primary task and one or more secondary tasks, the user can evaluate the one or more secondary tasks and provide such evaluation as user feedback for online retraining of the trained machine learning model. Advantageously, retraining the machine learning model using user feedback on the metric of interest for one or more secondary tasks improves the performance (e.g., accuracy) of the trained machine learning model for predicting the metric of interest for all tasks (i.e., the primary task and one or more secondary tasks).
[0019] Figure 1 A method 100 for online retraining of a trained machine learning model based on user feedback is shown according to one or more embodiments. The method 100 is performed using a trained machine learning model during an online or testing phase. The trained machine learning model was previously trained during an offline or training phase, for example, based on Figure 2 The method 200 is trained to perform multiple tasks: a primary task and one or more secondary tasks. The steps of the method 100 can be performed by any suitable computing device (such as Figure 6 The computer 602) executes.
[0020] At step 102, one or more input medical images are received. The one or more input medical images may be any images suitable for performing the primary task and the one or more secondary tasks. The one or more input medical images may be of any suitable modality or combination of modalities, such as, for example, computed tomography (CT), dynaCT, x-ray, magnetic resonance imaging (MRI), ultrasound (US), etc. The input medical images may be directly from an image acquisition device (e.g., a computer system) used to acquire the input medical images, by loading a previously acquired medical image from a storage device or memory of the computer system, or by receiving a medical image that has been transmitted from a remote computer system. Figure 6The image acquisition device 614 receives the one or more input medical images.
[0021] In one embodiment, the one or more input medical images include a pair (or more) of input medical images, each of which is from a corresponding image sequence. An image sequence refers to a plurality of images of an object of interest acquired within a period of time. For example, an image sequence may be a coronary angiography sequence. In one embodiment, the pair of input medical images are selected from their corresponding image sequences by synchronizing the images in each corresponding sequence and selecting synchronized or matching images from each sequence as the pair of input medical images, for example as follows Figure 4 described.
[0022] In one embodiment, additional data may also be received with the one or more input medical images. The additional data may include any suitable data for performing the primary task and the one or more secondary tasks. For example, the additional data may include patient data, such as, for example, a priori acquired medical (imaging or non-imaging) data, past medical examinations, etc. In another example, the additional data may include angle information associated with the image acquisition device during acquisition of the one or more input medical images (e.g., the angle of the C-arm of the image acquisition device used to acquire each time series).
[0023] At step 104, the metrics of interest for the primary task and one or more secondary tasks are predicted based on the one or more input medical images (and optionally based on the patient data, if any) using the trained machine learning model. The trained machine learning model may be based on, for example Figure 2 The method 200 is trained during a previous training phase. The trained machine learning model can be based on any suitable machine learning model, such as, for example, a neural network. In one embodiment, the trained machine learning model is a deep neural network. For example, the trained machine learning model can be based on a modified 3D recursive reconstruction neural network (3D-R2N2). In one embodiment, the trained machine learning model is trained using multi-task learning to predict metrics of interest for a primary task and one or more secondary tasks, and user feedback on the one or more secondary tasks is received for online retraining using interactive machine learning. According to one embodiment, the following reference is made to Figure 5 The network structure of the trained machine learning model is discussed in more detail.
[0024] The primary task and the one or more secondary tasks may be any suitable tasks that are related or associated. In one embodiment, the primary task and the one or more secondary tasks are related because they may be performed based on the one or more input medical images. Although the embodiments described herein may involve a primary task and one or more secondary tasks, it should be understood that any number of primary tasks and secondary tasks may be utilized.
[0025] A human user cannot directly verify the metric of interest for the primary task based on the one or more input medical images. In one embodiment, the primary task includes predicting a functional metric or quantification of coronary artery disease at a measurement location (e.g., after each stenotic segment). For example, the metric of interest may be a value of a virtual fractional flow reserve (FFR), which is a functional metric for quantifying the hemodynamic significance of a stenosis in an artery. FFR is typically determined based on the pressure drop of a coronary artery stenosis using invasive pressure line-based measurements during hyperemia. Virtual FFR attempts to reproduce the FFR value via less invasive means. It should be understood that the metric of interest may be any other hemodynamic indicator, such as, for example, coronary flow reserve (CFR), instantaneous wave-free ratio (iFR), basic stenosis resistance (BSR), hyperoxia stenosis resistance (HSR), microcirculatory resistance index (IMR), or any other metric or quantification of coronary artery disease. In another embodiment, the primary task is a clinical decision (e.g., made during or after an intervention), such as, for example, whether to perform percutaneous coronary intervention (PCI) or coronary artery bypass grafting (CABG), optimal medical treatment, date of next examination, etc.
[0026] A human user can directly examine the metrics of interest for the one or more secondary tasks based on the one or more input medical images. Examples of the one or more secondary tasks include: predicting the location of a standard measurement in the input medical image; predicting the location of one or more common image points (e.g., common anatomical landmarks) in the input medical image; predicting the location of a stenosis marker in the input medical image; predicting the location and anatomical significance of one or more stenoses in the input medical image; predicting the segmentation of one, several, or all blood vessels visible in the input medical image; predicting Thrombolysis in Myocardial Infarction (TIMI) frame counts in the input medical image as a surrogate for contrast agent velocity; predicting healthy radii at all locations in the input medical image (e.g., in the case of diffuse disease and bifurcation stenoses); predicting the centerline or tree structure of a blood vessel in the input medical image; predicting depth information of a blood vessel in the input medical image (as a surrogate for detecting vessel overlap); predicting the myocardial region associated with each main branch or side branch in the input medical image, etc. The one or more secondary tasks are determined during the training phase to maximize the performance of the primary task, as described below with respect to Figure 2 The method 200 is further described.
[0027] In step 106, the predicted metrics of interest for the primary task and the one or more secondary tasks are output. In one embodiment, outputting the predicted metrics of interest for the primary task and the one or more secondary tasks includes: for example, visually displaying the predicted metrics of interest on a display device of a computer system. For example, the predicted metrics of interest for the primary task and the one or more secondary tasks may be displayed together with the one or more medical images to facilitate a user's evaluation of the predicted metrics of interest (e.g., for the one or more secondary tasks). Outputting the predicted metrics of interest for the primary task and the one or more secondary tasks may also include: storing the predicted metrics of interest for the primary task and the one or more secondary tasks on a memory or storage device of the computer system or transmitting the predicted metrics of interest for the primary task and the one or more secondary tasks to a remote computer system.
[0028] At step 108, user feedback is received regarding the predicted metric of interest for the one or more secondary tasks. The user's ability to directly examine the one or more secondary tasks enables the user to provide such user feedback. The user feedback may take any suitable form. In one embodiment, the user feedback is a user's acceptance or rejection of the predicted metric of interest for the one or more secondary tasks. In another embodiment, the user feedback is a user input to correct or modify the predicted metric of interest for the one or more secondary tasks. For example, a user may interact with a user interface to select a common image point that was incorrectly predicted and move the common image point to a correct or desired position (e.g., in 2D or 3D space).
[0029] At step 110, based on the received user feedback on the predicted metrics of interest for the one or more secondary tasks, the trained machine learning model is retrained to predict the metrics of interest for the primary task and the one or more secondary tasks. In one embodiment, additional training data including the one or more input medical images and the user feedback (as ground truth values) is formed, and such additional training data is used, for example, to Figure 2Method 200 retrains a trained machine learning model. For example, if a user corrects the position of a common image point in one of the input medical images, the one or more input medical images with the corrected position of the common image point are used as ground truth values to retrain the trained machine learning model. In another example, if a user rejects the predicted metrics of interest for the one or more secondary tasks, the rejected predicted metrics of interest can be used as a negative example for retraining the trained machine learning model. Because the machine learning model has a shared layer, such retraining based on user feedback about the predicted metrics of interest for the one or more secondary tasks implicitly results in updating the machine learning model used to predict metrics of interest for all tasks (i.e., the primary task and one or more secondary tasks). The steps of method 100 can be performed any number of times for one or more newly received input medical images.
[0030] In one embodiment, the trained machine learning model may determine a measure of confidence in the predicted metric of interest based on how much the predicted metric of interest has changed after retraining. In particular, after predicting the metric of interest for the primary task and one or more secondary tasks using the trained machine learning model, the retrained machine learning model may be used to predict additional metrics of interest for the primary task and one or more secondary tasks. A measure of confidence in the metric of interest (predicted using the trained machine learning model) and / or the additional metric of interest (predicted using the retrained machine learning model) may be determined based on the change or difference between the metric of interest and the additional metric of interest. In one example, after retraining the machine learning model for the secondary task of predicting the location of a common image point, the primary task of predicting the FFR value may change. Depending on the change or difference between the FFR values, a confidence level of the result may be determined. In one embodiment, the confidence level is based on a threshold value, such that if the change in the FFR value after retraining is greater than a threshold amount (e.g., 0.1), the confidence level of the result is low.
[0031] In another embodiment, the trained machine learning model can provide a measure of confidence in the predicted metric of interest by applying the trained machine learning model multiple times to the one or more input medical images. In one example, the trained machine learning model can be applied multiple times, each time considering a different number of images from each time series. For example, a predicted first metric of interest can be obtained by considering only a single 2D image from each time series as the one or more input medical images, a predicted second metric of interest can be obtained by considering only two 2D images from each time series as the one or more input medical images, and so on. In another example, if the one or more input medical images include a sequence of images during multiple heartbeats, and the same behavior is assumed at each heartbeat, the trained machine learning model can be applied multiple times, each time to a different heartbeat. Depending on the change or difference between the predicted metrics of interest each time the trained machine learning model is applied, a confidence measure for the predicted first and / or second metrics of interest can be determined (e.g., based on a threshold).
[0032] In another embodiment, the measure of confidence can be based on patient characteristics (e.g., age or other demographics, patient history) as well as image characteristics (e.g., how clear is the transition from blood vessels to non-vessels in an angiographic image). Thus, different categories of patients and / or image characteristics can be identified, and different confidence levels can be defined for these different categories.
[0033] In one embodiment, if the primary task is to predict virtual CFR, the one or more input medical images (received at step 102) may include angiographic images acquired at relaxation and at hyperemia. The one or more secondary tasks in this embodiment may include TIMI frame counts determined for relaxation and hyperemia, respectively.
[0034] Figure 2 A method 200 for training a machine learning model for predicting a metric of interest for a primary task and one or more secondary tasks is shown in accordance with one or more embodiments. The method 200 is performed during an offline or training phase to train the machine learning model. In one embodiment, the machine learning model trained according to the method 200 may be applied during an online or testing phase to perform Figure 1 The steps of method 200 may be performed by any suitable computing device (such as Figure 6 The computer 602) is used to execute.
[0035] In step 202, training data is received. The training data includes training images labeled (or annotated) with ground truth values. The training images may be of any suitable modality or combination of modalities, such as, for example, CT, dynaCT, x-ray, MRI, US, etc. The training images may be directly obtained from an image acquisition device (e.g., Figure 6 The training image may also be a synthetically generated training image. In one embodiment, the training data also includes patient data.
[0036] In one embodiment, the training images include several pairs (or more) of training images and their mutual angle information (e.g., the angle of the C-arm of the image acquisition device used to acquire each time series), and each pair of training images is selected from a corresponding image sequence (e.g., angiograms). In one embodiment, each pair of training images is selected from their corresponding training image sequences by synchronizing the training images in each corresponding sequence and selecting synchronized or matched training images from each sequence as the pair of training images, for example, as described below with respect to Figure 4 as described.
[0037] For each pair of training images, annotations (i.e., ground truth values) are provided to identify metrics of interest in the training images. The metrics of interest are based on the task that the machine learning model will be trained to perform. For example, the metrics of interest may include the locations of common image points in the training images, stenosis markers, measurement locations, etc., and the FFR values or other hemodynamic metric values (e.g., CFR, iFR, BSR, HSR, IMR, etc.) associated with these locations in the training images. In one embodiment, the annotations are defined as a list of locations (e.g., landmarks) and the corresponding FFR values (or other hemodynamic metric values) for each location. An example of an annotation list is as follows:
[0038] Landmark 0: the location of the common image point of each image in the pair of training images;
[0039] Landmarks 1 to N: the location of the stenosis markers in each image of the pair of training images;
[0040] Landmarks N+1 to 2N+1: FFR measurement locations of each image in the pair of training images, ordered consistently with the locations of stenosis markers so that the FFR measurement locations can be associated with the corresponding stenosis; and
[0041] FFR values 1 to N: FFR values between 0 and 1, ordered consistently with the location of the stenosis markers so that the FFR value can be associated with the corresponding stenosis.
[0042] The annotated ground truth FFR values may be invasively measured or generated by a virtual FFR predictor (eg, based on computational modeling or based on regression / machine learning). In some embodiments, the training images may be synthetically generated from a synthetically generated 3D coronary artery model, for example, according to known methods.
[0043] Figure 3 Exemplary training data 300 according to one embodiment is shown. Training data 300 includes a pair of training images 302 and 304. Training images 302 can each be from a corresponding image sequence. For illustrative purposes, training images 302 and 304 show the following annotations: common image point 306, stenosis marker 312 and measurement location 308 and corresponding invasively measured FFR value 310.
[0044] exist Figure 2 At step 204, features of interest are extracted from the training data. Typically, during the method 200 for training a machine learning model, the machine learning model implicitly defines features of interest. For example, image data (e.g., image intensity, pixel values) can be directly provided as input to some dedicated network layer (e.g., convolution filters in a convolutional neural network), and as part of the training process, the parameters of the convolution filters will be optimized so that the resulting features (i.e., the output of the filters) will be the best predictor of the ground truth labeling. In addition or alternatively, "hand-made" features can be used. In one example, such hand-made features can be determined by segmenting the image into small blocks and (optionally) applying filters to such small blocks, for example, for enhancing edges. In another example, such hand-made features can be determined by extracting relevant information from patient data or medical history; for example, defining a vector of categorical variables (e.g., one variable for each pre-existing or related condition (such as hypertension, blood cholesterol concentration, etc.)) and assigning values to all variables according to the medical history or status of a particular patient. The network can use this as an additional "feature vector".
[0045] At step 206, metrics of interest are extracted from the training data. As described above, metrics of interest may include, for example, locations of common image points, stenosis markers, measurement locations, FFR values, or other hemodynamic metric values (e.g., CFR, iFR, BSR, HSR, IMR, etc.). For example, the metrics of interest may be extracted by parsing the annotation list. It should be understood that step 206 may be performed at any time prior to step 208 (e.g., before step 204, after step 204, or simultaneously with (e.g., in parallel with) step 204).
[0046] At step 208, the features of interest extracted from the training data are used to train a machine learning model for predicting the metrics of interest for the primary task and one or more secondary tasks. Any suitable method may be used to train the machine learning model, such as, for example, regression, instance-based methods, regularization methods, decision tree learning, Bayesian, kernel methods, clustering methods, association rule learning, artificial neural networks, dimensionality reduction, ensemble methods, etc. In one example, based on Figure 3 With the training data 300, a machine learning model can be trained for the primary task of predicting virtual FFR values and for multiple secondary tasks of predicting the locations of common image points, stenosis markers, and measurement locations.
[0047] The machine learning model can be based on any suitable machine learning model, such as, for example, a neural network. In one embodiment, the machine learning model is a deep neural network. For example, the deep neural network can be based on a modified 3D-R2N2 network. According to one embodiment, the following reference Figure 5 The network structure of the trained machine learning model is discussed in more detail.
[0048] In one embodiment, a machine learning model is trained using multi-task learning to predict metrics of interest for a primary task and one or more secondary tasks. Multi-task learning is based on the idea that focusing on a single task may prevent the machine learning model from including useful information that may come from learning related tasks. Multi-task learning is achieved by sharing parameters in the hidden layers of the machine learning model. Parameter sharing can be, for example: hard sharing, where hidden layers are shared between all tasks, while multiple output layers are task-specific; or soft sharing, where each task has a unique model with its own parameters, but constraints are added to maximize the similarity of parameters between tasks.
[0049] In multi-task learning, it is important to identify the one or more secondary tasks to maximize the performance of the primary task. In one embodiment, the one or more secondary tasks can be identified by, for example, proposing related tasks to the machine learning model (e.g., proposing a secondary task of predicting the severity of anatomical stenosis for the primary task of predicting virtual FFR), identifying the one or more secondary tasks as tasks that predict features that are not easily learned from the primary task (e.g., for the primary task of predicting virtual FFR, identifying a secondary task of predicting whether the stenosis is significant), or identifying the one or more secondary tasks used to focus attention on specific parts of the input medical image (e.g., for the primary task of predicting virtual FFR, identifying a secondary task of catheter tip detection, or identifying a secondary task that forces the machine learning model to learn to represent the distance from the coronary artery ostium).
[0050] In the context of predicting virtual FFR (or other hemodynamic indicators) as the primary task, multi-task learning enables training a unified machine learning model that maps image features of two or more input medical images to a virtual FFR for a detected stenosis. The one or more secondary tasks may include, for example, predicting the location of the stenosis and predicting the location of anatomical landmarks visible in the input medical image. The performance of the one or more secondary tasks may be evaluated by a user (e.g., visually) to evaluate the performance of the machine learning model.
[0051] User feedback on the performance of the one or more secondary tasks may be received for online retraining of the machine learning model using interactive machine learning. Interactive machine learning enables users to provide feedback to the machine learning model, thereby enabling online retraining. Using a multi-task learning approach, interactive machine learning may be applied to the one or more secondary tasks (for which user feedback is easier to provide and may be more robust to inter-user variability) to jointly provide feedback on the performance of the primary task and the one or more secondary tasks.
[0052] Figure 4 400 for synchronizing a first image sequence 406 and a second image sequence 408 according to one or more embodiments. In one embodiment, the workflow 400 may be performed to synchronize images from the image sequences to determine the Figure 1 One or more input medical images received at step 102 or Figure 2 In another embodiment, synchronization is one of the one or more secondary tasks, and workflow 400 is only executed for Figure 2 The training phase at step 202 prepares the training data. Although the workflow 400 is described as synchronizing a pair of image sequences, it should be understood that the workflow 400 can be applied to synchronize any number of image sequences.
[0053] In the workflow 400, electrocardiogram (ECG) signals 402 and 404 of a patient are measured or received during acquisition of a first image time series (or sequence) 406 and a second image time series 408, respectively. An index t is associated with each image in the first time series 406 and the second time series 408, representing a time relative to a heart cycle or cardiac cycle. In one embodiment, the index is a value between 0 and 1, where 0 represents the beginning of systole and 1 represents the end of diastole, to effectively subdivide the images of the time series 406 and 408 into a plurality of subsequences, each corresponding to a cardiac cycle.
[0054] The images within the subsequence of the selected cardiac cycles 412 in the first time series 406 and the second time series 408 are synchronized or matched based on their indices to provide synchronized images 410. In one embodiment, the images within the subsequence of the selected cardiac cycles 412 in the first time series 406 and the second time series 408 are matched where the difference between the indices t associated with the images is minimized. The images are synchronized such that each image in the first time series 406 has one and only one corresponding image in the second time series 408. Unmatched images may optionally be discarded.
[0055] In one embodiment, where an ECG signal is not available, relative times in the cardiac cycle (eg, systolic and diastolic times) are determined based on the vascular motion depicted in the image, for example, according to known methods.
[0056] Figure 5 FIG. 5 shows a network architecture 500 of a machine learning model according to one or more embodiments. According to one embodiment, the network architecture 500 may be a Figure 1 The method 100 is applied and Figure 2 The network architecture of the trained machine learning model trained in method 200.
[0057] The network architecture 500 illustrates machine learning models 502-A, 502-B, and 502-C (collectively, machine learning models 502). Although the machine learning models 502-A, 502-B, and 502-C are functionally illustrated as separate instances in the network architecture 500 for ease of understanding to illustrate the temporal analysis of input medical images 514-A, 514-B, and 514-C (collectively, input medical images 514), it should be understood that the same machine learning model 502 is applied to each input medical image 514 (i.e., the same machine learning model 502 with the same learned weights is used for each instance of the machine learning models 502-A, 502-B, and 502-C to analyze each corresponding input medical image 514).
[0058] The machine learning model 502 receives an input medical image 514 depicting a blood vessel and predicts corresponding segmentations of the blood vessel as outputs 516-A, 516-B, and 516-C (collectively, outputs 516). Although the network architecture 500 illustrates the machine learning model 502 as performing the task of predicting the segmentation of the blood vessel in the input medical image 514, it should be understood that the machine learning model 502 can be additionally or alternatively trained to perform one or more other tasks (e.g., a primary task and one or more secondary tasks). In one embodiment, the machine learning model 502 is a deep neural network. For example, the machine learning model 502 can be based on a modified 3D-R2N2 network.
[0059] The machine learning model 502 is composed of the following: 2D convolutional neural networks (CNNs) 504-A, 504-B and 504-C (collectively referred to as 2D CNNs 504), long short-term memory (LSTM) recurrent neural networks (RNNs) 506-A, 506-B and 506-C (collectively referred to as LSTM RNNs 506), 3D LSTM networks 510-A, 510-B and 510-C (collectively referred to as 3D LSTM networks 510), and decoder 3D deconvolutional neural networks 512-A, 512-B and 512-C (collectively referred to as decoders 512).
[0060] Each input medical image 514-A, 514-B, and 514-C includes an image sequence. For example, the corresponding input medical image 514 may include multiple coronary angiograms or all frames of coronary angiograms. The input medical images 514 are fed into the machine learning model 502, where they are encoded by the 2D CNN 504. The encoded features (from the 2D CNN 504) of multiple images from the same sequence 514 are aggregated by the LSTM RNN 506. The aggregated encoded features from the LSTM RNN 506 and the 2D view parameters 518-A, 518-B, and 518-C (collectively referred to as 2D view parameters 518) are combined into encoded features 508-A, 508-B, and 508-C (collectively referred to as encoded features 508), which are input into the 3D LSTM network 510. The 2D view parameters 518 are characteristics that describe the input medical image 514 , such as, for example, C-arm angle, source-to-detector distance, image resolution, and the like.
[0061] The 3D LSTM network 510 aggregates the encoded features 508 from different sequences (e.g., sequences 514-A, 514-B, 514-C), if any. For example, as shown in the network architecture 500, the 3D LSTM network 510-B aggregates the encoded features 508-B with the encoded features 508-A from the analysis of the input medical image 514-A. In another example, the 3D LSTM network 510-C aggregates the encoded features 508-C with the encoded features 508-A and 508-B from the analysis of the input medical images 514-A and 514-B. The aggregated encoded features from the 3D LSTM network 510 are decoded by the decoder 512 to generate the output 516.
[0062] Advantageously, by inputting a sequence of images rather than a single image as the input medical image 514 into the machine learning model 502, the machine learning model 502 is able to exploit temporal information from the sequence of images. For example, the coronary vessels change appearance during the cardiac cycle due to systole. The image sequence of the input medical images 514 enables the machine learning model 502 to learn how the effects of systole seen on the input medical images 514 (and their variation between subjects) relate to the 3D geometry of the coronary arteries. In another example, changes in lumen size during the cardiac cycle can also provide an indication of the health of the arterial wall. In cases of severe atherosclerosis, arterial compliance is nearly lost. The machine learning model 502 can learn from this information to further distinguish between cases with diffuse or focal disease.
[0063] Furthermore, by augmenting the encoded features from the LSTM RNN 506 with the 2D view parameters 518, the machine learning model 502 learns how to better distinguish the locations of the reconstructed blood vessels in 3D space.
[0064] In the multi-task learning framework, it is assumed that different tasks are learned by, for example, hard-shared layers of a neural network, while additional output layers are task-specific. As shown in the network architecture 500, the machine learning model 502 includes: a 2D CNN 504 and an LSTM RNN 506, forming an encoding portion of the machine learning model 502 with shared layers; and a 3D LSTM network 510 and a decoder 512, forming a decoding portion with task-specific layers. Therefore, in addition or alternatively, the machine learning model 502 can be trained to perform other tasks by separately training the task-specific decoding portion (i.e., the 3D LSTM network 510 and the decoder 512) for, for example, predicting virtual FFR, predicting the location of common image points, predicting measurement locations, predicting the location of stenosis, etc.
[0065] The predicted position in the corresponding 2D input medical image 514 can be determined by projecting the predicted 3D position onto the 2D image plane. To predict the virtual FFR, an additional layer can be trained to map the features of the 3D probability map to the FFR value in the entire volume. In one embodiment, the additional layer can be trained based on the method described in U.S. Patent No. 9,349,178 issued on May 24, 2016, the disclosure of which is incorporated herein by reference in its entirety.
[0066] The systems, devices, and methods described herein can be implemented using digital circuits or using one or more computers using well-known computer processors, memory units, storage devices, computer software, and other components. Typically, a computer includes a processor for executing instructions and one or more memories for storing instructions and data. A computer may also include or be coupled to one or more mass storage devices, such as one or more magnetic disks, internal hard disks and removable disks, magneto-optical disks, optical disks, and the like.
[0067] The systems, devices and methods described herein can be implemented using computers operating in a client-server relationship. Typically, in such a system, the client computer is located remotely from the server computer and interacts via a network. The client-server relationship can be defined and controlled by computer programs running on the respective client and server computers.
[0068] The systems, devices, and methods described herein can be implemented within a network-based cloud computing system. In such a network-based cloud computing system, a server or another processor connected to the network communicates with one or more client computers via the network. For example, a client computer can communicate with a server via a web browser application that resides on and operates on the client computer. The client computer can store data on the server and access the data via the network. The client computer can transmit a request for data or a request for an online service to the server via the network. The server can perform the requested service and provide the data to the (one or more) client computers. The server can also transmit data suitable for causing the client computer to perform a specified function (e.g., perform a calculation, display specified data on a screen, etc.). For example, the server can transmit data suitable for causing the client computer to perform one or more steps or functions (including Figure 1-2 Some steps or functions of the methods and workflows described herein (including Figure 1-2 One or more steps or functions of the method and workflow described herein may be performed by another processor in a server or network-based cloud computing system. Figure 1-2 One or more steps of the method and workflow described herein (including Figure 1-2 One or more steps of the process may be performed by a server and / or a client computer in a network-based cloud computing system in any combination.
[0069] The systems, apparatus, and methods described herein may be implemented using a computer program product tangibly embodied in an information carrier (e.g., in a non-transitory machine-readable storage device) for execution by a programmable processor; and may be implemented using one or more computer programs executable by such a processor including Figure 1-2 A computer program is a set of computer program instructions that can be used directly or indirectly in a computer to perform a specific activity or bring about a specific result. A computer program can be written in any form of programming language, including compiled or interpreted languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.
[0070] Figure 6 A high-level block diagram of an example computer 602 that can be used to implement the systems, apparatus, and methods described herein is depicted in FIG. The computer 602 includes a processor 604 that is operably coupled to a data storage device 612 and a memory 610. The processor 604 controls the overall operation of the computer 602 by executing computer program instructions that define such operation. The computer program instructions may be stored in the data storage device 612 or other computer-readable medium and loaded into the memory 610 when execution of the computer program instructions is desired. Thus, Figure 1-2 The methods and workflow steps or functions of the present invention may be defined by computer program instructions stored in the memory 610 and / or the data storage device 612 and controlled by the processor 604 executing the computer program instructions. For example, the computer program instructions may be implemented as a program programmed by a person skilled in the art to execute Figure 1-2 Thus, by executing the computer program instructions, the processor 604 performs Figure 1-2 The computer 602 may also include one or more network interfaces 606 for communicating with other devices via a network. The computer 602 may also include one or more input / output devices 608 (e.g., a display, keyboard, mouse, speaker, buttons, etc.) that enable a user to interact with the computer 602.
[0071] The processor 604 may include general-purpose microprocessors and special-purpose microprocessors, and may be the sole processor or one of multiple processors of the computer 602. The processor 604 may include, for example, one or more central processing units (CPUs). The processor 604, the data storage device 612, and / or the memory 610 may include, be supplemented by, or be incorporated in one or more application-specific integrated circuits (ASICs) and / or one or more field programmable gate arrays (FPGAs).
[0072] The data storage device 612 and the memory 610 each include a tangible, non-transitory computer-readable storage medium. The data storage device 612 and the memory 610 may each include a high-speed random access memory, such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a double data rate synchronous dynamic random access memory (DDR RAM), or other random access solid-state memory devices, and may include non-volatile memory, such as one or more magnetic disk storage devices (such as internal hard disks and removable disks), magneto-optical disk storage devices, optical disk storage devices, flash memory devices, semiconductor memory devices (such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)), compact disk read-only memory (CD-ROM), digital versatile disk read-only memory (DVD-ROM) disks, or other non-volatile solid-state storage devices.
[0073] The input / output devices 608 may include peripheral devices such as printers, scanners, display screens, etc. For example, the input / output devices 608 may include a display device (such as a cathode ray tube (CRT) or liquid crystal display (LCD) monitor) for displaying information to a user, a keyboard, and a pointing device (such as a mouse or trackball) through which a user can provide input to the computer 602.
[0074] The image acquisition device 614 can be connected to the computer 602 to input image data (e.g., medical images) to the computer 602. The image acquisition device 614 and the computer 602 can be implemented as one device. The image acquisition device 614 and the computer 602 can also be separate devices that communicate (e.g., wirelessly) over a network. In a possible embodiment, the computer 602 can be located at a remote location relative to the image acquisition device 614.
[0075] Any or all of the systems and devices discussed herein (including Figure 1 The elements of workstation 102 may be implemented using one or more computers (such as computer 602).
[0076] Those skilled in the art will recognize that an actual computer or computer system implementation may have other structures and may also include other components, and for illustrative purposes, Figure 6 is a high-level representation of some of the components of such a computer.
[0077] The foregoing detailed description is to be understood in every respect as illustrative and exemplary, rather than restrictive, and the scope of the invention disclosed herein is not determined by the detailed description, but by the claims interpreted in accordance with the maximum scope permitted by patent law. It should be understood that the embodiments shown and described herein are only illustrative of the principles of the invention, and that various modifications may be implemented by those skilled in the art without departing from the scope and spirit of the invention. Various other feature combinations may be implemented by those skilled in the art without departing from the scope and spirit of the invention.
Claims
1. A method for retraining a trained machine learning model, comprising: receiving one or more input medical images; using the trained machine learning model, predicting metrics of interest for the primary task and the secondary task from the one or more input medical images; outputting a predicted metric of interest for the primary task and the secondary task; receiving user feedback regarding the predicted metric of interest for the secondary task; and retraining the trained machine learning model for predicting the metric of interest for the primary task and the secondary task based on user feedback about the output for the secondary task, wherein the user cannot directly verify the metric of interest for the primary task based on the one or more input medical images, and the user can directly verify the metric of interest for the secondary task based on the one or more input medical images, The metrics of interest for the secondary task include at least one of the following: the location of a measurement point, the location of a common image point, the location of a stenosis, and the segmentation of a blood vessel, and the metrics of interest for the primary task include a hemodynamic index.
2. The method according to claim 1, wherein: The user feedback includes acceptance or rejection of the predicted metric of interest for the secondary task.
3. The method according to claim 1, wherein: The user feedback includes a modification to the predicted metric of interest for the secondary task.
4. The method according to claim 1, further comprising: predicting additional metrics of interest for the primary task and the secondary task from the one or more input medical images using the retrained machine learning model; as well as A measure of confidence in the predicted additional metrics of interest for the primary task and the secondary task is determined based on a difference between the metrics of interest predicted using the trained machine learning model and the additional metrics of interest predicted using the retrained machine learning model.
5. The method according to claim 1, wherein: Predicting the metrics of interest for the primary task and the secondary task based on the one or more input medical images using the trained machine learning model includes predicting the metrics of interest for the primary task and the secondary task based on a single input medical image from the first image sequence and a single input medical image from the second image sequence using the trained machine learning model, and the method further includes: using the trained machine learning model, predicting additional metrics of interest for the primary task and the secondary task based on two input medical images from the first image sequence and two input medical images from the second image sequence; and Determine a measure of confidence in the predicted metrics of interest and / or predicted additional metrics of interest for the primary task and the secondary task based on the difference between: 1) the metrics of interest predicted based on a single input medical image from a first image sequence and a single input medical image from a second image sequence using a trained machine learning model, and 2) the additional metrics of interest predicted based on the two input medical images from the first image sequence and the two input medical images from the second image sequence using the trained machine learning model.
6. The method according to claim 1, wherein: The hemodynamic index is the virtual fractional flow reserve.
7. The method of claim 1, further comprising selecting the one or more input medical images by: receiving corresponding electrocardiogram signals acquired during acquisition of the first image sequence and the second image sequence; associating an indicator with each image in the first image sequence and the second image sequence based on respective electrocardiogram signals of the first image sequence and the second image sequence; matching images of the first image sequence with images of the second image sequence based on an indicator associated with the first image sequence and the second image sequence; as well as The images of the first sequence and the images of the second sequence that match are selected as the one or more input medical images.
8. An apparatus for retraining a trained machine learning model, comprising: means for receiving one or more input medical images; means for predicting metrics of interest for a primary task and a secondary task from the one or more input medical images using the trained machine learning model; means for outputting a predicted metric of interest for said primary task and said secondary task; means for receiving user feedback regarding the predicted metric of interest for the secondary task; and means for retraining the trained machine learning model for predicting the metric of interest for the primary task and the secondary task based on user feedback regarding output for the secondary task, wherein the user cannot directly verify the metric of interest for the primary task based on the one or more input medical images, and the user can directly verify the metric of interest for the secondary task based on the one or more input medical images, The metrics of interest for the secondary task include at least one of the following: the location of a measurement point, the location of a common image point, the location of a stenosis, and the segmentation of a blood vessel, and the metrics of interest for the primary task include a hemodynamic index. 9 . The apparatus of claim 8 , wherein the user feedback comprises an acceptance or rejection of the predicted metric of interest for the secondary task.
10. The device according to claim 8, wherein: The user feedback includes a modification to the predicted metric of interest for the secondary task.
11. A non-transitory computer readable medium storing computer program instructions for retraining a trained machine learning model, the computer program instructions, when executed by a processor, causing the processor to perform operations comprising: receiving one or more input medical images; using the trained machine learning model, predicting metrics of interest for the primary task and the secondary task from the one or more input medical images; outputting a predicted metric of interest for the primary task and the secondary task; receiving user feedback regarding the predicted metric of interest for the secondary task; and retraining the trained machine learning model for predicting the metric of interest for the primary task and the secondary task based on user feedback about the output for the secondary task, in, The user cannot directly verify the metric of interest for the primary task based on the one or more input medical images, and the user can directly verify the metric of interest for the secondary task based on the one or more input medical images, The metrics of interest for the secondary task include at least one of the following: the location of a measurement point, the location of a common image point, the location of a stenosis, and the segmentation of a blood vessel, and the metrics of interest for the primary task include a hemodynamic index.
12. The non-transitory computer readable medium of claim 11, the operations further comprising: predicting additional metrics of interest for the primary task and the secondary task from the one or more input medical images using the retrained machine learning model; as well as A measure of confidence in the predicted additional metrics of interest for the primary task and the secondary task is determined based on a difference between the metrics of interest predicted using the trained machine learning model and the additional metrics of interest predicted using the retrained machine learning model.
13. The non-transitory computer readable medium of claim 11, wherein: Predicting the metrics of interest for the primary task and the secondary task based on the one or more input medical images using the trained machine learning model includes predicting the metrics of interest for the primary task and the secondary task based on a single input medical image from the first image sequence and a single input medical image from the second image sequence using the trained machine learning model, and the operations further include: using the trained machine learning model, predicting additional metrics of interest for the primary task and the secondary task based on two input medical images from the first image sequence and two input medical images from the second image sequence; and Determine a measure of confidence in a predicted metric of interest and / or predicted additional metric of interest for a primary task and a secondary task based on the difference between: 1) the metric of interest predicted based on a single input medical image from a first image sequence and a single input medical image from a second image sequence using a trained machine learning model, and 2) additional metrics of interest predicted based on the two input medical images from the first image sequence and the two input medical images from the second image sequence using a trained machine learning model.
14. The non-transitory computer readable medium of claim 11, the operations further comprising selecting the one or more input medical images by: receiving corresponding electrocardiogram signals acquired during acquisition of the first image sequence and the second image sequence; associating an indicator with each image in the first image sequence and the second image sequence based on respective electrocardiogram signals of the first image sequence and the second image sequence; matching images of the first image sequence with images of the second image sequence based on an indicator associated with the first image sequence and the second image sequence; as well as The images of the first sequence and the images of the second sequence that match are selected as the one or more input medical images.
