Interactive coronary artery labeling using interventional X-ray images and deep learning

By training and bias adjustment of the initial model, the problem of manual workload in the vasculature labeling process in coronary intervention is solved, and the accurate classification and labeling of the vasculature is achieved, and the efficiency and accuracy of diagnostic reports are improved.

CN112074912BActive Publication Date: 2024-11-01KONINKLIJKE PHILIPS NV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201980029300.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2018-05-03
Filing Date
2019-04-24
Publication Date
2024-11-01
Estimated Expiration
2039-04-24

AI Technical Summary

Technical Problem

Prior art In the process of coronary intervention, the vasculature marking process used for diagnosis and reporting relies on a large amount of manual work, and the standard model has limited accuracy in matching individual blood vessels, making it difficult to meet clinical needs.

Method used

By using the initial model to train the diagnostic image data, identify and adjust the deviations between the vascular trees, the model is gradually updated to improve the accuracy of vascular detection and labeling, and reduce the manual work of the physician.

Benefits of technology

It realizes accurate classification and labeling of the vasculature with minimal user interaction, improves the accuracy of vascular detection, reduces the size of the training data set, and improves the efficiency and accuracy of diagnostic reports.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112074912B_ABST
    Figure CN112074912B_ABST
Patent Text Reader

Abstract

A method for classifying a vascular system includes training a training device using diagnostic image data with an initial model of the vascular system, the diagnostic image data representing the geometry of a plurality of vessels for a vascular tree and including a corresponding vessel label for each vessel, providing at least one diagnostic image of the vascular tree of a patient, and identifying changes between the vascular tree represented by the initial model and the vascular tree of the patient. Such changes are examined and marked in order to improve the trained model. The process can be iteratively repeated until an accurate patient-specific model of the vascular system is reached.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a method for classifying vasculature, a corresponding system and a corresponding computer program. In particular, the present invention relates to an improved method of training a training device to classify vasculature with minimal user interaction while keeping the size of the training data set relatively small. Background Art

[0002] Percutaneous coronary intervention (PCI) is a minimally invasive procedure used to treat narrowing (such as stenosis) of the coronary arteries as found in coronary artery disease. It is usually performed as an (X-ray) image-guided treatment.

[0003] Image guidance for PCI usually includes visualization using corresponding diagnostic images. Before the PCI procedure, these diagnostic images allow the identification of potential stenoses. After the procedure, it is possible to evaluate the results of the procedure on the basis of the visualization.

[0004] The patient-specific (coronary artery) anatomy before and after treatment as derived from the diagnostic images is usually stored in the electronic medical record (EMR) together with additional patient-specific information. The information thus stored is then used for case reporting, for reference in the EMR and / or for interdisciplinary communication. In addition, the information is provided to so-called archives, which collect disease-specific information in order to analyze it for statistical evaluations regarding, for example, trends and treatments of specific diseases.

[0005] In order to establish a smooth and efficient storage and exchange process for such information, it is necessary to ensure that the information provided to the different entities is unambiguous and easy to interpret. This usually requires a predetermined information format / structure according to known medical terminology and coding standards (such as "Systematic Nomenclature of Medicine" (SNOMED) CT).

[0006] In PCI, the documentation of patient-specific pre- and post-treatment information related to the intervention is based on the patient-specific coronary vasculature, by annotating the collected images, by creating a customized schematic representation, or by providing a text file. Among these options, the use of a customized schematic representation is generally considered to be the preferred way to convey the route of the intervention. However, generating such a customized schematic representation for each patient individually is a very time-consuming process and requires a lot of manual work (typing, copying, etc.) by the physician. Summary of the invention

[0007] In order to reduce the manual work to be done by the physician, efforts have been made to find procedures that allow the automatic detection and marking of the vessels in the patient's vasculature. These methods facilitate the direct construction of reports with minimal involvement of the physician. The basic principle of these methods generally involves the extraction of centerlines, for example with the aid of morphological image processing operators. The centerlines thus extracted are considered to represent the geometry of the different vessels in the (coronary) vasculature, including length, course, etc. They are matched with a standard model of the vasculature and, based on the match, marking is performed.

[0008] However, there is a huge morphological variation in the coronary vasculature in different patients. That is, the geometry of the blood vessels is different for each patient, since the length, course, direction, etc. of the blood vessels and thus the topology of the vessel tree may vary from patient to patient. Therefore, the use of standard models of the vasculature that represent a "typical" vessel topology has only limited accuracy, which is particularly insufficient for use in clinical practice.

[0009] Therefore, it is an object of the present invention to provide an improved method for classifying the vasculature of a patient and a corresponding system. It is a further object of the present invention to provide an improved method for classifying the vasculature of a patient, which allows on the one hand to improve the accuracy of the detection and subsequent marking of the vessels in the vasculature, and on the other hand to avoid unnecessary manual work of the physician. More specifically, it is an object of the present invention to provide a method for training a training device to perform automated detection and marking using a small training data set.

[0010] Therefore, a method for classifying a vascular system is provided, the method comprising the steps of: a) training a training device with an initial model of the vascular system using diagnostic image data representing a first vessel tree, the diagnostic image data including corresponding vessel labels for at least one vessel of the first vessel tree, b) inputting at least one diagnostic image representing a second vessel tree, c) identifying at least one deviation between the first vessel tree and the second vessel tree, d) in response to the identifying, outputting an indication of the at least one deviation to a user and providing at least one label for the at least one deviation, and e) adjusting the initial model based on the at least one deviation and the at least one label to classify the vascular system.

[0011] According to this approach, it is possible to train a training device to automatically detect and label individual blood vessels in a patient's vasculature using only a small (initial) training data set. The training device may specifically implement a deep learning algorithm to derive associations between specific blood vessels of the vasculature and their corresponding labels from the training data set.

[0012] In this context, the term diagnostic image data may specifically refer to one or more images representing the vasculature of a patient, specifically the coronary vasculature of a patient. The diagnostic images may be acquired using any medical imaging modality suitable for imaging the vasculature. In some specific embodiments, the diagnostic images used for the initial training data set may be specifically acquired using an X-ray imaging modality. In other embodiments, the diagnostic images may also be acquired using different imaging modalities, such as magnetic resonance imaging, ultrasound imaging, etc. In some embodiments, the multiple diagnostic images in the diagnostic image data may specifically include a time series of the patient's vasculature, i.e., may correspond to a movie of the patient's vasculature captured over a period of time.

[0013] The term initial model of a patient's vasculature may specifically refer to a trained model of a patient's vasculature that has been taught to a training device using a set of diagnostic image data acquired for a plurality of patients. In this context, the set of diagnostic image data should be a relatively small set of diagnostic image data compared to prior art methods.

[0014] That is, the training device may initially be trained using diagnostic image data comprising multiple images or multiple image sequences that have been acquired for multiple different patients as a training data set. Using multiple images or multiple image sequences for different patients allows a wide variety of anatomical structures to be captured. The image sequence may specifically be a time series. By using all frames from an image (time) sequence showing a vascular tree, a large training data set may be provided that allows for free data augmentation.

[0015] The size of the training data set should be chosen to be as large as necessary, but as small as possible, in order to enable the training device to be trained with a sufficiently plausible first classification of the vessels. The resulting (not yet) all-encompassing initial model therefore adequately represents the different possible geometries of the vessel tree (i.e. the geometry of one or more of the vessels of the vessel tree) that can be found in different patients.

[0016] Here, the training may be performed in particular by identifying a vessel tree indicated in a diagnostic image dataset comprising diagnostic images. For this purpose, in these diagnostic images, a vessel tree comprising one or more (in particular a plurality of) vessels may be visible. Here, it should be understood that the term vessel tree may refer to one or more vessels forming the vascular system of a patient or a part thereof.

[0017] Additionally, the term geometry of a vessel tree may refer to the geometric properties of each of the vessels identified in the vessel tree, i.e., the route, length, etc. Alternatively, the term geometry may refer to the route, length, etc. of a subset of the plurality of vessels in the vessel tree. In some embodiments, the term geometry may also refer to the geometry of only a single vessel in the vessel tree. The term geometry may also be referred to as vessel geometry.

[0018] To this end, the geometric structure of the plurality of vessels in the vessel tree may also allow determining the vessel topology, ie the structural relationships, between the individual vessels in the vasculature.

[0019] In order to identify the blood vessels, the diagnostic image data may further include: a corresponding blood vessel label for at least one blood vessel in the blood vessel tree (more specifically for a subset or each of the blood vessels in the blood vessel tree). That is, the diagnostic image data includes at least one blood vessel label and specifically a plurality of blood vessel labels corresponding to the blood vessels represented by the diagnostic image, wherein each blood vessel label may be provided for one specific blood vessel.

[0020] In order to obtain such vessel labeling, the geometry of the vessels can be determined specifically using a centerline extraction method, wherein the centerline of each vessel is identified and locally refined, as described, for example, in US9129417B2. These centerlines allow the geometry of the vessels in the vascular system to be determined. The geometry of the vessel tree thus identified can then be (manually) labeled with the corresponding vessel labels for the vessels represented therein.

[0021] It should be understood that vessel labels should be provided specifically for each vessel represented by the initial model or for a subset thereof. Here, vessel labels can be provided specifically at least for vessels that are considered to form part of the standard anatomical model (i.e., vessels that are the most common vascular system in all patients). By identifying the most common vessels, it is possible to obtain the highest diagnostic insights into the model. In the case of the coronary vascular system, these most common vessels can specifically involve vessels such as the left coronary artery (LCA) with the left anterior descending artery (LAD) and the circumflex branch (CX), the right coronary artery with the posterior descending artery (PDA), etc. Once these vessels are marked, if necessary, the marking can also be continued for the second and third order branches. This ensures the high accuracy of the training of the patient-specific initial model, and due to this, the initial model corresponds to the starting point of the further training steps for the subsequent learning of the training device. Vessel labeling can be specifically performed manually by a user (such as a physician).

[0022] After the training device has been trained using the diagnostic image data acquired for the patient using the patient-specific initial model, at least one (additional) diagnostic image acquired for the patient may be input. The at least one diagnostic image may specifically have been acquired using the same imaging modality used for the plurality of images of the diagnostic image data.

[0023] The at least one diagnostic image may represent a second vessel tree and may also allow determining the geometry of one or more vessels of said second vessel tree. In order to appropriately derive this (second) geometry of the vessels, a centerline extraction method as mentioned above may be performed on the at least one diagnostic image. In this context, the at least one diagnostic image may in particular be input together with the centerlines that have been extracted therefrom. The at least one diagnostic image may thereby in particular be provided to an input unit. The input unit may be part of a training device. Alternatively, the input unit may be communicatively coupled to the training device.

[0024] After identifying the geometric structure of the blood vessels in the at least one diagnostic image, ie after identifying the second vessel tree, the first vessel tree and the second vessel tree may be compared with each other.

[0025] That is, it can be identified whether vessels that are part of the first vessel tree can also be found in the second vessel tree, and whether said vessels of the second vessel tree correspond geometrically to corresponding vessels of the first vessel tree, i.e. in length, course, position, direction, etc. This allows identifying geometric and / or topological differences - or deviations or changes - between the first vessel tree as represented by the initial model and the second vessel tree as determined from the at least one diagnostic image.

[0026] These deviations may specifically refer to anatomical variations: in some embodiments, a vessel may be present in the second vessel tree, but may not be present in the first vessel tree. This additional vessel may then be identified as a deviation between the first and second vessel trees. This additional vessel present in the second vessel tree may typically not yet have any vessel markings. Markings may accordingly have to be added to update the initial model.

[0027] In some embodiments, a vessel present in the first vessel tree may be missing in the second vessel tree. The initial model may therefore be updated as that particular vessel may be removed.

[0028] In some embodiments, the deviation may also correspond to differences in the geometry of the corresponding vessels in the first vessel tree and the second vessel tree. That is, the corresponding vessels may differ in length, route, position, direction, etc. Such changes in length, route, position, direction, etc. may therefore also be trained into the initial model in order to update the model.

[0029] The difference in the geometric structure can be determined here in particular by obtaining a difference value and comparing the difference value with a predetermined threshold value. If the difference value does not exceed the threshold value, no deviation is identified. If the threshold value is exceeded, a deviation exists and an indication of the deviation is output to the user. In addition, a marking is provided for the vessel or vessels involved.

[0030] Thresholding can be performed at different levels here. In one level, a first vascular topology of a first vascular tree and a second vascular topology of a second vascular tree can be determined. The first vascular topology and the second vascular topology can be compared to each other in order to identify the deviation between the first vascular topology and the second vascular topology. These deviations can then be identified using one or more difference values ​​in order to determine whether they are due to measurement and / or imaging errors or due to real geometric structure deviations and / or topological deviations between the first vascular tree and the second vascular tree. In another level, pixel data can be used to determine the deviations. That is, training on pixel data can be performed. Then, one or more pixels can receive confidence that they belong to a specific vessel (e.g., LCA or LAD). Subsequently, the received confidence can be thresholded and can help identify whether there are vessels missing in the first vascular tree and / or the second vascular tree compared to the second vascular tree and / or the first vascular tree. This allows the identification of missing and / or additional vessels.

[0031] Subsequently, a marker may be provided for the deviation in the at least one diagnostic image. That is, vessels that are present in the second vessel tree but not in the first vessel tree may be marked with a corresponding marker. Additionally or alternatively, vessels that have a different length, course, etc. in the second vessel tree than in the first vessel tree may be marked with their corresponding vessel markers, such as to identify a deviation in the topology in the at least one diagnostic image in this regard.

[0032] In this context, it should be understood that the instructions to the user may include suggestions to the user on how to mark and / or handle the corresponding deviation. Alternatively, the instructions may be a request to the user to manually mark and / or handle the deviation based on experience. The instructions may specifically be visual instructions, but may also be tactile or auditory.

[0033] The at least one diagnostic image may thus be provided with corresponding labels of one or more vessels in the second vessel tree. Subsequently, the at least one diagnostic image together with the newly added labels as corresponding vessel annotations is used to adjust the patient-specific initial model of the vasculature.

[0034] In this context, the term adjustment may specifically refer to changing or updating the initial model according to the new information provided. Such a change may specifically be performed by implementing a semi-supervised learning method in the training device. Additionally or alternatively, the change may be performed by adding at least one diagnostic image to the diagnostic image data after labeling and retraining the training device using the training data set updated thereby.

[0035] According to this method, instead of using the entire training data set from the outset, the training device trains with an initial model of the vasculature using a relatively small training data set consisting of diagnostic image data as a starting point, and subsequently trains only by deviations from said initial model using identified deviations corresponding to variations in vessel topology in the vasculature of different patients, i.e. differences found between vessels of a first vessel tree (as represented by the initial model) and a second vessel tree (as derivable from the at least one image). Thus, patient-specific anatomical variations are gradually introduced into the initial model. This allows improving the accuracy of the model of the vasculature by personalizing it for one specific patient, while keeping the initial training data set relatively small.

[0036] In some embodiments, providing the at least one marker includes receiving a first user input from the user indicating the at least one marker.

[0037] In some embodiments, the marking of deviations (specifically the marking of not yet identified vessels represented by at least one diagnostic image) can be performed using an interactive method with the user. That is, the user can be presented with the deviations and requested to manually enter a label for the deviations. In case the deviations correspond to one or more additional vessels, the user can specifically manually mark one or more additional vessels. If the deviations are differences in the geometry of previously marked vessels (i.e. different routes, lengths, locations, directions, etc.), the user can be requested to confirm that the vessel labeling is correct or to enter a new vessel labeling. This allows the overall topology of the vascular system to be updated based on patient-specific data.

[0038] In some embodiments, said adjusting the initial model to classify the vascular system comprises iteratively repeating steps b) to e) for a plurality of diagnostic images. By iteratively repeating the steps of: inputting at least one diagnostic image acquired from a patient, identifying deviations between a first vascular tree according to the initial model and a second vascular tree obtained from at least one diagnostic image acquired for the patient, providing a marker for the deviation / change between the first vascular tree and the second vascular tree and adjusting the initial model accordingly, the training data set can be gradually expanded.

[0039] This gradually improves the accuracy of the trained model of the vasculature of a specific patient. Thus, the amount of identified deviations / variations between the initial (i.e., gradually trained) model and the patient-specific vasculature will gradually decrease as the initial model approaches the patient-specific true vasculature. This allows only as many images as necessary to be introduced into the training data set used to train the training device until an accurate model of the patient's vasculature is reached, thereby reducing the amount of necessary input data while keeping the necessary user input to a minimum, as the user is only required to provide input after identification of the deviations / variations.

[0040] In some embodiments, adjusting the initial model comprises retraining the training device using the diagnostic image data representing the first vascular tree and the at least one diagnostic image representing the second vascular tree and the corresponding at least one label.

[0041] In some embodiments, the adjustment of the initial model of the vascular system can be performed specifically by retraining the training device using an updated training data set including the diagnostic image data originally used and at least one additional diagnostic image of the vascular system of the patient including the identified blood vessels and corresponding markings. That is, after marking the blood vessels identified in the at least one diagnostic image, the diagnostic image can be added to the training data set. After retraining, the initial model with which the training device has been trained can more closely resemble the patient-specific vascular system of the patient for which the at least one diagnostic image has been obtained. Thereby, the trained initial model is gradually adjusted to accurately represent the patient-specific vascular system.

[0042] In some embodiments, the method further comprises: identifying a geometric structure of the second vessel tree in the at least one diagnostic image by segmenting the at least one diagnostic image, identifying corresponding centerline information for a plurality of vessels of the second vessel tree based on the segmentation, and extracting the centerline information. In some embodiments, the centerline information is input together with the at least one diagnostic image.

[0043] In some embodiments, the geometry of the second vessel tree represented by at least one diagnostic image can be determined specifically using a centerline extraction method as mentioned above. That is, at least one diagnostic image can be received and segmented. Based on the segmentation, the presented geometry of the vessels in the vascular system imaged in the at least one diagnostic image can be predicted by providing an initial estimate of the centerline through the vessel and then refining the centerline. The refined centerline information can then be considered to be indicative of the geometry of the vessel.

[0044] In some embodiments, the centerline information thus extracted is used together with the at least one diagnostic image from which it has been extracted to adjust an initial model trained to the training device. That is, the at least one diagnostic image and the extracted centerline information are input to the training device so that the training device can use this additional information to update the initial model accordingly. This allows the training device to be trained iteratively by techniques such as semi-supervised learning or by retraining after adding the at least one diagnostic image and corresponding labels to the training data set.

[0045] According to some embodiments, the deviation may include a change between a geometry of the first vessel tree and a geometry of the second vessel tree, ie, a geometry change between the first vessel tree and the second vessel tree.

[0046] As indicated above, the deviation between the geometry of the first vessel tree and the geometry of the second vessel tree as represented by the initial model may be caused by geometric changes of the vessel tree which may result in topological changes of the vessel topology. Such geometric changes may specifically correspond to anatomical changes, such as missing vessels, missing branches, longer and / or shorter branches, highly tortuous branches, etc. In addition, the course and / or direction and / or position of a specific vessel may differ from the course and / or direction and / or position of a corresponding vessel in the initial model, or similar changes may occur.

[0047] In some embodiments, the diagnostic image data comprises a plurality of images, particularly between 10,000 and 100 images, even more particularly between 1,000 and 100 images.

[0048] According to some embodiments, the diagnostic image data constituting the initial training data set includes multiple images, such as diagnostic images that have been acquired using a diagnostic imaging modality. Due to the iterative method used to train the training device, only a small initial training data set is necessary. In some embodiments, between 1000 and 100, more specifically between 500 and 200 images are sufficient to train the training device using the initial model. Additional deviations / anatomical changes specific to a certain patient can then be gradually introduced with the aid of diagnostic images acquired for the patient that have been labeled based on expert input through user interaction.

[0049] In some embodiments, the training device is also trained to include externally induced changes into the initial model.

[0050] In some embodiments, external factors can affect the diagnostic image data and / or at least one diagnostic image. Examples of such external factors can involve acquisition errors and / or measurement inaccuracies, such as a relatively small and / or incomplete field of view or other disturbances that can negatively affect the acquisition accuracy of the imaging modality. In addition, image acquisition can be negatively affected by devices (such as stents, catheters, clips, etc.) that have been introduced into the patient. Devices such as stents can also change the shape and / or route of specific blood vessels in the vascular system, etc.

[0051] In order to take these external influencing factors into account, the changes resulting therefrom may be trained to the training device, such as to be included in the initial model, thereby further improving the accuracy of the classification.

[0052] According to another aspect, a classification system for classifying a vascular system is provided, the classification system comprising: a training device, an input unit, and an inference unit, the training device being configured to train using an initial model of the vascular system using diagnostic image data representing a first vascular tree, the diagnostic image data comprising a corresponding vascular label for each vessel of the first vascular tree, the input unit being configured to receive at least one diagnostic image representing a second vascular tree, the inference unit being configured to: identify at least one deviation between the first vascular tree and the second vascular tree, output an indication of the at least one deviation to a user in response to the identification, and provide at least one label for the at least one deviation, wherein the training device is configured to adjust the initial model based on the at least one deviation and the at least one label to classify the vascular system. In some embodiments, the classification system further comprises: a display unit and a user interface, the display unit being configured to display the indication of the at least one deviation to a user, the user interface being configured to receive a first user input from the user indicating the at least one label.

[0053] In yet another aspect, a classification system is provided, the classification system comprising a training device, an input unit, and an inference unit, the training device can be trained using an initial model of a first vascular tree, the input unit can receive at least one diagnostic image representing a second vascular tree, and the inference unit can derive a deviation between the first vascular tree and the second vascular tree and output such deviation to a user. The inference unit can also provide at least one tag for the deviation. In this context, the inference unit can specifically determine potential tags for specific deviations and suggest the potential tags to the user using corresponding instructions. Alternatively or additionally, the inference unit can be configured to request the user to manually input the tags using the instructions. In some embodiments, the inference unit can also be configured to provide the user with a list that can be ranked or alphabetically sorted using the suggested tags, and allow the user to optionally select from the list or manually input the tags.

[0054] Here, the system may specifically include a display unit, or be communicatively connected to a display unit for displaying instructions to a user. Alternatively or additionally, the system may also include an additional or integrated indication unit that provides auditory and / or tactile indications to the user to facilitate user interaction with the classification system.

[0055] The classification system may specifically include a user interface to receive user input. The user input may specifically be used to input a mark of one or more deviations between the first vascular tree and the second vascular tree. In addition, the user input can be used to obtain additional information, such as time, imaging conditions, etc. The user interface may specifically include a keyboard, a mouse, etc. In some embodiments, the user interface may be provided as part of a display unit, for example, as a touch screen. Alternatively, the user interface may be a touch screen provided in addition to the display unit. In some embodiments, the user interface may be provided as a combination of several interfaces, such as a combination of a touch screen, a keyboard and / or a mouse, etc. Additional user interfaces can be imagined as long as they allow the user to interact with the classification system.

[0056] In some embodiments, the inference unit is further configured to generate a medical representation, the representation comprising at least the vasculature (i.e., the blood vessels that have been identified for the patient), the blood vessel labels, and the at least one marker, and send the representation to a database. In some embodiments, the database comprises an electronic medical record (EMR), and the medical representation is generated according to a predetermined format of the electronic medical record.

[0057] In some embodiments, the inference unit can generate a medical representation based on the diagnostic image data, at least one diagnostic image and the corresponding markings of the at least one diagnostic image. In some embodiments, the medical representation can also be generated elsewhere, for example, by a special generation unit. In this context, the term medical representation can specifically refer to a customized schematic representation that allows the route of the intervention that can be performed on the patient to be transmitted to a third party that is not directly involved in the intervention. Alternatively or additionally, the term medical representation can also include a graphical representation (specifically a three-dimensional representation) of a model of the patient's vascular system. In some embodiments, the graphical representation can also include vascular markings for all blood vessels shown in the model of the patient's vascular system. In some embodiments, the graphical representation can be displayed on a display unit. During display, the vascular markings can be turned on or off, moved, changed in appearance, etc. by the user.

[0058] A medical representation (specifically a customized schematic representation) of the vascular system may be sent and stored in a corresponding (medical) database for further analysis of trends and treatments, for communication with other physicians, case reports, etc. In some embodiments, the medical representation is generated according to a predefined format (such as SNOMED CT). Such generation may be performed automatically, specifically based on a model trained to a training device. By automatically generating a patient-specific medical representation that conforms to a strict data structure imposed on the data by a third party (such as an archive or an organization that collects such information for trend analysis), a straightforward structured report may be established with minimal user intervention.

[0059] In a further aspect, a computer program is provided, which, when run by a processing unit, is adapted to perform a method according to the invention. In a further aspect, a computer readable medium is provided, having stored thereon the computer program.

[0060] It should be understood that the classification system for classifying the vascular system can be implemented by means of the processing unit. Thus, the training device, the input unit and the inference unit can be implemented as modules in the processing unit. The functionality of these modules can be implemented specifically by means of corresponding algorithms.

[0061] It shall be understood that the classification method of claim 1, the classification system of claim 10, the computer program of claim 14 and the computer readable medium of claim 15 have similar and / or identical preferred embodiments as specifically defined in the dependent claims.

[0062] It shall be understood that a preferred embodiment can also be the independent claim or any combination of the above embodiments with the respective dependent claims.

[0063] These and other aspects will be apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In the following figures:

[0065] Figure 1 A classification system for classifying the vasculature of a patient according to an embodiment is schematically illustrated.

[0066] Figure 2 A method for classifying vasculature according to an embodiment is illustrated.

[0067] Figure 3 A detailed method for identifying a geometry of a second vessel tree from at least one diagnostic image and for identifying changes (in geometry) between the second vessel tree and a first vessel tree according to an embodiment is schematically illustrated. DETAILED DESCRIPTION

[0068] The illustrations in the drawings are schematically. In different drawings, similar or identical elements are provided with the same reference signs.

[0069] Figure 1 A schematic illustration of a classification system 1 for classifying the vasculature of a patient is shown. Figure 1 In the exemplary embodiment of , the classification system 1 comprises a training device 100, an input unit 200, an inference unit 300, a display unit 400 and a user interface 500. In addition, the classification system 1 is communicatively coupled to a database 2.

[0070] In accordance with Figure 1 In an embodiment of the present invention, the classification system 1 comprises a training device 100, which receives diagnostic image data 10 representing a first vessel tree. The training device 100 implements a deep learning algorithm that is trained using the diagnostic image data 10 as a training data set. For this purpose, the diagnostic image data may specifically include about hundreds to thousands of diagnostic images of the vascular system of a plurality of patients. These are used to generate an initial model representing the vessels in the first vessel tree. Figure 1 In an exemplary embodiment of the invention, the diagnostic images included in the diagnostic image data have been acquired using X-ray imaging. However, other imaging modalities may also be used as long as they enable imaging of the blood vessels. In addition, the diagnostic image data include vessel markings for one or more blood vessels, in particular at least for blood vessels of a standard anatomical model of the vascular system. Using the diagnostic image data, the training device can then be trained with an initial model of the vascular system including blood vessels of the first vessel tree using a relatively small training data set.

[0071] The classification system 1 further comprises an input unit 200. The input unit 200 is configured to receive a diagnostic image 20 representing a second vessel tree. The diagnostic image 20 has been acquired for a patient whose vascular system is to be evaluated using the classification system. Figure 1 In an exemplary embodiment of the at least one diagnostic image 20 has been obtained using X-ray imaging. Other imaging modalities (such as ultrasound imaging) may likewise be used, as long as they allow for the identification of vessels in the patient's vasculature. In some embodiments, the diagnostic images 20 have been pre-processed elsewhere, and the input unit 200 also receives extracted centerline information for the second vessel tree (such as to allow for the identification of its geometry and topology) together with the at least one diagnostic image. In some embodiments, the input unit 200 receives only the diagnostic images 20, and provides the diagnostic images 20 to the inference unit 300, which then performs centerline extraction to identify the geometry of the vessels in the second vessel tree.

[0072] exist Figure 1 In an embodiment of the invention, the inference unit 300 therefore receives the diagnostic image 20 representing the second vessel tree from the input unit 200, and in addition, receives the initial model representing the first vessel tree from the training device 100. The inference unit 300 compares the first vessel tree and the second vessel tree with each other, and identifies one or more deviations between the first vessel tree and the second vessel tree. For this purpose, the inference unit 300 can specifically adopt a difference method, that is, the difference between the specific changes in the geometric structure of the vessels in the first vessel tree and the second vessel tree can be obtained respectively, and the difference is compared with a predetermined threshold. If the difference is below the threshold, the difference is considered to be within the error range of the imaging modality, and no deviation is identified. However, if the threshold is exceeded, it is considered that the deviation exists.

[0073] If the inference unit 300 determines that a deviation exists, the inference unit 300 provides a corresponding indication to the display unit 400. Figure 1 In a specific embodiment, the display unit 400 includes a computer screen on which a visual indication that a deviation has been detected can be displayed to the user. Thus, the display unit 400 can generate the visual indication. The display unit 400 can specifically provide a graphical representation of some or all of the vessels in the second vessel tree and a sign or other indicator indicating the identified deviation. The display unit 400 can also output to the user a request to provide a label for the deviation. The request can be a suggestion on how to label the deviation, which the user only needs to accept or reject, or a list of possible labeling suggestions which the user should select. The request can also be an indication to manually enter a label for the deviation. Additional requests that facilitate user interaction with the system 1 can also be envisioned.

[0074] The user may specifically provide a corresponding user input indicating a mark via the user interface 500. Figure 1 In an exemplary embodiment of the present invention, the user interface 500 may specifically include a keyboard. In addition, the user interface 500 may include a touch screen, a mouse, a remote control device, etc.

[0075] Once the user has entered the labels, the diagnostic image 20 thus labeled is then returned to the inference unit 300, which sends the diagnostic image 20 and the labels entered by the user to the training device 100 to expand the training set for the training device 100 by means of the diagnostic image 20 and the labels. Based on the thus expanded training set, the training device 100 can then use the diagnostic image 20 and the labels to adjust the (initial) model of the vascular system accordingly. It should be understood that the adjustment of the model can be performed in a variety of ways, such as by semi-supervised learning or by adding the labeled diagnostic image 20 to the diagnostic image data 10 and retraining the training device. Figure 1 In a particular embodiment of the present invention, the training device is retrained using the new model based on an updated training data set including the newly labeled diagnostic images 20 .

[0076] according to Figure 1 In an embodiment of the invention, the process is iteratively repeated for a plurality of diagnostic images 20 that have been acquired for a patient. Each diagnostic image 20 is processed and labeled as described above and subsequently used to retrain the training device. By means of this iterative retraining, the model trained to the training device is gradually adjusted to approximate the vasculature of the patient for which the diagnostic images were collected. Thus, while user interaction will be varied at the beginning of the process as many deviations may be identified, the required user input will decrease over time as the trained model more and more closely resembles the patient-specific vasculature. By means of this method, accurate automated labeling of blood vessels in the vasculature that also takes into account anatomical variations for each patient can gradually be achieved.

[0077] By means of this automated labeling method, a medical representation (customized schematic representation) of the patient's vascular system can be obtained. This medical representation can be constructed according to a predetermined format. Figure 1 In a specific embodiment, the medical representation can be constructed specifically according to medical terminology and according to the coding standards required by SNOMED CT.

[0078] The data thus constructed can then be sent to and stored in the database 2. The database 2 can be accessed by other physicians to obtain patient information and case reports and / or by third parties (such as archives or organizations) for treatment and trend analysis and prognosis. Since the medical representation is provided in a predetermined format, the information provided therein can be easily and in an unambiguous manner interpreted.

[0079] Figure 2 A method for classifying a patient's vascular system according to an embodiment is schematically illustrated. In step S101, a training device 100 receives diagnostic image data 10 representing a first vascular tree. In step S102, the training device 100 implementing a deep learning algorithm is trained using the diagnostic image data 10 as a training data set. The diagnostic image data may specifically include hundreds to thousands of diagnostic images of multiple patients, which represent multiple blood vessels and corresponding vascular labels for at least a subset of these blood vessels. In step S103, an initial model that has been trained using the diagnostic image data 10 representing the first vascular tree including the vascular labels is provided to the inference unit 300.

[0080] In step S201, the input unit 200 receives a diagnostic image 20 acquired from a patient and representing a second vascular tree. Figure 2 In the exemplary embodiment of , in step S202, the input unit 200 provides the diagnostic image 20 to the inference unit 300 for further processing. In this context, it can be understood that in alternative embodiments, the diagnostic image 20 received by the input unit 200 can be pre-processed and thus can include the extracted centerline information, which is then passed to the inference unit 300 together with the diagnostic image 20.

[0081] In step S301, the inference unit 300 receives the diagnostic image data 20 representing the second vascular tree from the input unit 200. In addition, also in step S301, the inference unit 300 receives the initial model representing the first vascular tree from the training device 100. In step S302, the inference unit 300 recognizes the geometric structure of the blood vessels in the second vascular tree represented by the diagnostic image 20 received from the input unit 200. Figure 2 In an exemplary embodiment of the invention, the inference unit 300 specifically uses a centerline extraction method to identify the geometric structure of the blood vessel. In step S303, the inference unit 300 then compares the first blood vessel tree with the second blood vessel tree (specifically their corresponding geometric structures) and identifies one or more deviations therebetween, for example by means of the difference method described above.

[0082] When a deviation is found by the inference unit 300, in step S304, the inference unit 300 provides a corresponding indication to the display unit 400. In step S401, the indication is received by the display unit 400, and in step S402, the display unit presents the indication to the user. Figure 2 In an embodiment of the invention, the indication provided to the user may specifically be a visual indication. The indication may also include: a tactile and / or an auditory component. For this purpose, in step S402, the display unit 400 may optionally generate a graphical representation of the vessels in the second vessel tree and a sign or other indicator indicating the identified deviation, and output a request to the user to mark the deviation by means of the user interface 500.

[0083] In step S501, an indication is output to a user to encourage the user to provide a corresponding interactive input for marking a deviation via a user interface. In step S502, the user may optionally manually mark the deviation in response to a request of the system.

[0084] In accordance with Figure 2 In a specific embodiment, providing a correspondingly labeled user input prompts the system to return to the initial training step S102. That is, at least one newly labeled diagnostic image 20 is input into the training device 100, which then adjusts the initial model according to the new label. Figure 2 In an exemplary embodiment of the present invention, this is achieved by adding the newly labeled diagnostic images to the diagnostic image data so as to expand the training data set and retraining the training device using the expanded training data set.

[0085] These steps can be iteratively repeated for multiple diagnostic images that have been acquired for the patient. By means of this iterative retraining, the model trained to the training device is gradually adjusted to approximate the patient-specific vasculature. This leads to a gradual reduction in the necessary user interaction while keeping the used training data set as small as possible.

[0086] Figure 3 A detailed method for identifying the geometry of blood vessels in a second vessel tree from at least one diagnostic image and for identifying deviations in geometry between blood vessels of the second vessel tree and blood vessels of a first vessel tree according to an embodiment is schematically illustrated.

[0087] Specifically, in step S301, the initial model as provided by the training device 100 and the diagnostic image 20 as provided by the input unit 200 are received at the inference unit 300. In step S302a, the inference unit segments the diagnostic image 20 and in step S302b, extracts centerlines based on the segmentation according to a known centerline extraction method. Based on these extracted centerlines, in step S302c, the inference unit identifies the geometry of the vessels in the second vessel tree. In step S303a, the inference unit 300 then compares the geometry of the vessels in the first vessel tree as inferred from the trained initial model with the geometry of the vessels in the second vessel tree as identified from the diagnostic image 20. The comparison may be performed specifically by determining the difference of selected points in the vessels in the first vessel tree and the second vessel tree, respectively, and comparing the difference with a corresponding threshold in order to determine whether the difference is due to an inaccuracy or to a real geometry (and therefore, anatomy) variation between the vessels of the first vessel tree and the second vessel tree.

[0088] Based on this comparison, in step S303b, the inference unit identifies at least one deviation between the vessels of the first and second vessel trees, such as a geometric or topological change. In step S304, the inference unit 300 indicates one or more of the identified deviations to the display unit 400 in order to indicate the deviations to the user and facilitate user interaction with the data. This allows interactively extending the data set used to define the model of the patient's vasculature, gradually improving the model accuracy, and allowing increasingly autonomous labeling and structure reporting with improved accuracy, while keeping the initial training data set relatively small.

[0089] Although in the embodiments described above, diagnostic images have been obtained using X-ray imaging, it should be understood that in other embodiments, diagnostic images can also be obtained by other imaging methods, such as positron emission tomography, single positron emission computed tomography, magnetic resonance imaging, X-ray scanning, ultrasound imaging, etc.

[0090] In addition, it should be understood that although in the above embodiments, the initial training data set includes diagnostic image data, which includes corresponding vessel labels and multiple diagnostic images, the initial training data set can also be derived from a small set of three-dimensional coronary artery atlases containing 3D centerlines of coronary arteries, which can be projected onto a two-dimensional image using the 3D geometric structure parameters of the X-ray system.

[0091] Additionally, although in the above embodiments, analysis has been performed on the coronary vasculature, in other embodiments, analysis may also be performed on vasculature in other parts of the human body, such as the peripheral vasculature.

[0092] It should also be understood that although in the above embodiments the data has been stored in a database according to SNOMED CT, other data structures that are easy to interpret and provide clear information may also be used.

[0093] Other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims.

[0094] In the claims, the word "comprising" does not exclude other elements or steps, and the word "a" or "an" does not exclude a plurality.

[0095] A single unit or device may fulfill the functions of several items recited in the claims.The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0096] Processes like receiving diagnostic image data, receiving at least one diagnostic image, segmenting the diagnostic image, extracting centerlines, identifying deviations between the first vessel topology and the second vessel topology, etc., which are performed by one or several units or devices, can be performed by any other number of units or devices. These processes according to the invention can therefore be implemented as program code units of a computer program and / or as dedicated hardware.

[0097] The computer program may be stored / distributed on suitable media, such as optical storage media or solid-state media supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless communication systems.

[0098] Any reference signs in the claims should not be construed as limiting the scope.

[0099] A method for classifying a vascular system comprises the steps of: a) training a training device with an initial model of the vascular system using diagnostic image data representing a first vessel tree, the diagnostic image data including corresponding vessel labels for at least one vessel of the first vessel tree, b) inputting at least one diagnostic image representing a second vessel tree, c) identifying at least one deviation between the first vessel tree and the second vessel tree, d) in response to the identifying, outputting an indication of the at least one deviation to a user and providing at least one label for the at least one deviation, and e) adjusting the initial model based on the at least one deviation and the at least one label to classify the vascular system.

[0100] With this approach, deviations from the initial model can be interactively resolved by presenting the deviations to an expert (such as a physician), who can interactively mark the deviations as represented in the image and thereby expand the training set with the necessary knowledge about anatomical variations between patients. This gradually leads to accurate automated labeling of the (coronary) vasculature.

Claims

1. A method for classifying a vascular system, the method comprising the following steps: a) training a training device using diagnostic image data representing a first vessel tree with an initial model of the vascular system, the diagnostic image data comprising a corresponding vessel label for at least one vessel of the first vessel tree, wherein the initial model is used to represent vessels in the first vessel tree; b) inputting at least one diagnostic image representing a second vessel tree; c) identifying at least one deviation between the first vessel tree and the second vessel tree; d) in response to said identifying, outputting an indication of said at least one deviation to a user and providing at least one label for said at least one deviation; and e) adjusting the initial model to classify the vasculature based on the at least one deviation and the at least one marker; The deviation includes a change between a geometric structure of the first vascular tree in the initial model and a geometric structure of the second vascular tree in the diagnostic image.

2. The method according to claim 1, wherein: Providing the at least one mark includes receiving a first user input from the user indicating the at least one mark.

3. The method according to claim 1, wherein: The adjusting the initial model to classify the vascular system comprises iteratively repeating steps b) to e) for a plurality of diagnostic images.

4. The method according to claim 1, wherein: The adjusting the initial model includes retraining the training device using the diagnostic image data representing the first vascular tree and the at least one diagnostic image representing the second vascular tree and the corresponding at least one label.

5. The method according to claim 1, further comprising: identifying a geometric structure of the second vessel tree in the at least one diagnostic image by segmenting the at least one diagnostic image; identifying corresponding centerline information for a plurality of vessels of the second vessel tree based on the segmentation; and The centerline information is extracted.

6. The method according to claim 5, wherein: The centerline information is input together with the at least one diagnostic image.

7. The method according to claim 1, wherein: The diagnostic image data comprises a plurality of images, specifically between 10,000 and 100 images, even more specifically between 1,000 and 100 images.

8. The method according to claim 1, wherein: The training device is also trained to include externally induced changes into the initial model.

9. A classification system for classifying a vascular system, comprising: a training device configured to train with an initial model of the vascular system using diagnostic image data representing a first vessel tree, the diagnostic image data comprising a corresponding vessel label for at least one vessel of the first vessel tree, wherein the initial model is used to represent a vessel in the first vessel tree; an input unit configured to receive at least one diagnostic image representing a second vessel tree; and An inference unit, which is configured to: identifying at least one deviation between the first vessel tree and the second vessel tree; outputting an indication of the at least one deviation to a user in response to the identifying; and providing at least one indicia for the at least one deviation; wherein the training device is configured to adjust the initial model to classify the vascular system based on the at least one deviation and the at least one marker; and The deviation includes a change between a geometric structure of the first vessel tree and a geometric structure of the second vessel tree.

10. The classification system according to claim 9, further comprising: a display unit configured to display said indication of said at least one deviation to a user; as well as A user interface is configured to receive a first user input from the user indicating the at least one mark.

11. The classification system according to claim 9, wherein: The inference unit is further configured to: generating a medical representation comprising at least the vasculature, the vessel marking, and the at least one marker; and The representation is sent to a database.

12. The classification system according to claim 11, wherein: The database includes an electronic medical record (EMR); and wherein the medical representation is generated according to a predetermined format of the electronic medical record.

13. A computer program product comprising instructions which, when executed by a processing unit, cause the processing unit to perform the method according to any one of claims 1 to 8 for controlling a classification system according to any one of claims 9 to 12.

14. A computer readable medium having instructions stored thereon, which when executed by a processing unit causes the processing unit to perform the method according to any one of claims 1 to 8 for controlling the classification system according to any one of claims 9 to 12.