System and method for processing images of catheterization procedure and providing alerts

Abstract

The present disclosure provides a solution for assisting in medical procedures, in particular in medical procedures involving angiogram or angioplasty procedure, such as in catheterization procedure. The solution provides alerts on risk states that may occur during the procedure related to undesired position of a medical device that is used in the medical procedure or parts thereof. This undesired position is detected in one or more images obtained from an imaging system that is used in the medical procedure, e.g. a fluoroscopy image system. The undesired position of the medical device is defined with respect to anatomy structure of the subject that is identified in the image. The solution involves the use of anatomy detection model, medical device detection model and anatomy to medical device analysis model. The anatomy detection model is configured to detect the anatomy of interest of the subject in the image, the medical device detection model is configured to detect the medical device or parts thereof in the image and the anatomy to medical device analysis model is configured to determine the relation between the medical device and the anatomy of interest of the subject to allow identification of misplacement of the medical device or parts thereof in the anatomy of interest. Upon misplacement identification of the medical device, an alert is triggered and output to the practitioner performing the medical procedure.

Description

[0001] SYSTEM AND METHOD FOR PROCESSING IMAGES OF CATHETERIZATION PROCEDURE AND PROVIDING ALERTS

[0002] TECHNOLOGICAL FIELD

[0003] The present disclosure is in the field of decision supporting technologies in medical procedures, in particular in catheterization procedures.

[0004] BACKGROUND ART

[0005] References considered to be relevant as background to the presently disclosed subject matter are listed below:

[0006] - WO 2012 / 107857

[0007] Acknowledgement of the above references herein is not to be inferred as meaning that these are in any way relevant to the patentability of the presently disclosed subject matter.

[0008] GENERAL DESCRIPTION

[0009] The present disclosure provides a solution for assisting in medical procedures, in particular in medical procedures involving angiogram or angioplasty procedure, such as in catheterization procedure. The solution provides alerts on risk states that may occur during the procedure related to undesired position of a medical device that is used in the medical procedure or parts thereof. This undesired position is detected in one or more images obtained from an imaging system that is used in the medical procedure, e.g. a fluoroscopy image system. The undesired position of the medical device is defined with respect to anatomy structure of the subject that is identified in the image. The solution involves the use of anatomy detection model, medical device detection model and anatomy to medical device analysis model. The anatomy detection model is configured to detect the anatomy of interest of the subject in the image, the medical device detection model is configured to detect the medical device or parts thereof in the image and the anatomy to medical device analysis model is configured to determine the relation between the medical device and the anatomy of interest of the subject to allow identification of misplacement of the medical device or parts thereof in the anatomy of interest. Upon misplacement identification of the medical device, an alert is triggered and output to the practitioner performing the medical procedure. A specific non-limiting example of the medical procedure is catheterization process that involves either angiogram or angioplasty procedure. The images that are used for detection may be images obtained from an X-ray imaging system or any other high energy imaging systems.

[0010] Therefore, an aspect of the present disclosure provides a computer-implemented system for processing a video stream of high energy imaging system, such as in an angiogram or angioplasty procedure or a fluoroscopy image system, and providing assisting information for the procedure, e.g. on-screen alerts, audible alerts or a combination thereof. The system comprises at least one processing circuitry; the system may comprise one or more memories coupled to the at least one processing circuitry and storing programming instructions for execution by the at least one processing circuitry. The at least one processing circuitry, in general or in particular together with the one or more memories coupled thereto, is configured for: receiving a stream of images constructing said video stream; applying on each image one or more risk state identifying models to identify one or more risk states and generate risk state data indicative of (i) an undesired position of a medical device, or parts thereof, with respect to an anatomical structure, or (ii) a damage to the anatomical structure caused in the procedure, typically by the medical device, or parts thereof; and generating instructions for outputting an alert based on said risk state data.

[0011] It is to be noted that any combination of the described embodiments with respect to any aspect of this present disclosure is applicable. In other words, any aspect of the present disclosure can be defined by any combination of the described embodiments.

[0012] In some embodiments of the system, the one or more risk state identifying models comprise machine learning model.

[0013] In some embodiments of the system, said applying comprises applying at least one of: (i) a first machine learning model trained to detect an anatomical structure of interest and generating anatomical structure location data indicative of the current most updated location of the anatomical structure of interest; and (ii) a second machine learning model trained to detect a medical device, including sub-parts of the medical device, and generating medical device location data indicative of the current most updated location and the orientation of the medical device, which can be real-time location or the last identified location before the contrast material injection is performed); performing anatomy to device analysis based on the anatomical structure location data and the medical device location data to identify one or more risk states and generate said risk state data indicative of an undesired position of the medical device, or parts thereof, with respect to the anatomical structure.

[0014] In some embodiments of the system, the processing circuitry is configured for applying the first machine learning model and the second machine learning model on real-time verified images. Namely, images that are verified as real-time images and not recorded images that are received from the imaging system of angiogram or angioplasty procedure, e.g. images that are presented in a loop of the last imaging session. Non-real- time images are less relevant for analysis as they do not present the current state of the medical device with respect to the anatomy of interest of the subject.

[0015] In some embodiments of the system, said applying further comprises applying the third machine learning model that is trained to detect a potential abnormal condition of the anatomical structure and generate abnormal condition data indicative of the type of the potential abnormal condition of the anatomical structure and the location of the abnormal condition. Said performing anatomy to device analysis is based on the anatomical structure location data, the medical device location data and the abnormal condition data to identify an abnormal condition of the anatomical structure and its location.

[0016] In some embodiments of the system the first machine learning model comprises the third machine learning model.

[0017] In some embodiments of the system, the processing circuitry is further configured to generate instructions for annotating the abnormal condition, based on the abnormal condition data, on an image being displayed, which is either the same image on which the detection of the anatomical structure of interest is made or another image that there is a spatial correlation between it and the image on which the detection was made. Namely, the position of the anatomical structure can be identified in the real-time image.

[0018] In some embodiments of the system, the abnormal condition is a perforation of the coronary artery or one of its branches. This is typically identified by identifying a contrast material that leaks from the artery.

[0019] In some embodiments of the system, the abnormal condition is dissection of the coronary artery or one of its branches. This typically occurs due to manipulation of a stent wire or balloon inflation in the artery. This condition is typically expressed in the image as a haze or blurriness in the dissection location that is caused due to irregular flow of blood and contrast material and standing contrast in the vessel lumen.

[0020] In some embodiments of the system, the processing circuitry is further configured to identify in each image a contrast material injection event, wherein only if a contrast material injection event is identified, the first machine learning model is applied.

[0021] In some embodiments of the system, the processing circuitry is further configured to generate instructions for annotating a detected anatomical structure of interest, based on the anatomical structure location data, on an image being displayed, which is either the same image on which the detection of the anatomical structure of interest is made or another image that there is a spatial correlation between it and the image on which the detection was made. Namely, the position of the anatomical structure can be identified in the real-time image.

[0022] In some embodiments of the system, the processing circuitry is further configured to generate instructions for annotating the medical device or parts thereof, based on the medical device location data, on an image being displayed, which is either the same image on which the detection of the anatomical structure of interest is made or another image that there is a spatial correlation between it and the image on which the detection was made. Namely, the position of the anatomical structure can be identified in the real-time image.

[0023] In some embodiments of the system, the anatomical structure of interest is a coronary artery of a subject and its branches, and the medical device is selected from any one of: catheter, wire, balloon, and stent, or any other non-anatomical component.

[0024] In some embodiments of the system, said performing comprises determining an angular relation between a distal end portion of a catheter body of the catheter, not including the wire that may extend from it, and a coronary artery portion in which the distal end portion is placed.

[0025] In some embodiments of the system, said performing comprises determining an overlap between the distal end portion of the catheter and the coronary artery to thereby determine how deep the catheter is inserted into the coronary artery and, for example, identify deep intubation or catheter misplaced.

[0026] In some embodiments of the system, said performing comprises determining an overlap between a catheter's wire or at least part thereof that comprises its distal end part and the coronary artery or its branches to thereby determine if the wire is misplaced, for example inserted to a small branch of the coronary artery.

[0027] In some embodiments of the system, said anatomical structure location data comprises data from a plurality of first sequential images and the medical device location data comprises data from a plurality of second sequential images. Namely, the correlation function that applies on the anatomical structure location data and the medical device location data is temporal function and takes into account data that is extracted from a plurality of sequential images. In other words, the anatomical structure location data and the medical device location data are generated based on images or real-time verified images that are within a selected time window.

[0028] In some embodiments of the system, said plurality of first sequential images comprise at least one common image with said plurality of second images and at times, the plurality of first sequential real-time verified images and the plurality of second sequential real-time verified images are the same. Namely, all the images that are used to generate the anatomical structure location data and the medical device location data are common.

[0029] In some embodiments of the system, said stream of images are received from an angiogram device, each image comprises an overlay of metadata of the image. The metadata can be presented on the anatomical image itself or aside the anatomical image, such as on a black pane.

[0030] In some embodiments, the system further comprises identifying in each image of said stream of images, whether it is a real-time image and classify it as a real-time verified image. Real-time verified images are images that are verified for presenting real-time state. Namely, the model is applied only on images that are real-time images and not a recording or a loop-recording of images that were taken before and are not presenting live, real-time imaged data.

[0031] In some embodiments of the system, said applying is performed only on real-time verified images.

[0032] In some embodiments of the system, said identifying comprises applying a realtime image identifying algorithm on each image of said stream of images to determine whether an image is a real-time image. Upon determination of a real-time image, the method further comprises classifying said image as a real-time verified image for further processing it by the machine learning models. It is to be noted that non-real-time images are not processed by the machine learning models as this does not bring any value for the physician performing the procedure.

[0033] In some embodiments of the system, said real-time image identifying algorithm comprises identifying in said overlay of metadata a pattern indicative of real-time imaging. This pattern is either a symbol or writing that identifies that the specific image is showing a real-time state.

[0034] In some embodiments of the system, said alert comprises an overlay presented on an image.

[0035] In some embodiments, the system further comprises a display unit for displaying said overlay.

[0036] In some embodiments of the system, said alert comprises an audible output.

[0037] In some embodiments, the system further comprises a speaker for outputting said audible output.

[0038] In some embodiments, the system further comprises a display unit for displaying an image that comprises the alert.

[0039] In some embodiments of the system, when said medical device data does not comprise a current location of the medical device, namely when the medical device is not detected by the second machine learning model, said processing circuitry is configured to generate instructions for outputting a no-device alert indicating that no device or parts thereof are identified in the image.

[0040] In some embodiments of the system, the processing circuitry is further configured to perform medical device tracking analysis for tracking the movement of the medical device or parts thereof in a selected time window. The medical device tracking analysis comprises determining a location of the medical device in sequential images in said time window and generate a movement profile of the device indicative of its temporal position in said time window, wherein if said movement profile deviates from a predetermined condition, the processing circuitry is further configured to generate instructions for outputting a movement alert that may include an annotation showing the movement of the device overtime.

[0041] Yet another aspect of the present disclosure provides a method for processing a video stream of high energy imaging system, e.g. a fluoroscopy image system, during an angiogram or angioplasty procedure. The method comprising: receiving a stream of images constructing said video stream; applying on each image one or more risk state identifying models to identify one or more risk states and generate risk state data indicative of (i) an undesired position of a medical device, or parts thereof, with respect to an anatomical structure, or (ii) a damage to the anatomical structure caused in the procedure, typically by the medical device, or parts thereof; and outputting an alert based on said risk state data.

[0042] In some embodiments of the method, said one or more risk state identifying models comprise machine learning model.

[0043] In some embodiments of the method, said applying comprises applying at least one of: (i) a first machine learning model trained to detect an anatomical structure of interest and generating anatomical structure location data indicative of the current, most updated, location of the anatomical structure of interest; and (ii) a second machine learning model trained to detect a medical device, including sub-parts of the medical device, and generating medical device location data indicative of the current, most updated, location and the orientation of the medical device, which can be real-time location or the last identified location before the contrast material injection is performed); and performing anatomy to device analysis based on the anatomical structure location data and the medical device location data to identify one or more risk states and generate said risk state data indicative of an undesired position of the medical device, or parts thereof, with respect to the anatomical structure.

[0044] In some embodiments of the method, the processing circuitry is configured for applying the first machine learning model and the second machine learning model on real-time verified images.

[0045] In some embodiments of the method, said applying further comprises applying the third machine learning model that is trained to detect a potential abnormal condition of the anatomical structure and generate abnormal condition data indicative of the type of the potential abnormal condition of the anatomical structure and the location of the abnormal condition. Said performing anatomy to device analysis is based on the anatomical structure location data, the medical device location data and the abnormal condition data to identify an abnormal condition of the anatomical structure and its location.

[0046] In some embodiments of the method the first machine learning model comprises the third machine learning model. In some embodiments, the method further comprising annotating the abnormal condition, based on the abnormal condition data, on an image being displayed, which is either the same image on which the detection of the anatomical structure of interest is made or another image that there is a spatial correlation between it and the image on which the detection was made. Namely, the position of the anatomical structure can be identified in the real-time image.

[0047] In some embodiments of the method, the abnormal condition is a perforation of the coronary artery or one of its branches. This is typically identified by identifying a contrast material that leaks from the artery.

[0048] In some embodiments of the method, the abnormal condition is dissection of the coronary artery or one of its branches. This typically occurs due to manipulation of a wire, stent or balloon inflation in the artery. This condition is typically expressed in the image as a haze or blurriness in the dissection location that is caused due to irregular flow of blood and contrast material, and standing contrast in the vessel lumen.

[0049] In some embodiments, the method further comprises identifying in each image a contrast material injection event, wherein only if a contrast material injection event is identified, the method comprises applying the first machine learning model.

[0050] In some embodiments, the method further comprises annotating a detected anatomical structure of interest, based on the anatomical structure location data, on an image being displayed, which is either the same image on which the detection of the anatomical structure of interest is made or another image that there is a spatial correlation between it and the image on which the detection was made. Namely, the position of the anatomical structure can be identified in an image or a verified real-time image.

[0051] In some embodiments, the method further comprises annotating the medical device or parts thereof, based on the medical device location data, on an image being displayed, which is either the same image on which the detection of the anatomical structure of interest is made or another image that there is a spatial correlation between it and the image on which the detection was made. Namely, the position of the anatomical structure can be identified in the image or the verified real-time image.

[0052] In some embodiments of the method, the anatomical structure of interest is a coronary artery of a subject and its branches, and the medical device is selected from any one of: catheter, wire, balloon, and stent, or any other non-anatomical component. In some embodiments of the method, said performing comprises determining an angular relation between a distal end portion of a catheter body of the catheter, not including the wire that may extend from it, and a coronary artery portion in which the distal end portion is placed.

[0053] In some embodiments of the method, said performing comprises determining an overlap between the distal end portion of the catheter and the coronary artery to thereby determine how deep the catheter is inserted into the coronary artery and, for example, identify deep intubation or catheter misplaced.

[0054] In some embodiments of the method, said performing comprises determining an overlap between a catheter's wire or at least part thereof that comprises its distal end part and the coronary artery or its branches to thereby determine if the wire is misplaced, for example inserted to a small branch of the coronary artery.

[0055] In some embodiments of the method, said anatomical structure location data comprises data from a plurality of first sequential images and the medical device location data comprises data from a plurality of second sequential images. Namely, the correlation function that applies on the anatomical structure location data and the medical device location data is temporal function and takes into account data that is extracted from a plurality of sequential images. In other words, the anatomical structure location data and the medical device location data are generated based on images or verified real-time verified images that are within a selected time window.

[0056] In some embodiments of the method, said plurality of first sequential images comprise at least one common image with said plurality of second images, and at times, the plurality of first sequential real-time verified images and the plurality of second sequential real-time verified images are the same. Namely, all the images that are used to generate the anatomical structure location data and the medical device location data are common.

[0057] In some embodiments of the method, said stream of images are received from an angiogram device, each image comprises an overlay of metadata of the image. The metadata can be presented on the anatomical image itself or aside the anatomical image, such as on a black pane.

[0058] In some embodiments, the method further comprises identifying in each image of said stream of images, whether it is a real-time image and classify it as a real-time verified image. Real-time verified images are images that are verified for presenting real-time state. Namely, the model may be applied only on images that are real-time images and not a recording or a loop-recording of images that were taken before and are not in realtime.

[0059] In some embodiments of the method, said applying is performed only on real-time verified images.

[0060] In some embodiments of the method, said identifying comprises applying a realtime image identifying algorithm on each image of said stream of images to determine whether an image is a real-time image, and upon determination of a real-time image classifying said image as a real-time verified image for further processing it by the machine learning models. It is to be noted that non-real-time images are not processed by the machine learning models as this, typically, does not bring any value for the physician performing the procedure.

[0061] In some embodiments of the method, said real-time image identifying algorithm comprises identifying in said overlay of metadata a pattern indicative of real-time imaging. This pattern is either a symbol or writing the identifies that the specific image is showing a real-time state.

[0062] In some embodiments of the method, said alert comprises an overlay presented on an image.

[0063] In some embodiments of the method, said alert comprises an audible output.

[0064] In some embodiments of the method, wherein when said medical device data does not comprise a current location of the medical device, namely when the medical device is not detected by the second machine learning model, the method further comprises outputting a no-device alert indicating that no device or parts thereof are identified in the image.

[0065] In some embodiments, the method further comprises performing medical device tracking analysis for tracking the movement of the medical device or parts thereof in a selected time window. The medical device tracking analysis comprises determining a location of the medical device in sequential images in said time window and generate a movement profile of the device indicative of its temporal position in said time window, wherein if said movement profile deviates from a predetermined condition, the method further comprises outputting a movement alert that may include an annotation showing the movement of the device over time. EMBODIMENTS

[0066] The following are optional embodiments and combinations thereof in accordance with aspects of the present disclosure:

[0067] 1. A computer-implemented system for processing a video stream of a fluoroscopy image system, the system comprising: at least one processing circuitry configured for: receiving a stream of images constructing said video stream; applying on each image one or more risk state identifying models to identify one or more risk states and generate risk state data indicative of (i) an undesired position of a medical device with respect to an anatomical structure, or (ii) a damage to the anatomical structure caused in the procedure, typically by the medical device, or parts thereof; and generating instructions for outputting an alert based on said risk state data.

[0068] 2. The system of embodiment 1, wherein said one or more risk state identifying models comprise machine learning model.

[0069] 3. The system of embodiment 1 or 2, wherein said applying comprises applying at least one of:

[0070] (i) a first machine learning model trained to detect an anatomical structure of interest and generating anatomical structure location data indicative of the current location of the anatomical structure of interest; and

[0071] (ii) a second machine learning model trained to detect a medical device and generating medical device location data indicative of the current location and the orientation of the medical device; performing anatomy to device analysis based on the anatomical structure location data and the medical device location data to identify one or more risk states and generate said risk state data indicative of an undesired position of the medical device with respect to the anatomical structure.

[0072] 4. The system of embodiment 3, wherein the at least one processing circuitry is configured for applying the first machine learning model and the second machine learning model on real-time verified images.

[0073] 5. The system of embodiment 3 or 4, wherein the at least one processing circuitry is further configured to identify in each image a contrast material injection event, wherein only if a contrast material injection event is identified, the first machine learning model is applied. 6. The system of any one of embodiments 3-5, wherein the at least one processing circuitry is further configured to generate instructions for annotating a detected anatomical structure of interest, based on the anatomical structure location data, on an image being displayed.

[0074] 7. The system of any one of embodiments 3-6, wherein the at least one processing circuitry is further configured to generate instructions for annotating the medical device or parts thereof, based on the medical device location data, on an image being displayed.

[0075] 8. The system of any one of embodiments 3-7, wherein the anatomical structure of interest is a coronary artery of a subject and its branches, and the medical device is selected from any one of: catheter, wire, balloon, and stent.

[0076] 9. The system of embodiment 8, wherein said performing comprises determining an angular relation between a distal end portion of a catheter body of the catheter and a coronary artery portion in which the distal end portion is placed.

[0077] 10. The system of embodiment 8 or 9, wherein said performing comprises determining an overlap between the distal end portion of the catheter and the coronary artery.

[0078] 11. The system of any one of embodiments 8-10, wherein said performing comprises determining an overlap between a catheter's wire or at least part thereof that comprises its distal end part and the coronary artery or its branches.

[0079] 12. The system of any one of embodiments 3-11, wherein said anatomical structure location data comprises data from a plurality of first sequential images and the medical device location data comprises data from a plurality of second sequential images.

[0080] 13. The system of embodiment 12, wherein said plurality of first sequential images comprise at least one common image with said plurality of second images.

[0081] 14. The system of any one of embodiments 1-13, wherein said stream of images are received from an angiogram device, each image comprises an overlay of metadata of the image.

[0082] 15. The system of embodiment 14, further comprising identifying in each image of said stream of images, whether it is a real-time image and classify it as a real-time verified image.

[0083] 16. The system of embodiment 15, wherein said applying is performed only on realtime verified images. 17. The system of embodiment 15 or 16, wherein said identifying comprises applying a real-time image identifying algorithm on each image of said stream of images to determine whether an image is a real-time image, and upon determination of a real-time image classifying said image as a real-time verified image.

[0084] 18. The system of embodiment 17, wherein said real-time image identifying algorithm comprises identifying in said overlay of metadata a pattern indicative of realtime imaging.

[0085] 19. The system of any one of embodiments 1-18, wherein said alert comprises an overlay presented on an image.

[0086] 20. The system of embodiment 19, comprising a display unit for displaying said overlay.

[0087] 21. The system of any one of embodiments 1-20, wherein said alert comprises an audible output.

[0088] 22. The system of embodiment 21, comprising a speaker for outputting said audible output.

[0089] 23. The system of any one of embodiments 1-22, comprising a display unit for displaying an image that comprises the alert.

[0090] 24. The system of any one of embodiments 1-23, wherein when said medical device data does not comprise a current location of the medical device, said at least one processing circuitry is configured to generate instructions for outputting a no-device alert indicating that no device or parts thereof are identified in the image.

[0091] 25. The system of any one of embodiments 1-24, wherein the at least one processing circuitry is further configured to perform medical device tracking analysis for tracking the movement of the medical device or parts thereof in a selected time window, said medical device tracking analysis comprises determining a location of the medical device in sequential images in said time window and generate a movement profile of the device indicative of its temporal position in said time window, wherein if said movement profile deviates from a predetermined condition, the at least one processing circuitry is further configured to generate instructions for outputting a movement alert.

[0092] 26. A method for processing a video stream of an angiogram procedure, comprising: receiving a stream of images constructing said video stream; applying on each image one or more risk state identifying models to identify one or more risk states and generate risk state data indicative of (i) an undesired position of a medical device with respect to an anatomical structure, or (ii) a damage to the anatomical structure caused in the procedure, typically by the medical device, or parts thereof; and outputting an alert based on said risk state data.

[0093] 27. The method of embodiment 26, wherein said one or more risk state identifying models comprise machine learning model.

[0094] 28. The method of embodiment 26 or 27, wherein said applying comprises applying at least one of:

[0095] (i) a first machine learning model trained to detect an anatomical structure of interest and generating anatomical structure location data indicative of the current location of the anatomical structure of interest; and

[0096] (ii) a second machine learning model trained to detect a medical device and generating medical device location data indicative of the current location and the orientation of the medical device; and performing anatomy to device analysis based on the anatomical structure location data and the medical device location data to identify one or more risk states and generate said risk state data indicative of an undesired position of the medical device with respect to the anatomical structure.

[0097] 29. The method of embodiment 26, comprising applying the first machine learning model and the second machine learning model on real-time verified images.

[0098] 30. The method of embodiment 28 or 29, further comprising identifying in each image a contrast material injection event, wherein only if a contrast material injection event is identified, the method comprises applying the first machine learning model.

[0099] 31. The method of any one of embodiments 28-30, comprising annotating a detected anatomical structure of interest, based on the anatomical structure location data, on an image being displayed.

[0100] 32. The method of any one of embodiments 28-31, comprising annotating the medical device or parts thereof, based on the medical device location data, on an image being displayed.

[0101] 33. The method of any one of embodiments 28-32, wherein the anatomical structure of interest is a coronary artery of a subject and its branches, and the medical device is selected from any one of: catheter, wire, balloon, and stent. 34. The method of embodiment 33, wherein said performing comprises determining an angular relation between a distal end portion of a catheter body of the catheter and a coronary artery portion in which the distal end portion is placed.

[0102] 35. The method of embodiment 33 or 34, wherein said performing comprises determining an overlap between the distal end portion of the catheter and the coronary artery.

[0103] 36. The method of any one of embodiments 28-35, wherein said performing comprises determining an overlap between a catheter's wire or at least part thereof that comprises its distal end part and the coronary artery or its branches.

[0104] 37. The method of any one of embodiments 28-36, wherein said anatomical structure location data comprises data from a plurality of first sequential images and the medical device location data comprises data from a plurality of second sequential images.

[0105] 38. The method of embodiment 37, wherein said plurality of first sequential images comprise at least one common image with said plurality of second images.

[0106] 39. The method of any one of embodiments 28-38, wherein said stream of images are received from an angiogram device, each image comprises an overlay of metadata of the image.

[0107] 40. The method of embodiment 39, further comprising identifying in each image of said stream of images, whether it is a real-time image and classify it as a real-time verified image.

[0108] 41. The method of embodiment 40, wherein said applying is performed only on realtime verified images.

[0109] 42. The method of embodiment 40 or 41 , wherein said identifying comprises applying a real-time image identifying algorithm on each image of said stream of images to determine whether an image is a real-time image, and upon determination of a real-time image classifying said image as a real-time verified image.

[0110] 43. The method of embodiment 42, wherein said real-time image identifying algorithm comprises identifying in said overlay of metadata a pattern indicative of realtime imaging.

[0111] 44. The method of any one of embodiments 34-43, wherein said alert comprises an overlay presented on an image.

[0112] 45. The method of any one of embodiments 34-44, wherein said alert comprises an audible output. 46. The method of any one of embodiments 34-45, wherein when said medical device data does not comprise a current location of the medical device, the method further comprises outputting a no-device alert indicating that no device or parts thereof are identified in the image.

[0113] 47. The method of any one of embodiments 34-46, comprising performing medical device tracking analysis for tracking the movement of the medical device or parts thereof in a selected time window, said medical device tracking analysis comprises determining a location of the medical device in sequential images in said time window and generate a movement profile of the device indicative of its temporal position in said time window, wherein if said movement profile deviates from a predetermined condition, the method further comprises outputting a movement alert.

[0114] BRIEF DESCRIPTION OF THE DRAWINGS

[0115] In order to better understand the subject matter that is disclosed herein and to exemplify how it may be carried out in practice, embodiments will now be described, by way of non-limiting example only, with reference to the accompanying drawings, in which:

[0116] Figs. 1A-1B are block diagrams exemplifying different non-limiting embodiments of the system according to an aspect of the present disclosure.

[0117] Fig. 2 is a flow diagram exemplifying a no-limiting embodiment of the method according to an aspect of the present disclosure.

[0118] Fig. 3 is a schematic illustration exemplifying annotation of roofing state of a catheter being identified by the method or the system of the present disclosure.

[0119] Fig. 4 is a schematic illustration exemplifying annotation of wire misplacement of a catheter being identified by the method or the system of the present disclosure.

[0120] Fig. 5 is a schematic illustration exemplifying annotation of deep intubation state of a catheter being identified by the method or the system of the present disclosure.

[0121] Figs 6-7 are flow diagrams exemplifying different non-limiting embodiments of the method according to an aspect of the present disclosure.

[0122] Fig. 8 is an example of an image with a real-time indication. DETAILED DESCRIPTION

[0123] The following figures are provided to exemplify embodiments and realization of the invention of the present disclosure.

[0124] Figs. 1A-1B are block diagrams of exemplary embodiments of the system of the present disclosure. Fig. 1A exemplifies a computer-implemented system 100 for processing a video stream of an angiogram or angioplasty procedure. The system 100 comprises a processing circuitry 102, coupled with memories (not in drawings) and stores programming instructions to be executed by the processing circuitry 102. These programming instructions include several modules, one of which is the input module 104, which is configured to receive an input of a stream of images, constructing the video stream, referred to herein as image data ID (i.e., the image data ID can be a stream of images, a single image, including any data and metadata associated with the images). This input of a stream of images may be received from an angiogram device during catheterization procedure. The processing circuitry 102 further comprises a risk state identification module 106 which receives image data ID from the input module 104 and applies on each image one or more risk state identification models, that may comprise a machine learning model, in order to identify one or more risk states. It is to be noted that the anatomical structure location identification module and the medical device location identification module are trained based on ground truth data collected from medical procedure and tagged with the corresponding features that are relevant for each module. An example for a model for identification of anatomical structure and medical device or part thereof is nnU-Net (no-new-Net) or a variation thereof, which is designed to automatically configure and adapt neural networks (NN) for various medical image segmentation. The risk state identification module 106 may be configured identifying in each image from the image data ID whether it is a real-time image and classifying it as a real-time verified image, that is, an image which is verified for presenting real-time state. The real-time verification can be made by detecting a real-time indicator that is presented on the image or an overlay panel associated with the image. An example of such indicators can be seen in Fig. 8 pointed by the arrows. A risk state data RSD is generated by the risk state identification module 106, which is indicative of an undesired position of a medical device, or parts thereof, with respect to an anatomical structure of the subject. An example for a model that operates the identification module is a ResNet (Residual Network) or a variation thereof. By using this model, many types of alerts can be automatically generated. The generated risk state data RSD is then being received by yet another module, which is the output module 108, which then generates instructions for outputting an alert based on the risk state data RSD. The alert may comprise an audible output and / or overlay presented on an image. The system may comprise one or more speakers (not in drawings) for outputting the audible output and / or a display unit (not in drawings) for displaying the overlay or possibly for displaying image that comprises the alert.

[0125] In another embodiment of the system 100, exemplified in Fig. IB, the system 100 differs from that of Fig. 1A by that the risk state identification module 106 is comprised of several different modules. The first is the anatomical structure location identification module 110, configured to receive image data ID and apply on each image a first machine learning model, trained to detect an anatomical structure of interest and generate anatomical structure location data ASLD, indicative of the current, most updated, location of the anatomical structure of interest. The anatomical structure of interest may be a coronary artery of a subject and its branches. Note that the processing circuitry 102 is further configured to identify a contrast material injection event (not presented in the drawings). In some embodiments, only the identification of a contrast material injection event triggers the first machine learning model. That is because the contrast material injection allows the anatomical structure to be visible and identifiable in the image, and thus makes the identification of the anatomical structure possible. Though, in some other embodiments, the first machine learning model is applied on each image, regardless of the identification of contrast material injection event. Another module is the medical device location identification module 112, configured to receive image data ID and apply on each image a second machine learning model, trained to detect a medical device, as well as sub-parts of medical device, and generate medical device location data MDLD, indicative of the current, most updated, location and the orientation of the medical device. Note that the identified location of the medical device can be real-time location or the last identified location before the contrast material injection is performed. In some embodiments, the medical device location data MDLD may include an estimation of the location of the medical device or parts thereof on a more recent image, in which the medical device is not identified, and the anatomical structure is identified. The medical device may be selected from any one of: catheter, wire, balloon and stent, or any other non-anatomical component. The medical device location identification module 112 is further configured to perform medical device tracking analysis for tracking the movement of the medical device or parts thereof in a selected time window. The medical device tracking analysis comprises determining a location of the medical device in sequential images in the time window and generating a movement profile of the device indicative of its temporal position in the time window. If the movement profile deviates from a predetermined condition, the output module 108 is further configured to generate instructions for outputting a movement alert (that may include an annotation showing the movement of the device over time). In addition to the risk state data RSD, as described earlier, the output module 108 is further configured to receive the anatomical structure location data ASLD and generate instructions for annotating a detected anatomical structure of interest, based on this data, on an image being displayed. In similar way, the output module is also configured to receive the medical device location data MDLD and generate instructions for annotating the medical device or parts thereof based on this data, on an image being displayed. In situations in which the medical device location data MDLD does not comprise a current location of the medical device (i.e., the medical device is not detected by the second machine-learning model), the output module 108 is configured to generate instructions for outputting a no-device alert indicating that no device or parts thereof are identified in the image. Another module of the risk state identification module 106 is the anatomical device module 114, which receives both the anatomical structure location data ASLD and the medical device location data MDLD, and based of these data, preforms anatomy to device analysis, to identify one or more risk states, and generate the risk state data RSD indicative of an undesired position of the medical device, or parts thereof, with respect to the anatomical structure. The anatomy to device analysis may comprise determining an angular relation between a distal end portion of a catheter body, not including the wire that may extend from it, and a coronary artery portion in which the distal end portion is placed. For example, this may include the detection of a "roofing" situation, as depicted in Fig. 3. Moreover, the anatomy to device analysis may comprise determining an overlap between the distal end potion of the catheter and the coronary artery. This is being done for example, to determine if the catheter is misplaced, or to identify if there is a risk of a deep intubation situation, as depicted in Fig. 5, in which the catheter is being inserted too deep inside the coronary artery. The anatomy to device analysis may also comprise determining an overlap between the catheter's wire, or at least a part thereof, that comprises its distal end part and the coronary artery of its branches. This is being done in order to determine if the wire is misplaced, or for example, if it's inserted into a small branch of the coronary artery, as depicted in Fig. 4.

[0126] Fig. 2 is a flow diagram presenting non-limiting example of an embodiment of the method according to the present disclosure. Fig. 2 shows a method that includes receiving a stream of images constructing a video stream 220, each image may comprise an overlay of metadata of the image. The method further comprises an optional step of identifying real-time images 222, that is identifying in each image of the stream of images, whether it is a real-time image, and classify it as a real-time verified image, which means that it is verified image for presenting real-time state. Note that the next steps of the method are applied only on real-time verified images, if step 222 is carried out, which means that these images are not taken from a recording or a loop-recording of images, or the following step are applied on all images, regardless they are real-time or non-real- time images. The method further comprises applying, on each image or each real-time verified image, a first machine learning model 224 trained to detect an anatomical structure of interest and generating anatomical structure location data 226 indicative of the most updated location of the anatomical structure of interest. The method further comprises applying on each image or real-time verified images, a second machinelearning model 228 trained to detect a medical device, including sub-parts of the medical device, and generating medical device location data 230 indicative of most updated location and the orientation of the medical device, which can be real-time location or the last identified location. The method further comprises of performing anatomy to device analysis 232 based on the anatomical structure location data and the medical device location data, to identify one or more risk states and generate risk state data 234. The risk state data is indicative of an undesired position of the medical device, or parts thereof with respect to the anatomical structure. The method further comprises generating instructions for outputting an alert 236.

[0127] Figs. 6-7 are flow diagrams of different examples of non-limiting examples of the method according to an aspect of the present disclosure. The figures are self-explanatory, and the method explained with respect to the Fig. 2 can be employed in any of the flow diagrams of Figs. 6-7.

Claims

CLAIMS:

1. A computer-implemented system for processing a video stream of a fluoroscopy image system, the system comprising: at least one processing circuitry configured for: receiving a stream of images constructing said video stream; applying on each image one or more risk state identifying models to identify one or more risk states and generate risk state data indicative of (i) an undesired position of a medical device with respect to an anatomical structure, or (ii) a damage to the anatomical structure caused in the procedure, typically by the medical device, or parts thereof; and generating instructions for outputting an alert based on said risk state data.

2. The system of claim 1, wherein said one or more risk state identifying models comprise machine learning model.

3. The system of claim 1, wherein said applying comprises applying at least one of:(i) a first machine learning model trained to detect an anatomical structure of interest and generating anatomical structure location data indicative of the current location of the anatomical structure of interest; and(ii) a second machine learning model trained to detect a medical device and generating medical device location data indicative of the current location and the orientation of the medical device; performing anatomy to device analysis based on the anatomical structure location data and the medical device location data to identify one or more risk states and generate said risk state data indicative of an undesired position of the medical device with respect to the anatomical structure.

4. The system of claim 3, wherein the at least one processing circuitry is configured for applying the first machine learning model and the second machine learning model on real-time verified images.

5. The system of claim 3, wherein the at least one processing circuitry is further configured to identify in each image a contrast material injection event, wherein only if a contrast material injection event is identified, the first machine learning model is applied.

6. The system of claim 3, wherein the at least one processing circuitry is further configured to generate instructions for annotating a detected anatomical structure of interest, based on the anatomical structure location data, on an image being displayed.

7. The system of claim 3, wherein the at least one processing circuitry is further configured to generate instructions for annotating the medical device or parts thereof, based on the medical device location data, on an image being displayed.

8. The system of claim 3, wherein the anatomical structure of interest is a coronary artery of a subject and its branches, and the medical device is selected from any one of: catheter, wire, balloon, and stent.

9. The system of claim 8, wherein said performing comprises determining an angular relation between a distal end portion of a catheter body of the catheter and a coronary artery portion in which the distal end portion is placed.

10. The system of claim 8, wherein said performing comprises determining an overlap between the distal end portion of the catheter and the coronary artery.

11. The system of claim 8, wherein said performing comprises determining an overlap between a catheter's wire or at least part thereof that comprises its distal end part and the coronary artery or its branches.

12. The system of claim 3, wherein said anatomical structure location data comprises data from a plurality of first sequential images and the medical device location data comprises data from a plurality of second sequential images.

13. The system of claim 12, wherein said plurality of first sequential images comprise at least one common image with said plurality of second images.

14. The system of claim 1, wherein said stream of images are received from an angiogram device, each image comprises an overlay of metadata of the image.

15. The system of claim 14, further comprising identifying in each image of said stream of images, whether it is a real-time image and classify it as a real-time verified image.

16. The system of claim 15, wherein said applying is performed only on real-time verified images.

17. The system of claim 15, wherein said identifying comprises applying a real-time image identifying algorithm on each image of said stream of images to determine whether an image is a real-time image, and upon determination of a real-time image classifying said image as a real-time verified image.

18. The system of claim 17, wherein said real-time image identifying algorithm comprises identifying in said overlay of metadata a pattern indicative of real-time imaging.

19. The system of claim 1, wherein said alert comprises an overlay presented on an image.

20. The system of claim 19, comprising a display unit for displaying said overlay.

21. The system of claim 1, wherein said alert comprises an audible output.

22. The system of claim 21, comprising a speaker for outputting said audible output.

23. The system of claim 1, comprising a display unit for displaying an image that comprises the alert.

24. The system of claim 1, wherein when said medical device data does not comprise a current location of the medical device, said at least one processing circuitry is configured to generate instructions for outputting a no-device alert indicating that no device or parts thereof are identified in the image.

25. The system of claim 1, wherein the at least one processing circuitry is further configured to perform medical device tracking analysis for tracking the movement of the medical device or parts thereof in a selected time window, said medical device tracking analysis comprises determining a location of the medical device in sequential images in said time window and generate a movement profile of the device indicative of its temporal position in said time window, wherein if said movement profile deviates from a predetermined condition, the at least one processing circuitry is further configured to generate instructions for outputting a movement alert.

26. A method for processing a video stream of an angiogram procedure, comprising: receiving a stream of images constructing said video stream; applying on each image one or more risk state identifying models to identify one or more risk states and generate risk state data indicative of (i) an undesired position of a medical device with respect to an anatomical structure, or (ii) a damage to the anatomical structure caused in the procedure, typically by the medical device, or parts thereof; and outputting an alert based on said risk state data.

27. The method of claim 26, wherein said one or more risk state identifying models comprise machine learning model.

28. The method of claim 26, wherein said applying comprises applying at least one of:(i) a first machine learning model trained to detect an anatomical structure of interest and generating anatomical structure location data indicative of the current location of the anatomical structure of interest; and(ii) a second machine learning model trained to detect a medical device and generating medical device location data indicative of the current location and the orientation of the medical device; and performing anatomy to device analysis based on the anatomical structure location data and the medical device location data to identify one or more risk states and generate said risk state data indicative of an undesired position of the medical device with respect to the anatomical structure.

29. The method of claim 26, comprising applying the first machine learning model and the second machine learning model on real-time verified images.

30. The method of claim 28, further comprising identifying in each image a contrast material injection event, wherein only if a contrast material injection event is identified, the method comprises applying the first machine learning model.

31. The method of claim 28, comprising annotating a detected anatomical structure of interest, based on the anatomical structure location data, on an image being displayed.

32. The method of claim 28, comprising annotating the medical device or parts thereof, based on the medical device location data, on an image being displayed.

33. The method of claim 28, wherein the anatomical structure of interest is a coronary artery of a subject and its branches, and the medical device is selected from any one of: catheter, wire, balloon, and stent.

34. The method of claim 33, wherein said performing comprises determining an angular relation between a distal end portion of a catheter body of the catheter and a coronary artery portion in which the distal end portion is placed.

35. The method of claim 33, wherein said performing comprises determining an overlap between the distal end portion of the catheter and the coronary artery.

36. The method of claim 28, wherein said performing comprises determining an overlap between a catheter's wire or at least part thereof that comprises its distal end part and the coronary artery or its branches.

37. The method of claim 28, wherein said anatomical structure location data comprises data from a plurality of first sequential images and the medical device location data comprises data from a plurality of second sequential images.

38. The method of claim 37, wherein said plurality of first sequential images comprise at least one common image with said plurality of second images.

39. The method of claim 28, wherein said stream of images are received from an angiogram device, each image comprises an overlay of metadata of the image.

40. The method of claim 39, further comprising identifying in each image of said stream of images, whether it is a real-time image and classify it as a real-time verified image.

41. The method of claim 40, wherein said applying is performed only on real-time verified images.

42. The method of claim 40, wherein said identifying comprises applying a real-time image identifying algorithm on each image of said stream of images to determine whether an image is a real-time image, and upon determination of a real-time image classifying said image as a real-time verified image.

43. The method of claim 42, wherein said real-time image identifying algorithm comprises identifying in said overlay of metadata a pattern indicative of real-time imaging.

44. The method of claim 34, wherein said alert comprises an overlay presented on an image.

45. The method of claim 34, wherein said alert comprises an audible output.

46. The method of claim 34, wherein when said medical device data does not comprise a current location of the medical device, the method further comprises outputting a no-device alert indicating that no device or parts thereof are identified in the image.

47. The method of claim 34, comprising performing medical device tracking analysis for tracking the movement of the medical device or parts thereof in a selected time window, said medical device tracking analysis comprises determining a location of the medical device in sequential images in said time window and generate a movement profile of the device indicative of its temporal position in said time window, wherein if said movement profile deviates from a predetermined condition, the method further comprises outputting a movement alert.

Images