Device for Tracking a Catheter
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- KONINKLIJKE PHILIPS NV
- Filing Date
- 2023-06-26
- Publication Date
- 2026-04-21
AI Technical Summary
The high number of chest X-ray examinations required for verifying the positioning of catheters and tubes, such as central venous catheters and endotracheal tubes, leads to increased workload and prolonged report turnaround times in radiology departments, with malpositioning incidents like pneumothorax being a significant concern.
A system that aligns medical images with real-time sensor data to provide accurate positioning of catheters and tubes by correlating patient data from sensors with a sensor coordinate system, allowing for real-time tracking and display of the catheter or tube position relative to the medical image, using technologies like cameras, lidar, and radar to ensure correct insertion.
Reduces the need for additional X-ray examinations by providing immediate feedback during the insertion process, improving patient safety and reducing radiologist workload through accurate, real-time tracking and visualization of catheters and tubes.
Smart Images
Figure 00000000_0000_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an apparatus for tracking a catheter, a system for tracking a catheter, a method for tracking a catheter, and a computer program element and a computer-readable medium.
Background Art
[0002] Chest X-ray examination (CXR) is one of the most common examinations in the radiology department because it is relatively low-cost and has a short acquisition time. As a result, the radiology department may have too many CXRs, significantly increasing the report turnaround time.
[0003] CXR is often used as the main modality for assessing the position of foreign bodies. Examples of foreign bodies include central venous catheters (CVCs) (e.g., see: Pikwer A, Baath L, Davidson B, Perstoft I, Akeson (2010). The incidence and risk of central venous catheter malpositioning: A prospective cohort study in 1619 patients. Anaesth Intensive Care. 2008;36:30 - 7, and Yi, X., Adams, S.J., Henderson, R.D., & Babyn, P. (2020). Computer - aided Assessment of Catheters and Tubes on Radiographs: How Good Is Artificial Intelligence for Assessment?. Radiology: Artificial Intelligence, 2(1), e190082), endotracheal tubes (e.g., see: Chen, S., Zhang, M., Yao, L., & Xu, W. (2016). Endotracheal tubes positioning detection in adult portable chest radiography for intensive care unit. International journal of computer assisted radiology and surgery, 11(11), 2049 - 2057), and feeding tubes (e.g., see: Singh, V., Danda, V., Gorniak, R., Flanders, A., & Lakhani, P. (2019). Assessment of critical feeding tube malpositions on radiographs using deep learning. Journal of digital imaging, 32(4), 651 - 655). Detecting malpositioned foreign bodies accurately and at the appropriate timing is very important for certain external objects.For example, the insertion of a CVC is notoriously difficult and can cause pneumothorax (PTX). In addition to the initial positioning of the device, routine verification using X-ray imaging is commonly performed. The incidence of malposition of CVCs is 3.6 - 14% (see: Pikwer A, Baath L, Davidson B, Perstoft I, Akeson (2010). The incidence and risk of central venous catheter malpositioning: A prospective cohort study in 1619 patients. Anaesth Intensive Care. 2008;36:30 - 7). The University Hospital Hamburg-Eppendorf, a clinical partner of Philips, currently reports that 20% of all CXRs present this clinical doubt. Conventionally, catheters and tubes have been inserted blindly or with the aid of ultrasound guidance. When the device is placed in this way, it is usually verified by one or more CXRs after the insertion procedure. This, as mentioned above, results in too many CXRs that have to be performed, extending the report turnaround time.
[0004] Therefore, there is a clinical need for an automated solution that can reduce the workload of hospital radiologists.
Summary of the Invention
Problems to be Solved by the Invention
[0005] It would be beneficial to provide an improved technique for reducing the need for X-ray examinations related to the insertion of catheters and tubes into patients.
Means for Solving the Problems
[0006] The object of the present invention is solved by the subject matter of the independent claims, and further embodiments are incorporated in the dependent claims. It should be noted that the aspects and examples of the present invention described below apply to a device for tracking a catheter, a system for tracking a catheter, a method for tracking a catheter, and computer program elements and computer-readable media.
[0007] In a first aspect, a device for tracking a catheter or a tube is provided. The device includes an input unit and a processing unit.
[0008] The input unit receives a medical image of the patient. The input unit provides the medical image of the patient to the processing unit. The input unit receives a live data stream from one or more sensors. The live data stream includes patient data, and a sensor coordinate system is defined for the one or more sensors. The input unit provides the live data stream to the processing unit. The processing unit receives the sensed position of the end of the catheter or tube to be inserted into the patient. The processing unit correlates the patient data received from the one or more sensors with the sensor coordinate system. The processing unit maps the medical image of the patient to the sensor coordinate system. The processing unit uses the sensed position of the end of the catheter or tube to determine the position of the end of the catheter or tube in the sensor coordinate system. The processing unit uses the medical image of the patient mapped to the sensor coordinate system and the position of the end of the catheter or tube in the sensor coordinate system to generate a representation of the sensed position of the end of the catheter or tube with respect to the medical image.
[0009] In other words, previous medical images of the patient, which may have been taken one year ago, one month ago, one week ago, one day ago, one hour ago, or a few minutes ago, are provided. The medical image shows details such as the patient's anatomical structure, for example, the patient's venous structure. Next, it is necessary to insert a catheter or a tube into the patient's body. For example, a central venous catheter (CVC) is inserted into the body. However, the insertion of a CVC is very difficult, and malposition of the CVC often occurs.
[0010] Therefore, new technologies have been developed for aligning a patient's medical image with the patient or, in fact, on the patient. Thus, previously collected X-ray images or MR images (or other medical images) are aligned with or associated with the patient, and when a healthcare provider inserts a catheter or tube into the patient, the anatomical structure of the patient that is actually mapped to the patient in front of the eyes is provided via the medical image, whereby the healthcare provider can correctly insert the catheter into the correct anatomical structure, such as a vein, that can be confirmed via the medical image aligned with the patient.
[0011] This new technology uses data indicating a patient from sensor systems such as a camera system, a lidar system, and a radar system. The sensor system generates a sensor coordinate system using anchors, antennas, or other markers in the room where the patient is located. The sensor system also provides patient data such as image data, depth data, and / or radar data in a real-time data stream and arranges the patient data in the real-time data stream in the sensor coordinate system. Then, previously collected medical images of the patient are mapped to the sensor coordinate system via image registration. Next, the position of the end of a catheter or tube to be inserted into the patient is sensed and is also arranged in the sensor coordinate system. Since both the medical image and the position of the end of the catheter or tube are coordinated with the sensor coordinate system, the position of the end of the catheter or tube can be represented with respect to the medical image. Therefore, the medical image or the warped medical image can be displayed on a visual display unit together with the position of the end of the catheter or tube. Then, when a medical professional moves the catheter or tube towards the patient, the end of the tube or catheter can be represented as moving with respect to the medical image on the visual display unit. Then, the medical professional can confirm the relevant anatomical structure (e.g., venous structure) of the patient and correctly insert the catheter or tube. As the patient moves, the medical image of the patient is mapped to the live data stream, so the medical image of the patient can be represented to account for its movement, and thus the medical professional can continuously confirm an accurate representation of, for example, the patient's venous structure and reliably and correctly insert the catheter or tube.
[0012] In one example, the processing unit utilizes the medical image of the patient mapped to the sensor coordinate system to generate a mapped medical image of the patient. Generating a representation of the sensed position of the end of the catheter or tube with respect to the medical image includes generating a representation of the sensed position of the end of the catheter or tube within the mapped medical image of the patient.
[0013] Accordingly, medical images such as X-ray images and MR images are warped to match the patient data collected by one or more sensors, such as the image data (defining the sensor coordinate system). Thus, if a medical image was collected two months ago when the patient was heavier than they are currently, this medical image is warped to match the patient's current appearance as sensed by one or more sensors.
[0014] In one example, the processing unit utilizes the patient's medical image mapped to the sensor coordinate system to generate a modified sensor coordinate system. In this case, determining the position of the end of the catheter or tube in the sensor coordinate system includes determining the position of the end of the catheter or tube in the modified sensor coordinate system. In this case, generating a representation of the sensed position of the end of the catheter or tube with respect to the medical image includes generating a representation of the sensed position of the end of the catheter or tube within the patient's medical image.
[0015] In other words, the patient's medical image forms the ground truth data, and although not warped, when aligning the medical image with the sensor coordinate system, the sensor coordinate system itself is warped, effectively warping the patient's live data feed. Thus, the medical image is always realistic and not warped, but the position of the end of the catheter or tube is displayed in the warped sensor coordinate system and makes movements that are slightly different from the movements made by the medical professional. This is because, for example, the patient can move, but the medical professional is always presented with a non-warped version, such as a medical image of the patient with non-warped venous structures.
[0016] In other words, patient data such as image data collected by one or more sensors (defining a sensor coordinate system) is warped, for example, to match medical images such as X-ray images or MR images collected several months ago when the patient's weight was heavier than it is currently. Thus, the medical images are effectively ground truth, and the space in which the patient is located is effectively warped. Thus, as described above, when the patient's weight is heavy when the medical image is collected, medical personnel will view this image on the screen. This image is larger than the current patient, but the representation of the space is warped so that as the catheter is moved towards the patient, the catheter reaches the same point simultaneously with the patient and the X-ray image, and at the correct point as indicated by the medical image.
[0017] Before returning to the warped medical image warped for actually inserting a catheter or tube into the patient, it is useful to switch to the unwarped medical image presented in the warped sensor coordinate system to verify the patient's anatomical structure such as the venous structure, and vice versa.
[0018] In one example, the one or more sensors include multiple sensors, and the live data stream includes data from each of the multiple sensors.
[0019] In one example, correlating the patient data with the sensor coordinate system utilizes patient data from each of the multiple sensors.
[0020] Thus, for example, image data from a 3D camera, radar, or lidar sensor can by itself determine the patient's position with respect to the position of such a sensor and place the patient in the sensor coordinate system. However, this information can be generated using multiple sensors. Thus, for example, multiple cameras actually provide 3D data of the patient, or multiple radar systems provide distance and direction information to provide detailed position information of the patient.
[0021] Thus, using triangulation, markers, and antennas, patient data within the sensor coordinate system, i.e., actually 3D data, can be found.
[0022] In one example, the live data stream includes data at the end of a catheter or tube, and the sensed position of the end of the catheter or tube inserted into the patient is determined from the data of the end of the catheter or tube from each of a plurality of sensors.
[0023] Thus, for example, using triangulation, the position of the end of the catheter or tube within the sensor coordinate system can be accurately sensed.
[0024] In one example, a radio positioning system senses the position of the end of a catheter or tube. The radio positioning system provides the sensed position of the end of the catheter or tube to a processing unit.
[0025] In other words, a radar system is used to sense the end of the catheter or tube. The radar system is at a known position and orientation in the camera coordinate system, and then the sensed position of the end of the catheter or tube with respect to the radar system is known with respect to the camera coordinate system.
[0026] In one example, the catheter or tube includes an optical fiber shaped sensing system that senses the position of the end of the catheter or tube. The optical fiber shaped sensing system provides the sensed position of the end of the catheter or tube to a processing unit.
[0027] In one example, mapping a medical image of a patient to the sensor coordinate system includes aligning the medical image of the patient with at least one frame of the live data stream.
[0028] Thus, for example, a frame of a data stream can be matched to a medical image, which includes finding and applying a warp transformation that correlates the patient's body data within the data frame with the patient's medical image. And this can be achieved there using classical computer vision algorithms or neural networks.
[0029] Thus, it is possible to map a camera image to a medical image (e.g., an X-ray image or an MR image), map a radar image to a medical image (e.g., an X-ray image or an MR image), or map depth data from a radar / RGB-D camera / lidar to a medical image (e.g., an X-ray image or an MR image).
[0030] In one example, aligning at least one frame of a live data stream with a medical image of a patient includes utilizing a warp transformation for correlating the medical image of the patient's body with the patient's data within at least one frame of the live data stream.
[0031] In a second aspect, a system for tracking a catheter or tube is provided. The system includes an input unit, a processing unit, one or more sensors, and a visual display unit.
[0032] The input unit receives the medical images of the patient. The input unit provides the medical images of the patient to the processing unit. The input unit receives a live data stream from one or more sensors. The live data stream includes patient data, and a sensor coordinate system is defined for one or more sensors. The input unit provides the live data stream to the processing unit. The processing unit receives the sensed position of the end of the catheter or tube to be inserted into the patient. The processing unit correlates the patient data received from one or more sensors with the sensor coordinate system. The processing unit maps the medical images of the patient to the sensor coordinate system. The processing unit uses the sensed position of the end of the catheter or tube to determine the position of the end of the catheter or tube in the sensor coordinate system. The processing unit uses the medical images of the patient mapped to the sensor coordinate system and the position of the end of the catheter or tube in the sensor coordinate system to generate a representation of the sensed position of the end of the catheter or tube with respect to the medical images on the visual display unit.
[0033] In one example, the processing unit uses the medical images of the patient mapped to the sensor coordinate system to generate a mapped medical image of the patient. The processing unit displays the mapped medical image on the visual display unit. Generating a representation of the sensed position of the end of the catheter or tube with respect to the medical images includes generating a representation of the sensed position of the end of the catheter or tube within the mapped medical image of the patient displayed on the visual display unit.
[0034] In one example, the processing unit uses the medical images of the patient mapped to the sensor coordinate system to generate a modified sensor coordinate system. In this case, determining the position of the end of the catheter or tube in the sensor coordinate system includes determining the position of the end of the catheter or tube in the modified sensor coordinate system. The processing unit displays the medical image on the visual display unit. In this case, generating a representation of the sensed position of the end of the catheter or tube with respect to the medical images includes generating a representation of the sensed position of the end of the catheter or tube within the medical image of the patient displayed on the visual display unit.
[0035] In a third aspect, a method for tracking a catheter or tube is provided. The method includes receiving a medical image of a patient, providing the medical image of the patient to a processing unit, receiving a live data stream from one or more sensors, the live data stream including patient data and a sensor coordinate system being defined for the one or more sensors, providing the live data stream to the processing unit, receiving, by the processing unit, a sensed position of an end of a catheter or tube to be inserted into the patient, associating, by the processing unit, the patient data received from the one or more sensors with the sensor coordinate system, mapping, by the processing unit, the medical image of the patient to the sensor coordinate system, determining, by the processing unit, a position of an end of the catheter or tube in the sensor coordinate system using the sensed position of the end of the catheter or tube, generating, by the processing unit, a representation of the sensed position of the end of the catheter or tube with respect to the medical image using the medical image of the patient mapped to the sensor coordinate system and the position of the end of the catheter or tube in the sensor coordinate system.
[0036] According to an aspect, as described above, a computer program element for controlling one or more of the aforementioned devices and / or systems is provided. When the computer program element is executed by a processor, the method is performed as described above.
[0037] In another aspect, a computer-readable medium storing the aforementioned program element is provided.
[0038] A computer program element can be, for example, a software program, but it can also be an FPGA, a PLD, or any other suitable digital means.
[0039] Advantageously, any advantages obtained by any of the above aspects are equally applicable to all other aspects, and vice versa.
[0040] The above aspects and examples will become apparent from the embodiments described below and will be described with reference to those embodiments.
Brief Description of the Drawings
[0041] Exemplary embodiments will be described below with reference to the following drawings.
[0042]
Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Modes for Carrying Out the Invention
[0043] Figure 1 shows an example of an apparatus 10 for tracking a catheter or a tube. This apparatus includes an input unit 20 and a processing unit 30. The input unit receives a medical image of a patient. The input unit provides the medical image of the patient to the processing unit. The input unit receives a live data stream from one or more sensors. The live data stream includes patient data, and a sensor coordinate system is defined with respect to the one or more sensors. The input unit provides the live data stream to the processing unit. The processing unit receives a sensed position of an end of a catheter or a tube to be inserted into the patient. The processing unit correlates the patient data received from the one or more sensors with the sensor coordinate system. The processing unit maps the medical image of the patient to the sensor coordinate system. The processing unit uses the sensed position of the end of the catheter or the tube to determine the position of the end of the catheter or the tube in the sensor coordinate system. The processing unit uses the medical image of the patient mapped to the sensor coordinate system and the position of the end of the catheter or the tube in the sensor coordinate system to generate a representation of the sensed position of the end of the catheter or the tube with respect to the medical image.
[0044] In one example, the medical image is an attenuation X-ray image.
[0045] In one example, the medical image is a CT X-ray image.
[0046] In one example, the medical image is a magnetic resonance image.
[0047] In one example, the one or more sensors are one or more cameras, and the live data stream is a live video image stream.
[0048] In one example, the one or more cameras include one or more 3D cameras, and the live data stream is a live video image stream or a live depth data stream.
[0049] In one example, the one or more cameras include one or more RGB-D cameras, and the live data stream is a live video image stream and / or a live depth data stream.
[0050] In one example, the one or more sensors are one or more lidar sensors, and the live data stream is a live lidar image stream and / or a live depth data stream.
[0051] For example, the one or more sensors are one or more radars, and the live data stream is a live radar data stream, a live image stream, and / or a live depth data stream.
[0052] In one example, the patient data received from the one or more sensors includes the patient's image data received from the one or more cameras.
[0053] In one example, the patient data received from the one or more sensors includes the patient's lidar image data received from the one or more lidar sensors.
[0054] In one example, the patient data received from the one or more sensors includes the patient's lidar depth data received from the one or more lidar sensors.
[0055] In one example, the patient data received from the one or more sensors includes the patient's depth data received from the one or more cameras.
[0056] In one example, the patient data received from the one or more sensors includes the patient's depth data received from the one or more radars.
[0057] In one example, the patient data received from the one or more sensors includes the patient's image data received from the one or more radars.
[0058] In one example, associating the patient data with the sensor coordinate system utilizes the patient data from the one or more sensors.
[0059] Accordingly, knowledge regarding the position of one or more sensors is used together with the patient's sensor data to determine the position of the portion of the patient related to the position of the sensor, whereby the patient can be placed in the sensor coordinate system.
[0060] In one example, the sensor coordinate system is a camera coordinate system.
[0061] In one example, the sensor coordinate system is a lidar sensor coordinate system.
[0062] In one example, the sensor coordinate system is a radar sensor coordinate system.
[0063] According to one example, the processing unit utilizes a medical image of the patient mapped to the sensor coordinate system to generate a mapped medical image of the patient. In this case, generating a representation of the sensed position of the end of the catheter or tube with respect to the medical image includes generating a representation of the sensed position of the end of the catheter or tube within the mapped medical image of the patient.
[0064] In one example, the mapped medical image of the patient is a warped version of the medical image that has been transformed to be coordinated with the patient's data from a live data stream.
[0065] In other words, the live data of the patient in the sensor coordinate system forms ground truth data, and the medical image is warped to match this ground truth when presented to a medical professional together with the position of the end of the catheter or tube in the camera coordinate system.
[0066] According to one example, the processing unit generates a modified sensor coordinate system using a medical image of the patient mapped to the sensor coordinate system. In this case, determining the position of the end of the catheter or tube in the sensor coordinate system includes determining the position of the end of the catheter or tube in the modified sensor coordinate system. In this case, generating a representation of the sensed position of the end of the catheter or tube with respect to the medical image includes generating a representation of the sensed position of the end of the catheter or tube within the medical image of the patient.
[0067] According to one example, the one or more sensors include a plurality of sensors, and the live data stream includes data from each of the plurality of sensors.
[0068] According to one example, correlating the patient's data with the sensor coordinate system utilizes the patient's data from each of the plurality of sensors.
[0069] According to one example, the live data stream includes data on the end of the catheter or tube, and the sensed position of the end of the catheter or tube inserted into the patient is determined from the data on the end of the catheter or tube from each of the plurality of sensors.
[0070] In one example, the data on the end of the catheter or tube includes image data collected by a plurality of cameras.
[0071] In one example, the data on the end of the catheter or tube includes depth data collected by a plurality of cameras.
[0072] In one example, the data on the end of the catheter or tube includes image data collected by a plurality of lidar sensors.
[0073] In one example, the data on the end of the catheter or tube includes depth data collected by a plurality of lidar sensors.
[0074] In one example, the data at the end of the catheter or tube includes image data collected by a plurality of radars.
[0075] In one example, the data at the end of the catheter or tube includes depth data collected by a plurality of radars.
[0076] According to one example, a wireless positioning system senses the position of the end of the catheter or tube. The wireless positioning system provides the sensed position of the end of the catheter or tube to a processing unit.
[0077] According to one example, the catheter or tube includes an optical fiber shape sensing system that senses the position of the end of the catheter or tube. The optical fiber shape sensing system provides the sensed position of the end of the catheter or tube to a processing unit.
[0078] According to one example, mapping a medical image of a patient to a sensor coordinate system includes aligning the medical image of the patient with at least one frame of a live data stream.
[0079] According to one example, aligning the medical image of the patient with at least one frame of a live data stream includes using a warp transformation to associate the medical image of the patient's body with the patient's data within at least one frame of the live data stream.
[0080] FIG. 2 shows an example of a system 100 for tracking a catheter or tube. The system 100 includes an input unit 120, a processing unit 130, one or more sensors 140, and a visual display unit 150. The input unit receives a medical image of a patient. The input unit provides the medical image of the patient to the processing unit. The input unit receives a live data stream from one or more sensors. The live data stream includes patient data, and a sensor coordinate system is defined with respect to the one or more sensors. The input unit provides the live data stream to the processing unit. The processing unit receives a sensed position of an end of a catheter or tube to be inserted into the patient. The processing unit correlates the patient data received from the one or more sensors with the sensor coordinate system. The processing unit maps the medical image of the patient to the sensor coordinate system. The processing unit determines the position of the end of the catheter or tube in the sensor coordinate system using the sensed position of the end of the catheter or tube. The processing unit uses the medical image of the patient mapped to the sensor coordinate system and the position of the end of the catheter or tube in the sensor coordinate system to generate a representation of the sensed position of the end of the catheter or tube with respect to the medical image on the visual display unit.
[0081] In one example, the medical image is an attenuation X-ray image.
[0082] In one example, the medical image is a CT X-ray image.
[0083] In one example, the medical image is a magnetic resonance image.
[0084] In one example, the one or more sensors are one or more cameras, and the live data stream is a live video image stream.
[0085] In one example, the one or more cameras include one or more 3D cameras, and the live data stream is a live video image stream or a live depth data stream.
[0086] In one example, the one or more cameras include one or more RGB-D cameras, and the live data stream is a live video image stream and / or a live depth data stream.
[0087] In one example, the one or more sensors are one or more lidar sensors, and the live data stream is a live lidar image stream and / or a live depth data stream.
[0088] For example, the one or more sensors are one or more radars, and the live data stream is a live radar data stream, a live image stream, and / or a live depth data stream.
[0089] In one example, the patient data received from the one or more sensors includes the patient's image data received from the one or more cameras.
[0090] In one example, the patient data received from the one or more sensors includes the patient's lidar image data received from the one or more lidar sensors.
[0091] In one example, the patient data received from the one or more sensors includes the patient's lidar depth data received from the one or more lidar sensors.
[0092] In one example, the patient data received from the one or more sensors includes the patient's depth data received from the one or more cameras.
[0093] In one example, the patient data received from the one or more sensors includes the patient's depth data received from the one or more radars.
[0094] In one example, the patient data received from the one or more sensors includes the patient's image data received from the one or more radars.
[0095] In one example, correlating the patient data with the sensor coordinate system utilizes the patient data from the one or more sensors.
[0096] Thus, knowledge about the position of one or more sensors can be used together with the patient's sensor data to determine the position of the patient's part relative to the position of the sensor, thereby enabling the patient to be placed in the sensor coordinate system.
[0097] In one example, the sensor coordinate system is a camera coordinate system.
[0098] In one example, the sensor coordinate system is a lidar sensor coordinate system.
[0099] In one example, the sensor coordinate system is a radar sensor coordinate system.
[0100] According to one example, the processing unit utilizes the medical image of the patient mapped to the sensor coordinate system to generate a mapped medical image of the patient. The processing unit displays the mapped medical image on a visual display unit. In this case, generating a representation of the sensed position of the end of the catheter or tube with respect to the medical image includes generating a representation of the sensed position of the end of the catheter or tube within the mapped medical image of the patient displayed on the visual display unit.
[0101] According to one example, the processing unit utilizes the medical image of the patient mapped to the sensor coordinate system to generate a modified sensor coordinate system. In this case, determining the position of the end of the catheter or tube in the sensor coordinate system includes determining the position of the end of the catheter or tube in the modified sensor coordinate system. The processing unit displays the medical image on a visual display unit. In this case, generating a representation of the sensed position of the end of the catheter or tube with respect to the medical image includes generating a representation of the sensed position of the end of the catheter or tube within the medical image of the patient displayed on the visual display unit.
[0102] In one example, the one or more sensors include a plurality of sensors, and the live data stream includes data from each of the plurality of sensors.
[0103] In one example, correlating a patient's data with a sensor coordinate system utilizes the patient's data from each of a plurality of sensors.
[0104] In one example, a live data stream includes data at the end of a catheter or tube, and the sensed position of the end of the catheter or tube inserted into the patient is determined from the data at the end of the catheter or tube from each of a plurality of sensors.
[0105] In one example, the data at the end of the catheter or tube includes image data collected by a plurality of cameras.
[0106] In one example, the data at the end of the catheter or tube includes depth data collected by a plurality of cameras.
[0107] In one example, the data at the end of the catheter or tube includes image data collected by a plurality of lidar sensors.
[0108] In one example, the data at the end of the catheter or tube includes depth data collected by a plurality of lidar sensors.
[0109] In one example, the data at the end of the catheter or tube includes image data collected by a plurality of radars.
[0110] In one example, the data at the end of the catheter or tube includes depth data collected by a plurality of radars.
[0111] In one example, a wireless positioning system senses the position of the end of the catheter or tube. The wireless positioning system provides the sensed position of the end of the catheter or tube to a processing unit.
[0112] In one example, a catheter or tube includes an optical fiber shaped sensing system that senses the position of the end of the catheter or tube. The optical fiber shaped sensing system provides the sensed position of the end of the catheter or tube to a processing unit.
[0113] In one example, mapping a medical image of a patient to a sensor coordinate system includes aligning the medical image of the patient with at least one frame of a live data stream.
[0114] In one example, aligning the medical image of the patient with at least one frame of the live data stream includes utilizing a warp transformation to correlate the medical image of the patient's body with the patient's data within at least one frame of the live data stream.
[0115] FIG. 3 shows the basic steps of a method 200 for tracking a catheter or tube. This method 200 includes receiving a medical image of the patient (210), providing the medical image of the patient to a processing unit (220), receiving a live data stream from one or more sensors (230), where the live data stream includes patient data and a sensor coordinate system is defined with respect to the one or more sensors, providing the live data stream to a processing unit (240), receiving, by the processing unit, the sensed position of the end of a catheter or tube to be inserted into the patient (250), correlating, by the processing unit, the patient data received from the one or more sensors with the sensor coordinate system (260), mapping, by the processing unit, the medical image of the patient to the sensor coordinate system (270), determining, by the processing unit, the position of the end of the catheter or tube in the sensor coordinate system using the sensed position of the end of the catheter or tube (280), The processing unit generates an expression of the sensed position of the end of the catheter or tube with respect to the medical image by using the medical image of the patient mapped in the sensor coordinate system and the position of the end of the catheter or tube in the sensor coordinate system (290).
[0116] In one example, the medical image is an attenuation X-ray image.
[0117] In one example, the medical image is a CT X-ray image.
[0118] In one example, the medical image is a magnetic resonance image.
[0119] In one example, one or more sensors are one or more cameras, and the live data stream is a live video image stream.
[0120] In one example, one or more cameras include one or more 3D cameras, and the live data stream is a live video image stream or a live depth data stream.
[0121] In one example, one or more cameras include one or more RGB-D cameras, and the live data stream is a live video image stream and / or a live depth data stream.
[0122] In one example, one or more sensors are one or more lidar sensors, and the live data stream is a live lidar image stream and / or a live depth data stream.
[0123] In one example, one or more sensors are one or more radars, and the live data stream is a live radar data stream, a live image stream, and / or a live depth data stream.
[0124] In one example, the patient data received from one or more sensors includes the patient image data received from one or more cameras.
[0125] In one example, the patient data received from one or more sensors includes the patient's lidar image data received from one or more lidar sensors.
[0126] In one example, the patient data received from one or more sensors includes the patient's lidar depth data received from one or more lidar sensors.
[0127] In one example, the patient data received from one or more sensors includes the patient's depth data received from one or more cameras.
[0128] In one example, the patient data received from one or more sensors includes the patient's depth data received from one or more radars.
[0129] In one example, the patient data received from one or more sensors includes the patient's image data received from one or more radars.
[0130] In one example, associating the patient data with the sensor coordinate system utilizes the patient data from one or more sensors.
[0131] Thus, knowledge of the positions of one or more sensors is used together with the patient's sensor data to determine the position of the patient's part relative to the sensor's position, thereby placing the patient in the sensor coordinate system.
[0132] In one example, the sensor coordinate system is a camera coordinate system.
[0133] In one example, the sensor coordinate system is a lidar sensor coordinate system.
[0134] In one example, the sensor coordinate system is a radar sensor coordinate system.
[0135] In one example, the method includes generating, by a processing unit, a mapped medical image of a patient using the medical image of the patient mapped to a sensor coordinate system, and generating a representation of a sensed position of an end of a catheter or tube with respect to the medical image includes generating a representation of the sensed position of the end of the catheter or tube within the mapped medical image of the patient.
[0136] In one example, the method includes generating, by a processing unit, a modified sensor coordinate system using the medical image of the patient mapped to a sensor coordinate system. In this case, determining the position of the end of the catheter or tube in the sensor coordinate system includes determining the position of the end of the catheter or tube in the modified sensor coordinate system, and generating a representation of the sensed position of the end of the catheter or tube with respect to the medical image includes generating a representation of the sensed position of the end of the catheter or tube within the medical image of the patient.
[0137] In one example, the one or more sensors include a plurality of sensors, and the live data stream includes data from each of the plurality of sensors.
[0138] In one example, correlating patient data with a sensor coordinate system includes using patient data from each of the plurality of sensors.
[0139] In one example, the live data stream includes data of the end of a catheter or tube, and the sensed position of the end of the catheter or tube inserted into the patient is determined from the data of the end of the catheter or tube from each of the plurality of sensors.
[0140] In one example, the data of the end of the catheter or tube includes image data collected by a plurality of cameras.
[0141] In one example, the data of the end of the catheter or tube includes depth data collected by a plurality of cameras.
[0142] In one example, the data at the end of the catheter or tube includes image data collected by a plurality of lidar sensors.
[0143] In one example, the data at the end of the catheter or tube includes depth data collected by a plurality of lidar sensors.
[0144] In one example, the data at the end of the catheter or tube includes image data collected by a plurality of radars.
[0145] In one example, the data at the end of the catheter or tube includes depth data collected by a plurality of radars.
[0146] In one example, the method includes receiving, by a processing unit, a sensed position of an end of a catheter or tube from a wireless positioning system that senses the position of the end of the catheter or tube.
[0147] In one example, the method includes receiving, by a processing unit, a sensed position of an end of a catheter or tube from an optical fiber shape sensing system of the catheter or tube that senses the position of the end of the catheter or tube.
[0148] In one example, mapping a medical image of a patient to a sensor coordinate system includes aligning the medical image of the patient with at least one frame of a live data stream.
[0149] In one example, aligning a medical image of a patient with at least one frame of a live data stream includes utilizing a warp transformation to correlate the medical image of the patient's body with the patient's data within at least one frame of the live data stream.
[0150] Therefore, this new technology provides automatic real-time tracking of tube-shaped foreign objects, which can reduce the number of incidents in the ED and ICU departments related to the accidental insertion of catheters and tubes, thereby improving the overall patient well-being and reducing the workload of radiologists. Furthermore, immediate feedback is provided to clinicians in the ED and ICU at the time of collection.
[0151] This is achieved by virtualizing the guidance process by inserting the real-time projection of the device into a previously collected CXR, MR, or other medical image that was taken immediately before or a significant time before the catheter or tube insertion process, and highlighting the device-specific anatomical regions to navigate to the correct insertion point.
[0152] Note that in the following detailed system description, the chest X-ray (CXR) modality is referred to, but the system is not limited to this modality and can be applied to multiple modalities such as CT, leg X-ray, and X-ray modalities of other body parts, as well as other medical imaging modalities such as magnetic resonance (MR) imaging and positron emission tomography (PET) imaging. Also, in the detailed embodiments described below, a sensor system in the form of one or more cameras is referred to, but this may be a radar sensor-based system, a lidar sensor-based system, a single 3D RGB-D-based system, or any other sensor-based system from which patient data is collected to identify the patient's position within a coordinate system defined with reference to these one or more sensors.
[0153] Therefore, referring to FIG. 4, the positions of the central venous catheter and the endotracheal tube are usually evaluated with respect to the sternum (the point where the lower ends of the right and left main bronchi meet), which is an anatomical point. Also, the superior vena cava (SVC) and the right atrium help the radiologist navigate to the expected regions for correct object placement. In FIG. 4, the "right clavicle" is represented as "A", the "right subclavian vein" as "B", the "pericardium" as "C", the "right internal jugular vein" as "D", the "left internal jugular vein" as "E", the "left clavicle" as "F", the "left subclavian vein" as "G", the "sternum" as "H", the "right brachiocephalic vein" as "I", the "left brachiocephalic vein" as "J", and the "catheter tip" as "K".
[0154] Present the existing workflow and contrast it with the new technology to help in understanding the new technology.
[0155] In the workflow performed at the clinical site for the evaluation of medical images (any modality), often, to rule out foreign objects with abnormal positions, it includes the step of inserting a specific tubular device into the patient, and the step of collecting a target verification examination (e.g., CXR, although it could also be MR).
[0156] The examination is sent to the PACS system and put in the queue at the end of the worklist. The radiologist examines the examination after time X. a. If an abnormal position of a foreign object is detected, report it directly to the clinician. b. Otherwise, a normal report is made.
[0157] The clinician receives the report after time Y and performs the necessary next steps.
[0158] Both times X and Y can be long, and if the foreign object is in the wrong position with respect to the correct region where it is expected, it can cause serious problems.
[0159] The new technology described in this specification eliminates steps 2 - 5 to enable clinicians to obtain direct feedback during the real - time process of device insertion. The new system can operate directly on an image acquisition device, but is not limited to this. For example, it can also be implemented on the cloud.
[0160] The main components of the new system are as follows: 1) A camera (such as a radar, lidar, etc.) for real - time video streaming 2) A target tube / catheter with a detectable tip (active / passive) 3) An antenna for a triangulation device and / or a wireless positioning device, and / or an optical fiber integrated into the catheter 4) A module for real - time alignment of the patient's body with CXR (MR, etc.) 5) An anatomical region segmentation / landmark detection module 6) A reporting and visualization module
[0161] In the new technology, the position of the main elements of the device (catheter / tube) can be identified and mapped to previously acquired X - ray images. Here, only X - rays are described, but as explained above, this can also be other medical imaging modalities such as MR. The position identification can be performed by wireless positioning, triangulation, optical fiber shape sensing, or other techniques known in the art. Thus, the position of the tip of the catheter / tube is identified in real - time within the patient coordinate system in conjunction with the patient's video stream. At the same time, an image alignment algorithm is used to map the video stream to the X - ray image. As a result, the coordinates of the tip can be displayed on the X - ray image.
[0162] The implementation of an automatic real-time system for tube and catheter tracking begins with pre-training of images for an X-ray alignment model and an anatomical region segmentation / landmark detection model. Both are encoder-decoder type deep convolutional neural networks. Both are pre-trained.
[0163] In a second step, a reference image is made available to guide the procedure. This is a pre-treatment X-ray image (e.g., a previous image of the patient), a template / atlas image, or another medical image such as an MR.
[0164] The third step is to calibrate the patient-specific coordinate system using the tip of the device and other major landmarks. The patient-specific coordinate system is connected to the video stream.
[0165] The fourth step is to infer anatomical beacons from the anatomical region segmentation / landmark detection model. These regions are displayed on a preliminary CXR.
[0166] The next step is to insert a tube-shaped device into the patient's body (see Figure 5). During insertion of the device, the coordinates of the tip are calculated in the patient-specific coordinate system and attached to the video frame. At the same time, the video frame is aligned with a pre-taken CXR image, and the patient-specific coordinate system is mapped to the CXR coordinate system. As a result, the coordinates of the tip are converted to the CXR coordinate system. The coordinates of the tip are overlaid on the CXR image using anatomical beacons and displayed on the monitor. A schematic of this process is shown in Figure 5.
[0167] Therefore, this new technology can be described as follows. A catheter / tube is about to be inserted into the patient's body, The patient's X-ray image is made available to the "system", The system has one or more cameras that observe the patient and effectively place the patient in a "camera coordinate system". The X-ray image is mapped to the "camera coordinate system". The X-ray image is mapped to the "camera coordinate system" by image registration.
[0168] This is done by matching two images, namely the X-ray photograph and a frame from the video, to the patient's body. In practice, a warp transformation that correlates the patient's body with the X-ray image is found. This can be done by means of a neural network or a classical computer vision algorithm.
[0169] The end of the catheter / tube is monitored. This is done using one or more cameras. This is done via another system such as a radar or fiber optic based system. The position of the catheter / tube is determined within the "camera coordinate system". Therefore, the position of the catheter / tube is known with respect to the X-ray image. The position of the catheter / tube is displayed with respect to the patient's X-ray image.
[0170] Therefore, medical personnel can in fact view the end of the catheter / tube with respect to the X-ray image of the patient representing the patient in front of them, and can thus correctly insert the catheter / tube as they approach the patient with respect to the patient's venous structure, and once the catheter / tube is inserted, can view the tip of the catheter / tube with respect to the patient's venous structure.
[0171] And medical personnel can correctly insert the catheter / tube into the patient.
[0172] Although the coordinate system has been referred to above, in reality there are the following three coordinate systems. 1) A patient-based coordinate system that represents the position of the device relative to some anchor, such as an antenna for triangulation. This coordinate system is not related to the patient. 2) A video stream coordinate system that describes the position of the patient over time. This can be connected (calibrated) to a patient-specific coordinate system, for example, using the known position of the antenna. 3) An X-ray coordinate system. This can be connected (calibrated) to the video stream coordinate system by image registration.
[0173] Therefore, the position of the device (catheter / tube) can be confirmed on the X-ray image. Therefore, in practice, the video stream serves as a proxy between the device and the patient.
[0174] In another exemplary embodiment, there is provided a computer program or computer program element characterized in that it executes the method steps of the method according to one of the above embodiments on a suitable device or system.
[0175] Therefore, the computer program element can be stored in a computer unit that can be part of an embodiment. This computing unit can execute or induce the execution of the steps of the above method. Furthermore, it can operate the components of the above system. The computing unit operates automatically and / or executes user commands. The computer program can be loaded into the working memory of a data processor. Therefore, the data processor may be configured to execute the method according to one of the above embodiments.
[0176] This exemplary embodiment of the invention is directed to both a computer program that uses the invention from the start and a computer program that converts an existing program into a program that uses the invention by an update.
[0177] Furthermore, the computer program element can provide all the steps necessary to perform the procedure of the exemplary embodiment of the above method.
[0178] According to a further exemplary embodiment of the present invention a computer readable medium such as a CD-ROM, a USB stick or the like is presented, on which computer program elements are stored, as described in the previous section.
[0179] The computer program may be stored and / or distributed on a suitable medium, such as an optical storage medium or a solid-state medium, supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless communication systems.
[0180] However, the computer program may also be presented via a network such as the World Wide Web and downloaded from such a network into the working memory of a data processor. According to a further exemplary embodiment of the invention, a medium making available a computer program element for downloading is provided, the computer program element being configured to perform a method according to one of the aforementioned embodiments of the invention.
[0181] It should be noted that the embodiments of the present invention are described with reference to different subject matters. In particular, some embodiments are described with reference to method type claims, while other embodiments are described with reference to device type claims. However, a person skilled in the art can infer from the above and following description that, unless otherwise specified, any combination of features belonging to one type of subject matter, as well as any combination of features related to different subject matters, are considered to be disclosed in the present application. However, all features can be combined if they provide a synergistic effect that is more than a mere collection of features.
[0182] Although the present invention has been illustrated and described in detail in the drawings and the above description, such illustration and description should be regarded as exemplary or exemplary and should not be regarded as limiting. The present invention is not limited to the disclosed embodiments. Other variations of the disclosed embodiments can be understood and implemented by those skilled in the art when implementing the invention according to the claims from the consideration of the drawings, the disclosure, and the dependent claims.
[0183] In the claims, the term "comprising" does not exclude other elements or steps, and a singular element does not exclude a plurality. A single processor or other unit can perform the functions of several items described in the claims. The mere fact that certain means are described in mutually different dependent claims does not mean that these means cannot be used advantageously in combination. Any reference signs in the claims should not be construed as limiting the scope.
Claims
1. A device for tracking a catheter or tube, including an input unit and a processing unit, The input unit receives a medical image of the patient. The input unit provides the medical image of the patient to the processing unit. The input unit receives a live data stream from one or more sensors, the live data stream includes the patient's data, and a sensor coordinate system is defined with respect to the one or more sensors. The input unit provides the live data stream to the processing unit. The processing unit receives the sensing position of the end of the catheter or tube to be inserted into the patient, The processing unit coordinates the patient data received from one or more sensors with the sensor coordinate system. The processing unit maps the medical image of the patient to the sensor coordinate system. The processing unit generates a modified sensor coordinate system using the medical image of the patient mapped to the sensor coordinate system. The processing unit determines the position of the end of the catheter or tube in the sensor coordinate system by utilizing the sensing position of the end of the catheter or tube. Determining the position of the end of the catheter or tube in the sensor coordinate system includes determining the position of the end of the catheter or tube in the modified sensor coordinate system. The processing unit generates a representation of the sensing position of the end of the catheter or tube relative to the medical image, using the medical image of the patient mapped to the sensor coordinate system and the position of the end of the catheter or tube in the sensor coordinate system. Generating the representation of the sensing position of the end of the catheter or tube in relation to the medical image includes generating the representation of the sensing position of the end of the catheter or tube in the medical image of the patient. Device.
2. The apparatus according to claim 1, wherein the processing unit generates a mapped medical image of the patient using the medical image of the patient mapped to the sensor coordinate system, and generates a representation of the sensing position of the end of the catheter or tube in the medical image, which includes generating a representation of the sensing position of the end of the catheter or tube in the mapped medical image of the patient.
3. The apparatus according to claim 1, wherein the one or more sensors include a plurality of sensors, and the live data stream includes data from each of the plurality of sensors.
4. The apparatus according to claim 3, wherein linking the patient's data with the sensor coordinate system includes utilizing the patient's data from each of the plurality of sensors.
5. The apparatus according to claim 3, wherein the live data stream includes data of the end of the catheter or tube, and the sensing position of the end of the catheter or tube inserted into the patient is determined from the data of the end of the catheter or tube from each of the plurality of sensors.
6. The apparatus according to claim 1, wherein a wireless positioning system senses the position of the end of the catheter or tube, and the wireless positioning system provides the sensed position of the end of the catheter or tube to the processing unit.
7. The apparatus according to claim 1, wherein the catheter or tube includes an optical fiber shape sensing system for sensing the position of the end of the catheter or tube, and the optical fiber shape sensing system provides the sensing position of the end of the catheter or tube to the processing unit.
8. The apparatus according to claim 1, wherein mapping the medical image of the patient to the sensor coordinate system includes aligning the medical image of the patient with at least one frame of the live data stream.
9. The apparatus according to claim 8, wherein aligning the medical image of the patient with at least one frame of the live data stream includes utilizing a warp transform to coordinate the medical image of the patient's body with the data of the patient in at least one frame of the live data stream.
10. A system for tracking a catheter or tube, comprising an input unit, a processing unit, one or more sensors, and a visual display unit, The input unit receives a medical image of the patient. The input unit provides the medical image of the patient to the processing unit. The input unit receives a live data stream from one or more sensors, the live data stream includes patient data, and a sensor coordinate system is defined with respect to one or more sensors. The input unit provides the live data stream to the processing unit. The processing unit receives the sensing position of the end of the catheter or tube to be inserted into the patient, The processing unit coordinates the patient data received from one or more sensors with the sensor coordinate system. The processing unit maps the medical image of the patient to the sensor coordinate system. The processing unit generates a modified sensor coordinate system using the medical image of the patient mapped to the sensor coordinate system. The processing unit determines the position of the end of the catheter or tube in the sensor coordinate system by utilizing the sensing position of the end of the catheter or tube. Determining the position of the end of the catheter or tube in the sensor coordinate system includes determining the position of the end of the catheter or tube in the modified sensor coordinate system. The processing unit uses the medical image of the patient mapped to the sensor coordinate system and the position of the end of the catheter or tube in the sensor coordinate system to generate a representation of the sensing position of the end of the catheter or tube relative to the medical image on the visual display unit. Generating the representation of the sensing position of the end of the catheter or tube in relation to the medical image includes generating the representation of the sensing position of the end of the catheter or tube in the medical image of the patient. system.
11. The system according to claim 10, wherein the processing unit generates a mapped medical image of the patient using the medical image of the patient mapped to the sensor coordinate system, the processing unit displays the mapped medical image on the visual display unit, and the generation of a representation of the sensing position of the end of the catheter or tube relative to the medical image includes generating a representation of the sensing position of the end of the catheter or tube in the mapped medical image of the patient displayed on the visual display unit.
12. The system according to claim 10, wherein the processing unit generates a modified sensor coordinate system using the medical image of the patient mapped to the sensor coordinate system, and determining the position of the end of the catheter or tube in the sensor coordinate system includes determining the position of the end of the catheter or tube in the modified sensor coordinate system, and the processing unit displays the medical image on the visual display unit, and generating a representation of the sensing position of the end of the catheter or tube relative to the medical image includes generating a representation of the sensing position of the end of the catheter or tube in the medical image of the patient displayed on the visual display unit.
13. A step of receiving a medical image of a patient, The steps include providing the medical image of the patient to a processing unit, A step of receiving a live data stream from one or more sensors, wherein the live data stream includes patient data and a sensor coordinate system is defined with respect to the one or more sensors. The steps include providing the live data stream to the processing unit, The processing unit performs the step of receiving the sensing position of the end of the catheter or tube to be inserted into the patient, The processing unit performs the steps of linking the patient data received from one or more sensors with the sensor coordinate system, The processing unit performs the steps of mapping the patient's medical image to the sensor coordinate system, The processing unit then performs the steps of generating a modified sensor coordinate system using the medical image of the patient mapped to the sensor coordinate system, A step of determining the position of the end of the catheter or tube in the sensor coordinate system using the sensing position of the end of the catheter or tube by the processing unit, wherein determining the position of the end of the catheter or tube in the sensor coordinate system includes determining the position of the end of the catheter or tube in the modified sensor coordinate system. A step of generating a representation of the sensing position of the end of the catheter or tube relative to the medical image using the medical image of the patient mapped to the sensor coordinate system and the position of the end of the catheter or tube in the sensor coordinate system, wherein generating the representation of the sensing position of the end of the catheter or tube relative to the medical image includes generating the representation of the sensing position of the end of the catheter or tube in the medical image of the patient, A method for tracking a catheter or tube, including the following.
14. A computer program for controlling the apparatus according to any one of claims 1 to 9, which, when executed by a processor, is configured to perform the method according to claim 13, and / or for controlling the system according to any one of claims 10 to 12, which, when executed by a processor, is configured to perform the method according to claim 13.