Device for tracking a catheter
By utilizing sensor systems and medical image registration technology, the frequent X-ray examination during catheter and tube insertion is solved, and the accurate insertion of catheter or tube and the reduction of workload of radiologists is achieved.
Patent Information
- Application Number
- CN202380052331.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-07-07
- Filing Date
- 2023-06-26
- Publication Date
- 2025-02-28
AI Technical Summary
In the prior art, multiple X-ray examinations are often required during the insertion of catheters and tubes, resulting in an extended report turnover time and an increased workload of radiologists.
Through a device and system, a sensor system is used to receive a real-time data stream of a patient and register previously acquired medical images with the patient, generating a representation of the end sensing position of the catheter or tube relative to the medical image, helping the physician to accurately insert the catheter or tube in real time.
Reduces dependence on X-ray examinations, improves the accuracy of insertion of catheters or tubes, reduces the workload of radiologists, and improves the efficiency of patients' health management.
Smart Images
Figure CN119546245A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a device for tracking a catheter, a system for tracking a catheter, a method for tracking a catheter and a computer program element and a computer readable medium. Background Art
[0002] Chest X-ray (CXR) is one of the most common examinations in radiology departments due to its relatively low cost and small acquisition time. As a result, radiology departments may be flooded with CXRs, which can significantly increase report turnaround time.
[0003] CXR is often used as the primary modality to assess the position of foreign bodies, such as central venous catheters (CVCs) (see, e.g., Pikwer A, L, Davidson B, Perstoft I, Akeson (2010). The incidence and risk of central venous catheter malpositioning: A prospective cohort study in 1619 patients. (Anaesth Intensive Care. 2008; 36: 30-7) and Yi, X., Adams, SJ, Henderson, RD, & Babyn, P. (2020). Computer-aided Assessment of Catheters and Tubes on Radiographs: How Good Is Artificial Intelligence for Assessment? (Radiology: Artificial Intelligence, 2(1), e190082)), endotracheal tubes (see, e.g., Chen, S., Zhang, M., Yao, L., & Xu, W. (2016). Endotracheal tubes positioning detection in adult portable chest radiography for intensive care unit.(Internationaljournal of computer assisted radiology and surgery, 2049-2057)), and feeding tubes (see, e.g., Singh, V., Danda, V., Gorniak, R., Flanders, A., & Lakhani, P. (2019).(Assessment ofcritical feeding tube malpositions on radiographs using deep learning.Journal of digital imaging, 32(4), 651-655)), etc. Accurate and correct time detection of misplaced foreign bodies may be very important for certain external objects, such as CVC insertion, which is notoriously difficult and may cause pneumothorax (PTX). In addition to the initial positioning of the device, daily verification using X-ray imaging is usually performed.The incidence of CVC misplacement is 3.6-14% (see: Pikwer A, L, Davidson B, Perstoft I, Akeson (2010). The incidence and risk of central venous catheter malpositioning: A prospective cohort study in 1619 patients. (Anaesth Intensive Care. 2008; 36: 30-7)). Philips' clinical partner (University Hospital Hamburg-Eppendorf) reported that 20% of all current CXRs have this clinical problem. Conventionally, catheters and tubes are inserted in a blind manner or with the help of ultrasound guidance. Then, the device is usually placed in this way after the insertion process on one or CXR, which, as mentioned above, may lead to the need to perform too many CXRs and increase the report turnaround time.
[0004] Therefore, there is a clinical need for automatic solutions that can reduce the workload of radiologists in hospitals. Summary of the invention
[0005] It would be advantageous to provide an improved technique to reduce the need for x-ray examinations associated with inserting catheters and tubes into patients.
[0006] The objects of the invention are solved by the subject-matter of the independent claims, wherein further embodiments are incorporated in the dependent claims. It should be noted that the following described aspects and examples of the invention apply to an apparatus for tracking a catheter, a system for tracking a catheter, a method for tracking a catheter, as well as to a computer program element and a computer readable medium.
[0007] In a first aspect, there is provided an apparatus for tracking a catheter or tube, comprising:
[0008] input unit; and
[0009] Processing unit.
[0010] The input unit is configured to receive a medical image of a patient. The input unit is configured to provide the medical image of the patient to the processing unit. The input unit is configured to receive a live data stream from one or more sensors. The live data stream includes data of the patient, and the sensor coordinate system is defined relative to the one or more sensors. The input unit is configured to provide the live data stream to the processing unit. The processing unit is configured to receive a sensed position of an end of a catheter or tube to be inserted into the patient. The processing unit is configured to align the data of the patient received from the one or more sensors with the sensor coordinate system. The processing unit is configured to map the medical image of the patient to the sensor coordinate system. The processing unit is configured to determine the position of the end of the catheter or tube in the sensor coordinate system, including using the sensed position of the end of the catheter or tube. The processing unit is configured to generate a representation of the sensed position of the end of the catheter or tube relative to the medical image, including using the medical image of the patient mapped to the sensor coordinate system and the position of the end of the catheter or tube in the sensor coordinate system.
[0011] In other words, a previous medical image of the patient that could have been taken a year ago, a month ago, a week ago, a day ago, an hour ago, or a few minutes ago is provided. The medical image will show details such as the patient's anatomical structure, such as, for example, the patient's venous structure. Patients now need to insert a catheter or tube or tubing into their body, such as a central venous catheter (CVC) to be inserted into their body. However, the insertion of a CVC is very difficult, and the CVC is often misplaced.
[0012] Therefore, a new technology has been developed to register or effectively register a medical image of a patient to the patient, thereby registering or aligning an X-ray image or MR image (or other medical image) that has been previously acquired to the patient so that when the practitioner is inserting a catheter or tube into the patient, they are provided via the medical image with the patient's anatomy, which has actually been mapped to the patient in front of them, so that they can correctly insert the catheter into the correct anatomy, such as a vein, which they can see via the medical image that has been registered to the patient.
[0013] The new technology uses data showing the patient from a sensor system (such as a camera system, a lidar system, a radar system). The sensor system can generate a sensor coordinate system using anchors, antennas or other markers in the room where the patient is located. The sensor system also provides the patient's data (such as image data and / or depth data and / or radar data in a real-time data stream), and places the patient's data in the real-time data stream into the sensor coordinate system. The medical image of the patient previously collected is then mapped to the sensor coordinate system via image registration. The position of the end of the catheter or tube to be inserted into the patient is then sensed, and it is also placed in the sensor coordinate system. Since both the position of the medical image and the end of the catheter or tube are aligned with the sensor coordinate system, the position of the end of the catheter or tube can be represented relative to the medical image. Therefore, a medical image or a distorted medical image can be presented on a visual display unit together with the position of the end of the catheter or tube. Then, as the medical expert moves the catheter or tube toward the patient, the end of the tube or catheter can be represented as moving relative to the medical image on the visual display unit. Then, the medical expert can, for example, see the relevant anatomical structure (such as venous structure) of the patient, and can correctly insert the catheter or tube. When mapping medical images to a live data stream as the patient moves, the medical images of the patient can be represented to account for that movement so that the medical professional has a continuously correct representation of, for example, the patient's venous structure to view and can ensure correct insertion of a catheter or tube.
[0014] In an example, the processing unit is configured to generate a mapped medical image of the patient, including using the medical image of the patient mapped to a sensor coordinate system. Generating a representation of the sensed position of the end of the catheter or tube relative to the medical image includes generating a representation of the sensed position of the end of the catheter or tube in the mapped medical image of the patient.
[0015] Thus, a medical image (such as an X-ray image or an MR image) is warped to match data of the patient acquired by one or more sensors, such as image data (which defines a sensor coordinate system). Thus, if a medical image was acquired 2 months ago when the patient was heavier than they are now, the medical image is warped to match how the patient looks now as sensed by the one or more sensors.
[0016] In an example, the processing unit is configured to generate a modified sensor coordinate system, including using a medical image of the patient mapped to the sensor coordinate system. Then, determining the position of the end of the catheter or tube in the sensor coordinate system includes determining the position of the end of the catheter or tube in the modified sensor coordinate system. Then, generating a representation of the sensed position of the end of the catheter or tube relative to the medical image includes generating a representation of the sensed position of the end of the catheter or tube in the medical image of the patient.
[0017] In other words, the medical image of the patient forms the ground truth data and is not distorted, but when aligning the medical image with the sensor coordinate system, the sensor coordinate system itself is distorted so that in effect the live data feed of the patient becomes distorted. Thus, the medical image is always realistic and undistorted, but the position of the end of the catheter or tube is now presented with a distorted sensor coordinate system and may then move slightly differently to the movements made by the medical specialist, since for example the patient may move, but the medical specialist is always presented with for example an undistorted version or a medical image of the patient with an undistorted venous structure.
[0018] In other words, the patient's data (such as image data) acquired by one or more sensors (which define a sensor coordinate system) is distorted to match a medical image, such as an X-ray image or MR image that may have been acquired a few months ago when the patient was more ill than he is now. Therefore, the medical image is actually the real situation, and the space in which the patient is located is actually distorted. Therefore, as mentioned before, if the patient was more ill when the medical image was acquired, the medical professional will see this image on their screen. This image is larger than the image of the patient at this moment, however the representation of space is distorted so that when the catheter moves towards the patient, it reaches the patient and the X-ray image at the same time and at the same point, and reaches the patient and the X-ray image at the correct point as indicated by the medical image.
[0019] It may be useful to switch to an undistorted medical image presented in a distorted sensor coordinate system to verify the patient's anatomy, such as venous structure, before switching back to a distorted medical image for actual insertion of a catheter or tube into the patient, or vice versa.
[0020] In an example, the one or more sensors include a plurality of sensors, and wherein the live data stream includes data from each of the plurality of sensors.
[0021] In an example, aligning the patient's data with the sensor coordinate system includes utilizing the patient's data from each of the plurality of sensors.
[0022] Thus, for example, image data from a 3D camera or radar or lidar sensor can by itself determine the position of the patient relative to the position of such a sensor and place the patient in the sensor coordinate system. However, multiple sensors can be utilized to generate this information. Thus, for example, multiple cameras can actually provide 3D data of the patient, or multiple radar systems can provide distance and direction information to provide detailed position information of the patient.
[0023] Thus, triangulation, markers, antennas can be used to locate the patient's data within the sensor coordinate system, ie in reality 3D data.
[0024] In an example, the live data stream includes data of an end of a catheter or tube, and wherein a sensed position of the end of the catheter or tube to be inserted into a patient is determined based on the data of the end of the catheter or tube from each of a plurality of sensors.
[0025] Thus, for example, triangulation may be used to accurately sense the position of the end of a catheter or tube within the sensor coordinate system.
[0026] In an example, the radio positioning system is configured to sense a position of an end of a catheter or tube, and wherein the radio positioning system is configured to provide the sensed position of the end of the catheter or tube to the processing unit.
[0027] In other words, a radar system is used to sense the end of a catheter or tube. The radar system is at a known position and orientation within the camera coordinate system, and then the sensed position of the end of the catheter or tube relative to the radar system is known relative to the camera coordinate system.
[0028] In an example, the catheter or tube comprises a fiber optic shape sensing system configured to sense the position of an end of the catheter or tube. The fiber optic shape sensing system is configured to provide the sensed position of the end of the catheter or tube to a processing unit.
[0029] In an example, mapping the medical image of the patient to the sensor coordinate system includes registration between the medical image of the patient and at least one frame of the live data stream.
[0030] Thus, for example, frames of a data stream may be matched to a medical image, which may involve applying a warp transformation that aligns data of a patient's body in the data frames with the medical image of the patient, and wherein this may be achieved using classical computer vision algorithms or by means of neural networks.
[0031] Thus, a camera image can be mapped to a medical image (e.g., an X-ray image or an MR image), or a radar image can be mapped to a medical image (e.g., an X-ray image or an MR image), or depth data from a radar / RGB-D camera / lidar can be mapped to a medical image (e.g., an X-ray image or an MR image).
[0032] In an example, the registration between the medical image of the patient and the at least one frame of the live data stream includes utilizing a warp transformation to align the medical image of the patient's body with data of the patient in the at least one frame of the live data stream.
[0033] In a second aspect, there is provided a system for tracking a catheter or tube, comprising:
[0034] Input unit;
[0035] Processing unit;
[0036] one or more sensors; and
[0037] Visual display unit.
[0038] The input unit is configured to receive a medical image of a patient. The input unit is configured to provide the medical image of the patient to the processing unit. The input unit is configured to receive a live data stream from one or more sensors. The live data stream includes data of the patient, and the sensor coordinate system is defined relative to the one or more sensors. The input unit is configured to provide the live data stream to the processing unit. The processing unit is configured to receive a sensed position of an end of a catheter or tube to be inserted into the patient. The processing unit is configured to align the data of the patient received from the one or more sensors with the sensor coordinate system. The processing unit is configured to map the medical image of the patient to the sensor coordinate system. The processing unit is configured to determine the position of the end of the catheter or tube in the sensor coordinate system, including using the sensed position of the end of the catheter or tube. The processing unit is configured to generate a representation of the sensed position of the end of the catheter or tube relative to the medical image on the visual display unit, including using the medical image of the patient mapped to the sensor coordinate system and the position of the end of the catheter or tube in the sensor coordinate system.
[0039] In an example, the processing unit is configured to generate a mapped medical image of the patient, including using the medical image of the patient mapped to the sensor coordinate system. The processing unit is configured to present the mapped medical image on a visual display unit. Then, generating a representation of the sensed position of the end of the catheter or tube relative to the medical image includes generating a representation of the sensed position of the end of the catheter or tube in the mapped medical image of the patient presented on the visual display unit.
[0040] In an example, the processing unit is configured to generate a modified sensor coordinate system, including using a medical image of a patient mapped to the sensor coordinate system. Then, determining the position of the end of the catheter or tube in the sensor coordinate system includes determining the position of the end of the catheter or tube in the modified sensor coordinate system. The processing unit is configured to present the medical image on a visual display unit. Then, generating a representation of the sensed position of the end of the catheter or tube relative to the medical image includes generating a representation of the sensed position of the end of the catheter or tube in the medical image of the patient presented on the visual display unit.
[0041] In a third aspect, there is provided a method for tracking a catheter or tube, comprising:
[0042] receiving medical images of patients;
[0043] providing a medical image of the patient to a processing unit;
[0044] receiving a live data stream from one or more sensors, wherein the live data stream includes data of the patient, and wherein a sensor coordinate system is defined relative to the one or more sensors;
[0045] providing the live data stream to the processing unit;
[0046] receiving, by the processing unit, a sensed position of an end of a catheter or tube to be inserted into the patient;
[0047] aligning, by the processing unit, the data of the patient received from the one or more sensors with the sensor coordinate system;
[0048] mapping, by the processing unit, the medical image of the patient to the sensor coordinate system;
[0049] determining, by the processing unit, a position of the end of the catheter or tube in the sensor coordinate system, including using a sensed position of the end of the catheter or tube; and
[0050] A representation of the sensed position of the end of the catheter or tube relative to the medical image is generated by the processing unit, including using the medical image of the patient mapped to the sensor coordinate system and the position of the end of the catheter or tube in the sensor coordinate system.
[0051] According to various aspects, a computer program element is provided for controlling one or more apparatuses and / or systems as described above, which computer program element, if executed by a processor, is adapted to perform a method as described above.
[0052] According to other aspects, a computer readable medium storing the computer unit as described above is provided.
[0053] The computer program element may for example be a software program, but may also be an FPGA, a PLD or any other suitable digital module.
[0054] Advantageously, the benefits provided by any of the above aspects apply equally to all other aspects, and vice versa.
[0055] The above-described aspects and examples will be apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS
[0056] Exemplary embodiments will be described below with reference to the following drawings:
[0057] Figure 1 A schematic representation of an apparatus for tracking a catheter is shown;
[0058] Figure 2 A schematic representation of a system for tracking a catheter is shown;
[0059] Figure 3 A method for tracking a catheter is shown;
[0060] Figure 4 An anatomical overview and important areas for a CVC are shown on the left, where each of the four veins can be used to insert a CVC, and Figure 4 A demonstration of a properly positioned CVC after insertion is shown on the right; and
[0061] Figure 5 A schematic representation of a detailed embodiment of a system for tracking a catheter is shown. DETAILED DESCRIPTION
[0062] Figure 1 An example of an apparatus 10 for tracking a catheter or tube is shown. The apparatus includes an input unit 20 and a processing unit 30. The input unit is configured to receive a medical image of a patient. The input unit is configured to provide the medical image of the patient to the processing unit. The input unit is configured to receive a live data stream from one or more sensors. The live data stream includes data of the patient, and a sensor coordinate system is defined relative to the one or more sensors. The input unit is configured to provide the live data stream to the processing unit. The processing unit is configured to receive a sensed position of an end of a catheter or tube to be inserted into a patient. The processing unit is configured to align the data of the patient received from the one or more sensors with the sensor coordinate system. The processing unit is configured to map the medical image of the patient to the sensor coordinate system. The processing unit is configured to determine the position of the end of the catheter or tube in the sensor coordinate system, including using the sensed position of the end of the catheter or tube. The processing unit is configured to generate a representation of the sensed position of the end of the catheter or tube relative to the medical image, including using the medical image of the patient mapped to the sensor coordinate system and the position of the end of the catheter or tube in the sensor coordinate system.
[0063] In an example, the medical image is an attenuation X-ray image.
[0064] In an example, the medical image is a CTX radiographic image.
[0065] In an example, the medical image is a magnetic resonance image.
[0066] In an example, the one or more sensors are one or more cameras and the live data stream is a live video image stream.
[0067] In an example, the one or more cameras include one or more 3D cameras, and the live data stream is a live video image stream or a live depth data stream.
[0068] In an example, the one or more cameras include one or more RGB-D cameras, and the live data stream is a live video image stream and / or a live depth data stream.
[0069] In an example, the one or more sensors are one or more lidar sensors and the live data stream is a live lidar image stream and / or a live depth data stream.
[0070] In an example, the one or more sensors are one or more radars, and the live data streams are live radar data streams and / or live image streams and / or live depth data streams.
[0071] In an example, the data of the patient received from the one or more sensors includes image data of the patient received from the one or more cameras.
[0072] In an example, the data of the patient received from the one or more sensors includes lidar image data of the patient received from one or more lidar sensors.
[0073] In an example, the data of the patient received from the one or more sensors includes lidar depth data of the patient received from one or more lidar sensors.
[0074] In an example, the data of the patient received from the one or more sensors includes depth data of the patient received from the one or more cameras.
[0075] In an example, the data of the patient received from the one or more sensors includes depth data of the patient received from the one or more radars.
[0076] In an example, the data of the patient received from the one or more sensors includes image data of the patient received from the one or more radars.
[0077] In an example, aligning the patient's data with the sensor coordinate system includes utilizing the patient's data from one or more sensors.
[0078] Thus, knowledge of the position(s) of one or more sensors can be used along with the patient's sensor data to determine the position of a part of the patient relative to the position(s) of the sensor, thereby placing the patient in the sensor coordinate system.
[0079] In the example, the sensor coordinate system is the camera coordinate system.
[0080] In the example, the sensor coordinate system is a lidar sensor coordinate system.
[0081] In an example, the sensor coordinate system is a radar sensor coordinate system.
[0082] According to an example, the processing unit is configured to generate a mapped medical image of the patient, including using the medical image of the patient mapped to the sensor coordinate system. Then, generating a representation of the sensed position of the end of the catheter or tube relative to the medical image may include generating a representation of the sensed position of the end of the catheter or tube in the mapped medical image of the patient.
[0083] In an example, the mapped medical image of the patient is a warped version of the medical image that has been transformed to align with the patient's data from the live data stream.
[0084] In other words, the patient's live data in the sensor coordinate system forms the ground truth data, and the medical image is warped to match this data when presented to the medical expert together with the position of the end of the catheter or tube in the camera coordinate system.
[0085] According to an example, the processing unit is configured to generate a modified sensor coordinate system, including using a medical image of the patient mapped to the sensor coordinate system. Then, determining the position of the end of the catheter or tube in the sensor coordinate system may include determining the position of the end of the catheter or tube in the modified sensor coordinate system. Then, generating a representation of the sensed position of the end of the catheter or tube relative to the medical image may include generating a representation of the sensed position of the end of the catheter or tube in the medical image of the patient.
[0086] According to an example, the one or more sensors include a plurality of sensors, and wherein the live data stream includes data from each of the plurality of sensors.
[0087] According to an example, the alignment of the patient's data with the sensor coordinate system includes utilizing the patient's data from each of the plurality of sensors.
[0088] According to an example, the live data stream includes data of an end of a catheter or tube, and wherein a sensed position of the end of the catheter or tube to be inserted into a patient is determined based on the data of the end of the catheter or tube from each of a plurality of sensors.
[0089] In an example, the data of the end of the catheter or tube includes image data acquired by a plurality of cameras.
[0090] In an example, the data of the end of the catheter or tube includes depth data acquired by multiple cameras.
[0091] In an example, the data of the end of the catheter or tube includes image data collected by multiple lidar sensors.
[0092] In an example, the data of the end of the catheter or tube includes depth data collected by multiple lidar sensors.
[0093] In an example, the data of the end of the catheter or tube includes image data acquired by multiple radars.
[0094] In an example, the data of the end of the conduit or tube includes depth data acquired by multiple radars.
[0095] According to an example, the radio positioning system is configured to sense a position of an end of a catheter or tube, and wherein the radio positioning system is configured to provide the sensed position of the end of the catheter or tube to the processing unit.
[0096] According to an example, the catheter or tube comprises a fiber optic shape sensing system configured to sense the position of an end of the catheter or tube. The fiber optic shape sensing system is configured to provide the sensed position of the end of the catheter or tube to a processing unit.
[0097] According to an example, mapping the medical image of the patient to the sensor coordinate system comprises registration between the medical image of the patient and at least one frame of the live data stream.
[0098] According to an example, the registration between the medical image of the patient and the at least one frame of the live data stream includes utilizing a warp transformation to align the medical image of the patient's body with data of the patient in the at least one frame of the live data stream.
[0099] Figure 2 An example of a system 100 for tracking a catheter or tube is shown. The system 100 includes an input unit 120, a processing unit 130, one or more sensors 140, and a visual display unit 150. The input unit is configured to receive a medical image of a patient. The input unit is configured to provide the medical image of the patient to the processing unit. The input unit is configured to receive a live data stream from one or more sensors. The live data stream includes data of the patient, and a sensor coordinate system is defined relative to the one or more sensors. The input unit is configured to provide the live data stream to the processing unit. The processing unit is configured to receive a sensed position of an end of a catheter or tube to be inserted into a patient. The processing unit is configured to align the data of the patient received from the one or more sensors with the sensor coordinate system. The processing unit is configured to map the medical image of the patient to the sensor coordinate system. The processing unit is configured to determine the position of the end of the catheter or tube in the sensor coordinate system, including using the sensed position of the end of the catheter or tube. The processing unit is configured to generate a representation of the sensed position of the end of the catheter or tube relative to the medical image on the visual display unit, including utilizing the medical image of the patient mapped to the sensor coordinate system and the position of the end of the catheter or tube in the sensor coordinate system.
[0100] In an example, the medical image is an attenuation X-ray image.
[0101] In an example, the medical image is a CT X-ray image.
[0102] In an example, the medical image is a magnetic resonance image.
[0103] In an example, the one or more sensors are one or more cameras and the live data stream is a live video image stream.
[0104] In an example, the one or more cameras include one or more 3D cameras, and the live data stream is a live video image stream or a live depth data stream.
[0105] In an example, the one or more cameras include one or more RGB-D cameras, and the live data stream is a live video image stream and / or a live depth data stream.
[0106] In an example, the one or more sensors are one or more lidar sensors and the live data stream is a live lidar image stream and / or a live depth data stream.
[0107] In an example, the one or more sensors are one or more radars, and the live data streams are live radar data streams and / or live image streams and / or live depth data streams.
[0108] In an example, the data of the patient received from the one or more sensors includes image data of the patient received from the one or more cameras.
[0109] In an example, the data of the patient received from the one or more sensors includes lidar image data of the patient received from one or more lidar sensors.
[0110] In an example, the data of the patient received from the one or more sensors includes lidar depth data of the patient received from one or more lidar sensors.
[0111] In an example, the data of the patient received from the one or more sensors includes depth data of the patient received from the one or more cameras.
[0112] In an example, the data of the patient received from the one or more sensors includes depth data of the patient received from the one or more radars.
[0113] In an example, the data of the patient received from the one or more sensors includes image data of the patient received from the one or more radars.
[0114] In an example, aligning the patient's data with the sensor coordinate system includes utilizing the patient's data from one or more sensors.
[0115] Thus, knowledge of the position(s) of one or more sensors can be used along with the patient's sensor data to determine the position of a part of the patient relative to the position(s) of the sensor, thereby placing the patient in the sensor coordinate system.
[0116] In the example, the sensor coordinate system is the camera coordinate system.
[0117] In the example, the sensor coordinate system is a lidar sensor coordinate system.
[0118] In an example, the sensor coordinate system is a radar sensor coordinate system.
[0119] According to an example, the processing unit is configured to generate a mapped medical image of the patient, including using the medical image of the patient mapped to the sensor coordinate system. The processing unit is configured to present the mapped medical image on a visual display unit. Then, generating a representation of the sensed position of the end of the catheter or tube relative to the medical image may include generating a representation of the sensed position of the end of the catheter or tube in the mapped medical image of the patient presented on the visual display unit.
[0120] According to an example, the processing unit is configured to generate a modified sensor coordinate system, including using a medical image of a patient mapped to the sensor coordinate system. Then, determining the position of the end of the catheter or tube in the sensor coordinate system may include determining the position of the end of the catheter or tube in the modified sensor coordinate system. The processing unit is configured to present the medical image on a visual display unit. Then, generating a representation of the sensed position of the end of the catheter or tube relative to the medical image may include generating a representation of the sensed position of the end of the catheter or tube in the medical image of the patient presented on the visual display unit.
[0121] In an example, the one or more sensors include a plurality of sensors, and wherein the live data stream includes data from each of the plurality of sensors.
[0122] In an example, aligning the patient's data with the sensor coordinate system includes utilizing the patient's data from each of the plurality of sensors.
[0123] In an example, the live data stream includes data of an end of a catheter or tube, and wherein a sensed position of the end of the catheter or tube to be inserted into a patient is determined based on the data of the end of the catheter or tube from each of a plurality of sensors.
[0124] In an example, the data of the end of the catheter or tube includes image data acquired by a plurality of cameras.
[0125] In an example, the data of the end of the catheter or tube includes depth data acquired by multiple cameras.
[0126] In an example, the data of the end of the catheter or tube includes image data collected by multiple lidar sensors.
[0127] In an example, the data of the end of the catheter or tube includes depth data collected by multiple lidar sensors.
[0128] In an example, the data of the end of the catheter or tube includes image data acquired by multiple radars.
[0129] In an example, the data of the end of the conduit or tube includes depth data acquired by multiple radars.
[0130] In an example, the radio positioning system is configured to sense the position of an end of a catheter or tube.The radio positioning system is configured to provide the sensed position of the end of the catheter or tube to the processing unit.
[0131] In an example, the catheter or tube comprises a fiber optic shape sensing system configured to sense the position of an end of the catheter or tube. The fiber optic shape sensing system is configured to provide the sensed position of the end of the catheter or tube to a processing unit.
[0132] In an example, mapping the medical image of the patient to the sensor coordinate system includes registration between the medical image of the patient and at least one frame of the live data stream.
[0133] In an example, the registration between the medical image of the patient and the at least one frame of the live data stream includes utilizing a warp transformation to align the medical image of the patient's body with data of the patient in the at least one frame of the live data stream.
[0134] Figure 3 A method 200 for tracking a catheter or tube is shown in its basic steps. The method 200 comprises:
[0135] Received medical images of 210 patients;
[0136] providing 220 a medical image of the patient to a processing unit;
[0137] receiving 230 a live data stream from one or more sensors, wherein the live data stream comprises data of the patient, and wherein a sensor coordinate system is defined relative to the one or more sensors;
[0138] Providing 240 a live data stream to a processing unit;
[0139] receiving 250, by a processing unit, a sensed position of an end of a catheter or tube to be inserted into a patient;
[0140] aligning, by the processing unit, the patient's data received from the one or more sensors with a sensor coordinate system 260 ;
[0141] mapping 270 the medical image of the patient to a sensor coordinate system by the processing unit;
[0142] determining 280, by the processing unit, a position of the end of the catheter or tube in the sensor coordinate system, including using a sensed position of the end of the catheter or tube; and
[0143] A representation of the sensed position of the end of the catheter or tube relative to the medical image is generated 290 by the processing unit, including utilizing the medical image of the patient mapped to the sensor coordinate system and the position of the end of the catheter or tube in the sensor coordinate system.
[0144] In an example, the medical image is an attenuation X-ray image.
[0145] In an example, the medical image is a CT X-ray image.
[0146] In an example, the medical image is a magnetic resonance image.
[0147] In an example, the one or more sensors are one or more cameras and the live data stream is a live video image stream.
[0148] In an example, the one or more cameras include one or more 3D cameras, and the live data stream is a live video image stream or a live depth data stream.
[0149] In an example, the one or more cameras include one or more RGB-D cameras, and the live data stream is a live video image stream and / or a live depth data stream.
[0150] In an example, the one or more sensors are one or more lidar sensors and the live data stream is a live lidar image stream and / or a live depth data stream.
[0151] In an example, the one or more sensors are one or more radars, and the live data streams are live radar data streams and / or live image streams and / or live depth data streams.
[0152] In an example, the data of the patient received from the one or more sensors includes image data of the patient received from the one or more cameras.
[0153] In an example, the data of the patient received from the one or more sensors includes lidar image data of the patient received from one or more lidar sensors.
[0154] In an example, the data of the patient received from the one or more sensors includes lidar depth data of the patient received from one or more lidar sensors.
[0155] In an example, the data of the patient received from the one or more sensors includes depth data of the patient received from the one or more cameras.
[0156] In an example, the data of the patient received from the one or more sensors includes depth data of the patient received from the one or more radars.
[0157] In an example, the data of the patient received from the one or more sensors includes image data of the patient received from the one or more radars.
[0158] In an example, aligning the patient's data with the sensor coordinate system includes utilizing the patient's data from one or more sensors.
[0159] Thus, knowledge of the position(s) of one or more sensors can be used along with the patient's sensor data to determine the position of a part of the patient relative to the position(s) of the sensor, thereby placing the patient in the sensor coordinate system.
[0160] In the example, the sensor coordinate system is the camera coordinate system.
[0161] In the example, the sensor coordinate system is a lidar sensor coordinate system.
[0162] In an example, the sensor coordinate system is a radar sensor coordinate system.
[0163] In an example, the method includes: generating, by a processing unit, a mapped medical image of a patient, including utilizing the medical image of the patient mapped to a sensor coordinate system; and generating a representation of a sensed position of an end of a catheter or tube relative to the medical image, including generating a representation of the sensed position of the end of the catheter or tube in the mapped medical image of the patient.
[0164] In an example, the method includes generating, by the processing unit, a modified sensor coordinate system, including using a medical image of a patient mapped to the sensor coordinate system. Determining a position of an end of the catheter or tube in the sensor coordinate system then includes determining the position of the end of the catheter or tube in the modified sensor coordinate system, and generating a representation of the sensed position of the end of the catheter or tube relative to the medical image then includes generating a representation of the sensed position of the end of the catheter or tube in the medical image of the patient.
[0165] In an example, the one or more sensors include a plurality of sensors, and wherein the live data stream includes data from each of the plurality of sensors.
[0166] In an example, aligning the patient's data with the sensor coordinate system includes utilizing the patient's data from each of the plurality of sensors.
[0167] In an example, the live data stream includes data of an end of a catheter or tube, and wherein a sensed position of the end of the catheter or tube to be inserted into a patient is determined based on the data of the end of the catheter or tube from each of a plurality of sensors.
[0168] In an example, the data of the end of the catheter or tube includes image data acquired by a plurality of cameras.
[0169] In an example, the data of the end of the catheter or tube includes depth data acquired by multiple cameras.
[0170] In an example, the data of the end of the catheter or tube includes image data collected by multiple lidar sensors.
[0171] In an example, the data of the end of the catheter or tube includes depth data collected by multiple lidar sensors.
[0172] In an example, the data of the end of the catheter or tube includes image data acquired by multiple radars.
[0173] In an example, the data of the end of the conduit or tube includes depth data acquired by multiple radars.
[0174] In an example, the method includes receiving, by the processing unit, a sensed position of an end of the catheter or tube from a radio positioning system configured to sense the position of the end of the catheter or tube.
[0175] In an example, the method includes receiving, by a processing unit, a sensed position of an end of a catheter or tube from a fiber optic shape sensing system of the catheter or tube, the fiber optic shape sensing system being configured to sense a position of the end of the catheter or tube of the catheter or tube.
[0176] In an example, mapping the medical image of the patient to the sensor coordinate system includes registering the medical image of the patient with at least one frame of the live data stream.
[0177] In an example, registering the medical image of the patient with the at least one frame of the live data stream includes aligning the medical image of the patient's body with data of the patient in the at least one frame of the live data stream using a warp transform.
[0178] Thus, the new technology solution provides automatic real-time tracking of foreign tubular objects. This enables a reduction in the number of incidents related to incorrectly inserted catheters and tubes in ED and ICU departments. Thus, it improves overall patient well-being and reduces radiologist workload. Furthermore, it provides direct feedback to ED and ICU clinicians at the time of acquisition.
[0179] This is accomplished by virtualizing the guidance process with a real-time projection of the device to be inserted into a previously acquired CXR or MR or other medical image that may have been acquired just prior to the catheter or tube insertion process or a significant time frame prior to the insertion process. Device specific anatomical regions may also be highlighted to aid navigation to the correct insertion point.
[0180] It should be noted that in the detailed system description below, reference is made to a chest X-ray (CXR) modality, however it should be understood that the system is not limited to this modality, and may be applied to a variety of modalities, such as CT, leg X-ray and other body parts and X-ray modalities, as well as other medical imaging modalities such as magnetic resonance (MR) imaging, positron emission tomography (PET) imaging, etc. In addition, reference is made below in the detailed embodiments described to a sensor system in the form of one or more cameras, but this may be a radar sensor based system, or a lidar sensor based system, or a single 3D RGB-D based system, or any other sensor based system from which data of a patient may be acquired to locate the patient within a coordinate system defined relative to those one or more sensors.
[0181] Therefore, reference Figure 4 , the position of central venous catheters and endotracheal tubes is usually assessed relative to the anatomical point of the carina (= the point where the lower edges of the left and right mainstem bronchi meet). Also, the superior vena cava (SVC) and the right atrium can help navigate the radiologist to the intended area for correct object placement. Figure 4 , the “right clavicle” is indicated by “A”, the “right subclavian vein” is indicated by “B”, the “pericardium” is indicated by “C”, the “right internal jugular vein” is indicated by “D”, the “left internal jugular vein” is indicated by “E”, the “left clavicle” is indicated by “F”, the “left subclavian vein” is indicated by “G”, the “carina” is indicated by “H”, the “right brachiocephalic vein” is indicated by “I”, the “left brachiocephalic vein” is indicated by “J”, and the “catheter tip” is indicated by “K”.
[0182] To help understand the new technology, the existing workflow is laid out and the new technology is contrasted against it.
[0183] A workflow commonly encountered in clinical practice for evaluating medical images (of any modality) to exclude misplaced foreign bodies involves the following steps:
[0184] A specific tubular device is inserted into the patient.
[0185] Acquire a target verification exam, such as CXR (but could be MR etc).
[0186] The study is sent to the PACS system and queued at the end of the work list.
[0187] The radiologist reviews the exam after some time X.
[0188] a. If some foreign body dislocation is detected, report it directly to the clinician.
[0189] b. Otherwise, perform normal reporting.
[0190] The clinician receives the report after some time Y and triggers the necessary next steps.
[0191] When the foreign body is misplaced relative to the intended correct area, both times X, Y may be long and may cause serious problems.
[0192] The new technology described here makes steps 2-5 obsolete, as the clinician can get direct feedback during the real-time process of device insertion. The new system can be run directly on the image acquisition machine, but is not limited to it. For example, it can also be implemented in the cloud.
[0193] The main components of the new system are:
[0194] 1) Camera for real-time video streaming (can be radar, lidar, etc.)
[0195] 2) Target tube / catheter with detectable tip (active / passive)
[0196] 3) Radio positioning equipment integrated into the catheter and / or optical fiber and / or antennas for triangulation
[0197] 4) Real-time patient body to CXR (MR, etc.) registration module
[0198] 5) Anatomical region segmentation / landmark detection module
[0199] 6) Reporting and visualization module
[0200] In the new technique, key elements of the device (catheter / tube) are located and the position is mapped to a previously acquired X-ray image. Only X-ray is discussed below, but as mentioned above, this can be other medical imaging modalities, such as MR. Positioning can be performed via radio positioning, triangulation, fiber optic shape sensing, or other techniques known in the art. Therefore, the catheter / tube tip position is located in real time in a patient coordinate system aligned with the patient's video stream. At the same time, the video stream is mapped to the X-ray image using an image registration algorithm. Therefore, the coordinates of the tip can be presented on the X-ray image.
[0201] The implementation of an automatic real-time system for tube and catheter tracking starts with pre-training the images to an X-ray registration model and an anatomical region segmentation / landmark detection model. Both are deep convolutional networks of the encoder-decoder type. Both are pre-trained.
[0202] In a second step, a reference image is made available to guide the procedure. This can be a pre-operative X-ray image (eg, a previous image of the patient) or a template / atlas image or other medical image such as MR.
[0203] The third step is to calibrate the patient-side coordinate system with the tip and other key landmarks of the device. The patient-side coordinate system is connected to the video stream.
[0204] The fourth step is to infer anatomical beacons from the segmentation / landmark detection model of the anatomical regions. These regions are shown on the preliminary CXR.
[0205] The next stage is to insert the tubular device into the patient (see Figure 5 ). As the device is being inserted, the tip coordinates are calculated in the patient's coordinate system and attached to the video frame. At the same time, the video frame is registered to the preliminarily captured CXR image, and the patient's coordinate system is mapped to the CXR coordinate system. As a result, the tip coordinates are converted to the CXR coordinate system. The tip coordinates are displayed on the monitor, overlaying the CXR image with anatomical beacons. Figure 5 A schematic overview of the process is shown in .
[0206] Therefore, the whole new technology can be described as follows:
[0207] The patient is about to have a catheter / tube inserted into his or her body.
[0208] An X-ray image of the patient is made available to the system.
[0209] The system has one or more cameras that view the patient and actually place the patient into a "camera coordinate system."
[0210] The X-ray image is mapped into a "camera coordinate system".
[0211] The imaged X-rays are mapped to the camera coordinate system via image registration. This can be done by matching 2 images: the X-rays and the frames from the video with the patient's body. In practice, a warping transformation is found that will align the patient's body with the X-ray image. It can be done with neural networks or classical computer vision algorithms.
[0212] Monitor the end of the catheter / tube.
[0213] This may be via one or more cameras.
[0214] This may be via another system, such as a radar or fiber optic based system.
[0215] The position of the catheter / tube is determined within the Camera Coordinate System.
[0216] Therefore, the position of the catheter / tube is known relative to the X-ray image.
[0217] The position of the catheter / tube is shown relative to the X-ray image of the patient.
[0218] Thus, the practitioner can actually see the end of the catheter / tube relative to an X-ray image of the patient which shows the patient in front of them, and they can see that they will be inserting the catheter / tube correctly as they approach the patient relative to the patient's venous structure, and they can see the tip of the catheter / tube relative to the patient's venous structure as they insert the catheter / tube.
[0219] The medical practitioner can then properly insert the catheter / tube into the patient.
[0220] The above references coordinate systems, however there are actually three coordinate systems:
[0221] 1) A patient-side coordinate system that describes the device position with respect to some anchor points (e.g., antennas for triangulation). This coordinate system may not be patient aware.
[0222] 2) A video stream coordinate system describing the patient's position in time. This can be connected (calibrated) to the patient's coordinate system using, for example, the known positions of the antennas.
[0223] 3) X-ray coordinate system. It can be connected (calibrated) to the video stream coordinate system via image registration.
[0224] Therefore, the position of the device (catheter / tube) can be viewed on the X-ray. So, in reality the video stream is a proxy between the device and the patient.
[0225] In another exemplary embodiment, a computer program or a computer program element is provided, characterized in that it is configured to execute the method steps of any method described in one of the above embodiments on a suitable device or system.
[0226] Therefore, the computer program element can be stored on a computer unit, which can also be part of an embodiment. The computing unit can be configured to perform the steps of the method described above or induce the execution of the steps of the method described above. In addition, it can be configured to operate the components of the system described above. The computing unit can be configured to automatically operate and / or execute the user's commands. The computer program can be loaded into the working memory of a data processor. The data processor can thus be equipped to perform the method according to one of the embodiments.
[0227] This exemplary embodiment of the invention covers both a computer program that right from the beginning uses the invention and a computer program that by means of an up-date turns an existing program into a program that uses the invention.
[0228] Furthermore, the computer program element can provide all necessary steps for implementing the procedures of an exemplary embodiment of the method as described above.
[0229] According to a further exemplary embodiment of the invention, a computer readable medium, such as a CD-ROM, a USB stick or the like, is proposed, wherein the computer readable medium has a computer program element stored on the computer readable medium, the computer program element being described in the preceding section.
[0230] The computer program may be stored / distributed on suitable media, such as optical storage media or solid-state media provided together with or as part of other hardware, but the computer program may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems.
[0231] However, the computer program may also be present on a network like the World Wide Web and can be downloaded from such a network into a working memory of a data processor. According to a further exemplary embodiment of the invention, a medium for making a computer program element available for downloading is provided, wherein the computer program element is arranged to perform a method according to one of the previously described embodiments of the invention.
[0232] It must be noted that embodiments of the present invention are described with reference to different subject matters. Specifically, some embodiments are described with reference to method type claims, while other embodiments are described with reference to device type claims. However, a person skilled in the art will appreciate from the above and following descriptions that, unless otherwise indicated, in addition to any combination of features belonging to one type of subject matter, any combination between features relating to different subject matters is also considered to be disclosed by the present application. However, all features can be combined to provide synergistic effects that exceed the simple sum of the features.
[0233] Although the present invention has been described and illustrated in detail in the drawings and the foregoing description, such description and illustration are to be considered illustrative or exemplary rather than restrictive. The present invention is not limited to the disclosed embodiments. Other variations of the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention by studying the drawings, the description, and the appended claims.
[0234] In the claims, the word "comprising" does not exclude other elements or steps, and the word "a" or "an" does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. Any reference signs in the claims should not be construed as limiting the scope.
Claims
1. A device (10) for tracking a catheter or tube, comprising: Input unit (20); as well as A processing unit (30); Wherein, the input unit is configured to receive a medical image of a patient; wherein the input unit is configured to provide the medical image of the patient to the processing unit; wherein the input unit is configured to receive a live data stream from one or more sensors, wherein the live data stream comprises data of the patient, and wherein a sensor coordinate system is defined relative to the one or more sensors; Wherein, the input unit is configured to provide the live data stream to the processing unit; wherein the processing unit is configured to receive a sensed position of an end of a catheter or tube to be inserted into the patient; wherein the processing unit is configured to align the data of the patient received from the one or more sensors with the sensor coordinate system; wherein the processing unit is configured to map the medical image of the patient to the sensor coordinate system; wherein the processing unit is configured to determine the position of the end of the catheter or tube in the sensor coordinate system, including using the sensed position of the end of the catheter or tube; and Wherein, the processing unit is configured to generate a representation of the sensed position of the end of the catheter or tube relative to the medical image, including utilizing the medical image of the patient mapped to the sensor coordinate system and the position of the end of the catheter or tube in the sensor coordinate system.
2. The device according to claim 1, wherein: The processing unit is configured to generate a mapped medical image of the patient, including utilizing the medical image of the patient mapped to the sensor coordinate system, and wherein generating the representation of the sensed position of the end of the catheter or tube relative to the medical image includes generating the representation of the sensed position of the end of the catheter or tube in the mapped medical image of the patient.
3. The device according to claim 1, wherein: The processing unit is configured to generate a modified sensor coordinate system, including using the medical image of the patient mapped to the sensor coordinate system, wherein determining the position of the end of the catheter or tube in the sensor coordinate system includes: determining the position of the end of the catheter or tube in the modified sensor coordinate system, and wherein generating the representation of the sensed position of the end of the catheter or tube relative to the medical image includes: generating the representation of the sensed position of the end of the catheter or tube in the medical image of the patient.
4. The device according to any one of claims 1 to 3, wherein: The one or more sensors include a plurality of sensors, and wherein the live data stream includes data from each of the plurality of sensors.
5. The device according to claim 4, wherein: Aligning the data of the patient with the sensor coordinate system includes utilizing the data of the patient from each of the plurality of sensors.
6. The device according to any one of claims 4 to 5, wherein: The live data stream includes data of the end of the catheter or tube, and wherein the sensed position of the end of the catheter or tube to be inserted into the patient is determined based on the data of the end of the catheter or tube from each of the multiple sensors.
7. The device according to any one of claims 1 to 5, wherein: A radio positioning system is configured to sense the position of the end of the catheter or tube, and wherein the radio positioning system is configured to provide the sensed position of the end of the catheter or tube to the processing unit.
8. The device according to any one of claims 1 to 5, wherein: The catheter or tube comprises a fiber optic shape sensing system configured to sense the position of the end of the catheter or tube, and wherein the fiber optic shape sensing system is configured to provide the sensed position of the end of the catheter or tube to the processing unit.
9. The device according to any one of claims 1 to 8, wherein: Mapping the medical image of the patient to the sensor coordinate system includes registration between the medical image of the patient and at least one frame of the live data stream.
10. The device according to claim 9, wherein: The registration between the medical image of the patient and at least one frame of the live data stream includes utilizing a warp transformation to align the medical image of the patient's body with the data of the patient in the at least one frame of the live data stream.
11. A system (100) for tracking a catheter or tube, comprising: Input unit (120); A processing unit (130); one or more sensors (140); as well as a visual display unit (150); Wherein, the input unit is configured to receive a medical image of a patient; wherein the input unit is configured to provide the medical image of the patient to the processing unit; wherein the input unit is configured to receive a live data stream from the one or more sensors, wherein the live data stream comprises data of the patient, and wherein a sensor coordinate system is defined relative to the one or more sensors; Wherein, the input unit is configured to provide the live data stream to the processing unit; wherein the processing unit is configured to receive a sensed position of an end of a catheter or tube to be inserted into the patient; wherein the processing unit is configured to align the data of the patient received from the one or more sensors with the sensor coordinate system; wherein the processing unit is configured to map the medical image of the patient to the sensor coordinate system; wherein the processing unit is configured to determine the position of the end of the catheter or tube in the sensor coordinate system, including using the sensed position of the end of the catheter or tube; and Wherein, the processing unit is configured to generate, on the visual display unit, a representation of the sensed position of the end of the catheter or tube relative to the medical image, including utilizing the medical image of the patient mapped to the sensor coordinate system and the position of the end of the catheter or tube in the sensor coordinate system.
12. The system according to claim 11, wherein: The processing unit is configured to generate a mapped medical image of the patient, including using the medical image of the patient mapped to the sensor coordinate system, wherein the processing unit is configured to present the mapped medical image on the visual display unit, and wherein generating the representation of the sensed position of the end of the catheter or tube relative to the medical image includes: generating the representation of the sensed position of the end of the catheter or tube in the mapped medical image of the patient presented on the visual display unit.
13. The system according to claim 11, wherein: The processing unit is configured to generate a modified sensor coordinate system, including using the medical image of the patient mapped to the sensor coordinate system, wherein determining the position of the end of the catheter or tube in the sensor coordinate system includes: determining the position of the end of the catheter or tube in the modified sensor coordinate system, wherein the processing unit is configured to present the medical image on the visual display unit, and wherein generating the representation of the sensed position of the end of the catheter or tube relative to the medical image includes: generating the representation of the sensed position of the end of the catheter or tube in the medical image of the patient presented on the visual display unit.
14. A method (200) for tracking a catheter or tube, comprising: receiving (210) a medical image of a patient; providing (220) the medical image of the patient to a processing unit; receiving (230) a live data stream from one or more sensors, wherein the live data stream includes data of the patient, and wherein a sensor coordinate system is defined relative to the one or more sensors; providing (240) the live data stream to the processing unit; receiving (250) by the processing unit a sensed position of an end of a catheter or tube to be inserted into the patient; aligning, by the processing unit, the data of the patient received from the one or more sensors with the sensor coordinate system (260); mapping (270) the medical image of the patient to the sensor coordinate system by the processing unit; determining (280), by the processing unit, a position of the end of the catheter or tube in the sensor coordinate system, including using the sensed position of the end of the catheter or tube; and A representation of the sensed position of the end of the catheter or tube relative to the medical image is generated (290) by the processing unit, including utilizing the medical image of the patient mapped to the sensor coordinate system and the position of the end of the catheter or tube in the sensor coordinate system.
15. A computer program element, wherein the computer program element is used to control the apparatus according to any one of claims 1 to 10, and when the computer program element is run by a processor, the computer program element is configured to perform the method according to claim 14, and / or, the computer program element is used to control the system according to any one of claims 11 to 13, and when the computer program element is run by a processor, the computer program element is configured to perform the method according to claim 14.