Systems and methods for visual image guidance during medical procedures

Abstract

A system for providing visualization during a medical procedure of a patient or a simulated medical procedure of a patient model, the system comprising: a. Means for receiving imaging data of a body of the patient or a first region of the patient model; b. Means for receiving tracking data of one or more instruments inserted in a second region of the patient's body or of the patient model; characterized in that: c. The system comprises means for receiving model data of a fourth region of the patient's body or of the patient model, and processing means adapted to collectively register the tracking data and the model data; d. The system comprises a display for simultaneous visualization of the co-registered data and the imaging data; e, the tracking data comprises three or more tracking points, and the three tracking points are not collinear; and / or one or more tracking points and roll, pitch and yaw angles or a set of Euler angles or a set of quaternion or angle-axis representations or a rotation matrix or a homogeneous transformation matrix; and / or a point cloud representation for at least one of the one or more instruments; f. The tracking data is used to determine the position and / or shape and / or orientation of at least one of the one or more instruments; and g. The visualization comprises a representation of the shape and / or position and / or orientation of at least one of the one or more instruments; and an associated method.

Description

Technical Field

[0001] The present disclosure relates to the field of visual image guidance during medical procedures or simulated medical procedures. Background Art

[0002] Advances in medical technology have enabled the development and routine use of minimally invasive procedures, such as laparoscopy or coronary catheterization. These procedures reduce surgical trauma, wound healing time, associated pain, and the risk of infection and other complications. However, providing accurate visual guidance to physicians performing these procedures remains a challenge. Typically, physicians rely on intraoperative endoscopic and / or fluoroscopic images, possibly combined with preoperative imaging techniques such as CT scans or MRIs, as well as their experience and knowledge of human anatomy. This approach is not without its drawbacks: endoscopes have a limited field of view, and fluoroscopy and preoperative imaging techniques require expensive equipment and expose patients to harmful radiation. Furthermore, the vast majority of these techniques provide 2D images. Therefore, the burden of correctly estimating the position and orientation (often referred to as pose) of 3D instruments within the patient's body cavity remains on the physician.

[0003] Document US2020188028A1 discloses a system and method for providing augmented reality visualization during a patient's medical procedure. According to US2020188028A1, a three-dimensional (3D) model of the patient's anatomical part is presented on a 3D display, aligned with the two-dimensional (2D) live imaging data obtained during the procedure. In addition, some embodiments of US2020188028A1 utilize 3D tracking data to locate the instruments used in the procedure in real time. The visualization of the position of the instrument can be presented on a 3D display in alignment with the 3D model and / or 2D live imaging data to guide the clinician in performing the medical procedure.

[0004] This approach to visual image guidance during medical procedures doesn't account for the fact that instruments can extend in three dimensions; it tracks a single point on the physical instrument and assumes that the corresponding portion of the instrument always fits within the patient's body cavity. However, for example, the tip of an endoscope can bend to provide different camera angles. If a clinician were to rotate such a curved tool around its own axis within the patient's body cavity, they could potentially injure the patient's internal organs and / or wall tissue.

[0005] Therefore, there remains a need for an alternative and improved system and method for visual image guidance during medical procedures. Summary of the Invention

[0006] The present disclosure is directed to remedying the above-referenced and other shortcomings.

[0007] According to a first aspect of the present disclosure, a system for providing visualization during a medical procedure of a patient or a simulated medical procedure of a patient model is disclosed, the system comprising:

[0008] means for receiving imaging data of a first region of a patient's body or a patient model;

[0009] means for receiving tracking data of one or more instruments inserted into a second region of the patient's body or a patient model;

[0010] means for receiving posture data of a patient's body or a patient model, said posture data belonging to a third area;

[0011] Processing means for co-registering imaging data, tracking data and posture data;

[0012] a display for simultaneously visualizing two or more of the first, second, and third regions and the co-registered data pertaining to the simultaneously visualized regions;

[0013] The system is characterized by:

[0014] The tracking data comprises three or more tracking points, wherein the three tracking points are not collinear; and / or one or more tracking points and roll, pitch, and yaw angles or a set of Euler angles or a set of quaternions or an angle-axis representation or a rotation matrix or a homogeneous transformation matrix; and / or a point cloud representation for at least one of the one or more instruments;

[0015] The tracking data is used to determine the position and / or shape and / or orientation of at least one of the one or more instruments; and

[0016] The visualization includes a representation of the shape and / or position and / or orientation of at least one of the one or more instruments.

[0017] According to a preferred embodiment, the system further comprises means for receiving model data of a fourth region of the patient's body or of a model of the patient, wherein the processing means is adapted to co-register the imaging data, the tracking data, the posture data and the model data, wherein the display is adapted to simultaneously visualise two or more of the first, second, third and fourth regions, and the co-registered data belonging to the simultaneously visualised regions.

[0018] According to a preferred embodiment:

[0019] Model data and tracking data include 3D data;

[0020] The display is a 3D display; and

[0021] The representation of the model data and the representation of the shape and / or position and / or orientation of the one or more instruments is a 3D representation.

[0022] According to a preferred embodiment, the imaging data are 2D data that are processed to allow visualization on a 3D display.

[0023] According to a preferred embodiment, there is at least a partial overlap between two or more regions, and wherein the visualization comprises a superposition of representations of data originating from the two or more regions.

[0024] According to a preferred embodiment, the overlay is presented aligned on the corresponding data, or wherein the overlay can be positioned and repositioned during the process in order to provide the user with the desired view.

[0025] According to a preferred embodiment, the system delay is equal to or less than 50 ms.

[0026] According to a preferred embodiment, co-registration of the model data, imaging data, tracking data and orientation data occurs in a continuous or intermittent manner during the process.

[0027] According to a preferred embodiment, the model data is generated prior to the process.

[0028] According to a preferred embodiment, the imaging data is generated during a surgical procedure.

[0029] According to a preferred embodiment, the imaging data is generated in an intermittent manner.

[0030] According to a preferred embodiment, one or more sensors for generating tracking data are embedded in or fixedly attached to at least one of the one or more instruments.

[0031] According to a preferred embodiment, at least one of the one or more sensors comprises an optical fiber, and / or at least one of the one or more sensors comprises one or more electromagnetic trackers, and / or at least one of the one or more sensors is a distributed sensor that conforms to the shape of the instrument.

[0032] According to a preferred embodiment, the one or more cameras or LIDAR systems have an unobstructed field of view of at least one of the one or more instruments to derive tracking data based on image processing techniques applied to the camera or LIDAR images.

[0033] According to a second aspect of the present disclosure, a method for providing visualization during a medical procedure of a patient or a simulated medical procedure of a patient model is disclosed, the method comprising:

[0034] receiving image data of a first region of a patient's body or a patient model;

[0035] receiving tracking data of one or more instruments inserted into a second region of the patient's body or a patient model;

[0036] receiving pose data of a patient body or a patient model, the pose data pertaining to a third region; co-registering the imaging data, the tracking data, and the pose data;

[0037] simultaneously visualizing two or more of the first, second, and third regions, and the co-registered data pertaining to the simultaneously visualized regions;

[0038] The method is characterized in that:

[0039] The tracking data comprises three or more tracking points, wherein the three tracking points are not collinear; and / or one or more tracking points and roll, pitch, and yaw angles or a set of Euler angles or a set of quaternions or an angle-axis representation or a rotation matrix or a homogeneous transformation matrix; and / or a point cloud representation for at least one of the one or more instruments;

[0040] The tracking data is used to determine the position and / or shape and / or orientation of at least one of the one or more instruments; and

[0041] The visualization includes a representation of the shape and / or position and / or orientation of at least one of the one or more instruments.

[0042] In some embodiments of the method, the method further comprises the following steps:

[0043] receiving model data of a fourth region of the patient's body or a model of the patient;

[0044] Co-register imaging data, tracking data, and pose data;

[0045] simultaneously visualizing two or more of the first, second, third, and fourth regions, and the co-registered data pertaining to the simultaneously visualized regions;

[0046] According to a third aspect of the present disclosure, a system for providing visualization during a medical procedure of a patient or a simulated medical procedure of a patient model is disclosed, the system comprising:

[0047] means for receiving imaging data of a first region of a patient's body or a patient model;

[0048] means for receiving tracking data of one or more instruments inserted into a second region of the patient's body or a patient model;

[0049] Its characteristics are:

[0050] The system comprises means for receiving model data of a fourth region of the patient's body or of a model of the patient, and processing means adapted to co-register the tracking data and the model data;

[0051] The system includes a display for simultaneous visualization of the co-registered data and the imaging data;

[0052] The tracking data comprises three or more tracking points, wherein the three tracking points are not collinear; and / or one or more tracking points and roll, pitch, and yaw angles or a set of Euler angles or a set of quaternions or an angle-axis representation or a rotation matrix or a homogeneous transformation matrix; and / or a point cloud representation for at least one of the one or more instruments;

[0053] The tracking data is used to determine the position and / or shape and / or orientation of at least one of the one or more instruments; and

[0054] The visualization includes a representation of the shape and / or position and / or orientation of at least one of the one or more instruments.

[0055] According to a preferred embodiment, the system further comprises means for receiving posture data of the patient body or of a patient model, said posture data belonging to the third area, wherein the processing means are adapted to further co-register the posture data, wherein the display is adapted for further visualization of the posture information.

[0056] According to a preferred embodiment, the model data and the tracking data comprise 3D data;

[0057] a. The shape of the device is determined based on tracking data including time-related data;

[0058] The display is a 3D display; and

[0059] The representation of the model data and the representation of the shape and / or position and / or orientation of the one or more instruments is a 3D representation.

[0060] According to a preferred embodiment, the imaging data are 2D data that are processed to allow visualization on a 3D display.

[0061] According to a preferred embodiment, there is at least a partial overlap between the two or more regions.The visualization comprises a superposition of representations of data originating from the two or more regions.

[0062] According to a preferred embodiment, the overlay is presented aligned on the corresponding data, or wherein the overlay can be positioned and repositioned during the process in order to provide the user with the desired view.

[0063] According to a preferred embodiment, the system delay is equal to or less than 50 ms.

[0064] According to a preferred embodiment, co-registration of the model data, imaging data, tracking data and orientation data occurs in a continuous or intermittent manner during the process.

[0065] According to a preferred embodiment, the model data is generated prior to the process.

[0066] According to a preferred embodiment, the imaging data is generated during a surgical procedure.

[0067] According to a preferred embodiment, the imaging data is generated in an intermittent manner.

[0068] According to a preferred embodiment, one or more sensors for generating tracking data are embedded in or fixedly attached to at least one of the one or more instruments.

[0069] According to a preferred embodiment, at least one of the one or more sensors comprises an optical fiber, and / or at least one of the one or more sensors comprises one or more electromagnetic trackers, and / or at least one of the one or more sensors is a distributed sensor that conforms to the shape of the instrument.

[0070] According to a preferred embodiment, the one or more cameras or LIDAR systems have an unobstructed field of view of at least one of the one or more instruments to derive tracking data based on image processing techniques applied to the camera or LIDAR images.

[0071] According to a fourth aspect of the present disclosure, a method for providing visualization during a medical procedure of a patient or a simulated medical procedure of a patient model is disclosed, the method comprising the following steps:

[0072] a. receiving image data of a first region of a patient's body or a patient model;

[0073] b. receiving tracking data of one or more instruments inserted into a second region of the patient's body or a patient model;

[0074] The method is characterized in that it comprises:

[0075] c. receiving model data of a patient's body or a patient model;

[0076] d. Co-register tracking data and model data;

[0077] e. Simultaneous visualization of imaging data and co-registered data;

[0078] And it is characterized by

[0079] f. The tracking data comprises three or more tracking points, wherein the three tracking points are not collinear; and / or one or more tracking points and roll, pitch, and yaw angles or a set of Euler angles or a set of quaternions or an angle-axis representation or a rotation matrix or a homogeneous transformation matrix; and / or a point cloud representation for at least one of the one or more instruments;

[0080] And it is characterized by

[0081] g. Tracking data is used to determine the position and / or shape and / or orientation of at least one of the one or more instruments; and

[0082] h. The visualization includes a representation of the shape and / or position and / or orientation of at least one of the one or more instruments.

[0083] According to a preferred embodiment, the method further comprises receiving posture data of the patient's body or of a patient model, said posture data belonging to the third region, wherein the method further comprises co-registering the posture data, the method further comprises visualizing the posture information.

[0084] According to a preferred embodiment:

[0085] a. Model data and tracking data including 3D data;

[0086] b. The display is a 3D display; and

[0087] c. The representation of the model data and the representation of the shape and / or position and / or orientation of the one or more instruments is a 3D representation.

[0088] According to a preferred embodiment, the imaging data are 2D data that are processed to allow visualization on a 3D display.

[0089] According to a preferred embodiment, there is at least a partial overlap between two or more regions, and wherein the visualization comprises a superposition of representations of data originating from the two or more regions.

[0090] According to a preferred embodiment, the overlay is presented aligned on the corresponding data, or wherein the overlay can be positioned and repositioned during the process in order to provide the user with the desired view.

[0091] According to a preferred embodiment, the system delay is equal to or less than 50 ms.

[0092] According to a preferred embodiment, co-registration of the model data, imaging data, tracking data and orientation data occurs in a continuous or intermittent manner during the process.

[0093] According to a preferred embodiment, the model data is generated prior to the process.

[0094] According to a preferred embodiment, the imaging data is generated during a surgical procedure.

[0095] According to a preferred embodiment, the imaging data is generated in an intermittent manner.

[0096] According to a preferred embodiment, one or more sensors for generating tracking data are embedded in or fixedly attached to at least one of the one or more instruments.

[0097] According to a preferred embodiment, at least one of the one or more sensors comprises an optical fiber, and / or at least one of the one or more sensors comprises one or more electromagnetic trackers, and / or at least one of the one or more sensors is a distributed sensor that conforms to the shape of the instrument.

[0098] According to a preferred embodiment, the one or more cameras or LIDAR systems have an unobstructed field of view of at least one of the one or more instruments to derive tracking data based on image processing techniques applied to the camera or LIDAR images.

[0099] According to yet another aspect of the present disclosure, a method for providing visualization of a target area on a patient during a medical procedure is disclosed, the method comprising the steps of:

[0100] receiving imaging data and model data of a target area;

[0101] receiving tracking data of the flexible device;

[0102] generating a 2D image based on the imaging data;

[0103] The method further comprises:

[0104] co-registering the model data and the tracking data to generate a 3D image; and

[0105] Simultaneous visualization of 2D and 3D images on a 3D display;

[0106] And wherein the tracking data is used to determine the position and / or shape and / or orientation of at least one of the one or more instruments.

[0107] According to a preferred embodiment, the method further comprises:

[0108] receiving posture data of the patient, wherein the posture data is used to determine a position and orientation of the patient;

[0109] The pose data is co-registered to the model data and the tracking data, wherein a 3D image is displayed on a 3D display in a parallel orientation to the patient.

[0110] According to a preferred embodiment, the 2D image and the 3D image are displayed side by side or superimposed on each other on a 3D display.

[0111] According to a preferred embodiment, co-registration of the model data, imaging data, tracking data occurs in a continuous or intermittent manner during the process.

[0112] According to a preferred embodiment, co-registration of the model data, pose data, tracking data occurs in a continuous or intermittent manner during the process.

[0113] According to a preferred embodiment, the displayed delay is equal to or less than 50 ms.

[0114] The following further explains the different features of the first and third aspects and the second and fourth aspects of the present disclosure, mutatis mutandis.

[0115] In the context of the present disclosure, "imaging data" refers to data representing one or more visual images of one or more parts of the interior of a patient's body or a patient model. Alternatively or additionally, imaging data may represent visual images of one or more parts of an instrument or a visual image of a physician operating an instrument. For example, imaging data may include a visual image of an interior region of a patient's body together with the part of an instrument inserted into the region and the physician's hand holding the instrument. Alternatively or additionally, imaging data may refer to recorded non-visual data of the patient's body or patient model, such as heart rate, blood pressure, or oxygen saturation level.

[0116] In the context of the present disclosure, "tracking data" refers to data that can be used to retrieve the spatial position and / or orientation of one or more instruments. Alternatively or in addition, tracking data may also include time-related data such as timestamps, chronology indications, propagation speeds, and the like.

[0117] In the context of the present disclosure, "posture data" refers to a collection of positions and orientations. For a human patient, the posture data refers in particular to the positions and orientations of the head, torso, and limbs. The posture data may relate to the patient's entire body or only a portion of the body. For an instrument, the posture data may refer to the position and orientation of a point on the instrument, such as the tip of the instrument, or the instrument handle, or any other convenient and representative point on the instrument. When one or more instruments are manipulable instruments, the combination of the posture data provides a representation of the shape of the instrument. The posture data at different points may vary relative to each other. By tracking each individual position and orientation point and fitting a spline curve or other representation, the shape of the manipulable instrument can be calculated and visualized accordingly.

[0118] In the context of this disclosure, "co-registration" refers to the process of aligning data originating from different reference frames. Such co-registration may require translating, rotating or scaling the data in the spatial and temporal domains.

[0119] Tracking data for at least one of the one or more instruments includes three or more tracking points, wherein the three tracking points are non-collinear. When the tracking points have a fixed geometric relationship to the instrument, knowledge of the positions of the three non-collinear tracking points allows the position and orientation of the instrument to be determined in an unambiguous manner. Preferably, the instrument includes tracking points distributed across its geometry such that the shape of the instrument can be estimated by knowledge of the tracking points.

[0120] Alternatively, the tracking data for at least one of the one or more instruments includes one or more tracking points and / or a corresponding orientation representation by means of corresponding roll, pitch, and yaw angles, a set of Euler angles, a set of quaternions, an angle-axis representation, a rotation matrix, a homogeneous transformation matrix, or any other orientation representation. Knowledge of these parameters also allows the position and orientation of the instrument to be determined unambiguously, provided that the tracking points have a fixed or known geometric relationship to the instrument. For instruments that are deformable and adjustable in shape, the shape of the instrument can be estimated as long as there are enough tracking points distributed over the instrument, or a point cloud representation of the instrument shape, or a mechanical or other type of relationship that allows the overall instrument shape to be estimated based on input parameters such as actuation forces, pressures, or other input parameters.

[0121] An advantage of this system is that not only the shape of the instrument but also its position and orientation can be visualized. If the shape, position, and / or orientation of the instrument are visualized on the display, the surgeon performing the procedure does not need to rely solely on knowledge, experience, and feel to assess whether the instrument is correctly positioned and fits within the anatomical structure into which it is inserted. This can make medical procedures faster, safer, and more effective.

[0122] In some embodiments of the system of the first aspect, the system further comprises means for receiving model data of a fourth region of the patient's body or a model of the patient. Furthermore, the processing means is adapted to co-register the imaging data, the tracking data, the pose data, and the model data, and the display is adapted to simultaneously visualize two or more of the first, second, third, and fourth regions, and the co-registered data pertaining to the simultaneously visualized regions.

[0123] In the context of this disclosure, "model data" refers to data that provides a visual representation of one or more regions within a patient's body or a model of a patient. Compared to "imaging data," model data can provide higher spatial resolution. Model data is often based on various imaging data sources that have been computer-processed to generate a single, more complex data set. Furthermore, model data may include visual indications of attributes that are not visually identifiable in real life.

[0124] In some embodiments of the system of the third aspect, the system further comprises means for receiving posture data of the patient's body or of a patient model, the posture data belonging to the third area, wherein the processing means is adapted to further co-register the posture data, and wherein the display is adapted for further visualization of the posture information.

[0125] In some embodiments of the system including model data, the model data and tracking data include 3D data, the display is a 3D display, and the representation of the model data and / or the representation of the shape and / or position and / or orientation of one or more instruments is a 3D representation. Preferably, the shape of the instrument is determined based on the tracking data including time-dependent data.

[0126] Anatomical structures and instruments used during medical procedures inherently possess 3D shapes. When these 3D shapes are projected onto a 2D plane, as occurs when visualizing them on a traditional 2D display, a significant amount of information is lost. Often, to properly assess the situation, 2D views from multiple angles are necessary. Even then, the geometric relationships between the different objects shown on the 2D display may not be immediately clear and unambiguous. The use of a 3D display, combined with a 3D model and tracking data, addresses these issues.

[0127] In some embodiments of systems employing a 3D display, the imaging data is 2D data that is processed to allow visualization on the 3D display.

[0128] Typically, live imaging data obtained during medical procedures is 2D in nature. Displaying 2D imaging data as 3D data may introduce visualization artifacts and / or give the false impression that the data contains additional geometric information. In order to visualize the data on a 3D display, such as an autostereoscopic display or XR glasses or any other type of 3D visualization system, two virtual cameras are placed in the scene to be rendered. Each of these cameras will provide an image for a single eye of the user. By aligning the position and orientation of these cameras, the user's two eyes are presented with the same image. Therefore, the resulting image visualized on the display is perceived as a 2D image. This technology allows 2D images to be correctly visualized on a 3D display. It also allows native 3D images to be converted into 2D images.

[0129] In some embodiments of the system, there is at least partial overlap between two or more identified regions, and the visualization includes a visualization of the data from the overlapping regions. For example, the amount of spatial overlap between the regions can be 0%, 20%, 50%, or 100% of the surface area of ​​the smallest region among the regions. In the case of overlap between three or more regions, the amount of overlap between each pair of regions in the three or more regions is not necessarily the same.

[0130] For example, there may be a partial spatial overlap between the second region and the fourth region, and the visualization includes superimposing a representation of the shape and / or position and / or orientation of at least one of the one or more instruments on the representation of the model data.

[0131] Preferably, there is a significant spatial overlap between the second area comprising the inserted instruments and the fourth area comprising the model data. Preferably, there is also a significant spatial overlap between these two areas and the target area of ​​the medical procedure or simulated medical procedure. Preferably, the superposition includes a representation of the shape, position and orientation of all instruments inserted into the patient or patient model, as well as a representation of the model data. The resulting visualization allows the physician to see in real time and in 3D how the instruments are positioned in the model representation of the patient's body or patient model. If the model data is collected from actual measurements of the patient's body or patient model and is sufficiently accurate, the resulting visualization allows the physician to interpret in real time and in 3D how the instruments are positioned within the patient's body or patient model.

[0132] For example, there may be a partial spatial overlap between the first region and the second region, and the visualization includes superimposing a representation of the shape and / or position and / or orientation of at least one of the one or more instruments on the representation of the imaging data.

[0133] Preferably, there is significant spatial overlap between the second region comprising the inserted instruments and the first region comprising the imaging data. Preferably, there is also significant spatial overlap between these two regions and the target region of the medical procedure or simulated medical procedure. Preferably, the overlay includes a representation of the shape, position, and orientation of all instruments inserted into the patient or patient model, as well as a representation of the imaging data. The resulting visualization allows the physician to see in real time how the instruments are positioned within the patient's body or patient model.

[0134] In some embodiments of the system, the visualization includes an overlay that is presented aligned on the corresponding data, or the overlay can be positioned and repositioned during the process to provide the user with a desired view.

[0135] Due to the co-registration of the different data streams, the transitions between them are known. Consequently, visualizations of two or more different data streams can be presented simultaneously, with visualizations of data points corresponding to the same physical features aligned. This allows for straightforward and unambiguous verification of where and how instruments are positioned within the patient or patient model, and whether they risk damaging any surrounding tissue. Alternatively, two visualizations can be positioned independently of one another, without aligning corresponding data points, to obtain a clearer view of the representation of one or more data streams.

[0136] Those skilled in the art will appreciate that more than two different data streams may be displayed simultaneously in an overlay. For example, both model data and tracking data may be overlaid on imaging data.

[0137] In some embodiments of the system, the system latency is equal to or less than 50 ms. Preferably, the system latency is less than 40 ms, more preferably less than 30 ms, and most preferably less than 20 ms. Those skilled in the art will appreciate that system latency can include the time required for raw data generation and acquisition, final pre-processing of the raw data, estimating or calculating the transformations between the various reference frames of the different data streams, the time required to co-register the different data streams based on the obtained transformations, and the duration of the rendering and visualization pipeline.

[0138] To provide real-time visual guidance during medical procedures, the system's latency must be minimal. Given the small size of certain anatomical structures, there is a noticeable delay between the physician's movements. Consequently, the receipt of raw data and its visualization could cause the physician to advance the instrument too far before receiving feedback, resulting in an unintended collision with the anatomy. This risk can be mitigated by keeping the system's maximum total latency on the same order of magnitude as the time it takes the human brain to acquire and process an image. Furthermore, perceptible system latency can lead to physician fatigue and / or frustration.

[0139] In some embodiments of the system, co-registration of the model data, imaging data, tracking data, and orientation data occurs in a continuous or intermittent manner during the process.

[0140] The time required to estimate or calculate the transformation between the various reference frames of different data streams contributes significantly to the overall system latency. Therefore, if the co-registration of the data streams is not updated for every frame displayed, but rather updated in an intermittent manner, the overall system latency and / or the required computing power can be significantly reduced. The intermittent co-registration process can be triggered in a variety of ways. For example, it can occur after a set number of frames have been visualized, or when prompted by the user of the system, or when a certain amount of movement of one of the reference frames is detected.

[0141] In some embodiments of the system, the model data is generated prior to the process.

[0142] The generation of model data typically relies on large amounts of imaging data, which is heavily processed. Therefore, generating model data is a computationally intensive task. Therefore, if model data is generated prior to a medical or simulated medical procedure, the overall system latency and / or required computing power can be significantly reduced.

[0143] In some embodiments of the system, the model data is generated using computed tomography, MRI, or any other suitable modality.

[0144] In some embodiments of the system, the imaging data is generated during a surgical procedure.

[0145] In some embodiments of the system, the imaging data is generated using fluoroscopy or intraoperative MRI (iMRI) or any other real-time imaging method.

[0146] In some embodiments of the system, imaging data is generated in an intermittent manner. In this context, generating imaging data "in an intermittent manner" should be understood as being in contrast to generating imaging data continuously or generating imaging data for each frame to be visualized. When operating in an intermittent manner, the generation of imaging data can be controlled, for example, by the elapsed time interval, the number of frames displayed, a predetermined amount of movement of one of the instruments, a predetermined amount of change between successively acquired data streams, a predetermined amount of change between successive images, or the actions of the surgeon performing the procedure. A person skilled in the art will understand what is occurring and will be able to derive appropriate criteria for determining the time intervals at which the imaging data are generated.

[0147] There are several reasons for generating imaging data intermittently rather than continuously or frame by frame. First, typical imaging techniques, such as fluoroscopy, expose the patient to a non-negligible amount of radiation. By generating imaging techniques only intermittently, the patient's total radiation exposure can be significantly reduced. Second, preprocessing of raw imaging data can be computationally intensive, which can negatively impact the overall system latency. Therefore, if imaging data is generated intermittently, the overall system latency and / or required computing power can be significantly reduced. Third, typical imaging techniques require a finite exposure time. This exposure time can be longer than the desired overall system latency.

[0148] In some embodiments of the system, one or more sensors for generating tracking data are embedded in or fixedly attached to at least one of the one or more instruments. These sensors may include accelerometers, capacitive sensors, inductive sensors, optical sensors, LIDAR, or any other type of sensor capable of generating data that allows estimation of position and / or orientation and / or shape.

[0149] This embedded or fixed attachment of the sensor ensures that the sensor cannot be separated from the device within the patient's body. Furthermore, it ensures that the sensor's position on the device is fixed in both space and time. Alternatively, the sensor can be removably attached to the device, provided that the geometric relationship between the sensor and the device is known or determinable. Thus, there is no need to periodically recalibrate the tracking points generated by the sensor relative to the shape or orientation of the device.

[0150] Preferably, the sensor is embedded in the device. This allows the device to have a smooth outer surface that does not abrade tissue within the patient's body. Furthermore, this ensures that the device surface does not need to exhibit any substantial undulations, gaps, or cracks that could serve as breeding grounds for pathogenic organisms.

[0151] In some embodiments of the system, at least one of the one or more sensors comprises an optical fiber, and / or at least one of the one or more sensors comprises one or more electromagnetic trackers, and / or at least one of the one or more sensors is a distributed sensor that conforms to the shape of the instrument.

[0152] Optical fibers are well-suited for implementation in medical devices. They are small in diameter, lightweight, flexible, chemically inert, and emit no electromagnetic noise. Furthermore, they offer high bandwidth for data transmission and can be used to introduce light sources into patients. When multiple fiber Bragg gratings (FBGs) are distributed along the length of an optical fiber, local strain on the fiber can be measured, allowing the fiber's shape to be reconstructed. Based on the estimated fiber shape, the position, orientation, and shape of the device can be determined.

[0153] Another option is to use an electromagnetic tracker (EMT) embedded in the device. For a detailed explanation of the combined use of distributed FBG sensors and EMT for catheter shape reconstruction, see:

[0154] Ha et al., Robust Catheter Tracking by Fusing Electromagnetic Tracking, Fiber Bragg Grating and Sparse Fluoroscopic Images, IEEE Sensors Journal, vol. 21, no. 20, pp. 23422–23434, DOI: 10.109 / JSEN.2021.3107036.

[0155] In some embodiments of the system, one or more cameras or LIDAR systems have an unobstructed field of view of at least one of the one or more instruments to derive tracking data based on image processing techniques applied to the camera or LIDAR images.

[0156] In some procedures and / or for certain instruments, one or more cameras or LIDAR systems can be positioned to maintain an unobstructed view of the instrument even while it is inserted into the patient or patient model. An advantage of this technology is that it does not require additional sensors to be installed in or on the instrument and can therefore be used with widely available standard instruments. BRIEF DESCRIPTION OF THE DRAWINGS

[0157] Figure 1 An embodiment of a system according to the present disclosure is schematically illustrated.

[0158] Figure 2(a) and 2(b) A simulated medical procedure is schematically illustrated.

[0159] FIG3( a ) shows a patient model used in a simulated cardiac catheterization procedure.

[0160] 3(b)-(f) illustrate various ways of visualizing a simulated medical procedure on the patient model of FIG3(a). DETAILED DESCRIPTION

[0161] The present disclosure will be described with reference to specific embodiments, which are illustrative embodiments of the present disclosure and should not be construed as limiting. It should be understood that the present disclosure is not limited by what has been particularly shown and / or described, and that alternative or modified embodiments can be developed based on the overall teachings of the present disclosure. The accompanying drawings described are merely illustrative and non-limiting.

[0162] Throughout this specification, reference to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the present disclosure. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment, but may be. Furthermore, in one or more embodiments, the particular features, structures, or characteristics may be combined in any suitable manner, as would be apparent to one of ordinary skill in the art from this disclosure.

[0163] Specific features, structures, or characteristics are indicated in the drawings by reference numerals. To avoid overloading the drawings, not every feature is shown in every drawing. Conversely, to avoid overloading the text, not every feature shown in a particular drawing is discussed in the context of that drawing.

[0164] Finally, unless expressly stated to the contrary, the use of ordinal numbers such as "first," "second," etc. in this disclosure in no way implies a hierarchical relationship between the features to which they are applied, whether in terms of importance, position, or timing. These ordinal numbers are only used to distinguish different but similar features, attributes, or structures.

[0165] Figure 1 An embodiment of a system according to the present disclosure is schematically illustrated. Figure 1 In some embodiments, a patient undergoes a medical procedure.

[0166] A first imager 12 provides imaging data 10 of a first region 11 of a patient's body 32. The first imager 12 may be, for example, an ultrasound, fluoroscope, or intraoperative magnetic resonance imaging (iMRI) device. Preferably, the first region 11 comprises the target area of ​​a medical procedure. The first imager 12 may be any transducer used to convert a type of energy signal into an image signal to generate the imaging data 10 during a medical procedure.

[0167] A second imager 34 provides pose data 30 of the patient's body 32 pertaining to the third area 31. The second imager can be, for example, a camera, or more preferably two cameras arranged to obtain a stereoscopic view of the patient's body 32. Preferably, the pose data 30 includes at least data relating to the position and orientation of a body part comprising the target area of ​​the medical procedure. Preferably, the pose data 30 includes data regarding the position and orientation of the patient's torso and head.

[0168] One or more instruments 22 are inserted into the second region 21 of the patient's body 32. Instruments 22 can include any type of surgical or non-surgical tool, such as a catheter, cannula, needle, endoscope, cutter, grasper, clamp, dilator, etc. Instruments 22 can be any type of flexible instrument that is inserted into the patient and tracked during the medical procedure. In some embodiments, instruments 22 can have a nonlinear shape or orientation during the medical procedure. Preferably, the second region 21 at least partially overlaps with the first region 11. Preferably, the second region 22 comprises the target area of ​​the medical procedure.

[0169] At least one of the instruments 22 is equipped with a device for generating tracking data 20. The tracking data 20 for a single instrument includes the positions of at least three non-collinear tracking points. Knowing the positions of the three non-collinear points allows the position and orientation of the instrument to be reconstructed without any doubt. In addition, the three non-collinear points can allow the shape of the instrument to be reconstructed. In the case where the instrument is flexible or includes joints, more tracking points may be required to reconstruct the position, orientation or shape of the instrument. Alternatively, when the shape of the instrument is known, the tracking data 20 for a single instrument may include the positions of one or more tracking points and the values ​​of the roll, pitch and yaw angles. Similarly, in the case where the instrument is flexible or includes joints, more tracking points and more angles may be required to reconstruct the position, orientation or shape of the instrument. The present disclosure will further discuss specific techniques for generating tracking data.

[0170] Imaging data 10, tracking data 20, and posture data 30 are received by a processor 50. Each data stream corresponding to the imaging data 10, tracking data 20, and posture data belongs to a reference frame. The reference frame is related to the position, orientation, and properties of the device generating the data stream. There may be important differences between different reference frames. For example, the imaging data 10 and tracking data 20 may be translated or rotated relative to each other, or may differ in scale. In addition, the data streams may be sampled continuously or intermittently at different frequencies. Furthermore, data streams collected continuously or intermittently may be sampled at different frequencies. The collection or sampling of data streams may be continuous or intermittent. For example, fluoroscopy is not always continuous, so the image may remain stationary for a certain period of time (e.g., seconds or minutes). For example, shape sensing is preferably real-time. The processor 50 co-registers at least two of the three different data streams to a single reference frame. Where necessary, the imaging data 10, tracking data 20, and posture data 30 are rotated, translated, and scaled—possibly in both the spatial and temporal domains—to properly align the three data streams. The co-registration process will be further described in detail in this disclosure.

[0171] Finally, two or more of the co-registered data streams are simultaneously visualized on a display 60. This disclosure further gives specific examples of simultaneously visualizing data.

[0172] Figure 2(a) and 2(b) An example of a simulated medical procedure is schematically illustrated. During this simulated medical procedure, when the instrument 22 is a (for example virtual) fetoscope 22 and is inserted into a patient model 33. Figure 2(a) and 2(b) In the embodiment of FIG. 3 , a patient model 33 is in the form of a simple cubic cavity. Fetoscope 22 is inserted into patient model 33 through cannula 22′ at incision point 35. Fetoscope 22 and incision point 35 are provided with respective electromagnetic (EM) trackers 25 and 36. Patient model 33 is placed on EM field generator 26, which is used to track the positions of EM trackers 25 and 36.

[0173] In some embodiments, multiple reference frames exist: the incision point reference frame {I}, the cannula reference frame {C}, the fetoscope reference frame {F}, the EM field generator reference frame {EM}, and the display reference frame {D}. To provide accurate visual guidance, data from these different reference frames must be co-registered. This requires knowledge of the transformation matrices between the different reference frames.

[0174] To calculate the transformation matrix between the electromagnetic field generator 26 and the display 60 The display 60 and the electromagnetic field generator 26 are each equipped with a tracker or marker 37. The camera 34 is positioned so that the camera 34 can observe both markers; the camera 34 itself has a reference frame {CAM}. Since one of the markers 37 is fixedly attached to the EM field generator 26, the transformation matrix between the EM field generator 26 and the marker 37 is By tracking the two markers 37 with the camera 34, the transformation matrix can be calculated and . Transformation Matrix and are the transformations from {CAM} to {D} and {EM} respectively. Then, the transformation matrix between the electromagnetic field generator reference frame {EM} and the display reference frame {D} can be obtained as Known transformation matrix This allows data originating from the EM generator reference frame {EM} to be co-registered in the reference frame of the display 60. In this exemplary embodiment, the data originating from the EM field generator reference frame {EM} may be pose data of a patient phantom 33 fixedly attached to the generator 26. By continuously tracking the markers 37, the transformation matrix is ​​always known, and calibration of the system is possible whenever the display 60 or the EM generator 26 is moved.

[0175] In the illustrated setup, data originating from the reference frame of the incision point, cannula, or fetoscope may first be transformed to the reference frame of the EM field generator 26. The incision point EM tracker 36 provides the transformation matrix The fetoscopic EM tracker 25 provides the transformation matrix The position and orientation of the cannula 22' and the fetoscope 22 are visualized relative to the incision point 35. When visualizing the 3D augmented reality (AR) content on the display 60, the incision point remains stationary and only the rotation and insertion of the cannula 22' and the fetoscope 22 are displayed. The rotation is preferably calculated in the display reference frame {D} to ensure that the AR instrument is always parallel to the real instrument. Since the fetoscope 22 is inserted through the cannula 22', the two instruments are collinear and therefore their rotation is the same. Therefore, the transformation matrix from the reference frame of the fetoscope and cannula to the reference frame of the display can be expressed as

[0176] The insertion length of the instrument is calculated as the distance between the calibrated incision point 35 and the point defined by the fetoscopic EM tracker 25. In some embodiments, the incision point can be registered at the beginning of the procedure using the EM tracker, for example, by registering a single point in the same {EM} reference frame. These transformations are expressed in the EM field generator reference frame {EM}. For the fetoscopic 22 and cannula 22', the distances obtained are the instrument lengths l outside the patient, l and l respectively. F,o and l F,oBy knowing the total length of these instruments, the insertion length is calculated as l F,i =l F –l F,o and l C,i =l C –l C,o , where l F and l C are the total lengths of the fetoscope 22 and the cannula 22', respectively.

[0177] The fetoscope 22 consists of two parts, a rigid shaft and a flexible tip. The flexible tip includes 10 cylinders, each of which can be rotated about the local z-axis. This property is used to visualize the bending of the fetoscope. The bending range of the fetoscope is 90 degrees. The bending angle is determined by measuring the input of the physician. It is assumed that the virtual fetoscope 22 is represented according to a constant curvature model, in which the rotation angle of each cylinder is equal and in which the flexible tip can only bend in one plane. Under this assumption, the bending angle is divided by the number of cylinders and given to each cylinder as a rotation. Since each successive cylinder is a sub-cylinder of the previous cylinder, the resulting overall bending angle of the 10th cylinder will be the same as the bending angle of the fetoscope: θ s =θ F / n s , where θ s and θ F are the rotation angle of each segment and the overall bending angle of the flexible tip, respectively, and n s is the number of segments that make up the flexible tip. The orientation of the plane in which the virtual fetoscope 22 can bend is given by the fetoscopic EM sensor 25 .

[0178] The insertion length, orientation, and bending angle of the fetoscope 22 (from which the shape is known) obtained, which constitute the tracking data, are transformed into the display reference frame. The real-time 2D fetoscope view - the image obtained by the tip of the fetoscope 22 - is the imaging data in this embodiment. This imaging data is also transformed into the display reference frame and is set as the background for the display visualization. To generate a 3D view, two virtual cameras are placed in the scene, with a horizontal offset of 65 mm to capture the virtual content. The offset corresponds to the average interpupillary distance. The offset can be adjusted based on user preferences. The two views from the cameras are stacked side by side, thereby generating a 3D image input for an autostereoscopic display. If the 2D fetoscope image is to be visualized on a 3D display, such as an autostereoscopic display or VR glasses, two additional virtual cameras are placed in the renderer. The image from one of these cameras will be visualized for the user's left eye, and the image from the other camera will be visualized for the user's right eye. By setting the two cameras to the same position and orientation, the same view can be provided to the left and right eyes even when using a 3D display, which results in the perception of a 2D image. Alternatively, only one additional virtual camera can be placed in the scene, and the image of the camera can be copied for the user's left and right eyes. Alternatively, the original fetoscopic image can be visualized for both the user's left and right eyes. In addition, the visualization of the instrument can be positioned outside the visualization of the imaging data or on top of the visualization of the imaging data because the user can freely move it in the scene. This mixed 2D and 3D visualization is preferably presented on an autostereoscopic display.

[0179] Figure 3(a) shows a picture of a patient model 33. In the embodiment of Figure 3(a), the patient model is a box trainer with a silicon coronary vessel model for simulation of cardiac catheterization. Figures 3(b)-3(f) All refer to the same embodiment.

[0180] In an exemplary embodiment in which instrument 22 is a catheter, catheter 22 is inserted into a silicon vascular model. Catheter 22 includes a distributed fiber Bragg grating (FBG) sensor that returns tracking data including the 3D positions of tracking points 23. The tracking data includes at least three or more tracking points. Preferably, the spacing of the tracking points along the length of the catheter is adapted to the mechanical properties of the catheter, so that every bending pattern of the catheter can be tracked and visualized. For example, the tracking data may include tracking points spaced apart along the length of the catheter. The distance between tracking points can vary depending on the instrument and the needs of the procedure. In some embodiments, the tracking points may be spaced 1 mm apart. Preferably, the tracking points are not located along the centerline of the catheter, but rather are distributed at lateral distances from the centerline on either side or at other sides. This distribution of tracking points ensures that three non-collinear tracking points are available at all times, even when the catheter is not bent. However, even if only a single optical fiber is located along the centerline of the catheter, it will return to a non-collinear point upon bending and to a collinear point when straightened, making this arrangement useful for determining the position, orientation, and shape of the catheter.

[0181] In some embodiments, the device, which is a flexible instrument, is introduced into a target area for the medical procedure. The insertion portion of the flexible instrument is tracked in real time during the medical procedure to generate tracking data. The tracking data includes tracking of bending of the flexible instrument during the medical procedure.

[0182] Typically, intraoperative imaging data for cardiac catheterization is provided by a C-arm fluoroscope, which provides 2D fluoroscopic images. For the purpose of simulating catheterization, a patient model 33 and a chest model can be used to simulate fluoroscopic images. In Figure 3 (b), this simulated fluoroscopic image is visualized. Based on the 3D position of the tracking point 23 obtained, the imaging data and the tracking data of the catheter are co-aligned, and a visualization of the catheter 22 is added. The visualization of the catheter shows the position, orientation, and shape of the catheter within the coronary vessels. Preferably, the view can be rotated along the x, y, and z axes to simulate the movement of the C-arm fluoroscope. In some embodiments, the first imager is configured to generate imaging data for forming a real-time image of the target area of ​​the medical procedure.

[0183] If the 2D fetoscopic image is to be visualized on a 3D display (such as an autostereoscopic display or VR glasses), two virtual cameras are placed in the renderer. In some other embodiments, the 3D display can be a traditional stereoscopic display, an autostereoscopic display, a head-mounted display, a binoculars of a stereo microscope or a holographic display. The image of one of these virtual cameras will be visualized for the user's left eye, and the image of the other camera will be visualized for the user's right eye. By setting the two cameras to the same position and orientation, even when using a 3D display, the same view can be provided to the left eye and the right eye, which obtains the perception of a 2D image. In this case, the visualization of the catheter is also converted into a 3D visualization.

[0184] Model data of the coronary vessels can be obtained through preoperative imaging, for example, by means of CT 3D reconstruction. Figure 3(c) shows a visualization of a 3D model of the patient model. Preferably, the color, transparency, orientation and scale of the visualization can be changed based on the user's preferences. The tracking data of the catheter is co-registered with the model data and visualized in the same scene. If the 3D model data and 3D tracking data are to be visualized on a 3D display, two virtual cameras are placed in the renderer. The image of one of these cameras will be visualized for the user's left eye, and the image of the other camera will be visualized for the user's right eye. A translation is introduced between the positions of the two cameras. Preferably, the translation can be adjusted based on the user's interpupillary distance (IPD). Preferably, the default translation between the positions of the two virtual cameras is 65mm, which corresponds to the average IPD of humans. Because the contents of the left and right images are offset relative to each other, the user perceives a 3D image.

[0185] In some embodiments, the model data can be used to generate a 3D model image of a target area or target organ for a medical procedure. A processor of the system can be used to co-register real-time tracking data with pre-stored model data. During the medical procedure, the tracking data is used to provide a 3D image rendering of an instrument corresponding to the 3D model image rendered using the model data. The 3D image is configured to provide placement of the instrument in the target area or target organ during the medical procedure. The 3D image, combined with a 2D image simultaneously rendered on the 3D display, can provide more accurate positioning of the instrument during the medical procedure.

[0186] It is also possible to combine the two visualization options mentioned above in a single visualization. In Figure 3(d), the background renderer includes a 2D visualization of 2D imaging data that is co-registered with the 3D tracking data. The foreground renderer includes a 3D visualization of 3D model data that is co-registered with the 3D tracking data. Alternatively, the two visualizations can be displayed side by side, as shown in Figure 3(e). This visualization mode is useful for avoiding possible occlusions. Also in this visualization mode, the 3D model data can be rotated and scaled, or aligned with the imaging data, according to user preferences. Figure 3(f) illustrates another visualization possibility. On the left, the foreground renderer includes a visualization of the posture data of the patient model, which can be captured using a 2D or 3D camera. The tracking data is co-registered with the posture data and is visualized on the same image. On the right, the background renderer includes a 2D visualization of the 2D imaging data that is co-registered with the tracking data. Preferably, the user can freely switch between the viewing modes shown.

[0187] According to one aspect of the present disclosure, a system for providing visualization during a medical procedure of a patient or a simulated medical procedure of a patient model is disclosed, the system comprising:

[0188] means for receiving imaging data of a first region of a patient's body or a patient model;

[0189] means for receiving tracking data of one or more instruments inserted into a second region of the patient's body or a patient model;

[0190] The system is characterized in that it comprises:

[0191] means for receiving model data of a fourth region of the patient's body or of a model of the patient, and processing means adapted to co-register the tracking data and the model data; and

[0192] a display for simultaneous visualization of the co-registered data and the imaging data;

[0193] And it is characterized by

[0194] The tracking data comprises three or more tracking points, wherein the three tracking points are not collinear; and / or one or more tracking points and roll, pitch, and yaw angles or a set of Euler angles or a set of quaternions or an angle-axis representation or a rotation matrix or a homogeneous transformation matrix; and / or a point cloud representation for at least one of the one or more instruments;

[0195] The tracking data is used to determine the position and / or shape and / or orientation of at least one of the one or more instruments; and

[0196] The visualization includes a representation of the shape and / or position and / or orientation of at least one of the one or more instruments.

[0197] Preferably, the system comprises:

[0198] means for receiving posture data of a patient's body or a patient model, said posture data belonging to a third area;

[0199] processing means for co-registering imaging data, tracking data and pose data;

[0200] A display for simultaneously visualizing two or more of the first, second, and third regions, and co-registered data pertaining to the simultaneously visualized regions.

[0201] According to yet another aspect of the present disclosure, a method for providing visualization during a medical procedure of a patient or a simulated medical procedure of a patient model is disclosed, the method comprising the steps of:

[0202] receiving image data of a first region of a patient's body or a patient model;

[0203] receiving tracking data of one or more instruments inserted into a second region of the patient's body or a patient model;

[0204] The method is characterized in that it comprises:

[0205] receiving model data of a patient's body or a patient model;

[0206] Co-register tracking data and model data;

[0207] Simultaneous visualization of imaging and co-registered data;

[0208] And it is characterized by

[0209] The tracking data comprises three or more tracking points, wherein the three tracking points are not collinear; and / or one or more tracking points and roll, pitch, and yaw angles or a set of Euler angles or a set of quaternions or an angle-axis representation or a rotation matrix or a homogeneous transformation matrix; and / or a point cloud representation for at least one of the one or more instruments;

[0210] The tracking data is used to determine the position and / or shape and / or orientation of at least one of the one or more instruments; and

[0211] The visualization includes a representation of the shape and / or position and / or orientation of at least one of the one or more instruments.

[0212] In some embodiments of the method, the method further comprises the following steps:

[0213] receiving model data of a fourth region of the patient's body or a model of the patient;

[0214] Co-register imaging data, tracking data, and pose data;

[0215] simultaneously visualizing two or more of the first, second, third, and fourth regions, and the co-registered data pertaining to the simultaneously visualized regions;

[0216] Reference numerals

[0217] Imaging data 10

[0218] First Area 11

[0219] Imager 12

[0220] Tracking data 20

[0221] Second Area 21

[0222] Equipment 22

[0223] Tracking point 23

[0224] Instrument electromagnetic tracker 25

[0225] Electromagnetic field generator 26

[0226] Posture data 30

[0227] Third Area 31

[0228] Patient's body 32

[0229] Patient Model 33

[0230] Camera 34

[0231] Incision point 35

[0232] Incision point electromagnetic tracker 36

[0233] Tracker or marker 37

[0234] Processor 50

[0235] Display 60

Claims

1. A system for providing visualization during a medical procedure on a patient or a simulated medical procedure on a patient model, the system comprising: a. a device for receiving imaging data (10) of a first region (11) of a patient's body (32) or of said patient model (33); b. means for receiving tracking data (20) of one or more instruments (22) inserted into a second region (21) of the patient's body (32) or of the patient model (33); Its characteristics are: c. The system comprises means for receiving model data of a fourth region of the patient's body (32) or of the patient model (33), and processing means adapted to co-register the tracking data (20) and the model data; d. The system comprises a display (60) for simultaneous visualization of the co-registered data and the imaging data (10); e. The tracking data comprises three or more tracking points (23), wherein the three tracking points (23) are not collinear; and / or one or more tracking points (23) and roll, pitch and yaw angles or a set of Euler angles or a set of quaternions or an angle-axis representation or a rotation matrix or a homogeneous transformation matrix; and / or a point cloud representation for at least one of the one or more instruments (22); f. The tracking data (20) is used to determine the position and / or shape and / or orientation of at least one of the one or more instruments (22); and g. The visualization includes a representation of the shape and / or position and / or orientation of at least one of the one or more instruments (22).

2. The system according to claim 1, wherein: The system further comprises means for receiving posture data (30) of the patient body (32) or of the patient model (33), the posture data (30) belonging to a third area (31), wherein the processing means is adapted to further co-register the posture data (30), wherein the display (60) is adapted for further visualization of the posture information (30).

3. The system according to claim 1, wherein: a. wherein the model data and the tracking data (20) include 3D data; b. wherein the shape of the device (22) is determined based on tracking data including time-related data; c. wherein the display (60) is a 3D display; as well as d. wherein the representation of the model data and the representation of the shape and / or position and / or orientation of the one or more instruments (22) are 3D representations.

4. The system according to claim 3, characterized in that The imaging data (10) is 2D data that is processed to allow visualization on the 3D display.

5. System according to any one of the preceding claims, characterized in that There is at least a partial overlap between two or more regions, and wherein the visualization comprises a superposition of representations of data originating from the two or more regions.

6. The system according to claim 5, characterized in that The overlays are presented aligned on the corresponding data, or wherein the overlays can be positioned and repositioned during the process to provide the user with a desired view.

7. System according to any one of the preceding claims, characterized in that The system delay is equal to or less than 50ms.

8. System according to any one of the preceding claims, characterized in that Co-registration of the model data, imaging data (10), tracking data (20) and orientation data occurs in a continuous or intermittent manner during the process.

9. System according to any one of the preceding claims, characterized in that The model data is generated prior to the process.

10. System according to any one of the preceding claims, characterized in that The imaging data (10) is generated during the surgical procedure.

11. The system according to claim 10, wherein: The imaging data (10) is generated in an intermittent manner.

12. System according to any one of the preceding claims, characterized in that One or more sensors for generating the tracking data (20) are embedded in or fixedly attached to at least one of the one or more instruments (22).

13. The system according to claim 12, wherein: At least one of the one or more sensors comprises an optical fiber, and / or wherein at least one of the one or more sensors comprises one or more electromagnetic trackers, and / or wherein at least one of the one or more sensors is a distributed sensor that conforms to the shape of the instrument (22).

14. System according to any one of the preceding claims, characterized in that One or more cameras or LIDAR systems have an unobstructed field of view of at least one of the one or more instruments (22) to derive the tracking data (20) based on image processing techniques applied to the camera (34) or LIDAR images.

15. A method for providing visualization during a medical procedure on a patient or a simulated medical procedure on a patient model, the method comprising the steps of: a. Receiving imaging data (10) of a first region (11) of a patient body (32) or a model of the patient (33); b. receiving tracking data (20) of one or more instruments (22) inserted into a second region (21) of the patient's body (32) or the patient model (33); The method is characterized in that: c. The method comprises the step of receiving model data of the patient body (32) or the patient model (33); d. The method comprises the step of co-registering the tracking data (20) and the model data; e. The method comprises the step of simultaneously visualizing the imaging data (10) and the co-registered data; And it is characterized by f. the tracking data (20) comprising three or more tracking points (23), wherein the three tracking points (23) are not collinear; and / or one or more tracking points (23) and roll, pitch and yaw angles or a set of Euler angles or a set of quaternions or an angle-axis representation or a rotation matrix or a homogeneous transformation matrix; and / or a point cloud representation for at least one of the one or more instruments (22); g. The tracking data (20) is used to determine the position and / or shape and / or orientation of at least one of the one or more instruments (22); and h. The visualization includes a representation of the shape and / or position and / or orientation of at least one of the one or more instruments (22).

16. The method according to claim 15, characterized in that The method further comprises receiving posture data (30) of the patient body (32) or the patient model (33), the posture data (30) belonging to a third region (31), wherein the method further comprises co-registering the posture data (30), and the method further comprises visualizing the posture information (30).

17. The method according to claim 15, characterized in that a. wherein the model data and the tracking data (20) include 3D data; b. wherein the display (60) is a 3D display; as well as c. wherein the representation of the model data and the representation of the shape and / or position and / or orientation of the one or more instruments (22) are 3D representations.

18. The method according to claim 15, characterized in that The imaging data (10) is 2D data that is processed to allow visualization on the 3D display.

19. The method according to any one of claims 15 to 18, characterized in that There is at least a partial overlap between two or more regions, and wherein the visualization comprises a superposition of representations of data originating from the two or more regions.

20. The method according to claim 19, characterized in that The overlays are presented aligned on the corresponding data, or wherein the overlays can be positioned and repositioned during the process to provide the user with a desired view.

21. The method according to any one of claims 15 to 20, characterized in that The system delay is equal to or less than 50ms.

22. The method according to any one of claims 15 to 21, characterized in that Co-registration of the model data, imaging data (10), tracking data (20) and orientation data occurs in a continuous or intermittent manner during the process.

23. The method according to any one of claims 15 to 22, characterized in that The model data is generated prior to the process.

24. The system according to any one of claims 15 to 23, characterized in that The imaging data (10) is generated during the surgical procedure.

25. The method according to claim 24, characterized in that The imaging data is generated in an intermittent manner.

26. The method according to any one of claims 15 to 25, characterized in that One or more sensors for generating the tracking data (20) are embedded in or fixedly attached to at least one of the one or more instruments (22).

27. The method according to claim 26, characterized in that At least one of the one or more sensors comprises an optical fiber, and / or wherein at least one of the one or more sensors comprises one or more electromagnetic trackers (25, 36), and / or wherein at least one of the one or more sensors is a distributed sensor that conforms to the shape of the instrument (22).

28. The method according to any one of claims 15 to 27, characterized in that One or more cameras or LIDAR systems have an unobstructed field of view of at least one of the one or more instruments (22) to derive the tracking data (20) based on image processing techniques applied to the camera (34) or LIDAR images.

29. A method for providing visualization of a target area on a patient during a medical procedure, comprising: receiving imaging data (10) and model data of the target area; receiving tracking data (20) of a flexible device (22); generating a 2D image based on the imaging data (10); Characterized in that the method comprises: co-registering the model data and the tracking data (20) to generate a 3D image; and visualizing the 2D image and the 3D image simultaneously on a 3D display; wherein the tracking data (20) is used to determine the position and / or shape and / or orientation of at least one of the one or more instruments (22).

30. The method according to claim 29, wherein Also includes: receiving posture data (30) of the patient, wherein the posture data (30) is used to determine the position and orientation of the patient; The pose data (30) is co-registered to the model data and the tracking data (20), wherein the 3D image is displayed on the 3D display in a parallel orientation to the patient.

31. The method according to any one of claims 29 and 30, characterized in that The 2D image and the 3D image are displayed side by side or overlapped with each other on the 3D display.

32. The method according to any one of claims 29 to 31, characterized in that The co-registration of the model data, imaging data (10), and tracking data (20) occurs in a continuous or intermittent manner during the process.

33. The method according to claim 32, characterized in that The co-registration of the model data, pose data (30), and tracking data (20) occurs in a continuous or intermittent manner during the process.

34. The method according to any one of claims 29 to 33, characterized in that The displayed latency is equal to or less than 50ms.

Images