Systems and methods for medical navigation

Abstract

To provide a guidance system for EUS navigation that can be used even by a beginner on the basis of image fusion.SOLUTION: A system for medical navigation includes: an echo endoscope configured to acquire an ultrasonic signal and / or a video signal in real time; a tracking system or device configured so as to spatially track a distal end of the echo endoscope; a user interface and a display; and calculation and storage means including preoperative radiological data on a patient. The calculation and storage means is configured so as to perform registration between real time intraoperative data and the preoperative radiological data. The display is configured so as to provide a user with a navigation view in which the real time intraoperative data view is integrated into the preoperative radiological data on the basis of the registration.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention is in the field of medical devices and systems for medical navigation including registration of several types of data. A first object of the present invention is a device for medical navigation including registration of radiological and ultrasound data. A second object of the present invention is a method for medical navigation including registration of radiological and ultrasound data. [Background technology]

[0002] Endoscopic ultrasound (EUS) is an important and increasingly used medical procedure for imaging and diagnosis of the upper gastrointestinal tract. Essentially, EUS is a gastroscope system with an ultrasound (US) sensor attached to its tip, and the endoscope tower contains the US unit. Both video and US feeds are displayed to the user. EUS provides high-resolution, minimally invasive, and low-cost multimodal imaging in close proximity to both surface and subsurface structures. However, EUS is a challenging technique for endoscopic sonographers to master and is highly operator-dependent.

[0003] This difficulty stems primarily from two navigational challenges. First, EUS probes generate images in a small spatial window (typically less than 10 cm wide). This provides excellent spatial resolution, but requires the endoscopic sonographer to mentally understand where the probe is relative to the patient, which is challenging. The second difficulty involves mechanical control: EUS requires a flexible scope that is mechanically controlled at the proximal end. This makes precise positioning and localization of the EUS tip challenging.

[0004] These difficulties combine to make EUS very challenging for non-experts, and it takes many years (>5000 procedures) to reach a high level of proficiency. Previously, fusion of pre-procedural computed tomography (CT) or magnetic resonance imaging (MRI) slice sets with live US images during the procedure has been proposed. These systems rely on the US probe being tracked in space and involve a registration step between the CT or MRI slice sets and the live US images. Siemens ACUSON S3000 TM In commercially available navigation systems with transperitoneal US probes, such as ultrasound systems, registration is completed as follows: the sonographer manually completes the registration by selecting the correct anatomical landmarks in the CT or MRI image set, displaying the anatomical landmarks on the patient using the US system, and then selecting the correct location of the landmarks in the displayed image. The system then registers the two images. This registration process is tailored for radiologists who are familiar with both the US system and the CT or MRI slice set: it requires the user to understand the CT or MRI slice set and to be able to display images of the requested anatomical region using the US system.

[0005] Although this has proven efficient for hand-controlled US probes, this approach is impossible to implement in the EUS environment. Indeed, navigating an EUS scope to a predefined area of ​​the body presents serious difficulties requiring expert skills, and a guidance system is most needed for novices.

[0006] Document US2015 / 0086956A1 describes a device including a co-registration and navigation system in which 3D and / or 2D ultrasound images are displayed together with virtual images of the patient and / or CT or MRI scans or other similar imaging techniques used in the medical field.

[0007] This system is useful for offline training of operators based on pre-registered US and radiological data. It does not provide a technical solution for real-time registration, and is especially not useful for real-time navigation of EUS scopes.

[0008] Document US2019 / 0307516 describes a system for guiding an instrument through an area of ​​a patient and a method for using the system.

[0009] Nevertheless, this system is not adapted to overcome the difficulty of guiding an EUS scope to a given location and does not solve the technical problems mentioned above. [Prior art documents] [Patent documents]

[0010] [Patent Document 1] US Patent Application Publication No. 2015 / 0086956 [Patent Document 2] US Patent Application Publication No. 2019 / 0307516 Summary of the Invention [Problem to be solved by the invention]

[0011] The aim of this disclosure is to overcome these limitations by introducing a novel guidance system for EUS navigation that is based on image fusion and is user-friendly even for beginners. [Means for solving the problem]

[0012] The object of the present invention is to provide a system for medical navigation comprising: an echoendoscope configured to acquire real-time ultrasound and / or video signals; A tracking system or device configured to spatially track the distal end of the echoendoscope; · User interface and display; Computing and storage means containing the patient's preoperative radiological data Includes: the computing and storing means is configured to perform registration between the real-time intraoperative data and the preoperative radiological data; The display is configured to provide a user with a navigation view that integrates real-time intraoperative data views within preoperative radiological data based on registration. It is characterized by:

[0013] The present disclosure presents a technical solution that overcomes the above difficulties and provides intraoperative guidance for EUS procedures.The purpose of the present disclosure is to describe a method and system for providing medical navigation.

[0014] An echoendoscope refers to an endoscope that includes an ultrasound transducer and is adapted to acquire real-time ultrasound signals. For example, an echoendoscope is adapted to acquire ultrasound images. An echoendoscope can also be configured to acquire real-time optical or video signals.

[0015] Real-time intraoperative data refers to data acquired during an echoendoscopic procedure. Real-time intraoperative data may include ultrasound and / or optical data acquired by an endoscope and / or tracking data acquired by a tracking device.

[0016] User interface and display means an interface adapted to provide information to a user and to gather information or commands from a user.

[0017] Preoperative data refers to data obtained prior to the endoscopic ultrasound examination.

[0018] Radiological data refers to images or other medical data acquired by a medical device. For example, radiological data can be provided by a 3D imaging tool, such as a computed tomography (CT) image, a magnetic resonance imaging (MRI), or a positron emission tomography (PET) image.

[0019] Registration between real-time intraoperative data and preoperative radiological data means identifying given anatomical landmarks or structures in both the intraoperative and preoperative data.

[0020] That is, the registration between real-time intraoperative data and preoperative data is a transformation of spatial coordinates, transforming the spatial coordinates of the endoscope or tracking system into the spatial coordinates of the patient.

[0021] Navigation view refers to a representation of intraoperative and / or preoperative radiological data. Thanks to the registration performed by the system according to the invention, intraoperative and preoperative data can be displayed simultaneously.

[0022] Advantageously, a medical navigation system according to the present invention performs such registration in an efficient and reliable manner, with minimal disruption to clinical workflow.

[0023] Advantageously, the medical navigation system according to the present invention displays an efficient navigation view that is easy to use even for an unskilled operator. The medical navigation view according to the present invention allows an unskilled operator to navigate the echoendoscope towards a predefined anatomical region.

[0024] For example, the system according to the present invention is configured to calculate the real-time position and orientation of the ultrasound signal relative to the radiological data, i.e., the system is adapted to transform the spatial coordinate system of the endoscope or tracking system into the spatial coordinate system of the patient, allowing real-time registration of the ultrasound signal and the radiological data.

[0025] Advantageously, the system for medical navigation according to the present invention is easy to use and does not require a highly skilled operator.

[0026] According to one embodiment, the calculation and storage means of the medical navigation system according to the invention are adapted to perform a first registration calculation and a registration refinement.

[0027] The system for medical navigation according to the invention may also have one or more of the following characteristics, considered individually or according to all technically possible combinations thereof: - The ultrasound signal is an ultrasound image; - the intraoperative data comprises: ultrasound signals acquired by the echoendoscope and / or video signals acquired by the echoendoscope and / or tracking signals acquired by the tracking device; - the computing means is configured to compute a real-time position and orientation of the ultrasound signal relative to the preoperative radiological data; - the display is configured to provide a user with a navigation view that integrates an ultrasound real-time view within preoperative radiological data based on said position and orientation; - the tracking system or device is further configured to define a tracking reference frame and provide a position and orientation of the distal end of the endoscope within said tracking reference frame; - The tracking system includes an electromagnetic tracking device or a fiber optic tracking device; - the tracking system includes means for selecting landmarks in the ultrasound signal and for identifying and marking said landmarks on the preoperative data; - the computing and storing means is configured to perform registration between the ultrasound signal and the preoperative data, said registration being based on selected landmarks; - The registration is further based on one additional piece of anatomical information; - Additional anatomical information includes anatomical axes or landmarks; - Additional anatomical information includes the location and orientation of the esophagus; - the system further includes a tracked pointer that is tracked by the tracking device; - the system is configured to perform a first registration calculation and a registration refinement; - the system is configured to perform a first registration calculation including localizing anatomical landmarks, identifying patient axes, and registering based on at least the anatomical landmarks and at least the patient axes, the anatomical landmarks being internal or external; - the system is configured to perform registration refinement, including ultrasound anatomical landmark localization, optical landmark localization, and locally-rigid alignment between the ultrasound anatomical landmarks and the optical landmarks; - the calculation and storage means is configured to perform a first registration calculation and a registration refinement; - the computing and storing means is configured to perform a first registration calculation including localizing external anatomical landmarks, identifying patient axes, and performing registration based on at least the external anatomical landmarks and at least the patient axes; The computing and storing means is configured to perform registration refinement including localization of ultrasound anatomical landmarks, localization of optical landmarks, and local rigid body alignment between the ultrasound anatomical landmarks and the optical landmarks.

[0028] Another object of the present invention is to provide a method for producing a method for manufacturing a semiconductor device comprising the steps of: - acquiring real-time ultrasound and / or video signals using an echo endoscope; - tracking the distal tip of the echoendoscope; - receiving and storing preoperative radiological data of the patient; - performing registration between the intraoperative data and the preoperative radiological data; - displaying a navigation view that integrates a real-time intraoperative data view within the preoperative radiological data based on the registration; A method for medical navigation, comprising:

[0029] The method for medical navigation according to the invention may also comprise one or more of the following steps, considered individually or according to all technically possible combinations thereof: - acquiring ultrasound signals or images with an ultrasound transducer or echoendoscope; - tracking the distal end of the ultrasound transducer; - calculating the position and orientation of the ultrasound signal or image relative to the preoperative radiological data; - displaying a navigation view that integrates ultrasound signals within preoperative radiological data based on the position and orientation. - the method includes registering the ultrasound signal and the preoperative radiological data; - the registration step includes a first registration calculation and a registration refinement; - the first registration calculation includes a step of localizing anatomical landmarks, a step of identifying patient axes, and a step of registration based on at least the anatomical landmarks and at least the patient axes, wherein the anatomical landmarks are external or internal; - the registration refinement includes a step of localizing ultrasound anatomical landmarks, a step of localizing optical landmarks, and a step of local rigid body alignment between the ultrasound anatomical landmarks and the optical landmarks; - performing an echoendoscopic procedure using the displayed navigation view.

[0030] Other characteristics and advantages of the present invention will become apparent from the description given below, given for guidance purposes and in no way limiting, with reference to the accompanying drawings, in which: [Brief explanation of the drawings]

[0031] [Figure 1] FIG. 1 is a diagram showing an outline of a medical navigation system according to the present invention. [Figure 2] FIG. 2 is a diagram showing an outline of the medical navigation method according to the present invention. [Figure 3] FIG. 3 shows a schematic diagram of the different spatial coordinate systems involved in the implementation of the method according to the invention. [Figure 4] FIG. 4 shows an overview of the initial registration calculation steps involved in the method according to the invention. [Figure 5] FIG. 5 shows an overview of the registration refinement steps involved in the method according to the invention. DETAILED DESCRIPTION OF THE INVENTION

[0032] Referring to FIG. 1 , a system for medical navigation described herein includes an echo-endoscope EES, a tracking system or device adapted to spatially track the distal end of the echo-endoscope, a user interface UI, a display D, and a computing unit CU and a storage unit SU containing at least an algorithm and a patient's preoperative radiological data RD. The computing unit and the storage unit may be fused into a computing and storage unit. The echo-endoscope EES is adapted to acquire ultrasound signals, such as ultrasound images. The system is adapted to calculate a live position and orientation of the real-time ultrasound image relative to the preoperative radiological data and display a navigation view to the user that integrates the live ultrasound view within the preoperative radiological data based on the position and orientation.

[0033] Referring to FIG. 2, the medical navigation method described herein includes: - providing a system for medical navigation comprising an echoendoscope, a tracking system or device adapted to spatially track the distal end of the echoendoscope, a user interface, a display, and computing and storage means including at least an algorithm; - Step STO of importing the patient's preoperative radiological data; - Step ACQ, which starts the echo-endoscopic procedure on the patient; - Step TRA to track the distal end of the endoscope; - a step COMP of calculating the live position and orientation of the live ultrasound image relative to the preoperative radiological data; - a step DISP of displaying to the user a navigation view integrating an ultrasound live view within the preoperative radiological data based on said position and orientation; - performing an echo-endoscopic procedure using the displayed navigation view; ENDO Includes.

[0034] The system disclosed herein comprises an echo endoscope EES and a tracking device or system that defines a tracking frame of reference and provides the 3D position and orientation of the distal tip of the endoscope within this frame of reference.

[0035] The tracking device or system is in communication with the calculation CU and storage SU unit in such a way that 3D position and orientation data can be sent to the calculation CU and storage SU unit and interpreted and stored by the calculation CU and storage SU unit. Since the shape and dimensions of the tip of the endoscope are known, the calculation and storage means is configured to associate any point contained in any live ultrasound image with the live position of this point in the tracking reference frame.

[0036] In the remainder of this document, selecting a point in a live US image using the user interface and having the computation and storage unit record the position of the selected point in the tracking reference frame will be referred to as "selecting a point in a US image."

[0037] In different embodiments, the tracking method of the EES probe is based on electromagnetic properties (EM tracking) or another type of tracking (i.e., fiber optic tracking). According to one embodiment, the tracking device comprises an EM tracking device or a fiber optic tracking device.

[0038] An object of the present invention is to provide a system that provides guidance during an EUS procedure by helping a user navigate a patient's anatomy based on 3D radiological data acquired before the EUS procedure. Navigating to a precise, predetermined location within a patient's body during an EUS procedure requires expert skill. Therefore, it would be unhelpful to implement a navigation system for EUS that requires a user to navigate to a precise, predetermined location within the body as part of the registration process.

[0039] Several embodiments of the registration process will now be described.

[0040] In one general embodiment, the system for medical navigation described herein includes a means for a user to select landmarks during an EUS procedure using a user interface and then identify and mark equivalent landmarks on the preoperative radiological data.

[0041] According to one embodiment, registration is performed in two steps, with anatomical landmark matching within each step. The anatomical landmark matching can be performed manually by an operator. Alternatively, the landmark matching is performed by a computational and storage means.

[0042] Algorithms stored in the calculation and storage unit complete the registration.

[0043] That is, the system according to the present invention is further configured to register preoperative radiological data with real-time ultrasound images acquired by the echo-endoscopic EES.

[0044] According to this embodiment, the system for medical navigation according to the present invention includes a tracked ultrasound endoscope EES, a display D, a user interface UI, and a computing unit CU, the computing unit including at least an algorithm and a patient's preoperative radiological data, and the system is configured so that a user selects landmarks in the patient's procedural US images using the user interface and selects the positions of matching anatomical landmarks in the patient's preoperative radiological data using the user interface. The computing and storage unit calculates registration between the two positions using at least one algorithm, calculates a 3D visualization of the patient's preoperative radiological data, calculates a live position and orientation of the US images relative to the patient's preoperative radiological data, and displays to the user a live view integrated into the 3D visualization of the preoperative radiological data and the live US images of the patient at the calculated position and orientation within the 3D visualization of the preoperative scan.

[0045] The medical navigation method according to the present invention comprises: - providing a guidance system including a computing unit and a tracked EUS probe; - storing preoperative radiological data of the patient in a large region of interest; - Navigating the US endoscope to a wide area of ​​interest and selecting a target anatomical point that is easy to identify both on the US images and on the preoperative radiological data; - selecting matching landmarks in the preoperative radiological data; - calculating a rigid registration between the reference frame of the tracked scope and the reference frame of the preoperative data set; - calculating a 3D visualization of the preoperative radiological data; - embedding the live US images into a 3D visualization based on the registration information; - Displaying 3D visualization of preoperative radiological data together with embedded US images Includes.

[0046] In another general embodiment, the registration process integrates additional anatomical information beyond the user-selected landmarks. Such embodiments are of interest because they can speed up or improve the accuracy of registration. For example, the system can use automatic recognition of anatomical structures. An example of an additional anatomical structure is the esophagus.

[0047] Recognition of anatomical structures or landmarks can be done using intraoperative data, including US data, endoscopic data such as optical or video data, and data from tracking devices.

[0048] This embodiment is illustrated by considering the case of upper gastrointestinal echoendoscopy. The concept here is to automatically recognize anatomical structures as a preliminary step of registration. The automatic recognition may be based on automatic recognition of anatomical shapes based on data from a tracking device. The automatic recognition may be based on recognition of anatomical landmarks using video images. In a subsidiary embodiment, the computing and storage unit includes an extraction algorithm adapted to automatically identify and extract the position and orientation of the esophagus in preoperative radiological data, and a recognition algorithm adapted to identify the position and orientation of the live esophagus by identifying the moment when the esophagus is explored by the endoscopist during the EUS procedure. The exploration of the esophagus is separated from the rest of the procedure based on anatomical and operational characteristics, for example: exploration occurs at the beginning of the procedure; and displacement is substantially linear along a 20 cm path.

[0049] The computation and storage unit then performs partial registration using a registration algorithm by matching the esophageal direction automatically extracted from the preoperative radiological scan with the position and orientation of the live esophagus identified during the procedure.

[0050] Registration can then be completed according to any other embodiment described herein. Those skilled in the art will appreciate that such embodiments can be applied to other anatomical structures by adapting separation criteria of shape, size, and / or timing of procedures.

[0051] In another embodiment, the computing and storage unit includes an extraction algorithm adapted to automatically extract the positions and orientations of landmarks in the preoperative radiological data and a video recognition algorithm adapted to identify the positions and orientations of landmarks during the EUS procedure by identifying video frames containing landmarks and recording the coordinates of the tracking device. The computing and storage unit then performs a partial registration or first registration calculation using a registration algorithm by matching the orientations and positions of the landmarks extracted from the preoperative radiological scan with the positions and orientations of the landmarks identified during the procedure. The registration can then be completed by a registration refinement step according to any other embodiment described herein.

[0052] In another general embodiment, the system integrates a means for completing part of the registration using external patient landmarks. Such an embodiment results in faster registration. In such an embodiment, the system further includes a tracked pointer that is tracked by a tracking device. A user can use a user interface to select the location of a landmark external to the patient's body from the preoperative radiological data and then use the tracked pointer to point to the corresponding landmark location. The computation and storage unit then performs partial registration using a registration algorithm by matching the location of the landmark selected from the preoperative radiological data with the location of the landmark marked with the tracked pointer. Registration can then be completed according to any other embodiment described herein.

[0053] The systems and methods for medical navigation described herein are intended for use by endoscopists who are not EUS specialists. These users are typically gastroenterologists or surgeons, and are typically not proficient in interpreting and understanding raw CT or MRI image slices and / or are not highly skilled in the visual interpretation of EUS and / or endoscopic images. Few widely adopted image file standards exist, and only raw CT or MRI images can be exchanged between systems.

[0054] In one general embodiment, the computing and storage unit includes a visualization algorithm adapted to compute a 3D visualization of the preoperative radiological raw data, which is then displayed to a user, who selects at least one landmark during the EUS procedure to complete registration. Once registration is complete, a live US image embedded in the 3D visualization of the preoperative radiological raw data is displayed to the user. The live US image embedded in the 3D visualization of the preoperative radiological raw data is hereinafter referred to as a navigation visualization.

[0055] In one dependent embodiment, the navigation visualization is oriented in real time to match the perspective seen from the tip of the scope.

[0056] In one dependent embodiment, the 3D visualization is calculated at the beginning of the procedure and only a subset of the 3D visualization is displayed to the user.

[0057] In one dependent embodiment, the navigation visualization includes a 3D visualization of the preoperative radiological image cut along the plane of the live US image to show the section of the 3D visualization currently being imaged.

[0058] In one general embodiment, the navigation visualization further integrates a live endoscopic view recorded by a camera mounted at the distal end of the echo-endoscope. During an EUS procedure with current systems, the position and orientation of the endoscope tip are known. After a registration process, the position and orientation of the endoscopic image relative to the preoperative radiological image are known. Thus, the live endoscopic view can be embedded in a realistic position and orientation within the 3D visualization of the preoperative radiological image.

[0059] In a typical embodiment, the preoperative radiological data can be a preoperative image set (CT or MR), or a 3D skin surface model automatically segmented from preoperative images (CT or MR), or a segmented preoperative 3D image (CT or MR), or a 3D skin surface model automatically created before the examination, or a biomechanical preoperative model (CT or MR) that accounts for soft-body changes.

[0060] In one embodiment, the method further includes a pre-operative planning step, in which the pre-operative radiological data, whether reconstructed or not, is annotated or enhanced by a user before being used as input for the system, which step may include, for example, allowing a user to carefully inspect the pre-operative data and identify lesions to be biopsied or resected and relevant structures, such as anatomical landmarks, to be used for registration purposes.

[0061] In one embodiment, the user interface allows a user to turn the display of the preoperative radiological data on and off. The user input interface can also be used to modify the visualization of the preoperative radiological data, for example, to switch between different visualization modes such as color map, label map, etc., adjust transparency, change the portion of the segment model that is displayed, and change the viewpoint characteristics of the visualization (angle, zoom, etc.). Those skilled in the art will be able to identify the display modes.

[0062] Those skilled in the art will recognize that any type of EUS known in the art can be used in the systems and methods for medical navigation described herein. For example, the EUS probe can vary between linear, two-dimensional, or three-dimensional EUS, and radial, frontal, or side-viewing EUS.

[0063] According to one embodiment, a system for medical navigation can be described as a system in which:

[0064] 1. A system for medical navigation that provides guidance during an endoscopic ultrasound examination procedure of an anatomical zone of a patient after a 3D radiological scan of the same anatomical zone of the same patient has been acquired, thus generating pre-operative radiological raw data, comprising: i) an endoscopic ultrasound system having an ultrasound endoscope that generates video and ultrasound images during an endoscopic ultrasound examination procedure; ii) Display and User Interface iii) A tracking device or system that defines a tracking reference frame and provides the 3D position and orientation of the tip of the ultrasound endoscope within this tracking reference frame. iv) Computation and storage means Including, said computing and storage means being configured to retrieve "pre-operative" radiological raw data and to associate any point contained in any of said ultrasound images with coordinates in said tracking frame of reference, and the medical system comprising: i) Compute 3D visualization of the preoperative radiological raw data; ii) providing an interface for a user to select and tag the location of at least one landmark in an ultrasound image during an endoscopic ultrasound examination procedure, and the system stores the 3D location of the US landmark with the associated marked ultrasound image; iii) providing an interface for the user to select the anatomical location of at least one landmark within the “pre-operative” radiological raw data or 3D visualization; iv) Calculate the spatial correspondence between the “pre-operative” radiological raw data and the tracking reference frame based on the location of landmarks in the pre-operative radiological raw data and the tracking reference frame. A live navigation view is calculated and displayed to the user in real time, integrating live US images within the 3D visualization of the preoperative radiological raw data, the live US images being positioned and oriented to match their actual live position and orientation. The present invention is characterized in that it is configured as follows.

[0065] FIG. 3 is a diagram illustrating the different coordinate systems involved in the application of the system and method according to the present invention.

[0066] An implementation of the registration system and method can be described based on Figure 3. The goal of this system is to solve a difficult computational problem: fast and reliable registration with minimal disruption to clinical workflow.

[0067] Figure 3 outlines the coordinate transformation between source data s, patient p, endoscope e, ultrasound image u, optical image o, tracking sensor t, and world coordinates w. World coordinates are defined as the coordinate system of the tracking system. In this document, source data, preoperative data, and preoperative radiological data refer to the same data.

[0068] Using the following coordinate system definition:

[0069] Source Data Coordinates: 3D source data is medical image data that is registered and visualized for computer-assisted procedural navigation. 3D source data is generated by 3D imaging systems, including but not limited to: Computed tomography (CT) images Magnetic resonance (MR) imaging Positron Emission Tomography (PET) imaging

[0070] The 3D source data can be acquired preoperatively during a diagnostic exam or intraoperatively during an EUS procedure. The source data can be pre-processed for enhanced visualization using one or more processes, including but not limited to: 3D segmentation calculated by human experts or automatically by AI-based segmentation systems Image transformations such as cropping, scaling, or intensity windowing Source data fusion, combining different source images in patient coordinates

[0071] World Coordinates: World coordinates are defined as the coordinate system of the endoscope's tracking device. At a minimum, the tracking device must produce a real-time 6-degrees-of-freedom (DoF) 3D pose (rotation and orientation) of the probe tip in the world coordinate system. Compatible tracking devices include, but are not limited to: An endoscope with a 6DoF EM tracking device embedded at the tip An endoscope with a 6DoF EM tracking device at the tip and an embedded chain of EM sensors within the scope An endoscope with a 6DoF EM tracking device fixed at the distal end of the auxiliary channel An endoscope with a 6DoF EM tracking device attached externally to the tip of the endoscope using a cap. Endoscope with an embedded 6DoF fiber optic tracking system Robotically controlled endoscopes where the position of the scope tip is known by kinematics.

[0072] Ultrasound coordinates: This is the coordinate frame of the endoscopic ultrasound image.

[0073] Optical Coordinates: This is the coordinate frame of the endoscopic optical image.

[0074] Tracking sensor coordinates: This is the coordinate frame of the tracking sensor at the tip of the endoscope.

[0075] Patient coordinates: This is the standard coordinate frame to which all data is registered. There are three main axes in patient coordinates: anterior / posterior, left / right, and superior / inferior.

[0076] The following transformations are known and can be calculated external to the registration process:

[0077] - Transformation (Te,t) from endoscope to tracking sensor. Without loss of generality, we define the coordinates of the endoscope to be the same as the coordinates of the tracking sensor, i.e. (Te,t) is the identity transformation. This is possible because we assume that the tracking sensor is rigidly fixed relative to the tip of the endoscope and therefore does not change over time.

[0078] - Transformation from endoscope to ultrasound image (Te,u). Because the endoscope tip is rigid, this does not change over time and is calculated in a one-time external calibration process, for example, hand-eye calibration [Plus Toolkit].

[0079] - Transformation from the endoscope to the optical image (Te,o). Because the endoscope tip is rigid, this does not change over time and is calculated in a one-time external calibration process, for example, hand-eye calibration [X].

[0080] - The tracking sensor to world transformation (Tt,w), which is provided by the tracking sensor. Unlike the above transformations, this is time-varying and is provided by the 3D tracking system.

[0081] - Source data to patient transformation (Ts,p). We assume that the patient is imaged in the source data supine position, i.e. the patient axes are approximately aligned with the scanner axes. This occurs in the overwhelming majority of cases. When there is a single source image, we define the source data and patient coordinates as the same. When there are multiple source images (e.g. MR and CT), they require co-registration.

[0082] The unknown coordinate transformation is a time-varying endoscope-to-patient transformation. It is determined by combining the endoscope-to-world transformation (known) with the world-to-patient transformation (Tw,p: unknown). Our registration process provides a novel, fast, and reliable solution for computing Tw,p with minimal disruption to clinical workflow.

[0083] According to one embodiment, the registration step comprises two further steps: Initial Registration Calculation: This process calculates the first world-to-patient transformation using external or internal anatomical landmarks and patient axis information. Registration refinement: This process improves the initial registration by updating it with information from internal anatomical or pathological landmarks identified during the procedure.

[0084] To explain these processes in detail, we first define the various physical and virtual entities:

[0085] Patient Graphical User Interface PGUI: Software component for the interactive fused imaging virtual reality display of all relevant information (source data, endoscopic, ultrasound and optical images) in patient coordinates. PGUI is used during registration for the interactive registration described below.

[0086] Picker device: A handheld device with a pointed tip used to localize external anatomical landmarks in world coordinates. This can be a dedicated picker device provided by a 3D tracking system, or it can be the tip of an endoscope. The advantage of using an endoscope as a picker device is that it eliminates additional devices in the operating room, but at the cost of lower accuracy compared to dedicated picker devices.

[0087] External Anatomical Landmark EAL: An anatomical landmark that can be repeatedly located both in the source data and on the patient's exterior during the procedure. For abdominal procedures, a good EAL is one that is relatively stable with respect to respiratory and other physiological movements. These include, but are not limited to, the sternal body and base.

[0088] Ultrasound Anatomical Landmarks UAL: Anatomical landmarks that can be located on both the source data and EUS ultrasound images during the procedure. For abdominal procedures, possible UALs include, but are not limited to, the cardia, esophageal junction, duodenal papilla, pancreatic head, pancreatic tail, aorta, and aortocoeliac axis junction.

[0089] Optical Anatomical Landmarks (OAL): Anatomical landmarks that can be located in both the source data and the endoscopic optical image during the procedure. For abdominal procedures, our OALs include, but are not limited to, the cardia and duodenal papilla.

[0090] Anatomical landmarks AL: Includes EAL, UAL, and OAL.

[0091] Intraoperative EAL, UAL and OAL position: 3D position of the EAL, UAL or OAL defined in the respective coordinate system (world, ultrasound image and optical image, respectively).

[0092] Intraoperative EAL, UAL and OAL covariance matrix: assessment of the reliability of the EAL, UAL or OAL position. This is implemented using a 3x3 covariance matrix, where larger values ​​in the covariance matrix indicate higher uncertainty in the landmark position.

[0093] Source Anatomical Landmark (SAL) Location: The 3D location of an EAL, UAL, or OAL placed in the source data, defined in source data coordinates.

[0094] SAL Covariance Matrix: An estimate of the reliability of the SAL position. This is implemented using a 3x3 covariance matrix.

[0095] Convolutional neural network (CNN): A well-known type of artificial neural network with neurons organized in a convolutional pattern.

[0096] The initial registration step or calculation is shown in Figure 4. The initial registration includes three steps:

[0097] Anatomical Landmark Localization - AL‐L. The purpose of this step is to determine a set of anatomical landmarks AL that match the source data and world coordinates. For landmarks outside the world coordinates, a picking or picker device is used. For landmarks inside or outside the source data coordinates, the landmarks are located using one of the following mechanisms:

[0098] - Manual localization: These are determined using a human operator using the PGUI.

[0099] - Automatically positioned ALs. These are determined using an AI system, typically a CNN, that can automatically localize anatomical landmarks in the source data. Compared to manual localization, the main advantage is reduced workflow interruptions.

[0100] The automatically placed EALs are visualized on the PGUI to a human operator who verifies their correctness. If they are determined to be incorrect, the human operator can reposition them manually.

[0101] Patient Axis Identification - PA-ID. When there are three or more matched EALs, it is possible to estimate the world-to-patient transformation using rigid alignment of landmarks using singular value decomposition (SVD). However, this is undesirable due to the time required for localization and the difficulty of finding three stable landmarks. To overcome this difficulty, we allow fewer EALs (at least one is required) and supplement this with additional patient axis information. The fewer the number of EALs, the more axis information is required to resolve the world-to-patient transformation.

[0102] The relationship between the number of EALs and the number of required patient axes that must be determined to uniquely calculate the world-to-patient coordinate transformation is as follows: If three EALs are matched, the number of required patient axes is 0. If two EALs are matched, one patient axis is required. If one EAL is patched, two patient axes are required.

[0103] Here we describe our approach to determining the patient axis given an EAL. We consider two cases based on the supine position.

[0104] It takes advantage of the fact that the normal vector of the patient table corresponds to the back / front axis. Therefore, to determine this axis, the normal vector of the patient table must be determined. Typically, this is calculated by a human operator using a picking device to touch three non-collinear points on the patient table. From these points, the plane normal is calculated using SVD. In the special case of using an EM tracking device, an even simpler solution is obtained by assuming that the field generator of the tracking device is placed on the patient table or on a plane parallel to the patient table. Once this is done, the back / front axis is immediately given by one of the axes of the tracking device system.

[0105] Then we calculate the top / bottom axis. Two solutions to this are explained with a very simple operator workflow. The operator is free to choose which solution they prefer.

[0106] Solution 1: Axis tracing A human operator uses a picking device to trace a line down the sternum. This generates a series of 3D points along the sternum, which are virtually projected onto the patient table surface to form a path of 2D points on the virtual table surface. Finally, a robust line fitting algorithm is used to estimate a best-fitting line passing through the 2D points. Random sample consensus, RANSAC), the direction of this line gives the superior / inferior axis.

[0107] Solution 2: Endoscope tip alignment The human operator aligns the US probe with the superior / inferior axis and places the tip of the endoscope on the patient's sternum. Once this is done, the 3D position of the endoscope is recorded using a tracking device, and the 3D position of the US image plane is determined in world coordinates. Finally, the superior / inferior axis is determined by intersecting the US image plane with the table surface.

[0108] Conversion Calculations - REG-COMP The left / right patient axes are then calculated by the cross product of the posterior / anterior axes and the superior / inferior axes. Given the EAL and the three patient axes, there is enough information (six geometric equations) to calculate the world-to-patient coordinate transformation (6 DoFs). This is calculated using Horn's absolute orientation algorithm.

[0109] In most cases, the source data is acquired with the patient in a supine position, so the patient's axes are aligned with the image coordinate system. In the special case where the patient is not scanned in a supine position, the axes are determined by one of two mechanisms. The first is for a human operator using NGI to interactively draw 3D vectors corresponding to two of the three primary axes onto the source data. From this data, the third axis can be automatically determined using the vector cross product. The second mechanism is to use a trained AI system, such as one based on CNN, to calculate the axes. Similar to EAL-L or Component 1, a visualization of this is provided to the human operator who validates the results.

[0110] In the non-supine position, the patient's axis is determined using tracking data from the EUS probe as it enters the GI tract. Specifically, the 3D position of the EUS probe is recorded over time as the tip of the EUS probe passes through the esophagus and reaches the stomach. This generates a 3D chain of points in world coordinates. In the source data, the central axis of the esophagus is determined manually using PGUI or automatically using an AI recognition system. This provides a 3D chain of points in patient coordinates. Finally, the point chain and at least one external landmark (e.g., the base of the sternum) are used to perform registration using robust linear least-squares point-to-path alignment.

[0111] The initial registration provides a coarse alignment, typically with an accuracy of less than 2 cm for organs that do not move significantly with respiratory motion, such as the pancreas. This may be sufficient for applications such as global positioning systems (GPS), where the intention is to provide only a rough visualization of the position and orientation of the EUS endoscope in patient coordinates. When a more precise registration is desired, a registration refinement system is implemented.

[0112] FIG. 5 illustrates the registration refinement steps or procedures.

[0113] Registration refinement involves three steps: ultrasound anatomical landmark matching UAL-M, optical anatomical landmark matching OAL-M, and local rigid body alignment LORA.

[0114] Ultrasound Anatomical Landmark Matching UAL-M: When a UAL is encountered during a procedure (e.g., see definition above), it can be localized and used to improve registration. Localization can be done manually (a human marks the landmark location in the US image using the PGUI) or automatically, provided by an AI system that automatically recognizes the UAL in the US footage. In addition to the landmark location, a confidence matrix is ​​also provided that determines the amount of confidence in the landmark location. In the case of manual localization, this is done by using the PGUI to mark zones around the landmark to indicate the confidence. In the case of automatic localization, the machine outputs the confidence matrix along with the landmark location.

[0115] Once the UAL is localized, it is then localized to the source data (source anatomical landmarks). Like the UAL, the SAL is either determined manually using the PGUI or automatically using an AI landmark recognition system.

[0116] Optical anatomical landmark matching OAL-M: Similar to UAL-M, landmarks can be located using optical images from the endoscope. The process is similar to UAL-M.

[0117] Local Rigid Alignment LORA: The goal of this step is to automatically calculate a time-varying registration matrix using all landmarks matched in the UAL-M and OAL-M steps, as well as the fixed set of matched landmarks used in the initial registration. The registration matrix is ​​calculated by solving a numerical optimization process. The registration is calculated as a rigid 3D matrix with two characteristics: 1. The influence of landmarks with higher confidence has a larger impact on the registration matrix compared to landmarks with lower confidence. 2. The influence of landmarks close to the current position of the endoscope has a larger effect on the registration matrix, which allows the physiological motion of distant landmarks to have a smaller effect on the registration matrix.

[0118] Next, an optimization process is performed. Specifically, we formulate a numerical search problem, implemented as a numerical loss function, aimed at finding a registration matrix that satisfies both of the above properties. To satisfy 1, we search for a registration matrix that minimizes the Jenson-Shannon divergence of each matched landmark using the landmark confidence matrix. To satisfy 2, we weight the loss function using exponential weighting based on the distance of each landmark to the endoscope's current 3D position in world coordinates.

Claims

1. 1. A system for performing an upper gastrointestinal echoendoscopy of a patient, the system comprising: an echoendoscope; a tracking device; a user interface; a display; and a computing and storage system, the computing and storage system including preoperative radiological images of the patient, the preoperative radiological images being radiological images of the patient obtained prior to the upper gastrointestinal endoscopy of the patient, the computing and storage system being configured to: receiving, through the user interface, coordinates of a first plurality of points in a spatial coordinate system of the preoperative radiological image, the first plurality of points being positioned along the esophageal central axis and representing a path of the esophageal central axis in the preoperative radiological image; - receiving coordinates of a second plurality of points in a second coordinate system related to the echoendoscope, the second plurality of points corresponding to different positions of the tip of the echoendoscope as it passes between the patient's esophagus and stomach; - receiving, via a user interface, the location of anatomical landmarks located on the patient's external surface in a preoperative radiological image; - receiving the localization of anatomical landmarks obtained through a tracking device; - performing a registration between a first coordinate system related to the preoperative radiological image and a second coordinate system related to the echoendoscope, the registration being performed based on the coordinates of the first plurality of points, the coordinates of the second plurality of points and the received localization of anatomical landmarks; It is configured as follows: the echo endoscope is further configured to acquire, after registration, real-time ultrasound images of the patient; the tracking device is further configured to acquire tracking data representative of the position of the tip of the echoendoscope in space during acquisition of the real-time ultrasound images; a computing and storage system further configured to compute, based on the registration and tracking data, a real-time position and orientation of the real-time ultrasound image relative to the preoperative radiological image based on the registration; The system, wherein the display is configured to display a navigation view that integrates real-time ultrasound views within preoperative radiological images based on the position and orientation.

2. The system of claim 1 , wherein the preoperative radiological image is a computed tomography image, a magnetic resonance image, or a positron emission tomography image.

3. The system of claim 1 or 2, wherein the tracking device is an electromagnetic tracking device or a fiber optic tracking device.

4. 4. The system of claim 1, wherein the localization of anatomical landmarks in preoperative radiological images is received through a user interface or is determined automatically using artificial intelligence.

5. 5. The system of claim 1, wherein the anatomical landmark located on the patient's external surface is the body of the sternum or the base of the sternum.

Images