Systems and methods for registering imaging data from different imaging modalities based on subsurface image scanning

Abstract

An example image registration system (100) identifies subsurface structures (502, 504) at a surgical site (306) based on subsurface imaging data from a subsurface image scan at the surgical site. The image registration system (100) uses the subsurface structures (502, 504) identified at the surgical site to register endoscopic imaging data (304-1) from an endoscopic imaging modality (302-1) with additional imaging data from an additional imaging modality (304-2). Corresponding systems and methods are also disclosed.

Description

[0001] Related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 893,040, filed August 28, 2019, entitled “Systems and methods for registering imaging data from different imaging modalities on vascular image scanning,” the contents of which are incorporated herein by reference in their entirety. Technical Field Background Technology

[0003] During surgical procedures, endoscopes can be used to capture endoscopic images of the surgical site. These endoscopic images can then be presented to the surgeon via a display device, allowing the surgeon to visualize the surgical site while performing the procedure.

[0004] In some scenarios, one or more additional imaging modalities (in addition to the endoscopic imaging modality) can be used to capture additional images of the surgical site that can also be presented to the surgeon. Such additional images can be captured preoperatively or intraoperatively and can be captured, for example, by ultrasound scanning, computed tomography (“CT”) scanning, magnetic resonance imaging (“MRI”) scanning, fluorescence imaging scanning, and / or another suitable imaging modality configured to capture images of the surgical site.

[0005] While images captured by different imaging modalities can be presented to facilitate surgeons' visualization of surgical sites, surgeons may still find it difficult and / or inconvenient to conceptualize surgical sites and synthesize conceptual models of surgical sites based on different types of images. This is especially true when images represent different content captured in different ways (e.g., comparison of surface anatomical content with subsurface anatomical content, comparison of preoperative content with intraoperative content, etc.). Therefore, there is still room for improvement in the processing and presentation of images captured by different imaging modalities. Summary of the Invention

[0006] The following description presents a simplified overview of one or more aspects of the systems and methods described herein. This overview is not a comprehensive summary of all anticipated aspects, and is neither intended to identify key or decisive elements of all aspects, nor to depict the scope of any or all aspects. Its sole purpose is to present one or more aspects of the systems and methods described herein as a prelude to the detailed descriptions that follow.

[0007] An exemplary system includes a memory storing instructions and a processor communicatively coupled to the memory and configured to execute instructions to identify subsurface structures at a surgical site based on subsurface imaging data from subsurface image scans at the surgical site; and to use the identified subsurface structures at the surgical site to register endoscopic imaging data from an endoscopic imaging modality with additional imaging data from an additional imaging modality.

[0008] Another exemplary system also includes a memory and a processor storing instructions, the processor being communicatively coupled to the memory and configured to execute instructions to perform various operations intraoperatively at the surgical site during a surgical procedure. For example, the various operations performed by the processor may include accessing intraoperative endoscopic imaging data from an endoscope; accessing intraoperative subsurface imaging data from a subsurface imaging module; accessing alignment parameters representing the registration of the intraoperative endoscopic imaging data with the intraoperative subsurface imaging data; accessing preoperative scan data captured via an additional imaging modality different from the endoscopic imaging modality, the preoperative scan data representing a three-dimensional (“3D”) model of the anatomical structure at the surgical site; identifying subsurface structures at the surgical site based on the intraoperative subsurface imaging data and the alignment parameters; and / or registering the intraoperative endoscopic imaging data with the preoperative scan data representing the 3D model of the anatomical structure at the surgical site using the identified subsurface structures at the surgical site.

[0009] An exemplary method includes an image registration system that identifies subsurface structures at a surgical site based on subsurface imaging data from subsurface image scans at the surgical site, and registers endoscopic imaging data from an endoscopic imaging modality with additional imaging data from an additional imaging modality using the identified subsurface structures at the surgical site. Attached Figure Description

[0010] The accompanying drawings illustrate various embodiments and are part of the specification. The illustrated embodiments are merely examples and do not limit the scope of this disclosure. Throughout the drawings, the same or similar reference numerals denote the same or similar elements.

[0011] Figure 1 An exemplary image registration system is illustrated for registering imaging data from different imaging modalities based on subsurface image scanning according to the principles described herein.

[0012] Figure 2 An exemplary aspect of the registration of a first set of imaging data with a second set of imaging data according to the principles described herein is illustrated.

[0013] Figure 3 The figure illustrates an exemplary configuration, in which Figure 1The image registration system described in this paper registers imaging data from different imaging modalities to generate a synthetic image of the surgical site.

[0014] Figure 4A The illustration depicts an exemplary aspect of an endoscopic imaging modality used to capture endoscopic imaging data depicting anatomical structures at a surgical site, based on the principles described herein.

[0015] Figure 4B The diagram illustrates the principles described in this article, derived from sources different from... Figure 4A An exemplary aspect of the representation of the surgical site captured by an additional imaging modality of the endoscopic imaging modality illustrated in the figure.

[0016] Figure 5 An exemplary aspect of registering endoscopic imaging data from an endoscopic imaging modality with additional imaging data from an additional imaging modality, according to the principles described herein, is illustrated.

[0017] Figures 6 to 9 Various exemplary synthetic images of surgical sites according to the principles described herein are illustrated, including depictions of anatomical structures captured by different imaging modalities and aligned within the synthetic image relative to the viewpoint.

[0018] Figure 10 An exemplary computer-assisted surgical system based on the principles described herein is illustrated.

[0019] Figure 11 and Figure 12 An exemplary method for registering imaging data from different imaging modalities based on subsurface image scanning according to the principles described herein is illustrated.

[0020] Figure 13 An exemplary computing device based on the principles described herein is illustrated. Detailed Implementation

[0021] This article describes systems and methods for registering imaging data from different imaging modalities based on subsurface image scanning. Various types of surgical procedures can be planned or performed at a surgical site, which may include the anatomy of the body to which the surgical procedure is being performed (or to be performed), anatomical structures adjacent to the anatomy being operated on, and other areas adjacent to that anatomy (e.g., open spaces). The anatomy being operated on and surrounding anatomy may include various subsurface structures located below or behind the surface anatomy being operated on during the procedure. For example, subsurface structures may include vascular systems consisting of multiple blood vessels (e.g., arteries, veins, etc.), bone structures, tissue masses (e.g., cancerous and / or non-cancerous tumors, cell growths, etc.), organs, tendons, muscles, ligaments, cartilage, nerves, fat structures, and / or other subsurface anatomy structures that can be identified by certain imaging modalities as located below or behind the surface being operated on.

[0022] The systems and methods described herein can use subsurface imaging data from subsurface image scans at a surgical site to identify the location of subsurface structures at the surgical site, and then use the identified locations of the subsurface structures to register data from different imaging modalities (e.g., register or refine the registration of data from different imaging modalities). As will be described in more detail below, any type of imaging scan configured to determine subsurface image data can be used as the subsurface image scan in the examples described herein. For example, a subsurface image scan can be a suitable ultrasound scan (e.g., standard ultrasound scan, contrast-enhanced ultrasound scan, harmonic imaging scan with or without contrast, scan employing ultrasound elastography, tomography, etc.) configured to detect invisible subsurface structures, such as vascular systems, bone structures, or tissue blocks, using a visible light modality such as an endoscope. As another example, vascular ultrasound scans, such as Doppler ultrasound scans, or standard ultrasound scans employing vascular segmentation algorithms can be used to specifically identify vascular systems. In other examples, subsurface imaging scans can be performed using non-ultrasound imaging modalities such as optical coherence tomography, fluorescence imaging, hyperspectral imaging, or another suitable type of image capable of mapping various subsurface structures of the body, such as vascular systems, skeletal structures, tissue blocks, etc.

[0023] The following describes in more detail the types of imaging modalities that can be registered with each other, the environments in which imaging modalities can be employed, and various exemplary embodiments. Furthermore, as will be further described herein, the registration of imaging data from different imaging modalities can be used to provide a synthetic image of the surgical site (e.g., a synthetic image displaying preoperative and intraoperative imaging data integrated together into a single image view) for display on a display device to facilitate surgical procedures.

[0024] An exemplary image registration system may include a memory storing instructions and a processor communicatively coupled to the memory and configured to execute the instructions to perform certain operations. For example, the processor may identify subsurface structures at a surgical site based on subsurface imaging data from subsurface image scans at the surgical site. The processor of the image registration system may also use the subsurface structures identified at the surgical site to register endoscopic imaging data from an endoscopic imaging modality with supplementary imaging data from an additional imaging modality. For example, the subsurface structures may be used to initially generate or correct (e.g., refine, correct, update, etc.) the registration of endoscopic imaging data with the supplementary imaging data.

[0025] Another exemplary image registration system may include an endoscope, a subsurface imaging module, and a processor. These components may operate together during a surgical procedure at the surgical site (i.e., “intraoperative”) to provide a user (e.g., a surgeon or surgical team member) with a combination of images captured intraoperatively and images captured prior to the surgical procedure (i.e., “preoperative”). For example, the endoscope may be configured to capture intraoperative endoscopic imaging data, the subsurface imaging module may be configured to capture intraoperative subsurface imaging data, and the processor may be configured to access, along with access to the intraoperative endoscopic imaging data from the endoscope and the intraoperative subsurface imaging data from the subsurface imaging module, preoperative scan data captured via an additional imaging modality. The additional imaging modality may differ from the endoscopic imaging modality used by the endoscope and may include, for example, a preoperative magnetic resonance imaging (“MRI”) scan, a preoperative computed tomography (“CT”) scan, a preoperative ultrasound scan, or the like. In some examples, the preoperative scan data may represent a three-dimensional (“3D”) model of the anatomical structures at the surgical site.

[0026] In addition to accessing endoscopic imaging data, subsurface imaging data, and preoperative scan data, the processor can be further configured to access alignment parameters representing the registration of intraoperative endoscopic imaging data with intraoperative subsurface imaging data. Therefore, the processor can identify subsurface structures at the surgical site associated with the intraoperative endoscopic imaging data based on the intraoperative subsurface imaging data and alignment parameters, and can use the identified subsurface structures at the surgical site to register the intraoperative endoscopic imaging data with preoperative scan data (e.g., a 3D model representing the anatomical structure at the surgical site).

[0027] The systems and methods described herein can offer a variety of advantages and benefits. For example, they can facilitate the initial registration or correction of the registration of different component image data captured through different imaging modalities to align imaging data that are otherwise difficult, impractical, or impossible to accurately align and register. For instance, it may be impossible to align and register endoscopic imaging data depicting surface anatomy with additional imaging data (e.g., MRI, CT, ultrasound, 3D anatomical models, etc.) representing subsurface anatomy using conventional techniques that identify and match common visual features depicted or represented in two imaging datasets. This is because endoscopic imaging data may depict features (e.g., surface anatomy features) that are entirely different from those represented in the additional imaging data (e.g., subsurface anatomy features).

[0028] Therefore, the systems and methods described herein can take advantage of the fact that: 1) subsurface imaging data from subsurface image scans at the surgical site can be accurately registered to endoscopic data in other ways (e.g., based on kinematic and / or visual data indicating that the subsurface imaging module is kinematic with respect to the endoscope); and 2) subsurface imaging data indicates the location of subsurface structures at the surgical site, which are also represented in the supplementary imaging data. Thus, subsurface structures identified based on subsurface image data can also be used to register endoscopic image data with supplementary imaging data, even if these datasets were captured at different times (e.g., preoperative and intraoperative) and include anatomical representations deformed (e.g., due to gravity or pressure of instruments applied differently when capturing different imaging data), etc.

[0029] Furthermore, the accurate and efficient registration of different types of imaging data provided by the systems and methods described herein can directly benefit users of computer-assisted surgical systems. For example, the systems and methods described herein can allow the integration of images captured by different imaging modalities into a synthetic image of the surgical site in a manner that produces an integrated and intuitive depiction of the surgical site images captured by different imaging modalities within the synthetic image. The systems and methods described herein can present the generated synthetic image to users of computer-assisted surgical systems, such as surgeons performing surgical procedures using computer-assisted surgical systems. The presented synthetic image may be visually intuitive for the surgeon; it can reduce the complexity of surgical procedures for the surgeon (e.g., by eliminating the need for the surgeon to mentally align images of the surgical site presented separately in a non-integrated manner); and / or it can allow the surgeon to instantly, conveniently, and intuitively visualize surface and subsurface anatomy integrated into the synthetic image. Furthermore, the presented synthetic image can be user-customizable, allowing the surgeon to provide input to conveniently and dynamically select the portion of the surgical site to be enhanced, enabling the selected portion to be viewed using an imaging modality different from that used to view another portion of the surgical site. For example, the system and method can enable surgeons to select a portion of the surgical site and display an image of the subsurface anatomy at that site as an enhancement of the image of the surface anatomy being displayed.

[0030] Various embodiments will now be described in more detail with reference to the accompanying drawings. The disclosed systems and methods may provide one or more of the benefits described above and / or various additional and / or alternative benefits that will become apparent herein.

[0031] Figure 1 An exemplary image registration system 100 (“System 100”) for registering imaging data from different imaging modalities based on subsurface image scanning is illustrated. System 100 may be included in computer-assisted surgical systems (such as those described below in conjunction with...). Figure 10 The system 100 is implemented by or connected to one or more components of the exemplary computer-assisted surgical system described herein. For example, system 100 may be implemented by one or more components of the computer-assisted surgical system, such as a manipulation system, a user control system, or an auxiliary system. As another example, system 100 may be implemented by a separate computing system communicatively coupled to the computer-assisted surgical system.

[0032] like Figure 1As shown, system 100 may include, but is not limited to, storage facility 102 and processing facility 104 selectively and communicatively coupled to each other. Facilities 102 and 104 may each include or be implemented by one or more physical computing devices, which include hardware and / or software components such as processors, memory, memory drives, communication interfaces, instructions stored in memory for execution by the processor, and the like. Although facilities 102 and 104 are... Figure 1 Facilities 102 and 104 are shown as separate facilities, but they can be combined into fewer facilities, such as a single facility, or divided into more facilities, as they can serve a particular implementation. In some examples, each of facilities 102 and 104 can be distributed among multiple devices and / or multiple locations, as they can serve a particular implementation.

[0033] Storage facility 102 may maintain (e.g., store) executable data used by processing facility 104 to perform any of the functions described herein. For example, storage facility 102 may store instructions 106 that can be executed by processing facility 104 to perform one or more of the operations described herein. Instructions 106 may be implemented by any suitable application, software, code, and / or other instance of executable data. Storage facility 102 may also maintain any data received, generated, managed, used, and / or transmitted by processing facility 104.

[0034] Processing facility 104 can be configured to perform (e.g., execute instructions 106 stored in storage facility 102) various operations associated with imaging data from different imaging modalities based on subsurface image scan registration. For example, processing facility 104 can be configured to identify subsurface structures at the surgical site based on subsurface imaging data from subsurface imaging scans at the surgical site (e.g., Doppler ultrasound scans, standard ultrasound scans using vascular segmentation algorithms, contrast-enhanced ultrasound scans, harmonic imaging scans, scans using ultrasound elastography, tomographic ultrasound, another type of ultrasound scan that allows the generation of two-dimensional (“2D”), 3D, or other types of subsurface structure maps, etc.) or based on data obtained through another suitable type of subsurface image scan (e.g., optical coherence tomography, fluorescence imaging, hyperspectral imaging, etc.). Processing facility 104 can use the subsurface structures identified at the surgical site to register endoscopic imaging data from an endoscopic imaging modality with additional imaging data from an additional (e.g., non-endoscopic) imaging modality.

[0035] Some embodiments of system 100 may be specifically configured to register imaging data from different imaging modalities based on real-time or near-real-time subsurface image scanning, such as by performing the above or other operations during surgery at the surgical site. For example, exemplary embodiments of system 100 may include a memory storing instructions and a processor communicatively coupled to the memory and configured to execute instructions to perform various functions during surgery at the surgical site. For example, the processing facility 104 may be configured in the following order or another suitable order as follows: 1) accessing intraoperative endoscopic imaging data from an endoscope; 2) accessing intraoperative subsurface imaging data from a subsurface imaging module; 3) accessing alignment parameters representing the registration of the intraoperative endoscopic imaging data with the intraoperative subsurface imaging data; 4) accessing preoperative scan data captured by an additional imaging modality different from the endoscopic imaging modality (e.g., the preoperative scan data may represent a 3D model of the anatomical structure at the surgical site); 5) identifying the subsurface structure at the surgical site based on the intraoperative subsurface imaging data and the alignment parameters; and 6) registering the intraoperative endoscopic imaging data with the preoperative scan data representing the 3D model of the anatomical structure at the surgical site using the subsurface structure identified at the surgical site.

[0036] This document describes these and other functions that can be performed by processing facility 104. In the following description, any reference to functions performed by system 100 shall be understood as being performed by processing facility 104 based on instructions 106 stored in storage facility 102.

[0037] Figure 2 The illustration depicts exemplary aspects of the registration of a first set of imaging data 200-1 and a second set of imaging data 200-2. In some examples, the registration of the first imaging data and the second imaging data may refer to a mapping from image data points in the first imaging data to corresponding image data points in the second imaging data, such that, from a particular viewpoint, the registration allows for alignment of the image datasets. For example, as... Figure 2 As shown, imaging data 200-1 represents a depiction 202-1 of an anatomical structure at a surgical site (e.g., an internal organ or part thereof), and imaging data 200-2 represents a depiction 202-2 of the same anatomical structure (although captured from slightly different viewpoints, making imaging data 200-1 and 200-2 similar but not identical). Registration of imaging data 200-1 with imaging data 200-2 can therefore involve determining whether various features 204 (e.g., features 204-1 to 204-6) in depiction 202-1 correspond to similar features 206 (e.g., features 206-1 to 206-6) in depiction 202-2.

[0038] As shown in the figure, for example, features 204-1 and 206-1 can be identified as a match (i.e., representing the same physical feature), and the same applies to feature pairs 204-2 and 206-2, 204-3 and 206-3, 204-4 and 206-4, and 204-5 and 206-5. In this example, features in each depiction that do not correspond to similar features in another depiction are also recalled. Specifically, no data point in depiction 202-2 of imaging data 200-2 corresponds to a data point representing feature 204-6 in depiction 202-1 of imaging data 200-1, nor does any data point in depiction 202-1 correspond to a data point representing feature 206-6 in depiction 202-2. Imaging data 200-1 can be registered with imaging data 200-2 by identifying a sufficient number of corresponding data point pairs, i.e., depiction 202-1 can be aligned with depiction 202-2 about a specific viewpoint (e.g., the viewpoint from which it captures imaging data 200-1, the viewpoint from which it captures imaging data 200-2, or another suitable viewpoint) (e.g., data point pairs representing similar features 204 and 206).

[0039] exist Figure 2 In the example, it will be understood that depictions 202-1 and 202-2 may look so similar because the corresponding images of each imaging data 200-1 and 200-2 were captured using the same imaging modality (e.g., in the same manner, by the same capture device, using the same imaging technique, etc.). Furthermore, and due to visual similarity, registration of imaging data 200-1 with imaging data 200-2 can be performed in a relatively straightforward manner by identifying feature 204 in depiction 202-1, identifying feature 206 in depiction 202-2, and matching features from each group to identify corresponding data points.

[0040] However, although Figure 2 The examples illustrate aspects of registering imaging data captured by a single imaging modality, but it will also be understood that in some examples, it may be desirable to register imaging data from one imaging modality (e.g., an endoscopic imaging modality) with imaging data from different imaging modalities (e.g., additional imaging modalities such as CT scans, MRI scans, or similar scans). In the latter case, registration may be accompanied by additional challenges and / or complexity.

[0041] As an example, different imaging modalities can capture depictions or other representations of anatomical structures at different points in time. For instance, an endoscopic imaging modality could include intraoperative scanning of the anatomy at the surgical site and thus could be performed in real time while the surgery is in progress. Similarly, ultrasound or fluorescence imaging scans (where a fluorescent chip is injected into the body to facilitate imaging at specific frequencies at which the chip exhibits fluorescence properties) could be similarly used during surgery, either in real time while the procedure is in progress or temporarily suspended during the procedure, with active surgical procedures temporarily suspended while imaging is being performed. Conversely, other types of imaging modalities can capture depictions or other representations of anatomical structures at a point in time prior to the procedure at the surgical site (e.g., immediately before, a day earlier, or so on). For example, imaging modalities including CT scans, MRI scans, ultrasound scans, X-ray scans, 3D modeling based on data from any such scans, or other suitable imaging modalities could be performed at different times as the body is in different states. For example, the patient's position during surgery may differ when using one imaging modality during the preoperative period and when using another imaging modality during the intraoperative period (e.g., supine versus lateral decubitus), or there may be other significant differences (e.g., fasting versus no fasting). In other examples, different modalities may be used simultaneously (e.g., preoperative, intraoperative, etc.) or at times different from this example (e.g., different preoperative times, different intraoperative times, preoperative and postoperative times, intraoperative and postoperative times, etc.).

[0042] As another illustrative complexity that may accompany registration when using different modalities, similar features such as features 204 and 206 may not be present in the corresponding depiction or other representation (e.g., 3D model, etc.) of the positively registered imaging data. This can happen because different imaging modalities may capture and represent anatomical structures in different ways. For example, one imaging modality, such as an endoscopic imaging modality, may capture data representing depictions of surface anatomy (i.e., anatomical structures that can be easily imaged by reflecting visible light from the anatomy), while additional imaging modalities, such as ultrasound scans, CT scans, MRI scans, or X-ray scans of the surgical site, may capture data representing subsurface anatomy (i.e., anatomical structures behind or below the surface anatomy that can only be imaged using advanced techniques involving sound waves, light waves beyond the visible spectrum, or similar techniques).

[0043] Because different visible features may exist on surface anatomy and subsurface anatomy, therefore regarding Figure 2 The feature matching registration shown and described may not be sufficient to register endoscopic imaging data depicting surface anatomy with additional imaging data representing subsurface anatomy.

[0044] To overcome these challenges, the systems and methods described herein can rely on certain subsurface structures, such as vascular systems, skeletal structures, tissue blocks, or other subsurface anatomy, which can be identified relative to various imaging modalities, even when used and / or representing different layers (e.g., surface or subsurface layers) at different times. For example, the location of vascular systems, as identified by vascular imaging data from vascular image scans registered with endoscopic images via kinematic, visual, or other suitable techniques, can be used to register endoscopic depictions of surface anatomy with 3D representations of subsurface anatomy contained in supplemental imaging data captured via non-endoscopic imaging modalities such as CT or MRI scans. Similarly, as another example, the location of certain bone structures, tissue blocks, and / or other subsurface structures can be identified by subsurface imaging data from subsurface image scans, which are also registered via kinematic, visual, and / or other suitable techniques. Similar to the vascular system structures described above, these subsurface structures can also be used to register endoscopic depictions of surface anatomy with 3D representations of subsurface anatomy contained in supplemental imaging data captured via non-endoscopic imaging modalities such as CT or MRI scans. Therefore, system 100 can use identified subsurface structures, such as vascular systems, bone structures, tissue blocks, or other suitable structures as implementations or alternatives to features 204 and 206, as system 100 generates or corrects (e.g., refines, updates, modifies, etc.) a set of alignment parameters representing the registration. For example, this set of alignment parameters can be associated with transformation matrices, translation matrices, and the like.

[0045] The alignment parameters included in a set of alignment parameters generated or corrected by system 100 can be configured to define a spatial transformation between endoscopic imaging data and supplementary imaging data. In various examples, the spatial transformation defined by this set of alignment parameters can involve rigid transformations from one set of imaging data to another (e.g., including translation and rotation operations), affine transformations from one set of imaging data to another (e.g., including general linear transformations, scaling, and tilting operations), deformable registrations from one set of imaging data to another (e.g., including nonlinear transformations with different parameterizations, etc.), or any other type of spatial transformation that may serve a particular implementation.

[0046] System 100 may register one set of imaging data with another set of imaging data for various purposes. For example, as already mentioned, one purpose of registering first imaging data from a first imaging modality with second imaging data from a second imaging modality is to align the first and second imaging data so that system 100 can generate and provide a synthetic image of the surgical site for display on a display device. This synthetic image includes aspects of the first and second imaging data viewed from a specific viewpoint (e.g., the viewpoint of a surgeon performing a surgical procedure at the surgical site, etc.). For example, such a synthetic image may be based on the registration of endoscopic imaging data from an endoscopic imaging modality with supplemental imaging data from an additional imaging modality, and may allow aspects of the endoscope and supplemental imaging data to be presented to the user in a convenient, customizable view to facilitate manipulation at the surgical site.

[0047] To illustrate, Figure 3 An exemplary configuration 300 is shown, in which system 100 registers imaging data from different imaging modalities to generate a synthetic image of the surgical site. As shown, configuration 300 may include multiple imaging modalities 302 (e.g., endoscopic imaging modal 302-1 and additional imaging modal 302-2) configured to capture imaging data 304 of the surgical site 306 (e.g., endoscopic imaging data 304-1 captured by endoscopic imaging modal 302-1 and additional imaging data 304-2 captured by additional imaging modal 302-2).

[0048] Surgical site 306 may include any volumetric space associated with a surgical procedure. For example, surgical site 306 may include any one or more parts of the patient's body, such as the patient's anatomical structures 308 (e.g., tissues, etc.) within the space associated with the surgical procedure. In some examples, surgical site 306 may be entirely located within the patient's body and may include spaces within the patient's body adjacent to the location where a surgical procedure is planned, is being performed, or has been performed. For example, for a minimally invasive surgical procedure performed on tissue within the patient's body, surgical site 306 may include surface tissue, anatomical structures beneath the surface tissue, and spaces surrounding the tissue, such as the location where surgical instruments are used to perform the surgical procedure. In other examples, surgical site 306 may be at least partially located outside the patient's body. For example, for an open surgical procedure being performed on a patient, a portion of surgical site 306 (e.g., the tissue being manipulated) may be inside the patient, while another portion of surgical site 306 (e.g., the space surrounding the tissue where one or more surgical instruments are located) may be outside the patient's body. Surgical site 306 may include a real workspace in which surgical procedures are performed, such as a real, real-world workspace associated with a patient and in which one or more surgical instruments are used to perform surgical procedures on the patient.

[0049] As used herein, a surgical procedure can include any medical procedure, including any diagnostic or therapeutic procedure in which manual and / or instrumental techniques are used on a patient to investigate or treat a patient’s physical condition. A surgical procedure can refer to any stage of a medical procedure, such as the preoperative, surgical (i.e., intraoperative), and postoperative stages of a surgical procedure.

[0050] Imaging mode 302 can be configured and / or used to capture imaging data 304 representing surgical site 306. This capture is performed by... Figure 3 The dashed line 310 in the diagram represents the imaging modality 302. Each of the imaging modalities 302 can capture imaging data 304 of the surgical site 306 in any suitable manner, and the imaging data 304 can take any suitable form. For example, the imaging data 304 can be implemented as data representing still frame images (e.g., grayscale images, color images, infrared images, etc.), videos (e.g., grayscale, color, infrared video, etc.), 3D models, or any other type of visualization or depiction that may be useful in some implementation to help the user visualize the surgical site 306. The imaging modalities 302 can also capture imaging data 304 at any suitable time. For example, one or more imaging modalities 302 can capture imaging data 304 of the surgical site 306 during one or more preoperative, intraoperative, and / or postoperative phases of a surgical procedure.

[0051] Endoscopic imaging modality 302-1 is a modality involving the capture of imaging data via an endoscope (e.g., or another suitable type of endoscopic instrument) configured to project light (e.g., visible frequency light) onto an anatomical structure at surgical site 306 and capture a photographic image of the anatomical structure when the light reflects from the anatomical structure to one or more image sensors associated with the endoscope. Conversely, in some examples, supplementary imaging modality 302-2 may be a different type of imaging modality (i.e., a modality different from the endoscopic imaging modality). For example, as described above, supplementary imaging modality 302-2 may include or involve, but is not limited to, ultrasound imaging performed by an ultrasound module or machine, CT imaging performed by a CT machine, MRI imaging performed by an MRI machine, or the like. Any other suitable supplementary imaging modality may be used in other examples.

[0052] In some examples, endoscopic imaging mode 302-1 can be configured to capture images of surface anatomy structures (e.g., the outer surface of tissue at the surgical site) included at the surgical site 306, and supplementary imaging mode 302-2 can be configured to capture images of subsurface anatomy structures (e.g., subsurface tissue behind the outer surface of tissue at the surgical site) included at the surgical site 306. For example, endoscopic imaging mode 302-1 can capture images of surface tissue within the patient's body, and supplementary imaging mode 302-1 can include ultrasound, CT, or MRI imaging capturing images of subsurface tissue that, from an endoscopic perspective, is behind the surface anatomy structures and hidden outside the endoscopic field of view.

[0053] As described above, imaging modal 302 can each capture imaging data 304 of surgical scene 306 at any suitable time, such as during any one or more stages of a surgical procedure or operation. In some examples, imaging modal 302 can capture imaging data 304 of surgical site 306 instantaneously. For example, endoscopic imaging modal 302-1 can capture endoscopic images during a surgical procedure (e.g., during the surgical phase of a surgical procedure), and supplementary imaging modal 302-1 can capture another type of image instantaneously during a surgical procedure. In other examples, imaging modal 302 can capture imaging data 304 of surgical site 306 at different times and / or different stages of a surgical procedure. For example, endoscopic imaging modal 302-1 can capture endoscopic images during the surgical phase of a surgical procedure, while supplementary imaging modal 302-2 can capture another type of image during the preoperative phase of a surgical procedure.

[0054] Imaging data 304 representing surgical site 306 may include images captured at surgical site 306 by imaging modality 302. For example, imaging data 304 may include endoscopic images, ultrasound images, CT images, MRI images, and / or any other suitable form of image of surgical site 306. Additionally or alternatively, imaging data 304 may include one or more surgical site models 306 generated based on imaging performed by the imaging modality. For example, additional imaging data 304-2 may include a 3D model of surgical site 306 generated based on imaging performed by an imaging modality, such as imaging performed by an ultrasound machine, CT machine, MRI machine, or other suitable imaging modality. The 3D model may be a full-volume model comprising voxels (i.e., volumetric pixels) having values ​​(e.g., color values, brightness values, etc.) representing the appearance of surgical site 306 at 3D points within the model. Such a volumetric model can facilitate the recognition and use of any slice of the 3D model by system 100 to generate images of 3D model slices.

[0055] Although Figure 3 Two imaging modalities 302-1 and 302-2 are depicted, respectively capturing imaging data 304-1 and 304-2 as input to system 100. However, other examples may include any suitable number and / or configuration of multiple different imaging modalities that capture images as input to system 100 for generating a synthetic image of surgical site 306. For example, three or more different imaging modalities may capture images input to system 100 for generating a synthetic image of surgical site 306.

[0056] System 100 can generate a synthetic image 312 of surgical site 306 (e.g., including one or more synthetic images 312) based on imaging data 304 captured by imaging modality 302. System 100 can perform this operation in any of the ways described herein to generate a synthetic image that includes a merged representation of portions of surgical site 306 captured by different imaging modalities 302. Examples of such synthetic images and how to generate synthetic images will be described in more detail below.

[0057] System 100 may instruct display device 314 to display composite image 312. For example, system 100 may provide display device 314 with data representing composite image 312, and display device 314 may be configured to display composite image 312 for viewing by users of the computer-assisted surgical system (e.g., surgeons or other surgical team members performing surgical procedures). Display device 314 may include any device capable of receiving and processing imaging data to display one or more images. For this purpose, display device 314 may include one or more displays on which images may be displayed. In some examples, display device 314 may be a component of the computer-assisted surgical system or communicatively connected to the computer-assisted surgical system, as will be described in more detail below.

[0058] Figure 4A The illustration depicts an exemplary aspect of an endoscopic imaging modality for capturing endoscopic imaging data depicting anatomical structures at the surgical site. More specifically, Figure 4A An exemplary depiction of a surgical site 306 captured by an endoscope 402 is shown. The endoscope 402 is implemented as any suitable endoscopic instrument and associated with a viewpoint 404 positioned relative to the surgical site 306. Although Figure 4A A 2D view is shown, but the principles described for the 2D view also apply to a 3D view of a surgical site with a viewpoint positioned relative to the surgical site.

[0059] like Figure 4A As shown, surgical site 400 includes anatomical structures 406, which may include any suitable anatomical structures (e.g., organs or other tissues), and are depicted in various layers or cross-sections. For illustrative reasons, it will become apparent that the individual layers or cross-sections represent the relative depth of the anatomical structures, rather than being depicted using, for example, accurate anatomical details of actual internal organs. It should be understood that surface anatomical structures directly visible to endoscope 402 are marked with a series of "1"s (i.e., "1 1 11 1 1 1 1...") and may represent the above... Figure 3 The surface anatomy structure 308 is shown. Furthermore, various subsurface anatomy structures (and thus in) are shown in cross-section behind or below the surface anatomy structure relative to viewpoint 404 (therefore in...). Figure 3The anatomical structures 308 (not visible in the surface) are marked with a series of "2"s (i.e., "2 2 2 2 2 2 2 2 2...") and "3"s (i.e., "3 3 3 3 3 3 3 3 3 3 3...") based on their relative depth from the surface. Subsurface anatomy can include any part of anatomical structure 406 located behind the surface anatomy layer marked with "1" from viewpoint 404 and / or hidden from viewpoint 404 by the surface anatomy. In some examples, surface anatomy can include the patient's outer layer of tissue, while subsurface anatomy can include anatomical structures embedded within the outer layer of tissue.

[0060] As shown, viewpoint 404 can be associated with endoscope 402, for example, as a viewpoint from which a user of endoscope 402 observes surgical site 306 (e.g., the viewpoint of one or more cameras of the endoscope). Therefore, it should be understood that viewpoint 404 can also be used as a viewpoint from which a composite image of surgical site 306 (e.g., composite image 312) can be presented. In some examples, instead of being the actual viewpoint of endoscope 402, viewpoint 404 can be a virtual viewpoint corresponding to a viewpoint of another imaging modality different from the endoscope's imaging modality, or it can be another arbitrary viewpoint relative to surgical site 306. Viewpoint 404 can be associated with and / or represent the intrinsic and extrinsic properties of an imaging device such as one or more cameras of an endoscope. Viewpoint 404 can have a field of view 408 within which endoscope 402 can capture an image of surgical site 306 and / or present a composite image of surgical site 306 within that field of view.

[0061] Figure 4A A subsurface imaging module 410, which can be associated with endoscope 402, is further described. For example, subsurface imaging module 410 can be implemented as an ultrasound module or another suitable subsurface imaging module as described herein. Both endoscope 402 and subsurface imaging module 410 can be controlled by the manipulation system of a computer-assisted surgical system (as will be described in more detail below), such that the computer-assisted surgical system can track endoscope 402 and subsurface imaging module 410 in a manner that allows knowledge of their relative positions (e.g., using kinematic, visual tracking, etc.). More specifically, the computer-assisted surgical system can generate, track, and maintain alignment parameters that represent the registration of endoscopic imaging data captured by endoscope 402 with subsurface imaging data captured by subsurface imaging module 410.

[0062] System 100 can access alignment parameters representing the registration of endoscopic imaging data with subsurface imaging data. Therefore, system 100 can accurately determine the position of any anatomical structure represented within the subsurface imaging data captured by subsurface imaging module 410 (e.g., any anatomical structure included within the capture field 412 associated with subsurface imaging module 410) relative to viewpoint 404. As an example, any subsurface structure represented within the subsurface imaging data representing anatomical structures in capture field 412 (e.g., any particular vascular system, particular bone structure, particular tissue block, etc.) can be automatically registered to endoscope 402 based on alignment parameters, such that system 100 has updated data indicating the position and geometry of the subsurface structure relative to viewpoint 404.

[0063] While endoscope 402 can be configured to capture only endoscopic imaging data depicting the surface of anatomical structure 406, and subsurface imaging module 410 can be configured to capture subsurface imaging data only at specific cross-sections within capture field 412, various other portions or view types of anatomical structure 406 can also be available and beneficial to the user performing procedures at surgical site 306. As described above, some of these additional portions or views of anatomical structure 406 can be captured simultaneously or at different times by other imaging modalities while the endoscope 402 and subsurface imaging module 410 are capturing imaging data. Such additional imaging data can be used in conjunction with information about... Figure 4A To the extent that the described endoscopic and / or subsurface imaging data are accurately registered, it may be desirable that these other anatomical portions or views of the anatomical structure 406 be presented to the user from viewpoint 404 (e.g., together with views depicted by endoscopic imaging data).

[0064] To illustrate, Figure 4B An exemplary aspect of the representation 414 of the anatomical structure 406 at the surgical site 306 is shown, which is composed of different Figure 4A Additional imaging modal capture of the endoscopic imaging modality shown. Because representation 414 can be captured using an imaging modality different from the endoscopic imaging data captured by endoscope 402, representation 414 can include data representing additional portions, views, and insights into anatomical structures 406. For example, representation 414 could represent a different or more comprehensive representation (e.g., a 3D model or the like) of anatomical structures 406 at surgical site 306. Specifically, as shown in the figure, Figure 4A and Figure 4BDifferent portions 416 of the anatomical structure 406 at the surgical site 306 are labeled (e.g., portions 416-A to 416-C), and different depths (labeled "1", "2", "3", etc.). However, while only portion 416-B and depths labeled "1" to "3" are shown as being imaged by endoscope 402 and subsurface imaging module 410, all portions 416-A to 416-C, along with depths labeled "2" to "6" and beyond (indicated by ellipses), are shown within representation 414. (It will be understood that the portions and depths of the illustrated anatomical structure 406 have arbitrary sizes, appearances, and numbers for illustrative purposes, and any additional portions and / or depths of suitable size or appearance may be included in other examples.) Therefore, representation 414 may not represent the surface of the anatomical structure 406 or may be presented in a different manner than indicated by other representations. Figure 4A The way the endoscopic imaging data represents surface anatomy makes direct comparison of surface anatomy features unlikely to be a feasible, efficient, effective, and / or convenient way to register representation 414 with endoscopic imaging data.

[0065] Representation 414 can be implemented as a 3D anatomical model or other such data structure, and can be generated based on additional imaging data captured via a non-endoscopic imaging modality. For example, representation 414 can be generated based on CT scan data, MRI scan data, ultrasound scan data, or any other imaging data that may serve a particular embodiment. In some examples, the data on which representation 414 is based may be captured at a different time than the endoscopic imaging data captured by endoscope 402. For example, as described above, endoscopic imaging data may be captured intraoperatively, while additional imaging data (and a 3D model of representation 414) may have been captured preoperatively.

[0066] return Figure 3 In some examples, endoscopic imaging data 304-1 can be implemented by data depicting the surface layer of anatomical structure 406 (i.e., anatomical structure 308) captured by endoscope 402 from viewpoint 404. In these examples, supplementary imaging data 304-2 can refer to data representing representation 414 (i.e., including subsurface regions of anatomical structure 406 captured by another imaging modality independent of endoscope 402). As shown, both endoscopic imaging data 304-1 and supplementary imaging data 304-2 can be considered by system 100 to generate or correct registration of endoscopic imaging data 304-1 and supplementary imaging data 304-2, and generate a synthetic image 312.

[0067] To perform this registration, system 100 can identify and match corresponding aspects of endoscopic imaging data 304-1 and supplementary imaging data 304-2. However, as stated above, while both imaging data 304-1 and 304-2 can represent certain common parts of anatomical structure 406, such as part 416-B (albeit in different ways and / or to different degrees or with different extents), there may be any visually identifiable features that are not shared by the depiction of endoscopic imaging data 304-1 and the representation 414 of supplementary imaging data 304-2. Therefore, based on the above regarding... Figure 2 The described method of performing registration of imaging data 304-1 and imaging data 304-2 through feature matching may be infeasible, inefficient, effective, convenient, or even impossible.

[0068] To illustrate, Figure 5 An exemplary aspect of the registration of endoscopic imaging data 304-1 (e.g., captured by endoscope 402 via an endoscopic imaging modality) with supplementary imaging data 304-2 (e.g., captured via an additional imaging modality different from the endoscopic imaging modality) is shown. As illustrated, endoscopic imaging data 304-1 depicts the surface anatomy at surgical site 306 (i.e., portions of anatomy 406 marked with "1"). This depiction of surface anatomy will be understood as being captured by endoscope 402 from viewpoint 404 at surgical site 306. Furthermore, Figure 5 Additional imaging data 304-2 is shown, representing the subsurface anatomy at surgical site 306 (i.e., the portion of anatomy 406 marked with "2"). This subsurface anatomy will be understood as being occluded from viewpoint 404 by surface anatomy, such that it is not represented within imaging data 304-1.

[0069] therefore, Figure 5 The depiction of endoscopic imaging data 304-1 in the data describes various "1"s to indicate the depiction of the surface layer, and further illustrates the representations as above regarding... Figure 2 Various surface features 204 of the type visible in the surface anatomy. In a similar manner, Figure 5 The additional imaging data 304-2 depicts various “2”s to indicate that a specific subsurface cross section is depicted (e.g., with...). Figure 4B (The cross section associated with depth marked "2" in the diagram). It will be understood that, in some examples, the additional imaging data 304-2 may include 3D representations of more than a single cross section. For example, the additional imaging data 304-2 may include 3D models combining various depths from representation 414.

[0070] Regardless of which subsurface portions are accurately represented in the supplementary imaging data 304-2, Figure 5The description of the additional imaging data 304-2 illustrates the corresponding feature 206 (above regarding...). Figure 2 The description corresponds to feature 204, which may not exist in the representation of supplementary imaging data 304-2. If it exists within supplementary imaging data 304-2, it indicates that the "X" of each feature 206 is present in... Figure 5 The image shows the location where the feature (relative to viewpoint 404) will be located, but the "X" is drawn with a dashed line to indicate that these features do not exist in the additional imaging data 304-2. Therefore, in this example, in order to register imaging data 304-1 with imaging data 304-2, it will be impossible to identify and match features 204 and 206, and it may be necessary to identify another relevant feature in the corresponding imaging dataset in order to perform registration accurately.

[0071] As already described, the features used for registration (e.g., as alternatives to visual features 204 and 206 when neither is available in the corresponding imaging data to be registered) may be related to identifiable subsurface structures. For example, such subsurface structures could be vascular systems at the surgical site, non-vascular anatomical structures including at least one of bone structures or tissue blocks, or another suitable subsurface structure. Such subsurface structures may be useful for performing the registration procedure because, as mentioned above, these subsurface structures can all be: 1) detected and registered to endoscopic imaging data 304-1 (e.g., through the subsurface imaging module 410 and its predefined and tracked relationship with the endoscope 402, such as regarding...). Figure 4A The descriptions 1) and 2) are included in the additional imaging data 304-2 because subsurface structures exist throughout the entire subsurface anatomy represented by 414. For illustration, multiple squares, each representing a specific feature (e.g., cross-section of a blood vessel, features of bone structure, features of tissue blocks, etc.) of a subsurface structure detectable within the two imaging data sets 304-1 and 304-2, are shown below. Figure 5 As shown. In endoscopic imaging data 304-1, the features of the subsurface structures represented by these squares will be individually referred to as features 502-1 to 502-6, and collectively referred to as subsurface structure 502. In supplementary imaging data 304-2, the corresponding subsurface features represented by squares will each be individually referred to as features 504-1 to 504-6, and collectively referred to as subsurface structure 504.

[0072] Subsurface structures 502 can be identified within endoscopic imaging data 304-1 based on subsurface imaging data detected by subsurface image scanning at surgical site 306 by subsurface imaging module 410. For example, subsurface structures 502 can be identified using vascular imaging scanning (e.g., Doppler ultrasound scanning or other types of 2D or 3D vascular imaging scanning) or other subsurface imaging scanning described herein, to identify and generate data representing subsurface structures 502. In examples of Doppler ultrasound scanning (e.g., in examples where subsurface structures 502 and 504 are realized by a vascular system and each of features 502-1 to 502-6 and 504-1 to 504-6 is realized by a separate blood vessel), the Doppler ultrasound device can employ the Doppler principle (e.g., the Doppler effect) to detect fluid flow in tissue. At locations where Doppler ultrasound detects fluid movement in a relatively small space through an anatomical structure, the presence of veins, arteries, or other blood vessels can be inferred or otherwise determined. For example, Doppler ultrasound can indicate the direction of fluid movement using different colors (e.g., red or blue), and these colors can be superimposed (or mixed or otherwise incorporated into) the endoscopic imaging data 304-1 at locations indicated by squares representing blood vessels (i.e., features 502-1 to 502-6). Based on the flow direction, the diameter of different blood vessels, the pattern of vascular geometry, and / or any other suitable criteria, system 100 can differentiate different types of blood vessels (e.g., arteries, veins, etc.) and / or can identify specific blood vessels as feature 502. While vascular systems and blood vessels are described in this instance as a specific example, it should be understood that other types of non-vascular subsurface structures (e.g., bone structures, tissue blocks, etc.) can be used similarly to their corresponding features in a manner similar to that used to describe the structural features of vascular systems and blood vessels.

[0073] Subsurface structures 504 can be identified within the supplementary imaging data 304-2 based on other methods suitable for the imaging modality used to capture the supplementary imaging data 304-2. For example, subsurface structures 504 can be identified from data captured in CT or MRI scans, or in any other suitable manner applicable to the specific imaging modality being used (such as other imaging modalities described herein). Where a comprehensive representation, such as a 3D model, has already been constructed from such data, that representation may include metadata that distinguishes different types of subsurface features 504 (e.g., vascular, skeletal, or tissue features) and / or identifies specific subsurface features 504.

[0074] System 100 can be configured to register endoscopic imaging data 304-1 with supplementary imaging data 304-2 using subsurface structures 502 and 504 in any suitable manner. For example, in some embodiments, registering imaging data 304-1 with imaging data 304-2 using the identified subsurface structures 502 and 504 may include initially generating the registration by, for example, generating appropriate alignment parameters that define any type of spatial transformation (e.g., rigid transformation, affine transformation, deformable registration, etc.) between the endoscopic imaging data and the supplementary imaging data as described herein. In other embodiments, or at a later time in the same embodiment, registering endoscopic imaging data 304-1 with imaging data 304-2 using the identified subsurface structures 502 and 504 may include correcting the registration of endoscopic imaging data 304-1 with supplementary imaging data 304-2 (e.g., the previously generated registration) after the initial registration generation. For example, this can be done by modifying, adjusting, correcting, updating, or otherwise calibrating one or more alignment parameters that define a specific spatial transformation from the endoscope to the additional imaging data.

[0075] More specifically, in some examples, the use of identified subsurface structures 502 and 504 for registration of endoscopic imaging data 304-1 with supplementary imaging data 304-2 can be performed by system 100 in the following manner: 1) accessing endoscopic imaging data 304-1 from endoscope 402 that captures endoscopic imaging data 304-1; 2) accessing subsurface imaging data from subsurface imaging module 410 that performs subsurface imaging scans; 3) accessing alignment parameters representing the registration of endoscopic imaging data 304-1 with subsurface imaging data; 4 Based on subsurface imaging data and alignment parameters, a first position of feature 502 present in the identified subsurface structure is determined relative to the anatomical structure 406 represented by endoscopic imaging data 304-1; 5) a second position of feature 504 corresponding to feature 502 (i.e., a feature representing the same as feature 502) is determined relative to the anatomical structure 406 represented by additional imaging data 304-2; and 6) a registration of endoscopic imaging data 304-1 and additional imaging data 304-2 is generated or corrected based on the first and second positions. Feature 502 present in the identified subsurface structure can represent blood vessels or other features in any way. For example, feature 502 can be the presence of a specific blood vessel, a specific point on a specific blood vessel (e.g., identified using bifurcation), registration of multiple blood vessels (e.g., without distinguishing individual blood vessels), or any other identifiable feature associated with one or more blood vessels or other suitable subsurface features. Similarly, feature 504 corresponding to feature 502 can be any of the same type of feature described above but represented as in additional imaging data 304-2.

[0076] When these operations are repeated on several subsurface features, very accurate registration between imaging data 304-1 and imaging data 304-2 can be achieved. Therefore, in some embodiments, identifying the correlation between feature pairs 502 and 504 in this way can be the primary or sole basis for registering endoscopic imaging data 304-1 with supplementary imaging data 304-2.

[0077] However, in other embodiments, the correlation between the subsurface structure 502 in endoscopic imaging data 304-1 and the subsurface structure 504 in supplementary imaging data 304-2 can be used as just one of several factors for generating or correcting registration. For example, in some examples, the correlated subsurface structure (e.g., a specific subsurface feature identified in both imaging data 304-1 and 304-2) can be used as an anchor for registration, and other factors such as other features or data detected by other imaging modalities can also be used to complete registration in the most efficient and accurate manner.

[0078] As an example, system 100 can further use the identified subsurface structures 502 and 504 for registration by accessing auxiliary data representing additional features present at the surgical site and different from the subsurface structures. In these embodiments, the generation or correction of registration includes: 1) anchoring the alignment of endoscopic imaging data 304-1 and additional imaging data 304-2 based on the first and second positions and relative to viewpoint 404, and 2) refining the alignment of endoscopic imaging data 304-1 and additional imaging data 304-2 based on the auxiliary data.

[0079] In some such examples, subsurface structures 502 and 504 may include a vascular system at the surgical site, and the additional features present may be additional, non-vascular system anatomy. In these examples, one or more vessels (or features thereof) implementing features 502 and 504 for such anchoring may be major vessels or groups of vessels (e.g., renal hilum, etc.) and may be selected as anchor points in any suitable manner. For example, vessels may be automatically selected as anchor points based on their size relative to other identified vessels (e.g., the selected vessel may be the largest), their importance relative to other identified vessels, the ease with which they are uniquely identified relative to other vessels, and / or based on other suitable criteria. In other examples, system 100 may rely on manual specification rather than automatic selection of vessels to be used as anchor points. Specifically, for example, registering endoscopic imaging data 304-1 with additional imaging data 304-2 using the identified vascular system may include system 100 detecting a user selection (e.g., by a surgeon or other user via an interactive user interface or the like) of a specific vessel in the identified vascular system to be used as an anchor point. The determination of the first and / or second location of a specific vessel can then be performed based on the user's selection of the specific vessel detected. Once the selected vessel is used for anchor registration (e.g., once a specific vessel is aligned with its corresponding vessel), the registration can be further refined relative to other degrees of freedom to fully align and register the image dataset based on other non-vascular system-related data (e.g., visible features, ultrasound data other than vascular imaging data, etc.).

[0080] In other examples employing non-vascular subsurface structures such as bone structures or tissue blocks, registering endoscopic imaging data 304-1 with supplementary imaging data 304-2 using the identified subsurface structure may include system 100 similarly detecting a user selection (e.g., by a surgeon or other user via an interactive user interface or similar) of a specific feature of the identified subsurface structure to be used as an anchor point. The determination of a first and / or second location of the specific feature can then be performed based on the detected user selection of that specific feature. Furthermore, once the selected feature is used for anchor registration (e.g., once a specific feature of the subsurface structure is aligned with a corresponding feature in another imaging data view), the registration can be further formulated relative to other degrees of freedom to fully align and register the image dataset based on supplementary features (e.g., surface anatomical features, other subsurface anatomical features visible in the subsurface imaging scan, etc.).

[0081] One example of additional features that registration refinement can be based on is cauterization marks applied by the surgeon to surface anatomy during surgical preparation to mark anatomical regions. For example, the surgeon may use a cauterization instrument to apply one or more cauterization marks (i.e., cauterization markers) to delineate or otherwise mark the area on which surgery will be performed (e.g., marking the boundaries of tissue blocks to be removed, etc.). Such cauterization marks are readily identifiable in endoscopic imaging data 304-1 and can be associated in a predefined manner with anatomical structures (e.g., tissue blocks, bone structures, etc.) readily identifiable in supplementary imaging data 304-2. Therefore, these marks may be useful in some embodiments for refining registrations that have already been anchored using the subsurface structures described herein.

[0082] As mentioned above Figure 3 Once the system 100 accurately registers the endoscopic imaging data 304-1 with the supplementary imaging data 304-2, the system 100 can generate one or more synthetic images (e.g., a sequence of synthetic images or a video) based on the registration. These synthetic images illustrate a description of the anatomical structure 406 derived from the endoscopic imaging data 304-1 and a view of the anatomical structure 406 represented in the supplementary imaging data 304-2. The system 100 can provide synthetic images of the surgical site 306 for display on a user's display device, such as that of a surgeon or other surgical team member.

[0083] To illustrate, Figure 6 An exemplary synthetic image 600 of a surgical site, such as surgical site 306, is shown. The synthetic image 600 may consist primarily of endoscopic images (e.g., real-time endoscopic image feeds) captured by the endoscope 402 from the viewpoint 404 (e.g., endoscopic imaging data 304-1). Therefore, Figure 6The composite image 600 is shown to largely depict anatomical structures 308 (e.g., the surface anatomical portion of anatomical structure 406) and other elements present at the surgical site 306, such as instruments 602 and subsurface imaging modules 410. However, as a composite image, image 600 shows more than just an image captured via an endoscopic imaging modality. Specifically, as shown, composite image 600 further includes image enhancement 604, implemented as overlay or other enhancements (e.g., hybrid or integrated enhancements) and includes representations of additional imaging data 304-2 captured via an additional imaging modality (e.g., cross-sectional depictions, 3D models, etc.). For example, the depiction of image enhancement 604 may represent a cross-section captured by an ultrasound scan (e.g., preoperative or intraoperative ultrasound scan), a portion of a 3D model generated based on data from a CT or MRI scan, an image derived from a fluorescence or X-ray scan, or any other suitable image from any suitable imaging modality as described herein or that may serve a particular implementation. Regardless of the type of image displayed in image enhancement 604 or the imaging modality used to capture it, the image in image enhancement 604 can display subsurface anatomy structures that are undetectable by the endoscope (e.g., visible) and correctly aligned with the endoscope's viewpoint 404 (e.g., as a result of the registration described above), so as to be located where the image would be detectable by the endoscope. Therefore, a viewer of synthetic image 600 can perceive image enhancement 604 as a "window" to see the desired view of subsurface anatomy structures beneath the surface anatomy.

[0084] Along with images detected by the endoscope (e.g., anatomical structures 308, instruments 602, etc.) and image enhancement 604, the synthesized image can also display other images or information in some embodiments (in... Figure 6 (Not explicitly shown in the example). For example, subsurface imaging data (e.g., red or blue Doppler ultrasound points representing blood vessel 502, etc.), additional image enhancements depicting similarly aligned subsurface anatomy captured by other additional imaging modalities, status information, and / or any other suitable data or images may be displayed in the synthetic image 600 as it may serve a particular implementation.

[0085] In some examples, system 100 may perform certain functions or checks before including image enhancement 604 in the composite image 600. For example, when subsurface imaging module 410 performs a subsurface imaging scan to provide subsurface imaging data, system 100 may verify the subsurface imaging data (e.g., by ensuring proper contact between the subsurface imaging module and anatomical structure 308 to obtain good readings (if applicable)). Thus, system 100 may register endoscopic imaging data 304-1 with additional imaging data 304-2 using the identified subsurface structure based on the verification of the subsurface imaging data. For example, for subsurface imaging scans using ultrasound modules that require contact with tissue to function properly, system 100 may generate registration only on the first contact, correct the registration only on each subsequent contact, and so on.

[0086] Figure 6 The synthetic image 600 showing surgical site 306 includes a first depiction (i.e., surface anatomy 308) of anatomical structure 406 represented by endoscopic imaging data 304-1 combined with a second depiction (i.e., subsurface anatomy) of anatomical structure 406 represented by additional imaging data 304-2, wherein the first and second depictions are aligned within the synthetic image relative to a viewpoint (e.g., viewpoint 404) associated with the synthetic image 600. However, as stated above, it should be understood that the identified subsurface structures 502 and 504 used for registering endoscopic imaging data 304-1 with additional imaging data 304-2 are independent of (i.e., may or may not be the same or related) the anatomical structures in the first and second depictions included in the synthetic image 600. In practice, while for some examples, typical subsurface structures 502 and 504 for registration may be depicted in the synthetic image 600 (e.g., as Doppler ultrasound color, as a 3D model of the vascular system, etc.), different subsurface structures or no subsurface structures at all may be shown in other examples of the synthetic image 600. More specifically, for any given implementation of system 100 and based on the environment of the surgical procedure, surgeon preferences, etc., the synthetic image 600 may or may not display the subsurface imaging data as an enhancement (e.g., image overlay), may or may not display the same region of the 3D model represented by the additional imaging data 304-2 as the region used for registration, may or may not display the same subsurface depth level as the level of the cross section on which the subsurface imaging data detection registration is based, and so on.

[0087] To illustrate, Figures 7 to 9 Additional exemplary synthetic images of surgical site 306 are shown, each including a depiction of anatomical structure 406, captured by different imaging modalities and aligned relative to the viewpoint associated with the image within the corresponding synthetic image. Figures 7 to 9In each of these, the image displayed in the corresponding image enhancement is different, in order to illustrate that image enhancement can be independent of the registration of the image data. For clarity, Figures 7 to 9 The above information is shown. Figure 4A and Figure 4B The anatomical parts and depths of the numbering scheme are introduced.

[0088] Figure 7 The illustration shows a composite image 700 depicting the surface anatomy 308 (represented by "1") at the surgical site, instrument 602, subsurface imaging module 410, and image enhancement 702. Image enhancement 702 can be any suitable shape, including the exemplary shape shown, and can be made to appear attached to instrument 602 or subsurface imaging module 410 (e.g., in a manner similar to a flag attached to a flagpole). In this example, image enhancement 702 depicts the subsurface anatomy at a depth represented by "2," which, as described above, can be at the same level as the cross-sectional view of the subsurface structure 502 provided by subsurface imaging module 410 for registering the additional imaging data of image enhancement 702 with endoscopic imaging data depicting anatomy 308.

[0089] and Figure 7 similar, Figure 8 A synthetic image 800 is shown, which also depicts surface anatomy 308 (represented by "1") at the surgical site, instrument 602, and subsurface imaging module 410, and can also be generated based on registration using subsurface imaging data representing anatomy at a depth represented by "2". However, with Figure 7 Compared to the synthetic image 700, the synthetic image 800 shows an image enhancement 802 depicting a different subsurface anatomy than that used for registration, namely the subsurface anatomy at a depth indicated by “3”.

[0090] and Figure 7 and Figure 8 similar, Figure 9 A synthetic image 900 is shown, which also depicts surface anatomy 308 (represented by "1") at the surgical site, instrument 602, and subsurface imaging module 410, and can also be generated based on registration using subsurface imaging data representing the anatomy at the same portion of the surgical site where the subsurface imaging module 410 is located. However, with Figure 7 Composite image 700 and Figure 8Compared to the synthetic image 800, the synthetic image 900 shows an image enhancement 902, which is of a different shape and depicts the surface anatomy at a portion 904 that is different from the portion scanned by the subsurface imaging data used for registration (i.e., portion 904 different from the portion scanned by the subsurface imaging module 410). Specifically, if the area of ​​the surgical site depicted in the synthetic image 900 is roughly divided into different portions 904 (i.e., portions 904-A to 904-C), it can be seen that the area 906 where the subsurface imaging module 410 and the anatomy were scanned during the subsurface image scanning is located in portion 904-A, while the image enhancement 902 is displayed in a different portion, i.e., portion 904-B. Therefore, to indicate the depth and portion displayed in the image enhancement 902, "2B" is indicated to indicate depth "2" and portion 904-B.

[0091] As already mentioned, system 100 can be implemented in or communicatively coupled to a computer-assisted surgical system. System 100 can receive input from the computer-assisted surgical system and provide output thereto. For example, system 100 can access images of the surgical site and / or any information about the surgical site and / or the computer-assisted surgical system from the computer-assisted surgical system, use the accessed images and / or information to perform any of the processes described herein to generate a synthetic image of the surgical site, and provide data representing the synthetic image to the computer-assisted surgical system for display (e.g., via a display device associated with the computer-assisted surgical system).

[0092] To illustrate, Figure 10 An exemplary computer-assisted surgical system 1000 (“Surgical System 1000”) is illustrated. System 100 may be implemented by Surgical System 1000, may be connected to Surgical System 1000, and / or may be used in combination with Surgical System 1000 in other ways.

[0093] As shown in the figure, the surgical system 1000 may include a control system 1002, a user control system 1004, and an auxiliary system 1006 that are communicatively coupled to each other. The surgical team can utilize the surgical system 1000 to perform computer-aided surgical procedures on the patient 1008. As shown, the surgical team may include a surgeon 1010-1, an assistant 1010-2, a nurse 1010-3, and an anesthesiologist 1010-4, all of whom can be collectively referred to as "surgical team members 1010". Additional or alternative surgical team members may be present during the surgical session as they may serve a specific implementation.

[0094] although Figure 10The illustration depicts a minimally invasive surgical procedure in progress; however, it should be understood that the surgical system 1000 can be similarly used to perform open surgical procedures or other types of surgical procedures, which can similarly benefit from the accuracy and convenience of the surgical system 1000. Furthermore, it should be understood that surgical sessions using the surgical system 1000 can include not only the operational phases of the surgical procedure, such as... Figure 10 As shown, it can also include preoperative, postoperative and / or other appropriate surgical procedures.

[0095] like Figure 10 As shown, the manipulation system 1002 may include a plurality of manipulator arms 1012 (e.g., manipulator arms 1012-1 to 1012-4), to which a plurality of surgical instruments (e.g., instruments 602 as shown above) may be coupled. Each surgical instrument may be implemented by any suitable surgical tool (e.g., a tool with tissue interaction capabilities), medical instrument, imaging device (e.g., endoscope, ultrasound tool, etc.), sensing device (e.g., force-sensing surgical instrument), diagnostic instrument, or analogues that can be used to perform computer-assisted surgical procedures on a patient (e.g., by being at least partially inserted into and manipulated to perform computer-assisted surgical procedures on the patient 1008). In some examples, the surgical instrument may be implemented by an ultrasound module (e.g., ultrasound module 410), or such an ultrasound module may be connected to or coupled to one of the other surgical instruments described above. Although the control system 1002 is depicted and described herein as comprising four manipulator arms 1012, it should be recognized that the control system 1002 may comprise only a single manipulator arm 1012 or any other number of manipulator arms, as may be used in a particular implementation.

[0096] The manipulator arm 1012 and / or surgical instruments attached to the manipulator arm 1012 may include one or more displacement transducers, orientation sensors, and / or position sensors for generating raw (i.e., uncorrected) kinematic information. One or more components of the surgical system 1000 may be configured to use kinematic information to track surgical instruments (e.g., determine their position) and / or control surgical instruments (and anything connected to the instruments, such as an ultrasound module).

[0097] User control system 1004 can be configured to facilitate surgeon 1010-1's control of manipulator arm 1012 and surgical instruments attached to manipulator arm 1012. For example, surgeon 1010-1 can interact with user control system 1004 to remotely move or manipulate manipulator arm 1012 and surgical instruments. To this end, user control system 1004 can provide surgeon 1010-1 with images of the surgical site associated with patient 1008 captured by an imaging system (e.g., any medical imaging system described herein), including high-resolution 3D images such as images 600, 700, 800, or 900, or other suitable synthetic images. In some examples, user control system 1004 may include a stereoscopic viewer with two displays, where surgeon 1010-1 can view stereoscopic images of the surgical site associated with patient 1008 and generated by a stereoscopic imaging system. In some examples, synthetic images generated by system 100 may be displayed by user control system 1004. The surgeon 1010-1 can use images displayed by the user control system 1004 to perform one or more procedures with one or more surgical instruments attached to the manipulator arm 1012.

[0098] To facilitate control of surgical instruments, the user control system 1004 may include a set of master controls. These master controls can be manipulated by the surgeon 1010-1 to control the movement of surgical instruments (e.g., by utilizing robotic and / or teleoperation technologies). The master controls can be configured to detect various hand, wrist, and finger movements of the surgeon 1010-1. In this way, the surgeon 1010-1 can intuitively perform procedures using one or more surgical instruments.

[0099] The auxiliary system 1006 may include one or more computing devices configured to perform primary processing operations of the surgical system 1000. In such a configuration, the one or more computing devices included in the auxiliary system 1006 may control and / or coordinate operations performed by various other components of the surgical system 1000 (e.g., the manipulation system 1002 and the user control system 1004). For example, the computing devices included in the user control system 1004 may transmit instructions to the manipulation system 1002 via one or more computing devices included in the auxiliary system 1006. As another example, the auxiliary system 1006 may receive (e.g., from the manipulation system 1002) and may process image data representing images captured by an imaging device attached to one of the manipulator arms 1012.

[0100] In some examples, the assistive system 1006 may be configured to present visual content to a surgical team member 1010 who may not have access to images provided to the surgeon 110-1 at the user control system 1004. To this end, the assistive system 1006 may include a display monitor 1014 configured to display one or more user interfaces, such as images of the surgical site (e.g., 2D images, 3D images, composite images such as images 600, 700, 800, 900, etc.), information associated with the patient 1008 and / or surgical procedures, and / or any other visual content that may serve a particular implementation. For example, the display monitor 1014 may display images of the surgical site (e.g., composite images generated by system 100) and additional content displayed concurrently with the images (e.g., graphical content, contextual information, etc.). In some embodiments, the display monitor 1014 is implemented as a touchscreen display that the surgical team member 1010 can interact with (e.g., via touch gestures) to provide user input to the surgical system 1000.

[0101] The operating system 1002, the user control system 1004, and the auxiliary system 1006 can be communicatively coupled to each other in any suitable manner. For example, such as Figure 10 As shown, the operating system 1002, user control system 1004, and auxiliary system 1006 can be communicatively coupled via control line 1016, which can represent any wired or wireless communication link suitable for a particular implementation. Therefore, the operating system 1002, user control system 1004, and auxiliary system 1006 can each include one or more wired or wireless communication interfaces, such as one or more local area network interfaces, Wi-Fi network interfaces, cellular interfaces, etc.

[0102] Figure 11 An exemplary method 1100 for registering imaging data from different imaging modalities based on subsurface image scanning is illustrated. Although Figure 11 The figure illustrates an exemplary operation according to one embodiment, but other embodiments may omit, add, reorder, combine, and / or modify it. Figure 11 Any of the operations shown. Figure 11 One or more of the operations shown can be performed by an image registration system such as system 100, any of the components included therein, and / or any implementation thereof.

[0103] In operation 1102, the image registration system can identify subsurface structures at the surgical site based on subsurface imaging data from subsurface image scans at the surgical site. Operation 1102 can be performed in any of the manner described herein.

[0104] In operation 1104, the image registration system can use the subsurface structure identified in operation 1102 to register endoscopic imaging data from the endoscopic imaging modality with additional imaging data from the additional imaging modality. Operation 1104 can be performed in any of the manner described herein.

[0105] Figure 12 Another exemplary method, method 1200, is illustrated for registering imaging data from different imaging modalities based on subsurface image scanning. Although Figure 12 The figure illustrates an exemplary operation according to one embodiment, but other embodiments may omit, add, reorder, combine, and / or modify it. Figure 12 Any of the operations shown. Figure 12 One or more of the operations shown can be performed by an image registration system such as system 100, any of the components included therein, and / or any implementation thereof.

[0106] In operation 1202, the image registration system can access intraoperative endoscopic imaging data from an endoscope associated with the image registration system. For example, the endoscope can be configured to capture intraoperative endoscopic imaging data. Operation 1202 can be performed in any of the manner described herein.

[0107] In operation 1204, the image registration system can access intraoperative subsurface imaging data from the subsurface imaging module associated with the image registration system. For example, the subsurface imaging module can be configured to capture intraoperative subsurface imaging data. Operation 1204 can be performed in any of the manner described herein.

[0108] In operation 1206, the image registration system can access alignment parameters representing the registration of intraoperative endoscopic imaging data with intraoperative subsurface imaging data. Operation 1206 can be performed in any of the manner described herein.

[0109] In operation 1208, the image registration system can access preoperative scan data. For example, the preoperative scan data can be captured using an additional imaging modality different from the endoscopic imaging modality, and the preoperative scan data can represent a 3D model of the anatomical structures at the surgical site. Operation 1208 can be performed in any of the manner described herein.

[0110] In operation 1210, the image registration system can identify subsurface structures at the surgical site. For example, the image registration system can identify subsurface structures based on intraoperative subsurface imaging data and alignment parameters. Operation 1210 can be performed in any of the manner described herein.

[0111] In operation 1212, the image registration system can use the subsurface structures identified in operation 1210 to register intraoperative endoscopic imaging data accessed in operation 1202 with preoperative scan data of a 3D model representing the anatomical structures at the surgical site, accessed in operation 1208. Operation 1212 can be performed in any of the manner described herein.

[0112] In some examples, a non-transitory computer-readable medium may be provided for storing computer-readable instructions, based on the principles described herein. When executed by a processor of a computing device, the instructions may direct the processor and / or the computing device to perform one or more operations, including one or more operations described herein. Such instructions may be stored and / or transmitted using any of a variety of known computer-readable media.

[0113] As used herein, a non-transitory computer-readable medium may include any non-transitory storage medium that contributes to providing data (e.g., instructions) that can be read and / or executed by a computing device (e.g., by a processor of the computing device). For example, a non-transitory computer-readable medium may include, but is not limited to, any combination of non-volatile storage media and / or volatile storage media. Exemplary non-volatile storage media include, but are not limited to, read-only memory, flash memory, solid-state drives, magnetic storage devices (e.g., hard disks, floppy disks, magnetic tapes, etc.), ferroelectric random access memory (“RAM”), and optical discs (e.g., compact discs, digital video discs, Blu-ray discs, etc.). Exemplary volatile storage media include, but are not limited to, RAM (e.g., dynamic RAM).

[0114] Figure 13 The figure illustrates an exemplary computing device 1300, which may be specifically configured to perform one or more processes described herein. Any system, unit, computing device, and / or other component described herein may be implemented by computing device 1300.

[0115] like Figure 13 As shown, computing device 1300 may include a communication interface 1302, a processor 1304, a storage device 1306, and an input / output (“I / O”) module 1308 that are communicatively connected to each other via communication infrastructure 1310. Although Figure 13 An exemplary computing device 1300 is shown, but Figure 13 The components shown are not intended to be limiting. Additional or alternative components may be used in other embodiments. A more detailed description will now follow. Figure 13 The components of the computing device 1300 shown.

[0116] Communication interface 1302 can be configured to communicate with one or more computing devices. Examples of communication interface 1302 include, but are not limited to, wired network interfaces (such as network interface cards), wireless network interfaces (such as wireless network interface cards), modems, audio / video connections, and any other suitable interfaces.

[0117] Processor 1304 generally refers to any type or form of processing unit capable of processing data and / or interpreting, executing, and / or directing the execution of one or more of the instructions, procedures, and / or operations described herein. Processor 1304 may perform operations by executing computer-executable instructions 1312 (e.g., applications, software, code, and / or other executable data instances) stored in storage device 1306.

[0118] Storage device 1306 may include one or more data storage media, devices, or configurations and may take any type, form, and combination of data storage media and / or devices. For example, storage device 1306 may include, but is not limited to, any combination of non-volatile media and / or volatile media described herein. Electronic data, including the data described herein, may be stored temporarily and / or permanently in storage device 1306. For example, data representing computer-executable instructions 1312 configured to boot processor 1304 to perform any of the operations described herein may be stored in storage device 1306. In some examples, data may be arranged in one or more databases residing within storage device 1306.

[0119] I / O module 1308 may include one or more I / O modules configured to receive user input and provide user output. I / O module 1308 may include any hardware, firmware, software, or a combination thereof that supports input and output capabilities. For example, I / O module 1308 may include hardware and / or software for capturing user input, including but not limited to a full keyboard or keypad, a touchscreen component (e.g., a touchscreen display), a receiver (e.g., an RF or infrared receiver), a motion sensor, and / or one or more input buttons.

[0120] I / O module 1308 may include one or more means for presenting output to a user, including but not limited to a graphics engine, a display (e.g., a screen), one or more output drivers (e.g., display drivers), one or more audio speakers, and one or more audio drivers. In some embodiments, I / O module 1308 is configured to provide graphical data to the display for presentation to the user. The graphical data may represent one or more graphical user interfaces and / or any other graphical content that may be used in a particular implementation.

[0121] In some examples, any of the facilities described herein may be implemented by or within one or more components of computing device 1300. For example, one or more applications 1312 residing in storage device 1306 may be configured to bootstrap processor 1304 to perform one or more operations or functions associated with processing facility 104 of system 100. Similarly, storage facility 102 of system 100 may be implemented by or within storage device 1306.

[0122] Various exemplary embodiments have been described in the foregoing description with reference to the accompanying drawings. However, it will be apparent that various modifications and changes can be made thereto, and additional embodiments can be implemented without departing from the scope of the invention as set forth in the appended claims. For example, certain features of one embodiment described herein may be combined with or substituted for features of another embodiment described herein. Therefore, the specification and drawings are to be considered illustrative rather than restrictive.

Claims

1. A system comprising: a memory storing instructions; and a processor communicatively coupled to the memory and configured to execute the instructions to: identify, at a surgical site, a set of differentiated blood vessels differentiated from other blood vessels based on sub-surface imaging data from a sub-surface image scan at the surgical site; and register endoscopic imaging data from an endoscopic imaging modality with additional imaging data from an additional imaging modality using the set of differentiated blood vessels identified at the surgical site, wherein registering the endoscopic imaging data with the additional imaging data using the set of differentiated blood vessels identified at the surgical site comprises: determining, based on the sub-surface imaging data and alignment parameters representing a registration of the endoscopic imaging data with the sub-surface imaging data, a first position of the set of differentiated blood vessels identified at the surgical site relative to an anatomical structure represented by the endoscopic imaging data; determining a second position of the set of differentiated blood vessels relative to the anatomical structure represented by the additional imaging data; and generating or correcting the registration of the endoscopic imaging data with the additional imaging data based on the first position and the second position.

2. The system of claim 1, wherein the processor is further configured to execute the instructions to provide, based on the registration of the endoscopic imaging data from the endoscopic imaging modality with the additional imaging data from the additional imaging modality, a composite image of the surgical site for display by a display device.

3. The system of claim 2, wherein: the composite image of the surgical site includes a first depiction of an anatomical structure represented by the endoscopic imaging data in combination with a second depiction of an anatomical structure represented by the additional imaging data, the first and second depictions aligned within the composite image relative to a viewpoint associated with the composite image; and the set of differentiated blood vessels identified at the surgical site and used for the registration is distinct from the anatomical structure in the first and second depictions included in the composite image.

4. The system of claim 1, wherein: the endoscopic imaging data depicts surface anatomical structure at the surgical site, the depiction of the surface anatomical structure captured from a viewpoint at the surgical site by the endoscopic imaging modality; and the additional imaging data represents sub-surface anatomical structure at the surgical site, the sub-surface anatomical structure occluded from the viewpoint by the surface anatomical structure such that the sub-surface anatomical structure is not represented in the endoscopic imaging data.

5. The system of claim 1, wherein: the endoscopic imaging modality includes an intraoperative scan of anatomical structure at the surgical site; and the additional imaging modality includes a preoperative scan of the anatomical structure at the surgical site.

6. The system of claim 1, wherein the additional imaging modality includes one of: an ultrasound scan of the surgical site; a computed tomography scan, CT scan, of the surgical site; a magnetic resonance imaging scan, MRI scan, of the surgical site; or a fluorescence imaging scan of the surgical site.

7. The system of any one of claims 1 to 6, wherein registering the endoscopic imaging data with the additional imaging data using the differentiated set of blood vessels identified at the surgical site comprises initially generating the registration of the endoscopic imaging data with the additional imaging data.

8. The system of any one of claims 1 to 6, wherein registering the endoscopic imaging data with the additional imaging data using the differentiated set of blood vessels identified at the surgical site comprises correcting the registration of the endoscopic imaging data with the additional imaging data after initially generating the registration.

9. The system of any one of claims 1 to 6, wherein registering the endoscopic imaging data with the additional imaging data using the differentiated set of blood vessels identified at the surgical site comprises generating or correcting a set of alignment parameters configured to define a spatial transformation between the endoscopic imaging data and the additional imaging data.

10. The system of any one of claims 1 to 6, wherein registering the endoscopic imaging data with the additional imaging data using the differentiated set of blood vessels identified at the surgical site further comprises: accessing the endoscopic imaging data from an endoscope that captured the endoscopic imaging data; accessing the subsurface imaging data from an ultrasound module that performed the subsurface image scan as an ultrasound scan; and accessing the alignment parameters representing the registration of the endoscopic imaging data with the subsurface imaging data.

11. The system of claim 1, wherein: registering the endoscopic imaging data with the additional imaging data using the differentiated set of blood vessels identified at the surgical site further comprises accessing auxiliary data representing additional features present at the surgical site and distinct from the differentiated set of blood vessels; and the generation or correction of the registration comprises: anchoring an alignment of the endoscopic imaging data and the additional imaging data based on the first location and the second location and relative to a viewpoint, and refining the alignment of the endoscopic imaging data and the additional imaging data based on the auxiliary data.

12. The system of claim 11, wherein the additional features represented by the auxiliary data and present at the surgical site comprise non-vasculature anatomical structures.

13. The system of claim 11, wherein the additional features represented by the auxiliary data and present at the surgical site comprise cauterization marks applied to surface anatomical structures in a preparatory operation to mark the surface anatomical structures.

14. The system of claim 1, wherein: registering the endoscopic imaging data with the additional imaging data using the differentiated set of blood vessels identified at the surgical site further comprises detecting a user selection of the differentiated set of blood vessels; and the generation or correction of the registration comprises: generating or correcting the registration based on the user selection. ​ performing a determination of at least one of the first location and the second location of the differentiated vascular group based on a user selection of the differentiated vascular group detected.

15. The system of any one of claims 1 to 6, wherein: the subsurface imaging data from the subsurface image scan is Doppler ultrasound data scanned by an ultrasound module; and the identification of the differentiated vascular group is based on at least one of blood flow direction, blood vessel diameter, or blood vessel pattern.

16. The system of any one of claims 1 to 6, wherein: the processor is further configured to execute the instructions to validate the subsurface imaging data as the subsurface imaging data is provided by an ultrasound module performing the subsurface image scan; and based on the validation of the subsurface imaging data, performing a registration of the endoscopic imaging data with the additional imaging data using the differentiated vascular group identified at the surgical site.

17. The system of any one of claims 1 to 6, wherein the differentiated vascular group identified at the surgical site comprises one of: arteries differentiated from other types of blood vessels at the surgical site; or veins differentiated from other types of blood vessels at the surgical site.

18. The system of any one of claims 1 to 6, wherein the differentiated vascular group identified at the surgical site comprises a geometric pattern of blood vessels differentiated from the other blood vessels at the surgical site.

19. A system comprising: a memory storing instructions; and a processor communicatively coupled to the memory and configured to execute the instructions intraoperatively during a surgical operation at a surgical site to: access intraoperative endoscopic imaging data from an endoscope, access intraoperative subsurface imaging data from a subsurface imaging module, access alignment parameters representing a registration of the intraoperative endoscopic imaging data with the intraoperative subsurface imaging data, access preoperative scan data captured by an additional imaging modality different from an endoscopic imaging modality, the preoperative scan data representing a three-dimensional model (3D model) of an anatomical structure at the surgical site, identify a differentiated vascular group differentiated from other blood vessels at the surgical site based on the intraoperative subsurface imaging data and the alignment parameters, and register the intraoperative endoscopic imaging data with the preoperative scan data representing the 3D model of the anatomical structure at the surgical site using the differentiated vascular group identified at the surgical site, wherein registering the intraoperative endoscopic imaging data with the preoperative scan data using the differentiated vascular group identified at the surgical site comprises: determining a first location of the differentiated vascular group identified at the surgical site relative to an anatomical structure represented by the intraoperative endoscopic imaging data based on the subsurface imaging data and alignment parameters representing a registration of the intraoperative endoscopic imaging data with the subsurface imaging data; determining a second location of the differentiated vascular group relative to the anatomical structure represented by the preoperative scan data; and generate or correct the registration of the intraoperative endoscopic imaging data with the preoperative scan data based on the first position and the second position.

20. The system of claim 19, wherein: the intraoperative subsurface imaging data from the subsurface imaging module is Doppler ultrasound data from an ultrasound module; and the identification of the differentiated set of blood vessels is based on at least one of blood flow direction, blood vessel diameter, or blood vessel pattern.

21. A method comprising: identifying, by an image registration system, a differentiated set of blood vessels differentiated from other blood vessels at a surgical site based on subsurface imaging data from a subsurface image scan at the surgical site; and registering, by the image registration system, endoscopic imaging data from an endoscopic imaging modality with additional imaging data from an additional imaging modality using the differentiated set of blood vessels identified at the surgical site, wherein registering the endoscopic imaging data with the additional imaging data using the differentiated set of blood vessels identified at the surgical site comprises: determining, based on the subsurface imaging data and alignment parameters representing a registration of the endoscopic imaging data with the subsurface imaging data, a first position of the differentiated set of blood vessels identified at the surgical site relative to an anatomical structure represented by the endoscopic imaging data; determining a second position of the differentiated set of blood vessels relative to the anatomical structure represented by the additional imaging data; and generating or correcting the registration of the endoscopic imaging data with the additional imaging data based on the first position and the second position.

22. The method of claim 21, further comprising providing, based on the registration of the endoscopic imaging data from the endoscopic imaging modality with the additional imaging data from the additional imaging modality, a composite image of the surgical site for display by a display device.

23. The method of claim 22, wherein: the composite image of the surgical site includes a first depiction of an anatomical structure represented by the endoscopic imaging data in combination with a second depiction of an anatomical structure represented by the additional imaging data, the first and second depictions aligned within the composite image relative to a viewpoint associated with the composite image; and the differentiated set of blood vessels identified at the surgical site and used for the registration is different from the anatomical structure included in the first and second depictions in the composite image.

24. The method of claim 21, wherein: the endoscopic imaging data depicts surface anatomical structure at the surgical site, the depiction of the surface anatomical structure captured by the endoscopic imaging modality from a viewpoint at the surgical site; and the additional imaging data represents subsurface anatomical structure at the surgical site, the subsurface anatomical structure occluded by the surface anatomical structure from the viewpoint such that the subsurface anatomical structure is not represented in the endoscopic imaging data.

25. The method of claim 21, wherein: the endoscopic imaging modality comprises an intraoperative scan of an anatomical structure at the surgical site; and the additional imaging modality comprises a preoperative scan of the anatomical structure at the surgical site.

26. The method of claim 21, wherein the additional imaging modality comprises one of: an ultrasound scan of the surgical site; a computed tomography (CT) scan of the surgical site; a magnetic resonance imaging (MRI) scan of the surgical site; or a fluorescence imaging scan of the surgical site.

27. The method of any one of claims 21-26, wherein the differentiated set of blood vessels identified at the surgical site comprises one of: arteries differentiated from other types of blood vessels at the surgical site; or veins differentiated from other types of blood vessels at the surgical site.

28. The method of any one of claims 21-26, wherein the differentiated set of blood vessels identified at the surgical site comprises a geometric pattern of blood vessels differentiated from other blood vessels at the surgical site.

Images