System and method for integrating images captured by different imaging modalities into composite image of surgical space

Abstract

Systems and methods for integrating images captured by different imaging modalities into a composite image of a surgical space are disclosed. The example system accesses images of the surgical space captured by the different imaging modalities, and generates a composite image that includes an integrated representation of the surgical space captured by the different imaging modalities based on the accessed images. An exemplary composite image includes a representation of the surgical space captured by the first imaging modality that is enhanced with an integrated representation of the surgical space captured by the second imaging modality. An integrated representation of the surgical space captured by the second imaging modality may be selectively moved and may be generated based on a first image of the surgical space captured by the first imaging modality and a second image of the surgical space captured by the second imaging modality in a manner that provides a visually real depth appearance.

Description

[0001] This application is a divisional application of Chinese patent application 202080052031.5, filed on May 29, 2020, entitled “System and method for integrating images captured by different imaging modalities into a synthetic image of a surgical space”.

[0002] Related Applications

[0003] This application claims priority to U.S. Provisional Patent Application No. 62 / 855,755, filed on May 31, 2019, entitled “SYSTEMS AND METHODS FOR INTEGRATING IMAGERY CAPTURED BY DIFFERENT IMAGING MODALITIES INTO COMPOSITE IMAGERY OF A SURGICAL SPACE,” the contents of which are incorporated herein by reference in their entirety. Background Art

[0004] During a surgical procedure, an endoscope may be used to capture endoscopic images of the surgical space. The endoscopic images may be presented to the surgeon via a display device so that the surgeon can visualize the surgical space while performing the surgical procedure. Endoscopy is an imaging modality used to capture images of the surgical space.

[0005] In some cases, one or more additional imaging modalities may be used to capture additional images of the surgical space that may also be presented to the surgeon. For example, ultrasound scanning, computed tomography (CT) scanning, and magnetic resonance imaging (MRI) scanning are other imaging modalities that may be used to capture images of the surgical space.

[0006] Images captured by different imaging modalities can be presented to a surgeon so that the surgeon can visualize the surgical space while performing a surgical procedure. However, there is still room for improvement in the techniques for processing and presenting images captured by different surgical space imaging modalities. Summary of the invention

[0007] The following description presents a simplified summary of one or more aspects of the systems and methods described herein. This summary is not a comprehensive overview of all contemplated aspects, and is neither intended to identify primary or critical elements of all aspects, nor to delineate the scope of any or all aspects. Its sole purpose is to present one or more aspects of the systems and methods described herein as a preface to the specific embodiments presented below.

[0008] An exemplary system includes a memory storing instructions; and a processor communicatively coupled to the memory and configured to execute the instructions to: determine an image rendering viewpoint, wherein an image of a surgical space is rendered from the image rendering viewpoint; determine a position of an enhancement region relative to the surgical space from the perspective of the image rendering viewpoint, the enhancement region selectively moving relative to the surgical space; generate a composite image of the surgical space from the perspective of the image rendering viewpoint and based on the determined position of the enhancement region relative to the surgical space; and instruct a display device to display the composite image. The composite image may include: an enhancement region at the determined position of the enhancement region relative to the surgical space; a representation of a first portion of the surgical space captured by a first imaging modality outside the enhancement region; and a representation of a second portion of the surgical space captured by a second imaging modality within the enhancement region, the representation of the second portion of the surgical space being generated based on a first image of the surgical space captured by the first imaging modality and a second image of the surgical space captured by the second imaging modality.

[0009] Another exemplary system includes a memory storing instructions; and a processor communicatively coupled to the memory and configured to execute the instructions to: access a first image of a surgical space captured by a first imaging modality; access a second image of the surgical space captured by a second imaging modality, the second imaging modality being different from the first imaging modality; generate a composite image of the surgical space based on the first image captured by the first imaging modality and the second image captured by the second imaging modality; and instruct a display device to display the composite image. The composite image may include: a representation of a first portion of the surgical space captured by the first imaging modality, the representation of the first portion of the surgical space generated based on the first image captured by the first imaging modality; an enhanced region integrated within the representation of the first portion of the surgical space; and within the enhanced region, a representation of a second portion of the surgical space captured by the second imaging modality, the representation of the second portion of the surgical space including a composition of an image of the surgical space captured by the second imaging modality modified by features of the image of the surgical space captured by the first imaging modality.

[0010] An exemplary method includes a computing system determining an image presentation viewpoint from which an image of a surgical space is presented; determining a position of an enhanced region relative to the surgical space from the perspective of the image presentation viewpoint, the enhanced region selectively moving relative to the surgical space; generating a composite image of the surgical space from the perspective of the image presentation viewpoint and based on the determined position of the enhanced region relative to the surgical space; and instructing a display device to display the composite image. The composite image may include: the enhanced region at the determined position of the enhanced region relative to the surgical space; a representation of a first portion of the surgical space captured by a first imaging modality outside the enhanced region; and a representation of a second portion of the surgical space captured by a second imaging modality within the enhanced region, the representation of the second portion of the surgical space being generated based on a first image of the surgical space captured by the first imaging modality and a second image of the surgical space captured by the second imaging modality. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings illustrate various embodiments and are part of the specification. The illustrated embodiments are examples only and do not limit the scope of the present disclosure. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements.

[0012] Figure 1 An exemplary imaging modality integration system according to the principles described herein is illustrated.

[0013] Figure 2 The diagram shows a schematic diagram of a Figure 1 An imaging modality integration system is provided, the imaging modality integration system being configured to generate a composite image of a surgical space based on images captured by different imaging modalities.

[0014] Figure 3 An exemplary synthetic image of a surgical space is illustrated according to the principles described herein.

[0015] Figure 4 Illustrated are exemplary positioning of image presentation viewpoints and enhancement regions relative to the surgical space, according to the principles described herein.

[0016] Figure 5A An exemplary real workspace included in a surgical space according to the principles described herein is illustrated.

[0017] Figure 5B The diagram shows a schematic diagram of a Figure 5A Example endoscopic image of the real working space.

[0018] Figure 5C The diagram shows a schematic diagram of a Figure 5BExample slope image extracted from an endoscopic image.

[0019] Fig. 6A An exemplary virtual workspace according to the principles described herein is illustrated.

[0020] Figure 6B The diagram shows a schematic diagram of a Fig. 6A An example image of a virtual workspace.

[0021] Figure 6C An exemplary mask image according to the principles described herein is illustrated.

[0022] Figure 7 Illustrated are exemplary composite images of a surgical space according to the principles described herein.

[0023] Figure 8 An exemplary computer-assisted surgical system according to the principles described herein is illustrated.

[0024] Fig. 9 An exemplary method according to the principles described herein is illustrated.

[0025] Fig.10 An exemplary computing device according to the principles described herein is illustrated. DETAILED DESCRIPTION

[0026] This article describes systems and methods for integrating images captured by different imaging modalities into a synthetic image of a surgical space. In some examples, an imaging modality integration system can be configured to integrate images captured by different imaging modalities by generating a synthetic image that includes an integrated representation of a surgical space captured by multiple different imaging modalities. For example, an imaging modality integration system can be configured to generate a synthetic image that includes a representation of a first portion of a surgical space captured by a first imaging modality and an integrated representation of a second portion of a surgical space captured by a second imaging modality. The imaging modality integration system can be configured to integrate the representation of the second portion of the surgical space with the representation of the first portion of the surgical space in a manner that enhances the image of the surgical space captured by one imaging modality (e.g., endoscopic imaging) with the image of the surgical space captured by a second imaging modality (e.g., ultrasound, CT, or MRI imaging).

[0027] In some examples, the imaging modality integration system can be configured to generate a representation of a second portion of the surgical space based on a first image of the surgical space captured by a first imaging modality and a second image of the surgical space captured by a second imaging modality. For example, a representation of the second portion of the surgical space can be generated to include a composite image of the features of the second image of the surgical space and the first image of the surgical space. For example, in order to generate a representation of the second portion of the surgical space, the second image of the surgical space can be combined with a ramp image representing gradient information extracted from the first image of the surgical space. Such a composite image can produce a visually realistic appearance (e.g., an appearance of depth that promotes depth perception) of the representation of the second portion of the surgical space integrated with the representation of the first portion of the surgical space. In some examples, the composite image is generated based on the actual organic colors of the first image and the second image, without using artificial or non-photorealistic colors. This article describes an example of how to generate a representation of the second portion of the surgical space and integrate it with a representation of the first portion of the surgical space.

[0028] In some examples, a first imaging modality may include endoscopic imaging (e.g., by endoscopic imaging) that captures endoscopic images of surface anatomical structures included in the surgical space, and a second imaging modality (e.g., ultrasound, CT, or MRI imaging) may capture images of subsurface anatomical structures included in the surgical space. In such examples, the imaging modality integration system may be configured to generate a composite imagery that includes a representation of a first portion of the surgical space generated based on an endoscopic image of the surface anatomical structure captured by the endoscope and an integrated representation of a second portion of the surgical space, the integrated representation being generated based on an image of the subsurface anatomical structure captured by the second imaging modality. A representation of the second portion of the surgical space may be generated to include a composite image of features of the image of the subsurface anatomical structure captured by the second imaging modality and the endoscopic image of the surface anatomical structure captured by the endoscope. For example, an image of the subsurface anatomical structure captured by the second imaging modality may be combined with a ramp image extracted from the endoscopic image to produce a representation of the second portion of the surgical space. When the representation is integrated into a composite image, such a composite image can produce a visually realistic appearance of imaged subsurface anatomy relative to imaged surface anatomy (eg, an appearance of depth that facilitates depth perception).

[0029] In some examples, the second portion of the surgical space can be dynamically selected by user input to the computer-assisted surgical system or automatically selected by the computer-assisted surgical system (e.g., by performing an automatic scan). For example, the imaging modality integration system can be configured to provide an enhanced region (e.g., a virtual object representing the enhanced region) that is selectively movable relative to the surgical space based on user input to the computer-assisted surgical system or based on automatic movement controlled by the computer-assisted surgical system (e.g., as part of an automatic scan). At any given time during the surgical procedure, the imaging modality integration system can be configured to determine the position of the enhanced region relative to the surgical space, and use the enhanced region relative to the determined position of the surgical space to determine that the first and second portions of the surgical space will be used to generate a synthetic image of the surgical space. For example, the imaging modality integration system can be configured to identify the first portion of the surgical space as the portion of the surgical space outside the enhanced region from the perspective of the image presentation viewpoint, and identify the second portion of the surgical space as the portion of the surgical space within the determined position of the enhanced region from the perspective of the image presentation viewpoint. The imaging modality integration system can use the identified first and second portions of the surgical space to generate a synthetic image of the surgical space. As described herein, a composite image may include an integrated representation of first and second portions of a surgical space captured by first and second imaging modalities, respectively.

[0030] To illustrate an example, an imaging modality integrated system can be configured to: determine an image rendering viewpoint from which an image rendering of a surgical space is rendered; determine a position of an enhancement region relative to the surgical space from the perspective of the image rendering viewpoint; and generate a composite image of the surgical space from the perspective of the image rendering viewpoint and based on the determined position of the enhancement region relative to the surgical space, such that the composite image includes: the enhancement region at the determined position of the enhancement region relative to the surgical space (e.g., a representation of the enhancement region); a representation of a first portion of the surgical space captured by a first imaging modality outside of the enhancement region; and a representation of a second portion of the surgical space captured by a second imaging modality within the enhancement region. The representation of the second portion of the surgical space can be generated in any manner described herein and can be based on a first image of the surgical space captured by the first imaging modality and a second image of the surgical space captured by the second imaging modality.

[0031] The systems and methods described herein can provide various advantages and benefits. For example, the systems and methods described herein can integrate images captured by different imaging modalities into a synthetic image of a surgical space in a manner that produces an integrated and visually realistic appearance of images of surgical scenes captured by different imaging modalities within the synthetic image. The systems and methods described herein can present the generated synthetic images to a user of a computer-assisted surgical system, such as a surgeon performing a surgical procedure using a computer-assisted surgical system. The synthetic images presented can be visually realistic and intuitive for the surgeon, can reduce the complexity of the surgical procedure for the surgeon (e.g., by eliminating the need for the surgeon to mentally align the images of the surgical space presented separately in a non-integrated manner), can allow the surgeon to simultaneously, conveniently and intuitively visualize the surface and subsurface anatomical structures integrated in the synthetic image, and / or can allow the surgeon to provide input to conveniently and dynamically select a portion of the surgical space to be enhanced, so that the selected portion can be viewed using an imaging modality different from that used to view another portion of the surgical space (e.g., by selecting a portion of the surgical space, where the image of the subsurface anatomical structure is displayed as an enhancement to the image of the surface anatomy being displayed). Furthermore, synthetic imagery generated based on the actual organic colors of the captured imagery may be more realistic in appearance (eg, facilitating better depth perception) than synthetic imagery generated based on artificial or non-photorealistic colors.

[0032] These and other advantages and benefits of the systems and methods described herein will become apparent herein.

[0033] Figure 1 An exemplary imaging modality integration system 100 ("system 100") is illustrated and is configured to integrate images captured by different imaging modalities, including generating a composite image using the captured images, the composite image comprising an integrated representation of portions of a surgical space captured by the different imaging modalities. System 100 may be included in, implemented by, or connected to one or more components of a computer-assisted surgical system. For example, system 100 may be implemented by one or more components of a computer-assisted surgical system. As another example, system 100 may be implemented by a stand-alone computing system communicatively coupled to the computer-assisted surgical system. An exemplary computer-assisted surgical system is further described below.

[0034] like Figure 1As shown, system 100 may include a storage facility 102 and a processing facility 104 that are selectively and communicatively coupled to each other. Each of facilities 102 and 104 may include or be implemented by one or more physical computing devices, one or more physical computing devices including hardware and / or software components such as processors, memories, storage drives, communication interfaces, instructions stored in memories for execution by processors, etc. Although facilities 102 and 104 are Figure 1 102 and 104, but the facilities 102 and 104 can be combined into fewer facilities, such as a single facility, or divided into more facilities to serve a specific implementation. In some examples, each of the facilities 102 and 104 can be distributed between multiple devices and / or multiple locations to serve a specific implementation.

[0035] The storage facility 102 may maintain (e.g., store) executable data used by the processing facility 104 to perform one or more operations described herein. For example, the storage facility 102 may store instructions 106 that may be executed by the processing facility 104 to perform one or more operations described herein. The instructions 106 may be implemented by any suitable application, software, code, and / or other executable data instance. The storage facility 102 may also maintain any data received, generated, managed, used, and / or transmitted by the processing facility 104.

[0036] The processing facility 104 may be configured to perform (e.g., execute instructions 106 stored in the storage facility 102 to perform) various operations associated with integrating images captured by different imaging modalities into a composite image of the surgical space. For example, the processing facility 104 may be configured to generate a composite image that includes an integrated representation of the surgical space captured by multiple different imaging modalities. For example, the processing facility 104 may be configured to generate a composite image that includes a representation of a first portion of the surgical space captured by a first imaging modality and a representation of a second portion of the surgical space captured by a second imaging modality. The processing facility 104 may be configured to integrate the representation of the second portion of the surgical space with the representation of the first portion of the surgical space so that the representations become an integrated whole within the composite image. The processing facility 104 may perform the integration in any manner described herein and in a manner that enhances an image of the surgical space captured by one imaging modality (e.g., endoscopic imaging) with an image of the surgical space captured by a second imaging modality (e.g., ultrasound, CT, or MRI imaging).

[0037] These and other operations are described herein that may be performed by the processing facility 104. In the following description, any reference to operations performed by the system 100 may be understood to be performed by the processing facility 104 of the system 100.

[0038] Figure 2 A configuration 200 is illustrated in which the system 100 is configured to generate a composite image of a surgical space based on images captured by different imaging modalities. As shown, the configuration 200 can include multiple imaging modalities 202 (e.g., imaging modalities 202-1 and 202-2) configured to capture images 204 of a surgical space 206 (e.g., images 204-1 captured by imaging modality 202-1 and images 204-2 captured by imaging modality 202-2).

[0039] The surgical space 206 may include any volume space associated with a surgical procedure. For example, the surgical space 206 may include any one or more parts of the patient's body, such as the patient's anatomical structure 208 (e.g., tissue, etc.) in the space associated with the surgical procedure. In some examples, the surgical space 206 may be completely disposed within the patient and may include a space within the patient near a location where a surgical procedure is planned to be performed, being performed, or having been performed. For example, for a minimally invasive surgical procedure performed on tissue within the patient's body, the surgical space 206 may include the tissue, the anatomical structure below the tissue, and the space around the tissue at a location where, for example, a surgical instrument is being used to perform the surgical procedure. In other examples, the surgical space 206 may be at least partially disposed outside the patient's body. For example, for an open surgical procedure performed on a patient, a portion of the surgical space 206 (e.g., the tissue on which it is operated) may be within the patient's body, while another portion of the surgical space 206 (e.g., a space where one or more surgical instruments may be disposed around the tissue) may be outside the patient's body. The surgical space 206 may include a real workspace in which a surgical procedure is performed, such as an actual real-world workspace associated with a patient and in which one or more surgical instruments are used to perform the surgical procedure on the patient.

[0040] As used herein, a surgical procedure may include any medical surgical procedure, including any diagnostic or therapeutic procedure in which manual and / or instrumental techniques are applied to a patient to investigate or treat a physical condition of the patient. A surgical procedure may refer to any stage of a medical procedure, such as the preoperative, intraoperative (i.e., during surgery), and postoperative stages of a surgical procedure.

[0041] Imaging modality 202 may be configured and / or used to capture image 204 of surgical space 206. Such capture may be performed by Figure 2204 of the surgical space 206 in any suitable manner at any suitable time. Thus, one or more imaging modalities 202 may capture images 204 of the surgical space 206 during one or more preoperative, intraoperative, and / or postoperative stages of a surgical procedure.

[0042] Imaging modalities 202 may include any set of different imaging modalities that can be used to capture images of the surgical space. Examples of imaging modalities 204 include, but are not limited to, endoscopic imaging performed by an endoscope, ultrasound imaging performed by an ultrasound machine, CT imaging performed by a CT machine, and MRI imaging performed by an MRI machine. Any suitable additional or alternative imaging modalities may be used in other examples. In some embodiments, imaging modality 202-1 may include endoscopic imaging performed by an endoscope, and imaging modality 202-2 may include any different imaging modalities, such as ultrasound imaging performed by an ultrasound machine, CT imaging performed by a CT machine, or MRI imaging performed by an MRI machine. In such embodiments, imaging modality 202-1 may capture image 204-1 as an endoscopic image of surgical space 206, and imaging modality 202-2 may capture image 204-2 as an ultrasound image, a CT image, or an MRI image of surgical space 206.

[0043] In some examples, imaging modality 202-1 can be configured to capture images of surface anatomical structures included in the surgical space (e.g., the outer surface of tissue included in the surgical space), and imaging modality 202-2 can be configured to capture images of subsurface anatomical structures included in the surgical space (e.g., subsurface tissue behind the outer surface of tissue included in the surgical space). For example, imaging modality 202-1 can include endoscopic imaging performed by an endoscope that captures images of surface tissue within a patient, and imaging modality 202-1 can include ultrasound, CT, or MRI imaging that captures images of subsurface tissue that is located behind the endoscope from the perspective of the endoscope and is hidden from the view of the endoscope by the surface tissue within the patient.

[0044] As described above, each imaging modality 202 can capture images 204 of a surgical scene 206 at any suitable time (e.g., during one or more any stages of a surgical procedure). In some examples, the imaging modalities 202 can simultaneously capture images 204 of a surgical space 206. For example, the imaging modality 202-1 can capture endoscopic images during a surgical procedure (e.g., during an operating phase of a surgical procedure), and the imaging modality 202-1 can simultaneously capture another type of image during the surgical procedure. In other examples, the imaging modalities 202 can capture images 204 of a surgical space 206 at different times and / or during different stages of a surgical procedure. For example, the imaging modality 202-1 can capture endoscopic images during an intraoperative phase of a surgical procedure, and the imaging modality 202-2 can capture another type of image during a preoperative phase of a surgical procedure.

[0045] Image 204 of surgical space 206 may include an image of surgical space 206 captured by imaging modality 202. For example, image 204 may include an endoscopic image, an ultrasound image, a CT image, an MRI image, and / or any other suitable form of image of surgical space 206. Image 204 may include any suitable type of image represented by data in any suitable data format. For example, image 204 may include a still frame image, a video, a color image, an infrared image, and / or any other type of image that may visually represent surgical space 206. The image captured by the imaging modality may include a grid of pixels having values ​​(e.g., color values, brightness values, etc.) that represent the appearance of surgical space 206 captured by the imaging modality. The color values ​​of the pixels in the captured image may represent the actual organic colors of the surgical space captured by the imaging modality.

[0046] Additionally or alternatively, the image 204 may include one or more models of the surgical space 206 generated based on imaging performed by an imaging modality. For example, the image 204 may include a three-dimensional (3D) model of the surgical space 206 generated based on imaging performed by an imaging modality, such as imaging performed by an ultrasound machine, a CT machine, an MRI machine, or other suitable imaging modality. The 3D model may be a full volume model including voxels (i.e., volume pixels) having values ​​(e.g., color values, brightness values, etc.) representing the appearance of the surgical space 206 at a 3D point within the model. Such a volume model may facilitate any slice of the 3D model to be recognized and used by the system 100 to generate an image of the slice of the 3D model. The color values ​​of the pixels in the slice image may represent the actual organic colors of the surgical space captured by the imaging modality.

[0047] Although Figure 2While depicted with two imaging modalities 202-1 and 202-2 capturing images 204-1 and 204-2, respectively, that are provided as input to system 100, other examples may include any suitable number and / or configuration of multiple different imaging modalities capturing images that are provided as input to system 100 for use in generating a composite image of surgical space 206. For example, three or more different imaging modalities may capture images that are input to system 100 for use in generating a composite image of surgical space 206.

[0048] System 100 may generate a composite image 212 of surgical space 206 based on images 204 captured by imaging modalities 202. System 100 may do this in any manner described herein to generate a composite image that includes an integrated representation of portions of surgical space 206 captured by different imaging modalities 202. Examples of such composite images and how to generate the composite images are described herein.

[0049] The system 100 may instruct the display device 214 to display the synthetic image 212. For example, the system 100 may provide data representing the synthetic image 212 to the display device 214, and the display device 214 may be configured to display the synthetic image 212 for viewing by a user of the computer-assisted surgical system. The display device 214 may include any device capable of receiving and processing image data to display one or more images. To this end, the display device 214 may include one or more display screens on which images may be displayed. In some examples, the display device 214 may be a component of the computer-assisted surgical system or be communicatively connected to the computer-assisted surgical system.

[0050] Figure 3 An exemplary composite image 300 that may be generated by system 100 and displayed by a display device is illustrated. Composite image 300 may be an image of a surgical space and may include a representation 302 of an augmented region positioned relative to the surgical space. The positioning of augmented region 302 within composite image 300 may represent the position of augmented region 3002 relative to the surgical space and / or be determined based on the position of augmented region 302 relative to the surgical space (e.g., the position of a virtual object representing augmented region 302). Examples of positioning of augmented region 302 relative to the surgical space are described herein.

[0051] Enhanced region 302 may be any suitable shape that defines a region within composite image 300. For example, enhanced region 302 may be circular, elliptical, quadrilateral (eg, rectangular, sector-shaped), triangular, or any other suitable shape.

[0052] The positioning of enhancement region 302 within composite image 300 may define two portions of composite image 300—a first portion outside of enhancement region 302, and a second portion within enhancement region 302. In the first portion of composite image 300 outside of enhancement region 302, composite image 300 may include a representation 304 of a first portion of a surgical space captured by a first imaging modality. For example, representation 304 may include an image of the surgical space captured by the first imaging modality, such as image 204-1 captured by first imaging modality 202-1. In the second portion of composite image 300 within enhancement region 302, composite image 300 may include a representation 306 of a second portion of the surgical space captured by a second imaging modality. For example, representation 306 may include an image of the surgical space captured by the second imaging modality and modified by features of an image of the surgical space captured by the first imaging modality, such as image 204-2 captured by second imaging modality 202-2 and modified by features of image 204-1 captured by first imaging modality 202-1. Thus, representation 306 of the second portion of the surgical space may be generated based on both image 204-1 and image 204-2 and may include a composite image of features (e.g., gradient information) of image 204-2 and image 204-1. As indicated herein, when representation 306 is integrated with representation 304 in composite image 300, the composite image may create a visually realistic appearance of depth for representation 306. Figure 3 , representation 304 is illustrated as including a diagonal line and representation 306 is illustrated as including a vertical line to indicate that representations 304 and 306 represent surgical spaces captured by different imaging modalities.

[0053] In composite image 300, representation 304 of a first portion of the surgical space captured by a first imaging modality is augmented with an integrated representation 306 of a second portion of the surgical space captured by a second imaging modality. Thus, a surgeon and / or other surgical team member viewing composite image 300 may simultaneously visualize integrated representations 304 and 306 of portions of the surgical space captured by different imaging modalities. Because representation 306 is positionally integrated within representation 304, the surgeon may visualize the surgical space captured by the different imaging modalities without having to mentally align representations 304 and 306 with one another and the surgical space would be as desired if representations 304 and 306 were presented separately and not positionally integrated with one another.

[0054] In some examples, the augmented region 302 can be movable relative to the surgical space via user input to the computer-assisted surgical system. The computer-assisted surgical system can be configured to receive any suitable user input that can be used to move the augmented region 302 relative to the surgical space. Such input can include actuation of a button, movement of a controller (e.g., a joystick controller, a master controller, etc.), movement of a surgical instrument connected to the computer-assisted surgical system (e.g., movement of an ultrasound probe or other surgical instrument from which the augmented region 302 is projected), and / or any other suitable user input.

[0055] This movement of the enhancement region 302 can allow a user of the computer-assisted surgical system to instantly select a specific portion of the surgical space during the surgical procedure that will be viewed as captured by the second imaging modality rather than by the first imaging modality. This can allow the surgeon to dynamically "focus" on the selected portion of the surgical space in order to view the selected portion captured by the second imaging modality. For example, representation 304 can represent the surface anatomical structure of the patient captured by an endoscope, and representation 306 can represent the subsurface anatomical structure of the patient captured by a different imaging modality (e.g., ultrasound, CT, or MRI equipment). In this example, the surgeon can position the enhancement region 302 to view the subsurface anatomical structure at the selected portion of the surgical space while still viewing the surface anatomical structure at another portion of the surgical space. In this embodiment, the enhancement region 302 can be used as a virtual resection region (e.g., a virtual resection plane) that the surgeon can use to select a portion of the representation of the surface anatomical structure to be virtually resected from the view to reveal the representation of the subsurface anatomical structure behind the surface anatomical structure (e.g., a virtual cutting plane in a preoperative 3D model of the surgical space that is aligned with the endoscopic view of the surgical space).

[0056] Movement of the enhancement region 302 relative to the surgical space may include movement in one or more any suitable directions relative to the surgical space. For example, the movement may include lateral movement of the enhancement region 302 across the image translation of the surgical space. Additionally or alternatively, the movement may include depth movement that changes the distance of the enhancement region 302 from the perspective viewpoint from which the image of the surgical space is presented. Such depth movement of the enhancement region 302 may position the enhancement region 302 at a different depth relative to the surgical space, which position may be used to identify a slice of the virtual representation of the surgical space to be presented (e.g., a slice of a 3D model mapped to the virtual representation of the surgical space). Such free movement of the enhancement region 302 may provide flexibility to a user of the computer-assisted surgical system to instantly select specific portions of the surgical space to be enhanced and images to be used for enhancement during surgery.

[0057] To generate a synthetic image of the surgical space, the system 100 may determine an image rendering viewpoint, wherein the image rendering of the surgical space is rendered from the image rendering viewpoint, and the position of the enhanced region relative to the surgical space is determined from the perspective of the image rendering viewpoint. The system 100 may generate a synthetic image of the surgical space from the perspective of the image rendering viewpoint and based on the determined position of the enhanced region relative to the surgical space. The synthetic image may include: an enhanced region at the determined position of the enhanced region relative to the surgical space; a representation of a first portion of the surgical space captured by a first imaging modality outside the enhanced region; and a representation of a second portion of the surgical space captured by a second imaging modality within the enhanced region. The representation of the second portion of the surgical space may be generated in any manner described herein and may be based on a first image of the surgical space captured by the first imaging modality and a second image of the surgical space captured by the second imaging modality.

[0058] Figure 4 An exemplary depiction of a surgical space 400 is illustrated having an image rendering viewpoint 402 and an augmented region 404 positioned relative to the surgical space 400. Although the depiction shows a two-dimensional (2D) view, the principles described with respect to the 2D view also apply to a 3D view of the surgical space having an image rendering viewpoint 402 and an augmented region 404 positioned relative to the surgical space.

[0059] like Figure 4As shown, surgical space 400 includes anatomical structure 406, which includes surface anatomical structure 408 and subsurface anatomical structure 410. Subsurface anatomical structure 410 may include any anatomical structure that is located behind surface anatomical structure 408 from the perspective of image rendering viewpoint 402 and / or is hidden by surface anatomical structure 408 from the perspective of image rendering viewpoint 402. In some examples, surface anatomical structure 408 may include an outer layer of tissue of a patient, and subsurface anatomical structure 410 may include anatomical structure embedded within the outer layer of tissue.

[0060] Image presentation viewpoint 402 ("viewpoint 402") can be any viewpoint in which an image of surgical space 400 can be presented. Viewpoint 402 can include an actual viewpoint of an imaging modality such as an endoscope (e.g., a viewpoint of one or more cameras of an endoscope). Alternatively, viewpoint 402 can include a virtual viewpoint corresponding to an actual viewpoint of an imaging modality such as an endoscope. Viewpoint 402 can be associated with and / or represent intrinsic and extrinsic characteristics of an imaging device, such as one or more cameras of an endoscope. Viewpoint 402 can have a field of view in which an image of surgical space 400 can be presented. The space within the solid arrow 412 extending from viewpoint 400 represents the field of view of viewpoint 402.

[0061] As shown, viewpoint 402 is located relative to surgical space 400 and defines a perspective from which images of surgical space 400 may be presented. Figure 4 The illustrated position of viewpoint 402 is exemplary only. Viewpoint 402 may be located at any position relative to surgical space 400 from which an image of surgical space 400 may be presented.

[0062] Enhancement region 404 may be positioned relative to surgical space 400 to define a portion of surgical space 400 to be enhanced. Enhancement region 404 may be defined to include any suitable shape, area, or volume that may be positioned relative to surgical space 400 to delineate a portion of surgical space 400 to be enhanced from the perspective of viewpoint 402. Figure 4 , the enhanced region 404 is represented as a side view of a planar shape, which may be a circle, an ellipse, a quadrilateral, or any other suitable planar shape.

[0063] In some examples, augmented area 404 may be a virtual object or a view of a virtual object from the perspective of viewpoint 402. For example, a virtual object may be defined and positioned relative to the surgical space. The virtual object may be a 2D or 3D object. The virtual object may be moved relative to surgical space 400 through user input to the computer-assisted surgical system.

[0064] At a given point in time, system 100 may determine the position of viewpoint 402 and enhancement region 404 relative to surgical space 400, and generate a composite image of surgical space 400 based on the determined position of viewpoint 402 and enhancement region 404 relative to surgical space 400. In the composite image, a first portion of surgical space 400 may be represented by an image captured by a first imaging modality, and a second portion of surgical space 400 may be represented by an image captured by a second imaging modality.

[0065] To this end, system 100 may define a first portion and a second portion of surgical space 400 using the determined positions of viewpoint 402 and enhancement region 404 relative to surgical space 400. For illustration, Figure 4 A dashed line is shown extending from viewpoint 402 and intersecting the boundary of enhancement region 404 to define alignment boundary 414 .

[0066] Portions of surgical space 400 outside of the space within alignment boundary 414 are referred to as unaligned portions 416 of surgical space 400 because they are not aligned with enhancement region 404 from the perspective of viewpoint 402. Unaligned portion 416 of surgical space 400 may constitute a first portion of surgical space 400 in a composite image of surgical space 400.

[0067] The portion of surgical space 400 that is within the space within alignment boundary 414 is referred to as an aligned portion 418 of surgical space 400 because, from the perspective of viewpoint 402, the portion is aligned with enhancement region 404. Aligned portion 418 of surgical space 400 may constitute a second portion of surgical space 400 in the composite image of surgical space 400.

[0068] In the composite image of surgical space 400, a representation of a first portion of surgical space 400 (a representation of misaligned portion 416 of surgical space 400) may be generated based on images captured by a first imaging modality. For example, the first imaging modality may include an endoscope positioned at viewpoint 402 to capture endoscopic images of surgical space 400, and the representation of the first portion of surgical space 400 may include an endoscopic image of surface anatomical structure 408 in misaligned portion 416 of surgical space 400.

[0069] In the composite image of surgical space 400, a representation of a second portion of surgical space 400 (a representation of aligned portion 418 of surgical space 400) may be generated based on images captured by a second imaging modality different from the first imaging modality. For example, the second imaging modality may include an ultrasound, CT, or MRI device that captures ultrasound, CT, or MRI images of surgical space 400, and the representation of the second portion of surgical space 400 may include an ultrasound, CT, or MRI image of surface anatomical structure 408 or subsurface anatomical structure 410 in aligned portion 418 of surgical space 400. In an example where the representation of the second portion of surgical space 400 is generated based on an image of subsurface anatomical structure 410 in aligned portion 418 of surgical space 400, the image may be an image of subsurface anatomical structure 410 and one or more images of any depth behind surface anatomical structure 408. In some embodiments, the depth of the image of the subsurface anatomical structure 410 captured by the second imaging modality can be selected in a manner (e.g., on-the-fly during the surgical procedure) by user input to the computer-assisted surgery system (e.g., by user input to move the enhancement region 404). ， This approach changes the distance between viewpoint 402 and enhancement region 404 and / or moves enhancement region 404 to select a depth within subsurface anatomical structure 410 .

[0070] As described herein, the representation of the second portion of the surgical space 400 in the composite image can be generated based on the imagery captured by the second imaging modality and the imagery captured by the first imaging modality. For example, the representation of the second portion of the surgical space 400 can include a composite map of features of the imagery of the subsurface anatomical structure 410 within the alignment region 418 captured by the second imaging modality and the imagery of the surface anatomical structure 408 within the alignment region 418 captured by the first imaging modality. For example, the system 100 can extract gradient information from the imagery of the surface anatomical structure 408 within the alignment region 418 and generate a slope imagery representing the extracted gradient information. The system 100 can modify the imagery of the subsurface anatomical structure 410 within the alignment region 418 captured by the second imaging modality with the slope imagery, such as by summing the extracted slope imagery and the imagery of the subsurface anatomical structure 410 within the alignment region 418 captured by the second imaging modality to generate a composite map for the representation of the second portion of the surgical space 400 in the composite image. As described, the composite image may provide a visually realistic representation of the depth of subsurface anatomical structures 410 relative to surface anatomical structures 408 in the composite image.

[0071] In some examples, the combination of the gradient image extracted from the image of the surface anatomy 408 and the image of the subsurface anatomy 410 can be performed using actual organic color values ​​of the image of the surface anatomy 408 and / or the subsurface anatomy 410, rather than using artificial or non-photorealistic colors. This can facilitate a visually realistic representation of the surgical space, including a visually realistic representation of the depth of the subsurface anatomy 410 relative to the surface anatomy 408 in the composite image.

[0072]

[0066] An exemplary manner of generating a composite image of a surgical space will now be described. Figure 5A A real workspace 502 that can be included in a surgical space is illustrated. The real workspace 502 can be a real-world physical workspace located in front of an endoscope 504 configured to capture endoscopic images of the workspace. In some examples, the real workspace 502 can include a patient's anatomical structure 506 and one or more surgical instruments 508 (e.g., surgical instruments 508-1 and 508-2) positioned relative to the anatomical structure 506 in the real workspace 502. In the example shown, surgical instrument 508-1 is a grasping tool and surgical instrument 508-2 is an ultrasound probe.

[0073] The system 100 may access a real image of the real workspace 502. For example, the system 100 may access a real image of the real workspace 502 captured by the endoscope 504. Figure 5B An example of a real image (R) of a real workspace 502 captured by an endoscope 504 is illustrated.

[0074] The system 100 may generate a slope image from a real image (R). For example, the system 100 may extract gradient information from the real image (R) and use the extracted gradient information to generate a slope image representing the gradient information extracted from the real image (R). The gradient information may represent a directional change in a feature of the real image (R), such as a directional change in intensity, color, or another feature of the real image (R). The gradient information may represent a change in one or more directions.

[0075] Figure 5C The figure shows an example of a slope image (S) extracted from a real image (R). Figure 5C , shown in black and white. However, the slope image (S) may include various shades of black, white, and / or gray representing the slope. The slope image (S) may represent any suitable gradient information, including a horizontal gradient, a vertical gradient, one or more other directional gradients, or any combination or sub-combination thereof.

[0076] The system 100 can generate and maintain a virtual workspace that represents the real workspace 502. The virtual workspace can be a 3D space (e.g., a 3D coordinate space) to which images of the real workspace 502 captured by different imaging modalities can be mapped. For example, an endoscopic image captured by an endoscope 504 and other images captured by one or more other imaging modalities can be mapped to the virtual workspace so that the endoscopic image and the other images are registered with each other in the virtual workspace.

[0077] The registration may be performed in any suitable manner. For example, depth values ​​may be determined and associated with pixels in the real image (R) to generate a 3D grid of 3D coordinate points associated with the color values ​​of the pixels. The 3D grid may be mapped to the virtual workspace.

[0078] In an example where the other imagery captured by another imaging modality includes a 3D model of the surgical space, the system 100 can map the 3D model to the virtual workspace. This can be performed in any suitable manner and can include the system 100 registering features in the 3D model to a 3D mesh generated from the endoscopic imagery and matching features of depth information associated with the endoscopic imagery. Thus, the 3D model can be registered to the 3D mesh in the virtual workspace.

[0079] In examples where the other imagery captured by another imaging modality includes a 2D image of the surgical space and depth data for the 2D image is available, system 100 can map the 2D image to the virtual workspace, similar to how system 100 maps endoscopic images to the virtual workspace. In examples where the other imagery captured by another imaging modality includes a 2D image of the surgical space but depth data for the 2D image is not available, system 100 can project the 2D image onto any suitable surface in the virtual workspace, such as a surface of a virtual object representing an augmented area in the virtual workspace.

[0080] Fig. 6A An exemplary virtual workspace 602 is illustrated, which may be generated by the system 100 based on and / or representing the real workspace 502. The virtual workspace 602 may include virtual representations of elements of the real workspace 502, such as a virtual representation 606 of an anatomical structure 506 and virtual representations 608 (e.g., virtual representations 608-1 and 608-2) of surgical instruments 508 (e.g., surgical instruments 508-1 and 508-2). The system 100 may generate virtual representations in the virtual workspace in any suitable manner, including based on a real image (R) of the real workspace 502 and depth information associated with the real image (R).

[0081] Virtual workspace 602 may also include an image rendering viewpoint 604 ("viewpoint 604"), which may be a virtual viewpoint corresponding to a viewpoint of endoscope 504 included in real workspace 502. Viewpoint 604 may be configured based on intrinsic and extrinsic characteristics of endoscope 504. Viewpoint 604 may be positioned relative to other elements of virtual workspace 602 and may represent a viewpoint perspective from which an image of virtual workspace 602 may be rendered.

[0082] The virtual workspace 602 may also include an enhanced area 610 positioned relative to other elements of the virtual workspace 602. Fig. 6A In the example shown, the enhanced region 610 is projected from the ultrasound probe 608-2 in a specific manner (e.g., in a specific direction, orientation, posture, shape, etc.), and the position of the enhanced region 610 is determined based on the ultrasound probe 608-2, so that the enhanced region 610 is positioned relative to the ultrasound probe 608-2, the viewpoint 604, and other elements of the virtual workspace 602.

[0083] The system 100 can project the image captured by the ultrasound probe 608-2 onto the enhanced area 610 in the virtual workspace 602. The ultrasound image projected onto the enhanced area 610 is Fig. 6A The horizontal line fill pattern 612 in FIG. Fig. 6A The example illustrated in depicts the projection of an ultrasound image onto the enhanced region 610, but any other image captured by another imaging modality may be projected onto the enhanced region 610 in the virtual workspace 602. For example, the system 100 may determine a slice of a 3D model (e.g., a 3D model generated from a CT or MRI image) based on the position of the enhanced region 610 relative to the 3D model registered to the virtual workspace 602 and project the slice image onto the enhanced region 610. In such an example, the enhanced region 610 projected from the ultrasound probe 608-2 may be used as a placeholder onto which an image of a slice of the registered 3D model of the surgical space 602 may be projected in the virtual workspace 602.

[0084] In some examples, the image projected onto the augmented region 610 in the virtual workspace may be selected by way of user input to the computer-assisted surgical system. For example, the user of the computer-assisted surgical system may provide input to switch from one imaging modality image projected onto the augmented region 610 to another imaging modality image projected onto the augmented region 610 (e.g., switching from an ultrasound image to a CT or MRI phantom image, or vice versa).

[0085] System 100 may be configured to generate a composite image of the surgical space using virtual workspace 602 in any suitable manner. For example, system 100 may generate an image of virtual workspace 602 from the perspective of viewpoint 604. Figure 6B An example of an image (V) of a virtual workspace 602 presented by system 100 from the perspective of viewpoint 604 is illustrated, which image (V) may be referred to as a virtual image (V). As described herein, system 100 may be configured to generate a synthetic image of a surgical space using the virtual image (V).

[0086] The system 100 may generate a mask image, such as a binary rendering mask image, based on the virtual workspace 602 and / or the virtual image (V). The mask image may correspond in size to the virtual image (V) and may include a first portion aligned with the location of the enhancement region 610 (e.g., a region within the enhancement region 610) and assigned a first binary value, and a second portion not aligned with the location of the enhancement region 610 (e.g., a region outside the enhancement region 610) and assigned a second binary value different from the first binary value. Figure 6C An example of a mask image (M) that can be generated based on a virtual image (V) by the system 100 is illustrated. As shown, the mask image (M) includes a first portion 622 including a white fill representing a binary value, and a second portion 624 including a black fill representing another binary value.

[0087] The system 100 may be configured to generate a composite image (C) of a surgical space using a real image (R), a ramp image (S), a virtual image (V), and a mask image (M). For example, the system 100 may perform a blending function that generates a composite image (C) based on the real image (R), the ramp image (S), the virtual image (V), and the mask image (M). In some embodiments, the system 100 may perform the following blending function for i and j, respectively, iterating over the image width and height:

[0088] C=blend(R,S,V,M)=M (i,j) *(S (i,j) +V (i，j) +(1-M (i,j )*R (i，j)

[0089] In the blending function, the mask image (M) may have a binary value of "1" in a first portion aligned with the enhancement region 610 and a binary value of "0" in a second portion not aligned with the enhancement region 610. Accordingly, the system 100 may use the real image (R) for all pixel positions set to "0" in the mask image (M) and may use a blended output of the virtual image (V) and the ramp image (S) for all pixel positions set to "1" in the mask image (M).

[0090] Based on the blending function, a first portion of the composite image (C) may include a representation of a surgical scene captured by the endoscope 504, and a second portion of the composite image (C) may include a representation of a surgical scene captured by another imaging modality. The representation of the second portion of the surgical scene captured by the other imaging modality may include a composite image of features of an image of the surgical space captured by the other imaging modality and an endoscopic image of the surgical space. For example, to generate a representation of the second portion of the surgical space, the system 100 may combine an image of the second portion of the surgical space captured by the other imaging modality (e.g., which may include an image as mapped to the enhanced region 610 in the virtual workspace 602 and / or as represented in the virtual image (V)) and a slope image of gradient information extracted from the endoscopic image of the second portion of the surgical space. The combination may be performed by the system 100 executing a blending function iterating over pixel positions of the composite image (C) and selectively using values ​​at corresponding pixel positions in the real image (R), the slope image (S), the virtual image (V), and the mask image (M) (as indicated by the blending function) to generate the composite image (C). In some examples, this may be performed without using artificial colors to generate the composite image (C).

[0091] Figure 7 An example of a synthetic image (C) that can be generated by the system 100 is illustrated, which performs a blending function to generate the synthetic image (C) based on the real image (R), the slope image (S), the virtual image (V), and the mask image (M). As shown, the synthetic image (C) can include a representation 702 of the enhanced area 610. Outside the representation 702 of the enhanced area 610, the synthetic image (C) includes a representation 704 of a first portion of the surgical space, which includes an endoscopic image of the surface anatomical structure captured by the endoscope 504. Within the representation 702 of the enhanced area 610, the synthetic image (C) includes a representation 706 of a second portion of the surgical space captured by another imaging modality (e.g., an ultrasound probe), which includes a composite image of features of the image captured by the other imaging modality and the endoscopic image of the second area of ​​the surgical space (e.g., the slope of the endoscopic image).

[0092] A composite image, such as composite image (C), may include any suitable representation of the enhanced region. Such a representation may include any visual representation of a boundary or transition between representations of the first and second portions of the surgical scene in the composite image.

[0093] Although FIG. 5A to FIG. 7 An example is illustrated in which the first imaging modality includes an endoscope and the second imaging modality includes an ultrasound probe, but images captured by other different imaging modalities can be similarly processed to generate a synthetic image of the surgical space. In other examples, for example, the first imaging modality can include an endoscope, and the second imaging modality can include a CT or MRI machine. In such an example, the image can be captured by a CT or MRI machine during the preoperative stage of the surgical procedure and processed to generate a 3D model of the surgical space. The system 100 can align the 3D model of the surgical space to the endoscopic image of the surgical space in the virtual workspace. The system 100 can determine the position of the image presentation viewpoint relative to the surgical space, determine the position of the enhanced region relative to the surgical space, and use the position of the image presentation viewpoint and the position of the enhanced region relative to the surgical space to identify an image of the 3D model (e.g., a planar slice of the 3D image) for generating a representation of the surgical space within the enhanced region. For example, the system 100 can project the image of the identified 3D model onto the enhanced region in the virtual workspace and perform a blending function as described herein to generate a synthetic image of the surgical space.

[0094] In some embodiments, for example, the system 100 can project an enhancement region from an ultrasound probe positioned in a surgical space. To this end, the system 100 can access tracking information (e.g., position information, orientation information, movement information, mobility information, etc.) of the ultrasound probe from a computer-assisted surgical system to which the ultrasound probe is connected and use the tracking information to identify the pose of the ultrasound probe within the surgical space (e.g., the position and orientation of the ultrasound probe). The system 100 can project the enhancement region into the surgical space (e.g., into a virtual workspace representing a real workspace within the surgical space) based on the pose of the ultrasound probe.

[0095] In some examples, system 100 can generate a composite image of the surgical space, which includes a representation of an ultrasound image of the surgical space captured by the ultrasound probe within the enhanced region. In other examples, system 100 can generate a composite image of the surgical space, which includes a representation of a portion of the surgical space captured by different imaging modalities within the enhanced region. For example, the representation within the enhanced region may include or may be based on a CT or MRI image of the surgical space (e.g., a 3D model of the surgical space generated from CT or MRI imaging of the surgical space). In such an example, the enhanced region projected from the ultrasound probe can be used as a virtual resection region and / or placeholder on which a CT-based or MRI-based representation of the surgical space is projected. Therefore, a user of the computer-assisted surgical system can provide input to position the ultrasound probe within the surgical space to select the location of the enhanced region to define a portion of the surgical space that will be enhanced with an image of the registered 3D model of the surgical space. As described above, in some examples, system 100 can be configured to switch the representation within the enhanced region between representations of ultrasound images captured by an ultrasound probe and representations of other images captured by another imaging modality (e.g., CT or MRI images captured by a CT or MRI machine).

[0096] Using an ultrasound probe to define and move an augmentation region relative to a surgical space is illustrative of certain examples. Other examples may implement different real-world surgical instruments, virtual objects, or any other suitable mechanism for a user to use to define and move an augmentation region relative to a surgical space.

[0097] As described above, system 100 can be implemented in or communicatively coupled to a computer-assisted surgical system. System 100 can receive input from and provide output to the computer-assisted surgical system. For example, system 100 can access images of a surgical space and / or any information about the surgical space and / or the computer-assisted surgical system from the computer-assisted surgical system, use the accessed images and / or information to perform any of the processing described herein to generate a synthetic image of the surgical space, and provide data representing the synthetic image to the computer-assisted surgical system for display (e.g., via a display device associated with the computer-assisted surgical system).

[0098] Figure 8 An exemplary computer-assisted surgical system 800 ("surgical system 800") is illustrated. System 100 may be implemented by, connected to, and / or otherwise used in conjunction with surgical system 800.

[0099] As shown, surgical system 800 may include a manipulation system 802, a user control system 804, and an auxiliary system 806 that are communicatively coupled to each other. Surgical system 800 may be used by a surgical team to perform a computer-assisted surgical procedure on a patient 808. As shown, the surgical team may include a surgeon 810-1, an assistant 810-2, a nurse 810-3, and an anesthesiologist 810-4, all of whom may be collectively referred to as "surgical team members 810." Additional or alternative surgical team members may be present in a surgical meeting to serve a particular embodiment.

[0100] Although Figure 8 A minimally invasive surgical procedure is shown being performed, but it will be appreciated that the surgical system 800 may similarly be used to perform open surgical procedures or other types of surgical procedures that may similarly benefit from the accuracy and convenience of the surgical system 800. Furthermore, it will be appreciated that a surgical conference, in which the surgical system 800 may be employed throughout the surgical conference, may include not only the intraoperative phase of a surgical procedure, such as Figure 8 s shown, but may also include preoperative, postoperative, and / or other suitable stages of the surgical procedure.

[0101] like Figure 8 As shown, the manipulation system 802 may include a plurality of manipulator arms 812 (e.g., manipulator arms 812-1 to 812-4), and a plurality of surgical instruments may be coupled to the plurality of manipulator arms 812. Each surgical instrument may be implemented by any suitable surgical tool (e.g., a tool having tissue interaction functionality), a medical tool, an imaging device (e.g., an endoscope, an ultrasonic tool, etc.), a sensing instrument (e.g., a force-sensing surgical instrument), a diagnostic instrument, etc., which may be used for a computer-assisted surgical procedure performed on a patient 808 (e.g., by being at least partially inserted into the patient 808 and being manipulated to perform a computer-assisted surgical procedure on the patient 808). Although the manipulation system 802 is depicted and described herein as including four manipulator arms 812, it should be appreciated that the manipulation system 802 may include only a single manipulator arm 812 or any other number of manipulator arms that may serve a particular embodiment.

[0102] Manipulator arm 812 and / or a surgical instrument attached to manipulator arm 812 may include one or more displacement sensors, orientation sensors, and / or position sensors for generating raw (i.e., uncorrected) kinematic information. One or more components of surgical system 800 may be configured to use kinematic information to track (e.g., determine its position) and / or control a surgical instrument.

[0103] The user control system 804 can be configured to facilitate the surgeon 810-1 to control the manipulator arm 812 and the surgical instruments attached to the manipulator arm 812. For example, the surgeon 810-1 can interact with the user control system 804 to remotely move or manipulate the manipulator arm 812 and the surgical instruments. To this end, the user control system 804 can provide the surgeon 810-1 with an image (e.g., a high-definition 3D image) of the surgical space associated with the patient 808 captured by an imaging system (e.g., any medical imaging system described herein). In some examples, the user control system 804 can include a stereoscopic viewer with two displays, wherein the stereoscopic image of the surgical space associated with the patient 808 and generated by the stereoscopic imaging system can be viewed by the surgeon 810-1. In some examples, the synthetic image generated by the system 100 can be displayed by the user control system 804. The surgeon 810-1 can perform one or more procedures using the image displayed by the user control system 804 with one or more surgical instruments attached to the manipulator arm 812.

[0104] To facilitate control of surgical instruments, user control system 804 may include a set of master controllers. These master controllers may be manipulated by surgeon 810-1 to control the movement of surgical instruments (e.g., by utilizing robotics and / or teleoperation techniques). The master controllers may be configured to detect various hand, wrist, and finger movements of surgeon 810-1. In this way, surgeon 810-1 may intuitively perform procedures using one or more surgical instruments.

[0105] The auxiliary system 806 may include one or more computing devices configured to perform the primary processing operations of the surgical system 800. In such a configuration, the one or more computing devices included in the auxiliary system 806 may control and / or coordinate operations performed by various other components of the surgical system 800 (e.g., the manipulator system 802 and the user control system 804). For example, a computing device included in the user control system 804 may transmit instructions to the manipulator system 802 via one or more computing devices included in the auxiliary system 806. As another example, the auxiliary system 806 may receive and process image data from the manipulator system 802 representing images captured by an imaging device attached to one of the manipulator arms 812.

[0106] In some examples, the auxiliary system 806 can be configured to present visual content to a surgical team member 810 who may not have access to images provided to the surgeon 810-1 at the user control system 804. To this end, the auxiliary system 806 can include a display monitor 814 that is configured to display one or more user interfaces, such as an image of the surgical space (e.g., a 2D image), information associated with the patient 808 and / or the surgical procedure, and / or any other visual content that may be used for a particular embodiment. For example, the display monitor 814 can display an image of the surgical space (e.g., a composite image generated by the system 100) and additional content (e.g., graphical content, contextual information, etc.) displayed simultaneously with the image. In some embodiments, the display monitor 814 is implemented by a touch screen display with which the surgical team member 810 can interact (e.g., via touch gestures) to provide user input to the surgical system 800.

[0107] The operating system 802, the user control system 804, and the auxiliary system 806 can be communicatively coupled to each other in any suitable manner. Figure 8 As shown, the operating system 802, the user control system 804, and the auxiliary system 806 can be communicatively coupled via a control line 816, which can represent any wired or wireless communication link that can serve a particular implementation. To this end, the operating system 802, the user control system 804, and the auxiliary system 806 can each include one or more wired or wireless communication interfaces, such as one or more local area network interfaces, Wi-Fi network interfaces, cellular interfaces, etc.

[0108] Fig. 9 An exemplary method 900 is shown. Fig. 9 Exemplary operations according to one embodiment are illustrated, but other embodiments may omit, add, reorder, combine and / or modify Fig. 9 Follow any of the steps shown in . Fig. 9 One or more of the operations shown in may be performed by a computing system such as system 100, any components included therein, and / or any implementation thereof.

[0109] In operation 902, the computing system determines an image rendering viewpoint from which an image of the surgical space is rendered. Operation 902 may be performed in any manner described herein.

[0110] In operation 904, the computing system determines the position of the enhancement region relative to the surgical space from the perspective of the image rendering viewpoint. Operation 904 may be performed in any manner described herein.

[0111] In operation 906, the computing system generates a composite image of the surgical space from the perspective of the image rendering viewpoint and based on the determined position of the augmented region relative to the surgical space. Operation 906 may be performed in any manner described herein.

[0112] In operation 908, the computing system instructs the display device to display the composite image. Operation 908 may be performed in any manner described herein.

[0113] The composite image generated in operation 806 can be generated in any suitable manner, including in any of the manners described herein. The composite image can include any of the exemplary elements described herein. For example, the composite image can include an enhanced region at a determined position of the enhanced region relative to the surgical space. Outside the enhanced region, the composite image can include a representation of a first portion of the surgical space captured by a first imaging modality. Within the enhanced region, the composite image can include a representation of a second portion of the surgical space captured by a second imaging modality.

[0114] As described herein, the representation of a first portion of the surgical space captured by a first imaging modality may include a first image of the surgical space captured by the first imaging modality (e.g., an endoscope) and / or may be generated based on the first image of the surgical space captured by the first imaging modality (e.g., an endoscope), and the representation of a second portion of the surgical space may be generated based on the first image of the surgical space captured by the first imaging modality and a second image of the surgical space captured by a second imaging modality (e.g., an ultrasound probe, a CT device, or an MRI device). For example, as described herein, the representation of the second portion of the surgical space may include a composite image of features of the second image of the surgical space captured by the second imaging modality and the first image of the surgical space captured by the first imaging modality.

[0115] In some examples, a non-transitory computer-readable medium storing computer-readable instructions can be provided according to the principles described herein. When executed by a processor of a computing device, the instructions can direct the processor and / or computing device to perform one or more operations, including one or more operations described herein. Any of a variety of known computer-readable media can be used to store and / or transmit such instructions.

[0116] As referred to herein, non-transitory computer-readable media may include any non-transitory storage medium that participates in providing data (e.g., instructions) that can be read and / or executed by a computing device (e.g., by a processor of a computing device). For example, non-transitory computer-readable media may include, but are not limited to, any combination of non-volatile storage media and / or volatile storage media. Exemplary non-volatile storage media include, but are not limited to, read-only memory, flash memory, solid-state drives, magnetic storage devices (e.g., hard disks, floppy disks, magnetic tapes, etc.), ferroelectric random access memory ("RAM"), and optical disks (e.g., compact disks, digital video disks, Blu-ray disks, etc.). Exemplary volatile storage media include, but are not limited to, RAM (e.g., dynamic RAM).

[0117] Fig.10 An exemplary computing device 1000 is shown, which may be specifically configured to perform one or more processes described herein. Any system, unit, computing device, and / or other component described herein may be implemented by the computing device 1000.

[0118] like Fig.10 As shown, computing device 1000 may include a communication interface 1002, a processor 1004, a storage device 1006, and an input / output ("I / O") module 1008 communicatively coupled to each other via a communication infrastructure 1010. Fig.10 shown in Fig.10 The components shown in the drawings are not intended to be limiting. Additional or alternative components may be used in other embodiments. Fig.10 Components of computing device 1000 are shown in .

[0119] The communication interface 1002 may be configured to communicate with one or more computing devices. Examples of the communication interface 1002 include, but are not limited to, a wired network interface (e.g., a network interface card), a wireless network interface (e.g., a wireless network interface card), a modem, an audio / video connection, and any other suitable interface.

[0120] The processor 1004 generally represents any type or form of processing unit capable of processing data and / or interpreting, executing, and / or directing the execution of one or more of the instructions, processes, and / or operations described herein. The processor 1004 may perform operations by executing computer-executable instructions 1012 (e.g., applications, software, code, and / or other executable data instances) stored in the storage device 1006.

[0121] Storage device 1006 may include one or more data storage media, devices or configurations and may take any type, form and combination of data storage media and / or devices. For example, storage device 1006 may include, but is not limited to, any combination of non-volatile media and / or volatile media described herein. Electronic data including data described herein may be temporarily and / or permanently stored in storage device 1006. For example, data representing computer executable instructions 1012 configured to instruct processor 1004 to perform any operation described herein may be stored within storage device 1006. In some examples, the data may be arranged in one or more databases residing within storage device 1006.

[0122] The I / O module 1008 may include one or more I / O modules configured to receive user input and provide user output. The I / O module 1008 may include any hardware, firmware, software, or combination thereof that supports input and output capabilities. For example, the I / O module 1008 may include hardware and / or software for capturing user input, including but not limited to a keyboard or keypad, a touch screen component (e.g., a touch screen display), a receiver (e.g., an RF or infrared receiver), a motion sensor, and / or one or more input buttons.

[0123] The I / O module 1008 may include one or more devices for presenting output to a user, including, but not limited to, a graphics engine, a display (e.g., a display screen), one or more output drivers (e.g., a display driver), one or more audio speakers, and one or more audio drivers. In some embodiments, the I / O module 1008 is configured to provide graphical data to the display for presentation to the user. The graphical data may represent one or more graphical user interfaces and / or any other graphical content that may serve a particular implementation.

[0124] In the foregoing description, various exemplary embodiments have been described with reference to the accompanying drawings. However, it will be apparent that various modifications and changes may be made thereto, and additional embodiments may be implemented without departing from the scope of the invention as set forth in the appended claims. For example, certain features of one embodiment described herein may be combined or substituted with features of another embodiment described herein. Therefore, the description and drawings are to be regarded as illustrative rather than restrictive.

Claims

1. A system comprising: a memory storing instructions; as well as a processor communicatively coupled to the memory and configured to execute the instructions to: determining an image rendering viewpoint from which the image rendering of the surgical space is rendered; determining a position of an enhancement region relative to the surgical space from the perspective of the image presentation viewpoint, the enhancement region being selectively movable relative to the surgical space; generating a composite image of the surgical space from the angle of the image rendering viewpoint and based on the determined position of the enhancement region relative to the surgical space, the composite image comprising: the enhanced region at the determined position of the enhanced region relative to the surgical space, a representation of a first portion of the surgical space captured by a first imaging modality outside of the enhancement region, and a representation of a second portion of the surgical space captured by a second imaging modality within the enhanced region, the representation of the second portion of the surgical space being generated based on a first image of the surgical space captured by the first imaging modality and a second image of the surgical space captured by the second imaging modality; and A display device is instructed to display the composite image.

2. A system according to claim 1, wherein the representation of the second portion of the surgical space comprises a composite image of features of the second image of the surgical space captured by the second imaging modality and the first image of the surgical space captured by the first imaging modality.

3. The system of claim 2, wherein the features of the first image of the surgical space captured by the first imaging modality include gradient information extracted from the first image of the surgical space.

4. The system of claim 3, wherein the composite image is generated by summing the second image of the surgical space captured by the second imaging modality and a slope image representing the gradient information extracted from the first image of the surgical space.

5. The system of claim 1, wherein: The first image includes an image of surface anatomy contained in the first portion of the surgical space; and The second image includes an image of subsurface anatomical structures contained in the second portion of the surgical space.

6. The system of claim 1, wherein: The representation of the first portion of the surgical space includes an endoscopic image of surface anatomy contained in the first portion of the surgical space; and The representation of the second portion of the surgical space includes a composite image of: including an image of subsurface anatomical structures in said second portion of said surgical space, and A slope image is extracted from an endoscopic image of surface anatomical structures included in the second portion of the surgical space.

7. The system of claim 1, wherein: The first imaging modality comprises an endoscopic imaging modality; and The second imaging modality includes one of an ultrasound imaging modality, a computed tomography imaging modality (CT imaging modality) and a magnetic resonance imaging imaging modality (MRI imaging modality).

8. The system of claim 1, wherein: From the angle of the image presentation viewpoint, the first portion of the surgical space is not aligned with the enhancement region; as well as From the perspective of the image presentation viewpoint, the second portion of the surgical space is aligned with the enhancement region.

9. The system of claim 1, wherein the augmented region is projected from a surgical instrument that is selectively movable relative to the surgical space based on user input to a computer-assisted surgical system.

10. The system of claim 1, wherein: The enhancement region is projected from an ultrasound probe that is selectively movable relative to the surgical space based on user input to a computer-assisted surgical system; The first image comprises an endoscopic image of the surgical space; and The second image includes slice images of a three-dimensional model (3D model) generated from a pre-operative computed tomography image (CT image) or a pre-operative magnetic resonance imaging image (MRI image) of the surgical space.

Images