An interactive augmented reality system for laparoscopic and video-assisted surgery
An interactive augmented reality system for laparoscopic surgery enables surgeons to enhance their vision and situational awareness by aligning preoperative and intraoperative images, improving surgical precision and safety through supervised image registration and real-time overlays.
Patent Information
- Application Number
- JP2024526912
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-11-09
- Filing Date
- 2022-11-08
- Publication Date
- 2024-11-08
AI Technical Summary
The limited viewing angle of laparoscopic cameras and the need to mentally merge real-time video feeds with preoperative images increases the mental burden on surgeons, making it difficult to accurately navigate and perform surgeries.
An interactive augmented reality system that allows surgeons to provide supervised information for image registration, aligning preoperative and intraoperative images, and overlays relevant anatomical structures and risk alerts, using a physical console and camera-captured images.
Enhances surgical precision by providing high-accuracy image registration and situational awareness, allowing surgeons to visualize hidden anatomical structures and avoid risks during laparoscopic procedures.
Smart Images

Figure 2024541293000001_ABST
Abstract
Description
[Technical field]
[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 277,511, filed November 9, 2021, the entire disclosure of which is incorporated herein by reference in its entirety for all purposes.
[0002] The present disclosure relates to an augmented reality system for video-assisted surgery, and more particularly, to an interactive augmented reality system for laparoscopic surgery. Video-assisted surgery can include robotic and non-robotic surgery, including robotic and non-robotic laparoscopic surgery. Video-assisted surgery includes a wide variety and range of surgeries, including but not limited to the examples specifically mentioned herein. [Background technology]
[0003] One major challenge in laparoscopic or video-assisted surgery is the limited viewing conditions for the surgeon due to the small viewing angle of the laparoscopic camera or other types of cameras, as shown in the exemplary situation of the liver in Figure 1. Furthermore, the surgeon may need to view separate monitors, one displaying a real-time video feed from the scope, and the other showing preoperative diagnostic images, including CT (X-ray computed tomography) images, MRI (magnetic resonance imaging), PET (positron emission tomography) images, or sonography ("ultrasound") images, which are often used for surgical planning and navigation to the surgical site. The surgeon must then mentally fuse such separate preoperative images with the live laparoscopic view, creating a large mental load. Summary of the Invention
[0004] The present disclosure relates to an augmented reality system for video-assisted surgery. The augmented reality system may be interactive, and the video-assisted surgery may include laparoscopic surgery, including robotic laparoscopic surgery and non-robotic laparoscopic surgery. Some device embodiments may include a device comprising: a physical console configured to receive user-provided information indicating a plurality of matching points between an anatomical structure in a reference image and a corresponding anatomical structure in the camera-captured image captured by the camera, a camera-captured image of a target field of view of a camera attached to a surgical instrument, perform image registration based on the received user-provided information indicating a plurality of matching points between the anatomical structure in the reference image and the corresponding anatomical structure in the camera-captured image, and overlay augmented reality content on the camera-captured image based on the image registration; and a storage for recording the image registration, the storage being included in the physical console or the storage being separate from the physical console, and the camera-captured image overlaid with the augmented reality content is displayed on a display. In some device embodiments, the physical console is configured to check the accuracy of the image registration and recalibrate the image registration. In some device embodiments, the physical console is configured to generate a warning of a risk situation.
[0005] Some method embodiments can include a method including capturing an image of a camera target field of view with a camera attached to a surgical instrument; receiving user-provided information indicating a plurality of matching points between anatomical structures in the reference image and corresponding anatomical structures in the camera-captured image; performing image registration based on the received user-provided information indicating the plurality of matching points between anatomical structures in the reference image and corresponding anatomical structures in the camera-captured image; overlaying augmented reality content on the camera-captured image based on the image registration; and displaying the camera-captured image overlaid with the augmented reality content.
[0006] Some non-transitory machine-readable media embodiments can include a non-transitory machine-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method, the method including: receiving user-provided information indicating a plurality of matching points between anatomical structures in a reference image and a camera-captured image of a target field of view of a camera attached to a surgical instrument, the camera-captured image being captured by the camera; performing image registration based on the received user-provided information indicating a plurality of matching points between the anatomical structures in the reference image and the corresponding anatomical structures in the camera-captured image; and overlaying augmented reality content on the camera-captured image based on the image registration, wherein the camera-captured image overlaid with the augmented reality content is displayed on a display.
[0007] In some embodiments, the surgical instrument comprises a laparoscope. In some embodiments, the reference image is a computed tomography (CT) image, a magnetic resonance imaging (MRI) image, a positron emission tomography (PET) image, or an ultrasound image. In some embodiments, the augmented reality content is based on a 3D model that is based on the reference image. In some embodiments, the augmented reality content comprises visualization content that is hidden from the target field of view of the camera. [Brief description of the drawings]
[0008] [Figure 1] 1 illustrates the viewing angles of a laparoscopic camera or other type of camera in the exemplary context of a liver.
[0009] [Diagram 2] 1 shows fiducials placed on the human skull for navigation.
[0010] [Diagram 3] 1 illustrates an exemplary embodiment of a laparoscopic surgery system with an AR system.
[0011] [Figure 4] 1 illustrates an example embodiment of an AR system implemented in an AR console.
[0012] [Diagram 5] 1 illustrates interactive image registration for initial calibration and intraoperative recalibration of an augmented reality system.
[0013] [Figure 6] 13 shows the screen layout of AR software that a surgeon can use to label matching points or features between pre-operative images / models and intra-operative images.
[0014] [Figure 7] 1 shows the tracking phase after the initial calibration.
[0015] [Figure 8] 1 shows a pre-operative (or intra-operative) 3D model that is transformed into a later intra-operative 3D model.
[0016] [Figure 9] 1 illustrates an exemplary embodiment of a system with two imaging devices with an AR system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0017] The present disclosure is not limited to the specific systems, devices, and methods of the present disclosure, which may vary. The terms used in the present disclosure are only for the purpose of describing a particular version or embodiment, and are not intended to limit the scope. Various examples are described below. The description provides specific details for a thorough understanding and enabling description of these examples. However, those skilled in the art will understand that various examples can be implemented without many of these details. Similarly, those skilled in the art will also understand that the embodiments may include many other obvious features that are not described in detail herein. Furthermore, some well-known structures or functions may not be shown or described in detail herein to avoid unnecessarily obscuring the relevant description.
[0018] Augmented reality (AR) can merge multiple information sources into a computer-generated integrated view, where the multiple sources are aligned in real time. In addition to fusing preoperative images or models into the live surgical view, other important structures can be highlighted to make localization easier, and hidden anatomical structures (e.g., blood vessels or tumors below the organ surface) can be visualized as an overlay to improve safety and avoid surgical errors and complications.
[0019] Such augmented reality systems can be used in surgical navigation, where the exact location and size of structures to be removed (e.g., tumors) are often not directly visible in laparoscopic images. AR systems that provide accurate visualization of such hidden structures can help select optimal incision plans that minimize damage to healthy tissue. In addition to fusing various imaging modalities, AR systems can also alert the surgeon to risk situations, either via voice or on-screen pop-ups, for example, when an instrument gets too close to a risk structure.
[0020] The present disclosure describes an interactive augmented reality system for improving the surgeon's vision and situational awareness during laparoscopic and video-assisted surgery. Instead of relying purely on computer vision algorithms for image registration between pre-operative images / models and intra-operative scope images, the system implements an interactive mechanism where the surgeon provides supervised information in an initial calibration phase of the augmented reality function, thus achieving highly accurate image registration. The interaction between the surgeon and the system occurs during surgery, in addition to the initial phase before the start of surgery. In particular, the patient's tissues may move or deform during surgery, for example due to cutting. The augmented reality system can be recalibrated during surgery when the accuracy of the image registration decreases, by seeking additional supervised labeling from the surgeon.
[0021] The augmented reality system improves the surgeon's view during surgery by achieving high image registration accuracy with sporadic use of the surgeon's guidance. The AR system of the present disclosure can extend what the surgeon sees even when the viewing angle of the laparoscopic camera or other type of camera is small. For example, if the camera is placed very close to liver tissue, the surgeon can only see red tissue without the AR system. However, the AR system can reveal what is underneath the red liver tissue, allowing more information to be visible to the surgeon.
[0022] Surgical navigation through augmented reality has been widely adopted in neurosurgery and spine surgery, where trackers can be placed on rigid structures such as the skull or bones. For example, FIG. 2 shows fiducials placed on the human skull for navigation. On the other hand, augmented reality for surgical navigation in laparoscopic surgery has not been well applied due to frequent and irregular movements of tissues / organs during laparoscopic surgery, tissue deformation caused by the surgeon's actions, and the difficulty of placing trackers on target tissues. The present disclosure goes beyond neurosurgery and spine surgery and describes an augmented reality system for laparoscopic surgery, where it is difficult to place trackers on rigid structures.
[0023] A key technique in augmented reality for surgical navigation is image registration between pre-operative images / models and intra-operative images. This has traditionally been achieved by innovations in computer vision algorithms without the involvement of the surgeon. Pure computer vision-based techniques have been unable to achieve high image registration accuracy due to tissue movement and deformation during laparoscopic surgery. Also, these image registration models have not gained the trust of surgeons due to their black-box nature. To overcome these challenges, this disclosure describes an interactive mechanism where the surgeon assists the image registration algorithm by providing some matching points between pre-operative and intra-operative images as anchor points. The system then uses these anchor points to perform image registration to achieve high accuracy.
[0024] This disclosure describes an augmented reality system for improving the surgeon's visibility and situational awareness during laparoscopic and video-assisted surgery by fusing preoperative (or intraoperative) images and reconstructed models with live laparoscopic views, visualizing hidden anatomical structures as overlays on the live images, and alerting the surgeon to risk situations. A key module of such an AR system is image registration, which aligns the live laparoscopic video with the preoperative (or intraoperative) data, i.e., fusing them with a common coordinate system. A major challenge in image registration in such an AR system is the fact that soft tissues are not rigid and move and deform preoperatively and intraoperatively. To address such challenges, this disclosure describes a human-in-the-loop interactive mechanism that reduces the search space in which multiple modalities of images may match, thereby increasing the accuracy of image registration. Specifically, the system captures the surgeon's labeling of important anatomical points, such as the location of the aorta, during surgery and utilizes such additional information to help match the live laparoscopic images with the preoperative (or intraoperative) images.
[0025] Here, video assisted surgery may include robotic and non-robotic surgery, including robotic and non-robotic laparoscopic surgery. Video assisted surgery may include video assisted thoracoscopic surgery (VATS). Video assisted surgery may include endoscopic procedures, percutaneous procedures, and colonoscopic procedures. Video assisted surgery is not limited to the specific examples listed herein, but includes a wide variety of procedures.
[0026] FIG. 3 illustrates an exemplary embodiment of a laparoscopic surgery system 300 with an AR system. The AR system can be implemented in an AR console 310, or other form factors and implementations including hardware or software, or a combination of both hardware and software. The AR console 310 can capture video image frames from a laparoscope 360, for example, by a video capture card. A camera 362 can be attached to, included as part of, or otherwise integrated with the laparoscope 360. The camera 362 can include a video camera that captures images that can be transmitted from the laparoscope 360 to the AR console 310. The AR console 310 can store, for example, by a hard drive or other storage medium, a pre-operative (or intra-operative) 3D model that can be reconstructed from CT, MRI, PET, or ultrasound images. The AR console 310 can run AR software, which can align the pre-operative (or intra-operative) 3D model to the intra-operative video image frames from the laparoscope 360, for example, by using a computer vision algorithm to perform the alignment. The AR monitor 330 can visualize augmented reality results, such as an augmented video in which a deformed pre-operative (or intra-operative) 3D model is overlaid on the laparoscopic video. The AR monitor 330 can be a basic monitor that displays the augmented video received from the AR console 310, or it can be a more robust monitor system that includes a display component capable of displaying the augmented video, as well as circuitry capable of performing some or all of the functions of the AR console 310 to generate the augmented video. Also, a display, such as a panel display that is part of the AR console 310 or a separate display that can be communicatively connected to the AR console 310, for example, via a wired or wireless connection, can perform the functions of the AR monitor 330. A user of the AR system, for example, a surgeon or another surgical team member, can operate the laparoscopic console 340 to adjust or monitor the settings of the laparoscope 360.A surgeon or other surgical team member can proceed with the AR image-based surgical procedure by, for example, cutting, removing, and sealing tissue and / or blood vessels using a stapling device or an energy device.
[0027] 4 illustrates an exemplary embodiment of the AR system implemented in an AR console. The AR console 310 may include circuitry 412, e.g., a video capture card, for capturing video frames from a laparoscope 360. The AR console 310 may include storage 414, e.g., a hard drive or other storage medium, for storing pre-operative (or intra-operative) 3D models, CT images, MRI images, PET images, or ultrasound images, and AR software. The storage may include one or more storage media devices. The AR console 310 may include circuitry 416, e.g., one or more CPUs or other types of processors, for executing software or firmware or other types of programs that cause the circuitry of the AR console 310 to perform the functions of the AR console 310. The AR console 310 may include circuitry 418, e.g., one or more GPUs, for performing real-time inference of the AI model. The AR console 310 may include or be connected to a user interface 420 (e.g., manual operators such as buttons, rotary dials, switches, touch surfaces, touch screens, stylus, track pads, mice, scroll wheels, keyboard keys, etc.; audio devices such as microphones, speakers, etc.; visual devices such as cameras, lights, light sensors, etc.; any other conventional user interface devices) to receive input from and / or provide output to a user. Some or all of the components and / or circuits of the AR console 310 may be interconnected via one or more connections 422, such as buses, cables, wires, traces, etc.
[0028] In some embodiments, the AR console 310 may include an operating system, such as a program, e.g., Linux, for performing the operations of the AR console 310. In some embodiments, the AR console 310 may include circuitry, e.g., an FPGA or an ASIC, or some combination of hardware circuitry and software, for performing the operations of the AR console 310. The AR console 310 may perform real-time video capture, image processing, and / or augmented reality visualization through some or all of the above components. In some embodiments, instead of implementing a separate AR console 310, the AR system may be implemented with hardware components, circuits, and / or software in the laparoscopic console 340. In some embodiments, instead of implementing a separate AR console 310, the AR system may be implemented with hardware components, circuits, and / or software in the monitor 330. For example, the monitor 330 may include elements of a user interface 420, such as a touch screen, microphone, speaker, and camera, to receive user input and provide system output to the user.
[0029] The AR system may have some or all of the following features:
[0030] Fusion of pre-operative (or intra-operative) reconstructed 3D model with live laparoscopic image. 3D reconstruction is widely used as a surgical planning tool where a 3D model of an anatomical structure or tumor is constructed from volumetric images such as CT or MRI imaging, which may be stored in storage 414, which may be located inside or outside the AR console 310. The 3D model may also be constructed from PET or ultrasound images. The 3D model may be reconstructed during surgery, i.e., an intraoperative reconstructed 3D model. FIG. 8 shows a pre-operative (or intra-operative) version 810 of a 3D model reconstructed from CT or MRI images. The AR system may match such a reconstructed 3D model with a live laparoscopic image to visualize the tumor as an overlay on the live laparoscopic image. If the laparoscope's target field of view includes an anatomical structure or tumor (e.g., soft tissue / organ) that is deforming during laparoscopic surgery, the deformation will appear in the intraoperative video frames from the laparoscope 360. To match the deformations captured in the intraoperative video frames, the AR console 310 can use an AI model that correspondingly deforms a preoperative (or intraoperative) 3D model 810 to a later intraoperative 3D model 820, as shown in Figure 8. The AR console 310 can then overlay the deformed 3D model 820 onto the video frames from the laparoscope 360 to generate an augmented video that is output to the monitor 330. The monitor 330 can display the augmented video in which the AR information is more closely aligned with the deformed anatomical structure or tumor, enhancing the surgeon's vision in the laparoscope's field of view.
[0031] Visualization of visible and hidden blood vessels, lymph nodes and nerves on live laparoscopic images. The system can detect and locate blood vessels, lymph nodes and nerves from pre-operative (or intra-operative) images stored in storage 414, for example, using neural network-based object detection and segmentation algorithms of computer vision algorithms executed by AR console 310. During surgery, these pre-operative (or intra-operative) detected tissues are or can be fused with live laparoscopic images from laparoscope 360, for example, by image registration executed by AR console 310. The surgeon can then see tissues hidden from these laparoscopic images, for example, by augmented reality of augmented video from AR console 310, displayed on monitor 330. The system can also detect and locate blood vessels, lymph nodes and nerves from live laparoscopic images from laparoscope 360, using the same neural network-based object detection and segmentation algorithms of computer vision algorithms executed by AR console 310, for example. For example, in the augmented video from the AR console 310, these visible tissues may be or may be highlighted on the live laparoscopic image for better visualization on the monitor 330.
[0032] Detection and location of biomarkers used in surgery. These biomarkers can be used to match pre-operative (or intra-operative) images with later intra-operative images. Biomarkers refer to any visible features on organ tissue, such as vascular intersections or organ edges. Pre-operative (or intra-operative) images and later intra-operative images can be stored in storage 414. Later intra-operative images can come from laparoscope 360 during surgery. AR console 310 can detect the presence of biomarkers and identify the location of the biomarkers in the images, for example, via computer vision algorithms. Biomarkers can be useful as fiducial markers for AR console 310 and its computer vision algorithms and / or surgeons, especially when the laparoscope's target field of view includes a dynamically changing environment in real time, such as a deforming anatomical structure or a tumor.
[0033] Interactive image registration between pre-operative (or intra-operative) and post-operative images. Figure 5 shows a flowchart 500 detailing the interactive mechanism whereby surgeon labeling during surgery is used for image registration. In addition to such surgeon-provided supervised information, visual appearances on tissues such as vascular junctions and surface textures, as well as biomarkers placed pre-operatively (or intra-operatively) and detected during surgery are used in the registration process.
[0034] 6 shows a screen layout 600 of AR software executable on the AR console 310 that the surgeon can use to label the matching points. The screen layout may be displayed on the monitor 330, or on a different display that is part of the AR console 310, or on a different display that is separate from either the AR console 310 and / or the monitor 330. In the exemplary screen layout 600, the left side can show CT images, MRI images, PET images, ultrasound images and / or 3D models 610 (e.g., reconstructed 3D models), 612 (e.g., CT axial views), 614 (e.g., CT sagittal views), 616 (e.g., CT coronal views) stored in the storage 414, and the right side can show live laparoscopic images 620 from the laparoscope 360. The user surgeon is familiar with both the content of the left images and / or models 610, 612, 614, 616 (e.g., pre-operative or intra-operative information) within the target field of view of the laparoscope 360 and the content of the live laparoscopic video feed 620 on the right, such that the user surgeon has sufficient knowledge to be able to label correspondences or features between the left (e.g., still images and / or live video) content and the right (e.g., live video) content shown in the screen layout 600.
[0035] By recognizing the two pairs of matching points or features on the left side 612 and the right side 620, the user surgeon can label a first pair of matching points or features by setting a first left point 632 and a first right point 634 on the screen (e.g., touching these two points on a touch screen with a finger or stylus, directing the mouse pointer and clicking a mouse button when the pointer is on these two points, pressing an arrow key to direct the pointer and then pressing a set key (e.g., space bar) when the pointer is on these two points, etc.). The user surgeon can then label a second pair by similarly setting a second left point 642 and a second right point 644 on the screen.
[0036] Figure 5 can be further explained with reference to the example of Figure 6. In step 510, the user surgeon labels the matches between the pre-operative (or intra-operative) and intra-operative images, as illustrated by the two pairs of matches or features between the left side 612 and right side 620 of Figure 6 above.
[0037] In step 520, the system performs image registration using surgeon-provided labels, which may correspond to the AR console 310 labeling the right-side points 634 and 644 using the left-side points 632 and 642 (along with any other surgeon-provided labeling input data), and then aligns the right-side live laparoscopic video image 620 with the left-side images and / or models 610, 612, 614, and / or 616 based on this labeling. The AR system can fuse the left-side (e.g., still and / or live video) content 610, 612, 614, and / or 616 and the right-side (e.g., live video) content 620 into a common coordinate system. The system uses input from a knowledgeable or informed user (e.g., surgeon, other surgical team member), which can facilitate the image registration algorithm, as these inputs can reduce the search space of possible matching features between multiple modalities of images, and can increase the accuracy and speed of image registration. In performing image registration, the AR system can use matching points or features as anchor points, which can facilitate increased speed and greater accuracy. As described above, the system can perform image registration based not only on surgeon-provided supervised information, but also on visual appearances on tissues, such as vascular junctions and surface textures, as well as biomarkers placed pre-operatively and detected intraoperatively.
[0038] In step 530, the system checks the image registration accuracy. If the image registration accuracy is not good enough, the system returns to step 510 and the user surgeon can relabel the matching points between the pre-operative and intra-operative images, as illustrated in FIG. 6. For example, there may have been an error by the system or the user surgeon in previous steps 510 and / or 520, or the system may lack enough matching points from the user surgeon to reach a good level of image registration accuracy. On the other hand, if the image registration is good for the system, the system can proceed to step 540.
[0039] In step 540, the surgeon checks the image registration accuracy. If the image registration accuracy is not good enough, the system returns to step 510 and the user surgeon can re-label the matching points between the pre-operative (or intra-operative) image and the later intra-operative image, as illustrated in FIG. 6. For example, the surgeon may want a higher level of accuracy, or the surgeon may want to redo the entire calibration phase, or the surgeon may want to modify or cancel previously set labels or anchor points. On the other hand, if the image registration is good enough for the surgeon, the system can proceed to step 550.
[0040] In step 550, an initial calibration phase of the system is performed and the augmented reality system can begin operation. For example, the AR system can perform some or all of the functions provided above, such as displaying augmented video on monitor 330 to enhance the surgeon's vision at the target field of view of laparoscope 360.
[0041] The operational phase of the system after the initial calibration can be understood according to FIG. 7, which shows a flow chart 700 of the details of the interactive mechanism, where tracking takes place after the initial registration of FIG. 5. During surgery, the surgeon may move the laparoscope 360 to a variety of different positions and orientations. The AR system can track these scope movements such that the AR overlay follows the camera movements and adjusts for corresponding changes in the laparoscope's target field of view. During surgery, the surgeon can also move tissues and organs while keeping the position of the laparoscope 360 constant. The AR system can also track these movements and adjust for corresponding changes in the laparoscope's target field of view.
[0042] In step 710, the system is tracking. In this tracking phase, the AR overlay of the system follows the camera movement and / or tissue and / or organ movement by the surgeon. During surgery, image registration accuracy may decrease due to tissue movement or deformation caused, for example, by cutting. In step 730, the augmented reality system can periodically or constantly check the image registration, for example via a computer vision algorithm, and if the image registration accuracy is equal to or exceeds a pre-set threshold, the system returns to or remains in the tracking phase step 710. If such accuracy is below a pre-defined threshold, the system proceeds to step 740, where the system prompts the surgeon to re-execute the interactive mechanism detailed in FIG. 5 to recalibrate the image registration, and then proceeds to step 510 of FIG. 5. In step 720, the surgeon can also manually override the system's decision and request a recalibration of the image registration if he feels the accuracy is low, and then the system proceeds to step 510 of FIG. 5. Thus, even during surgery, the system can capture the surgeon's labeling of important anatomical points, such as the location of the aorta, and can utilize such additional information to facilitate matching of points or features in the live laparoscopic images with reference images, e.g., pre-operative (or intra-operative) images.
[0043] The surgeon can define risk structures that should not be approached during surgery, such as major vessels or nerves. Separately from the AR system, the surgeon can define the risk structures on a preoperative (or intraoperative) 3D model. The 3D model can be input into the AR system. With image registration to identify the location of hidden anatomical structures by fusion with the preoperative (or intraoperative) image, and automatic detection and localization of visible anatomical structures from the live laparoscopic video, the AR system can warn or alert the surgeon during surgery, either by audio (e.g., from the speaker of the AR console 310, the speaker of the monitor 330, the speaker of the laparoscopic console 340, or a standalone placed speaker) or by a pop-up on a screen (e.g., on the display on the AR console 310, the monitor 330, the display on the laparoscopic console 340, or a standalone placed display) if an instrument (e.g., a surgical instrument) gets too close to a predefined risk structure.
[0044] The AR system is deployed or may be deployed in a console that is physically located in the operating room and connected to the laparoscopic surgery system. The console may perform real-time inference of the model. The console may record image registration and detection results generated during the procedure and may also save or store such information on disk as a system log. In some embodiments, some or all components of the AR system may be remotely located outside the operating room and connected to the laparoscopic surgery system via a wired connection (e.g., a data communication cable) or a wireless connection (e.g., a Wi-Fi network or connection). In some embodiments, some or all components of the AR system may be physically integrated into the housing of the laparoscopic surgery console. In some embodiments, some or all components of the laparoscopic surgery system may be physically integrated into the AR console. In some embodiments, some or all components of the AR system may be physically integrated into the housing of the monitor. In some embodiments, some or all components of the monitor may be physically integrated into the AR console. In some embodiments, the AR system may be implemented as a software program or software module executed by one or more processors in the laparoscopic surgery console or in the monitor. In some embodiments, the AR console can be connected to other consoles for other types of video-assisted surgery, such as video-assisted thoracic surgery, video-assisted endoscopic surgery, video-assisted percutaneous surgery, video-assisted colonoscopic surgery, etc. Any or all of the disclosure regarding laparoscopes and laparoscopic surgery may be similarly applied to other video-assisted surgical instruments and procedures, such as endoscopes and endoscopic surgery, percutaneous scopes and percutaneous surgery, colonoscopes and colonoscopic surgery, etc.
[0045] FIG. 9 illustrates an exemplary embodiment of a system 900 with two imaging devices with an AR system. The teachings of the AR system described above can be similarly applied to the AR console 310 of FIG. 9. The AR console 310 can receive and / or capture images from a first imaging device 960, such as by receiving them via one or more video capture cards and storing them in one or more storage media, and can receive and / or capture images from a second imaging device 970. A scanner (e.g., camera, image generator) 962 can be attached to, included as part of, or integrated with the first imaging device 960 (e.g., surgical scope, CT machine, MRI machine, ultrasound machine). The scanner 962 can capture or generate images that can be transmitted from the first imaging device 960 to the AR console 310, for example, as a first live image feed. The scanner (e.g., camera, image generator) 972 may be attached to, included as part of, or integrated into the second imaging device 970 (e.g., surgical scope, CT machine, MRI machine, ultrasound machine). The scanner 972 may capture or generate images that may be transmitted from the second imaging device 970 to the AR console 310, for example, as a second live image feed. The AR console 310 may fuse a first live image feed (e.g., a real-time live endoscopic image feed for a bronchoscopy or colonoscopy procedure) with a second live image feed (e.g., a real-time live CT / ultrasound image feed). The AR console 310 may store, for example, by a hard drive or other storage medium, a pre-operative (or intra-operative) 3D model, which may be reconstructed from CT, MRI, PET, or ultrasound images from the second imaging device 970. The AR console 310 may run AR software that may align the pre-operative (or intra-operative) 3D model to the intra-operative image from the first imaging device 960, for example, by using a computer vision algorithm to perform the alignment.The AR monitor 330 can visualize augmented reality results, such as an augmented video in which a deformed pre-operative (or intra-operative) 3D model is overlaid on an image feed from the first imaging device 960. The AR monitor 330 can display the augmented video received from the AR console 310 or generate and display the augmented video. The display can also perform the functions of the AR monitor 330. A user of the AR system, for example, a surgeon or another surgical team member, can operate the first imaging device console 940 to adjust or monitor settings of the first imaging device 960. A user of the AR system, for example, a surgeon or another surgical team member, can operate the second imaging device console 980 to adjust or monitor settings of the second imaging device 970. In some embodiments, the first imaging device 960 and the first imaging device console 940 can operate as a real-time 2D or 3D camera image feed scope, and the second imaging device 970 and the second imaging device console 980 can operate to provide diagnostic imaging driven by x-ray or ultrasound imaging. A surgeon or other surgical team member can proceed with the AR image-based surgical procedure by, for example, cutting, removing, and sealing tissue and / or blood vessels using a stapling device or an energy device.
[0046] Exemplary embodiments are shown and described in this disclosure. It should be understood that the embodiments can be used in various other combinations and environments and can be changed or modified within the scope of the concepts expressed herein. Some such modifications can include using a program stored in a non-transitory computer-readable medium to enable a computer and / or computer system to perform some or all of the above-described method modifications. Such modifications should not be considered as departures from the spirit and scope of the present invention, and all such modifications as would be apparent to one skilled in the art are intended to be included within the scope of the following claims.
Claims
1. receiving user-provided information indicating a plurality of matching points between anatomical structures in the reference image and corresponding anatomical structures in a camera-captured image of a target field of view of a camera attached to a surgical instrument, the camera-captured image being captured by the camera; performing image registration based on the received user-provided information indicative of the plurality of matching points between anatomical structures in a reference image and corresponding anatomical structures in the camera-captured image; overlaying augmented reality content onto the camera-captured images based on the image registration; A physical console configured as follows: a storage for recording the image registration, the storage being included in the physical console or the storage being separate from the physical console; and An apparatus comprising: the camera-captured image overlaid with the augmented reality content is displayed on a display; Device.
2. The apparatus of claim 1 , wherein the surgical instrument comprises a laparoscope.
3. The apparatus of claim 1 , wherein the reference image is a computed tomography (CT) image, a magnetic resonance imaging (MRI) image, a positron emission tomography (PET) image, or an ultrasound image.
4. The physical console includes: Checking the accuracy of the image registration; Recalibrating the image registration The apparatus of claim 1 , configured as follows:
5. The device of claim 1 , wherein the augmented reality content is based on a 3D model that is based on the reference image.
6. The device of claim 1 , wherein the augmented reality content includes visualization content that is hidden from the target field of view of the camera.
7. The physical console includes: Generate risk situation alerts The apparatus of claim 1 , configured as follows:
8. capturing, by a camera attached to a surgical instrument, an image of a target field of view of the camera; receiving user-provided information indicative of a plurality of matching points between anatomical structures in the reference image and corresponding anatomical structures in the camera-captured image; performing image registration based on the received user-provided information indicative of the plurality of matching points between anatomical structures in a reference image and corresponding anatomical structures in the camera-captured image; overlaying augmented reality content onto the camera-captured images based on the image registration; displaying the camera-captured image overlaid with the augmented reality content; and A method comprising:
9. The method of claim 8 , wherein the surgical instrument comprises a laparoscope.
10. The method of claim 8 , wherein the reference image is a computed tomography (CT) image, a magnetic resonance imaging (MRI) image, a positron emission tomography (PET) image, or an ultrasound image.
11. The method comprises: verifying the accuracy of the image registration; and recalibrating the image registration; and The method of claim 8 , comprising:
12. The method of claim 8 , wherein the augmented reality content is based on a 3D model that is based on the reference image.
13. The method of claim 8 , wherein the augmented reality content includes visualization content that is hidden from the target field of view of the camera.
14. The method comprises: Generate warnings of risk situations The method of claim 8 , comprising:
15. A non-transitory machine-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method, comprising: The method comprises: receiving user-provided information indicating a plurality of matching points between anatomical structures in the reference image and corresponding anatomical structures in a camera-captured image of a target field of view of a camera attached to a surgical instrument, the camera-captured image being captured by the camera; performing image registration based on the received user-provided information indicative of the plurality of matching points between anatomical structures in a reference image and corresponding anatomical structures in the camera-captured image; overlaying augmented reality content onto the camera-captured images based on the image registration; Including, the camera-captured image overlaid with the augmented reality content is displayed on a display; Non-transitory machine-readable media.
16. 16. The non-transitory machine-readable medium of claim 15, wherein the reference image is a computed tomography (CT) image, a magnetic resonance imaging (MRI) image, a positron emission tomography (PET) image, or an ultrasound image.
17. The method comprises: verifying the accuracy of the image registration; and recalibrating the image registration; and 20. The non-transitory machine-readable medium of claim 15, comprising:
18. The non-transitory machine-readable medium of claim 15 , wherein the augmented reality content is based on a 3D model that is based on the reference image.
19. The non-transitory machine-readable medium of claim 15 , wherein the augmented reality content includes visualization content that is hidden from the target field of view of the camera.
20. The method comprises: Generate warnings of risk situations 20. The non-transitory machine-readable medium of claim 15, comprising:
Citation Information
Patent Citations
Image processor
JP1999000309A
Surgery support device and surgery support program
JP2013202313A
