Surgical robotic system and method for intra-operative fusion of different imaging modalities

Abstract

An imaging system includes a laparoscopic camera configured to capture a video stream of tissue and an intraoperative imaging device configured to be inserted through an access port and to obtain a plurality of signals from the tissue. The system also includes an image processing device configured to: generate a 3D reconstruction of the surgical site from the laparoscopic camera video stream to estimate a 3D position of the intraoperative imaging device in a reference frame of the laparoscopic camera; and positioning the laparoscopic camera and the intraoperative imaging device in a world coordinate system based on the 3D reconstruction of the surgical site. The image processing device is further configured to receive a volumetric image of tissue formed by the preoperative imaging modality and to generate a multi-frame representation from the plurality of signals from the intra-operative imaging device. The image processing device is further configured to: register the multi-frame representation with a volumetric image of tissue; deforming the volumetric image of the tissue according to the multi-frame representation; and generating an overlay of the volumetric image of the tissue and the multi-frame representation. The system further includes a screen configured to display a video stream showing data based on the plurality of signals from the intraoperative imaging device and an overlay extending from the intraoperative imaging device.

Description

Background Art

[0001] Surgical robotic systems are currently used for a variety of surgical procedures, including minimally invasive medical procedures. Some surgical robotic systems include a surgeon's console that controls a surgical robotic arm and a surgical instrument having an end effector (e.g., a clamp or grasping instrument) coupled to and actuated by the robotic arm. In operation, the robotic arm moves to a position above the patient and then guides the surgical instrument into a small incision via a surgical port or natural orifice of the patient to position the end effector at a working site in the patient's body. A system is needed for combining preoperative imaging with intraoperative imaging (e.g., endoscopic video feed) and another intraoperative imaging modality (e.g., ultrasound imaging, gamma probe, Raman spectroscopy, etc.) to provide the surgeon with precise instrument placement and tissue edge identification. Summary of the invention

[0002] According to one embodiment of the present disclosure, an imaging system is disclosed. The imaging system includes a laparoscopic ultrasound probe configured to be inserted through an access port and obtain multiple 2D ultrasound images of tissue. The system also includes a laparoscopic camera configured to capture a video stream of the tissue. The system further includes an image processing device configured to receive a volume image of the tissue formed by a first modality image, generate an ultrasound volume from multiple 2D ultrasound images, align the ultrasound volume with the volume image of the tissue, and generate an overlay of the volume image of the tissue and the 2D ultrasound image in the multiple 2D ultrasound images. The system further includes a screen configured to display a video stream showing the laparoscopic ultrasound probe and an overlay extending from the laparoscopic ultrasound probe.

[0003] In addition to 2D intraoperative ultrasound imaging, other intraoperative imaging modalities are also considered, including gamma radiation, Raman spectroscopy, multispectroscopy, time-resolved fluorescence spectroscopy (ms-TRFS), autofluorescence, etc. The corresponding probes can be embedded or tool-integrated.

[0004] Implementations of the above embodiments may also include one or more of the following features. According to one aspect, an ultrasound image may be obtained using an ultrasound probe without using physical fiducial markers, i.e., using a visual-only approach to position the ultrasound probe. A vision-based approach may utilize a deep learning model to estimate the degrees of freedom (DoF) pose of a rigid object (i.e., an ultrasound probe) from a stereo or monocular laparoscopic camera image. The probe may have any number of DoFs, which may be 6DoF. Realistic training data for probe positioning of the deep learning model may be provided by a custom synthetic data generation pipeline. A synthetic 3D anatomically accurate surgical site may be developed based on real data from a surgical procedure. The ultrasound probe may be presented at the surgical site using a 3D virtual (e.g., computer-assisted drafting) model of the probe and a stereo laparoscopic camera geometry from camera calibration. The data may include multiple synthetic images (e.g., about 100,000) to develop a deep learning network to estimate the 6DoF pose of the ultrasound probe directly from the image without modifying the probe in any way, i.e., without physical fiducial markers on the probe. This deep learning model can be trained on each image in a pair of stereo images individually rather than in pairs.

[0005] According to one aspect of the above embodiment, a laparoscopic or robotic ultrasound probe can be positioned in the field of view of a laparoscopic monocular or stereo camera by a visual approach only, without modifying the laparoscopic or robotic ultrasound probe to include any physical fiducial markers. The image processing device can be further configured to position the laparoscopic ultrasound probe in the video stream based on key points or virtual fiducial markers. The image processing device can be further configured to estimate multiple DoF (e.g., 6) poses and orientations of the laparoscopic or robotic ultrasound probe from the video stream based on key points or fiducial markers.

[0006] The image processing device may further be configured to estimate the articulation pose and orientation of a grasper instrument holding the ultrasound probe.Finally, the image processing device may be configured to estimate the pose and orientation of the ultrasound probe by combining the pose and orientation of the probe and the pose and orientation of the grasper holding the probe.

[0007] In addition, the image processing device can be further configured to generate a dense depth map of the surgical site based on a laparoscopic monocular or stereo camera to estimate the 3D position of instruments, probes and anatomical structures in the laparoscopic camera reference frame. In addition, the image processing device can also be configured to implement a deformable visual simultaneous localization and mapping (DV-SLAM) pipeline to locate the laparoscopic camera in 3D space relative to a world coordinate system (WCS) at each acquired image frame over time. The WCS can be associated with a trocar from which the laparoscopic endoscopic camera is inserted, in which case the position of the trocar on the patient's anatomy is estimated based on multiple external cameras mounted on a robotic cart or tower. The WCS can also be associated with one of the instruments (e.g., a grasper instrument that manipulates a laparoscopic ultrasound probe), in which case the position of the trocar on the patient's anatomy is estimated by splitting the axis of the grasper instrument into multiple images captured from the laparoscopic camera and calculating the intersection between the lines fitted to the axis, thereby locating the remote center of motion (RCM) of the grasper instrument trocar. The image processing system may be configured to set the position and orientation of the ultrasound probe in each laparoscopic camera image relative to the WCS, thereby transmitting all images relative to a fixed reference frame.

[0008] The image processing apparatus may also be configured to generate the ultrasound volume by calculating a value of each ultrasound voxel by interpolating between values ​​of ultrasound image slice pixels overlapping with the corresponding voxel after placing each ultrasound image in the world coordinate system.

[0009] As described above, before the ultrasound volume is generated, the laparoscopic camera is positioned relative to the WCS using depth mapping from stereo vision or a pair of monocular images and DV-SLAM. In particular, the positioning of the camera includes depth mapping from the stereo pair, followed by DV-SLAM of consecutive stereo image pairs over time. The depth map provides a single-frame snapshot of how far the object is from the camera, while DV-SLAM provides the positioning of the camera in the WCS.

[0010] Depth mapping can be performed for either a monocular camera or a stereo camera. Stereo camera depth estimation is easier and more reliable than monocular camera depth estimation. DV-SLAM can be performed using either monocular camera input or stereo camera input. DV-SLAM using monocular camera input cannot reliably resolve the scale factor (how far the camera is from the scene) because there are multiple solutions along the same line of 3D points.

[0011] The combination of deformable visual SLAM and depth maps from stereo reconstruction provides the most reliable method for positioning the camera relative to the WCS. The default operating modes may include: 1) depth estimation by stereo reconstruction using a calibrated stereo endoscope or by monocular depth estimation (i.e., depth mapping) using a monocular laparoscope; and 2) DV-SLAM at each frame using stereo pairs in the images from a calibrated stereo camera pair.

[0012] An ultrasound volume can be generated using a method that relies on 6 degrees of freedom (DoF) probe pose estimation based on calibrated stereoscopic endoscopic images. The ultrasound probe is positioned in 3D space by performing 6DoF probe pose estimation in a stereoscopic endoscopic reference frame based on stereoscopic endoscopic images. The stereoscopic endoscopic camera itself can be located in 3D space in a reference WCS associated with the trocar into which the camera is inserted. Stereo reconstruction and DV-SLAM are used to update the position of the camera based on the images provided by the camera itself. Once the ultrasound probe is positioned in the endoscopic image, the 3D position of the probe is tracked in a stationary reference WCS associated with a landmark (e.g., one or more trocars). For 6DoF ultrasound probe positioning, virtual fiducial markers on the probe are estimated based on two images of the stereo pair at each time step (i.e., frame). Then, pose regression of the detected key points is run through a deep learning model. At run time, a neural network based on a deep learning model is executed separately on each channel of the stereoscopic video stream to estimate the DoF pose from each stream. Stereo calibration is used to combine the estimated poses from each channel (i.e., the left or right image in the image pair). In one embodiment, a neural network can be trained directly on the stereo pair to estimate the 6DoF pose of the probe directly from the rectified stereo pair images. This locates the probe DoF in 3D. In addition, the image processing device can also be configured to enhance the ultrasound volume by matching multiple key points in each 2D ultrasound image of the multiple ultrasound images. The image processing device can be further configured to generate a 3D model based on the volumetric image of the tissue formed by the first modality image, and deform the 3D model to conform to the ultrasound volume.

[0013] The 3D model can be deformed as follows: segmenting the 2D / 3D anatomical target surface and tracking the instrument to isolate the motion of the instrument from the anatomical structure; compensating for patient motion, such as respiratory motion compensation, by tracking the target anatomy and estimating the movement of the anatomy while masking out the movement of the instrument; and performing biomechanical modeling to estimate the physically realistic movement of the organs of interest and the anatomical structures surrounding the tracked tissue.

[0014] In the context of 3D model deformation, the image processing device can be configured to segment all instruments at the surgical site so as to mask out non-anatomical areas of interest from the organ surface deformation estimation. The image processing device can be further configured to perform instance segmentation masking on the organ for each frame in the laparoscopic camera image to estimate respiratory motion and surface deformation. The surface deformation can further involve generating a depth map to estimate the ultrasound probe pressure on the tissue and the resulting deformation in the tissue surface. The respiratory motion estimation component can also involve an interface in a pulse oximetry system to predict tissue surface and sub-tissue structure deformation associated with the respiratory cycle. The image processing device can be further configured to identify common sub-tissue landmarks between the ultrasound image and the original preoperative image or the corresponding 3D model, and calculate a dense displacement field registration map between the ultrasound volume and the 3D model. The image processing device can also be further configured to use a neural network to migrate slices of the 3D model to corresponding 2D ultrasound images of the ultrasound volume.

[0015] Prior to registering the 3D model to the ultrasound image / volume, common landmarks in the preoperative (e.g., CT / MRI) images and intraoperative images (e.g., ultrasound and laparoscopic video feeds) are identified. This involves ultrasound sub-tissue structure identification (vessels, tumors, etc.) by ultrasound segmentation, followed by matching and aligning corresponding landmarks in the preoperative imaging model (i.e., same vessels, tumors, etc.). Preoperative imaging model registration includes the following: identification of anatomical landmarks in CT (organ surface, subsurface internal key structures (e.g., vessels, tumors)); identification of surface anatomical landmarks in stereoscopic endoscopic images; and segmentation of subsurface internal key structures (e.g., vessels, tumors).

[0016] According to another embodiment of the present disclosure, a surgical robot system is disclosed. The surgical robot system includes: a first robotic arm that controls a laparoscopic ultrasound probe that is configured to be inserted through an access port and obtain multiple 2D ultrasound images of tissue during surgery; and a second robotic arm that controls a laparoscopic camera that is configured to capture a video stream of the tissue. The system also includes an image processing device that is configured to receive a preoperative image of the tissue, generate a 3D model of the tissue from the preoperative image, and generate an ultrasound volume from a plurality of 2D ultrasound images. The image processing device is further configured to align the ultrasound volume with the 3D model and generate an overlay of the 3D model and the 2D ultrasound images in the plurality of 2D ultrasound images. The system also includes a screen that is configured to display a video stream that shows the laparoscopic ultrasound probe and an overlay that includes a 3D model and a 2D ultrasound image extending along an imaging plane of the laparoscopic ultrasound probe.

[0017] Implementations of the above embodiments may further include one or more of the following features. According to one aspect of the above embodiments, a laparoscopic or robotic ultrasound probe may include multiple physical fiducial markers on the probe so as to robustly estimate its pose and orientation based on the laparoscopic camera image. This can be used to identify probes that lack any discernible visual features on the outside for pose and orientation estimation. According to another aspect of the above embodiments, the laparoscopic or robotic ultrasound probe may not have any physical fiducial markers, thereby eliminating the need for any modification of the ultrasound probe. The image processing device may be further configured to locate the laparoscopic ultrasound probe in the video stream based on multiple key points or virtual fiducial markers. This configuration may involve a training phase in which multiple stereo or monocular laparoscopic camera images are generated from a combination of real surgical sites or in a synthetic environment using synthetic images generated using computer graphics. The training image set is used to train a neural network with a key point detection subnetwork and a pose regression subnetwork. The neural network is used to process each channel of a monocular laparoscopic camera image or a stereo laparoscopic camera image to generate the pose and orientation of the laparoscopic ultrasound probe in the laparoscopic camera image. In this case, the image processing device can be configured to combine the estimated pose from each channel (i.e., the left or right image of the stereo image pair) using a stereo calibration for the final 6DoF pose of the ultrasound probe. Alternatively, the neural network can be trained directly on the stereo image pair and trained for end-to-end 6DoF pose estimation directly from the stereo pair input images.

[0018] The image processing device may also be configured to position the laparoscopic ultrasound probe based on the kinematic data of the first manipulator. The image processing device may be further configured to estimate the posture and orientation of the laparoscopic ultrasound probe from the video stream based on key points or virtual reference markers. The posture and orientation estimation of the laparoscopic ultrasound probe may be completed by a combination of the kinematic data of the first manipulator and the positioning of multiple key points or virtual reference markers according to the laparoscopic camera video stream. The image processing device may be further configured to generate an ultrasound volume by calculating the value of each ultrasound voxel and interpolating between the values ​​of ultrasound image slice pixels overlapping with the corresponding voxel after placing each ultrasound image in the world coordinate system. In addition, the image processing device may also be configured to enhance the ultrasound volume by matching multiple key points in each 2D ultrasound image of the multiple ultrasound images. The image processing device may also be further configured to migrate the slices of the 3D model to the corresponding 2D ultrasound image of the ultrasound volume using a neural network.

[0019] According to another embodiment of the present disclosure, a method for intraoperative imaging of tissue is disclosed. The method includes: generating a 3D model of the tissue from a plurality of preoperative images; generating and updating a depth-based surface map of the tissue using a monocular or stereoscopic laparoscopic camera; and generating an ultrasound volume from a plurality of 2D ultrasound images obtained from a laparoscopic ultrasound probe. The method further includes a visual positioning and mapping pipeline that places the laparoscopic camera in a world coordinate system according to each image of the camera. The method further includes performing ultrasound probe pose and orientation estimation according to a monocular or stereoscopic laparoscopic camera in a world coordinate system to generate an ultrasound volume from a plurality of registered ultrasound image slices. The method further includes registering the ultrasound volume with the 3D model and generating an overlay of the 3D model and a 2D ultrasound image in a plurality of 2D ultrasound images. The method additionally includes displaying a video stream and an overlay obtained from a laparoscopic camera. The video stream includes a laparoscopic ultrasound probe, and the overlay includes a 3D model and a 2D ultrasound image extending along an imaging plane of the laparoscopic ultrasound probe.

[0020] The implementation of the above embodiment may further include one or more of the following features. According to one aspect of the above embodiment, the method may further include: positioning the laparoscopic ultrasound probe in the video stream based on a plurality of key points or virtual fiducial markers set on the laparoscopic ultrasound probe. The method may further include: moving the laparoscopic ultrasound probe by a robotic arm, and positioning the laparoscopic ultrasound probe based on the kinematic data of the robotic arm. The method may further include estimating the posture and orientation of the laparoscopic ultrasound probe from the video stream based on a combination of key points or virtual fiducial markers, robotic arm kinematic data, stereo reconstruction, and visual simultaneous positioning and mapping of the laparoscopic camera relative to a world coordinate system. The method may further include: generating an ultrasound volume by calculating the value of each ultrasound voxel, and interpolating between the values ​​of ultrasound image slice pixels overlapping with the corresponding voxel after placing each ultrasound image in the world coordinate system. In addition, the image processing device may also be configured to enhance the ultrasound volume by matching a plurality of key points in each 2D ultrasound image of the plurality of ultrasound images. The method may further include migrating a slice of the 3D model to a corresponding 2D ultrasound image of the ultrasound volume using a neural network. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Various embodiments of the present disclosure are described herein with reference to the accompanying drawings, in which:

[0022] Figure 1 is a schematic diagram of a surgical robotic system according to an embodiment of the present disclosure, the surgical robotic system comprising a control tower, a control console, and one or more surgical robotic arms each disposed on a movable cart;

[0023] Figure 2According to the embodiments of the present disclosure Figure 1 A three-dimensional diagram of a surgical robot arm of a surgical robot system;

[0024] Figure 3 is a perspective view of a mobile cart having a mounting arm according to an embodiment of the present disclosure, the mounting arm having Figure 1 The surgical robot arm of the surgical robot system;

[0025] Figure 4 According to the embodiments of the present disclosure Figure 1 A schematic diagram of the computer architecture of a surgical robot system;

[0026] Figure 5 is positioned around an operating table according to an embodiment of the present disclosure Figure 1 A plan view of a movable cart;

[0027] Figure 6 is a method for intraoperative fusion of different imaging modalities according to an embodiment of the present disclosure;

[0028] Figure 7 is a method of acquiring and registering multiple imaging modalities according to an embodiment of the present disclosure;

[0029] Fig. 8A is a computed tomography image according to an embodiment of the present disclosure;

[0030] Figure 8B is a 3D model according to an embodiment of the present disclosure;

[0031] Figure 8C is an endoscopic video image according to an embodiment of the present disclosure;

[0032] Fig.8D is an ultrasound image according to an embodiment of the present disclosure;

[0033] Fig. 9 is a stereoscopic view of a laparoscopic ultrasound probe according to an embodiment of the present disclosure;

[0034] Fig.10 is a schematic diagram showing multiple coordinate systems and transformation matrices for converting between them according to an embodiment of the present disclosure;

[0035] Fig.11 shows an ultrasound segmentation image according to an embodiment of the present disclosure;

[0036] Fig.12 shows an enhanced laparoscopic video stream superimposed with a 3D model and an ultrasound imaging plane extending from an ultrasound probe according to an embodiment of the present disclosure;

[0037] Fig.13A and Fig. 13B is a schematic diagram of a laparoscopic ultrasound probe according to an embodiment of the present disclosure, the laparoscopic ultrasound probe identifying the margins of a tumor in tissue and identifying a resection path;

[0038] Fig.14 is a schematic diagram showing generation of an ultrasound volume from a plurality of 2D ultrasound slices according to an embodiment of the present disclosure; and

[0039] Fig.15 is a schematic flow chart showing the registration between an intraoperative ultrasound volume and a preoperative CT volume according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0040] Embodiments of the surgical robotic system disclosed herein are described in detail with reference to the drawings, in which like reference numerals designate like or corresponding elements in each of the several views.

[0041] As will be described in detail below, the present disclosure relates to a surgical robotic system that includes a surgeon's console, a control tower, and one or more movable carts having a surgical robotic arm coupled to a mounting arm. The surgeon's console receives user inputs through one or more interface devices, which are processed by the control tower into movement commands for moving the surgical robotic arm and instruments and / or cameras coupled thereto. Thus, the surgeon's console enables remote operation of the surgical arm and attached instruments / cameras. The surgical robotic arm includes a controller that is configured to process movement commands and is configured to generate torque commands for enabling one or more actuators of the robotic arm, which in turn move the robotic arm in response to the movement commands.

[0042] refer to Figure 1 , the surgical robotic system 10 includes a control tower 20 that is connected to all components of the surgical robotic system 10, including a surgeon's console 30 and one or more mobile carts 60. Each mobile cart 60 includes a robotic arm 40 having a surgical instrument 50 removably coupled thereto. The robotic arm 40 is also coupled to the mobile cart 60. The robotic system 10 may include any number of mobile carts 60 and / or robotic arms 40.

[0043] The surgical instrument 50 is configured for use during minimally invasive surgical procedures. In an embodiment, the surgical instrument 50 can be configured for open surgical procedures. In other embodiments, the surgical instrument 50 can be an electrosurgical clamp configured to seal tissue by compressing the tissue between the jaw members and applying an electrosurgical current thereto. In yet other embodiments, the surgical instrument 50 can be a surgical stapler comprising a pair of jaws configured to grasp and clamp tissue, simultaneously deploy multiple tissue fasteners (e.g., staples) and cut stapled tissue. In yet other embodiments, the surgical instrument 50 can be a surgical clip applicator comprising a pair of jaws configured to apply a surgical clip to the tissue.

[0044] One of the robotic arms 40 may include a laparoscopic camera 51 configured to capture a video of the surgical site. The laparoscopic camera 51 may be a stereo endoscope configured to capture two side-by-side (i.e., left and right) images of the surgical site to produce a video stream of the surgical scene. The laparoscopic camera 51 is coupled to an image processing device 56, which may be disposed within the control tower 20. The image processing device 56 may be any computing device as described below, which is configured to receive a video feed from the laparoscopic camera 51 and output a processed video stream.

[0045] The surgeon console 30 includes a first screen 32 that displays a video feed of the surgical site provided by a camera 51 of a surgical instrument 50 disposed on the robotic arm 40, and a second screen 34 that displays a user interface for controlling the surgical robotic system 10. The first screen 32 and the second screen 34 may be touch screens that allow various graphical user inputs to be displayed.

[0046] The surgeon console 30 also includes a plurality of user interface devices, such as a foot pedal 36 and a pair of hand controls 38a and 38b, which are used by the user to remotely control the robotic arm 40. The surgeon console further includes an armrest 33 for supporting the clinician's arms while operating the hand controls 38a and 38b.

[0047] The control tower 20 includes a screen 23 (which may be a touch screen) and outputs on a graphical user interface (GUI). The control tower 20 also serves as an interface between the surgeon's console 30 and one or more robotic arms 40. In particular, the control tower 20 is configured to control the robotic arms 40, such as moving the robotic arms 40 and corresponding surgical instruments 50 based on a set of programmable instructions and / or input commands from the surgeon's console 30, in a manner such that the robotic arms 40 and surgical instruments 50 perform a desired sequence of movements in response to inputs from the foot pedals 36 and hand controllers 38a and 38b. The foot pedals 36 may be used to enable and lock the hand controllers 38a and 38b, reposition camera movement, and electrosurgical activation / deactivation. In particular, the foot pedals 36 may be used to perform clutch actions on the hand controllers 38a and 38b. The clutch is activated by pressing one of the foot pedals 36, which disconnects the hand controllers 38a and / or 38b from the robotic arm 40 and the corresponding instrument 50 or camera 51 attached thereto (i.e., prevents movement input). This allows the user to reposition the hand controllers 38a and 38b without moving the robotic arm(s) 40 and instruments 50 and / or camera 51. This is useful when reaching the control boundaries of the surgical space.

[0048] Each of the control tower 20, surgeon console 30 and robotic arm 40 includes a corresponding computer 21, 31, 41. The computers 21, 31, 41 are interconnected to each other using any suitable communication network based on wired or wireless communication protocols. The term "network", whether plural or singular, as used herein, refers to a data network, including but not limited to the Internet, an intranet, a wide area network or a local area network, and is not limited to the full scope of the definition of communication networks covered by the present disclosure. Suitable protocols include but are not limited to the Transmission Control Protocol / Internet Protocol (TCP / IP), the Datagram Protocol / Internet Protocol (UDP / IP) and / or the Datagram Congestion Control Protocol (DC). Wireless communication can be achieved through one or more wireless configurations, such as radio frequency, light, Wi-Fi, Bluetooth (an open wireless protocol for exchanging data from fixed and mobile devices over short distances using short-length radio waves), creating a personal area network (PAN), (A set of specifications for advanced communication protocols using small, low-power digital radios based on the IEEE 122.15.4-1203 Wireless Personal Area Network (WPAN) standard).

[0049] Computer 21,31,41 may include any suitable processor (not shown), the processor is operably connected to a memory (not shown), the memory may include one or more of a volatile, non-volatile, magnetic, optical or electrical medium, such as a read-only memory (ROM), a random access memory (RAM), an electrically erasable programmable ROM (EEPROM), a non-volatile RAM (NVRAM) or a flash memory. The processor may be any suitable processor (e.g., a control circuit) suitable for performing the operations, calculations and / or instruction sets described in the present disclosure, including but not limited to a hardware processor, a field programmable gate array (FPGA), a digital signal processor (DSP), a central processing unit (CPU), a microprocessor and combinations thereof. It will be appreciated by those skilled in the art that a processor may be replaced by using any logical processor (e.g., a control circuit) suitable for performing the algorithms, calculations and / or instruction sets described herein.

[0050] refer to Figure 2 , each robotic arm 40 may include a plurality of links 42a, 42b, 42c that are interconnected at joints 44a, 44b, 44c, respectively. Other configurations of links and joints may be used as known to those skilled in the art. Joint 44a is configured to secure the robotic arm 40 to the mobile cart 60 and defines a first longitudinal axis. Figure 3 , the movable cart 60 includes a lift 67 and a mounting arm 61, which provides a base for mounting the robotic arm 40. The lift 67 allows the mounting arm 61 to move vertically. The movable cart 60 also includes a display 69, which is used to display information about the robotic arm 40. In an embodiment, the robotic arm 40 can include any type and / or number of joints.

[0051] The installation arm 61 includes a first link 62a, a second link 62b and a third link 62c, which provide lateral maneuverability of the robot arm 40. The links 62a, 62b, 62c are interconnected at joints 63a and 63b, and each joint may include an actuator (not shown) for rotating the links 62b and 62b relative to each other and relative to the link 62c. In particular, the links 62a, 62b, 62c can move in their corresponding lateral planes parallel to each other, thereby allowing the robot arm 40 to extend relative to the patient (e.g., an operating table). In an embodiment, the robot arm 40 can be connected to an operating table (not shown). The installation arm 61 includes a control device 65, which is used to adjust the movement of the links 62a, 62b, 62c and the elevator 67. In an embodiment, the installation arm 61 can include any type and / or number of joints.

[0052] The third link 62c may include a rotatable base 64 having two degrees of freedom. In particular, the rotatable base 64 includes a first actuator 64a and a second actuator 64b. The first actuator 64a can rotate around a first fixed arm axis perpendicular to the plane defined by the third link 62c, and the second actuator 64b can rotate around a second fixed arm axis transverse to the first fixed arm axis. The first actuator 64a and the second actuator 64b allow the robot 40 to be fully oriented in three dimensions.

[0053] The actuator 48b of the joint 44b is connected to the joint 44c via the belt 45a, and the joint 44c is further connected to the joint 46b via the belt 45b. The joint 44c can include a transfer case connecting the belts 45a and 45b, so that the actuator 48b is configured to rotate each of the links 42b, 42c and the holder 46 relative to each other. More specifically, the links 42b, 42c and the holder 46 are passively connected to the actuator 48b, which is forced to rotate around the pivot point "P", which is located at the intersection of the first axis defined by the link 42a and the second axis defined by the holder 46. In other words, the pivot point "P" is the remote center of motion (RCM) of the robot 40. Therefore, the actuator 48b controls the angle θ between the first axis and the second axis, thereby allowing the surgical instrument 50 to be oriented. Due to the interconnection of the links 42a, 42b, 42c and the retainer 46 via the straps 45a and 45b, the angle between the links 42a, 42b, 42c and the retainer 46 is also adjusted to achieve the desired angle θ. In an embodiment, some or all of the joints 44a, 44b, 44c may include actuators to eliminate the need for mechanical linkages.

[0054] The joints 44a and 44b include actuators 48a and 48b that are configured to drive the joints 44a, 44b, 44c relative to each other through a series of belts 45a and 45b or other mechanical linkages, such as drive rods, cables, levers, etc. In particular, the actuator 48a is configured to rotate the robotic arm 40 about a longitudinal axis defined by the link 42a.

[0055] refer to Figure 2 The holder 46 defines a second longitudinal axis and is configured to receive an instrument drive unit (IDU) 52 ( Figure 1). The IDU 52 is configured to be coupled to the actuation mechanisms of the surgical instrument 50 and the camera 51 and is configured to move (e.g., rotate) and actuate the instrument 50 and / or the camera 51. The IDU 52 transmits an actuation force from its actuator to the surgical instrument 50 to actuate components of the end effector 49 of the surgical instrument 50. The holder 46 includes a sliding mechanism 46a configured to move the IDU 52 along a second longitudinal axis defined by the holder 46. The holder 46 also includes a joint 46b that rotates the holder 46 relative to the link 42c. During an endoscopic procedure, the instrument 50 can pass through an endoscope access port 55 ( Figure 3 ) is inserted. The holder 46 also includes a port lock 46c ( Figure 2 ).

[0056] The robot arm 40 also includes a plurality of manual override buttons 53 ( Figure 1 ), the plurality of manual override buttons can be used in manual mode. The user can press one or more of these buttons 53 to move the component associated with the button 53.

[0057] refer to Figure 4 , each of the computers 21, 31, 41 of the surgical robot system 10 may include multiple controllers, which may be implemented in hardware and / or software. The computer 21 of the control tower 20 includes a controller 21a and a safety viewer 21b. The controller 21a receives data from the computer 31 of the surgeon's console 30 about the current position and / or orientation of the hand controllers 38a and 38b and the status of the foot pedal 36 and other buttons. The controller 21a processes these input positions to determine the desired drive commands for each joint of the robotic arm 40 and / or the IDU 52, and transmits these desired drive commands to the computer 41 of the robotic arm 40. The controller 21a also receives the actual joint angles measured by the encoders of the actuators 48a and 48b, and uses this information to determine force feedback commands, which are transmitted back to the computer 31 of the surgeon's console 30 to provide tactile feedback through the hand controllers 38a and 38b. The safety observer 21b performs validity checks on data entering and exiting the controller 21a and, if an error in data transmission is detected, notifies the system fault handler to place the computer 21 and / or surgical robot system 10 into a safe state.

[0058] The computer 41 includes a plurality of controllers, namely, a cart main controller 41a, a mounting arm controller 41b, a robot arm controller 41c, and an instrument drive unit (IDU) controller 41d. The cart main controller 41a receives and processes joint commands from the controller 21a of the computer 21, and transmits them to the mounting arm controller 41b, the robot arm controller 41c, and the IDU controller 41d. The cart main controller 41a also manages instrument exchange and the overall status of the movable cart 60, the robot arm 40, and the IDU 52. The cart main controller 41a also transmits the actual joint angles back to the controller 21a.

[0059] Each of the joints 63a and 63b and the rotatable base 64 of the mounting arm 61 is a passive joint (i.e., there is no actuator therein) that allows manual adjustment by the user. The joints 63a and 63b and the rotatable base 64 include brakes that are disengaged by the user to configure the mounting arm 61. When the brake is engaged, the mounting arm controller 41b monitors the sliding of each of the joints 63a and 63b and the rotatable base 64 of the mounting arm 61, or when the brake is disengaged, the mounting arm controller can be freely moved by the operator without affecting the control of other joints. The robot arm controller 41c controls each joint 44a and 44b of the robot arm 40 and calculates the desired motor torque required for gravity compensation, friction compensation, and closed-loop position control of the robot arm 40. The robot arm controller 41c calculates a movement command based on the calculated torque. The calculated motor command is then transmitted to one or more of the actuators 48a and 48b in the robot arm 40. The actual joint positions are then transmitted by actuators 48a and 48b back to the robot controller 41c.

[0060] The IDU controller 41d receives the desired joint angles (such as wrist and jaw angles) of the surgical instrument 50 and calculates the desired current for the motors in the IDU 52. The IDU controller 41d calculates the actual angles based on the motor positions and transmits the actual angles back to the cart master controller 41a.

[0061] The robotic arm 40 is controlled in response to the posture of a hand controller (e.g., hand controller 38a) controlling the robotic arm 40, which is transformed into the desired posture of the robotic arm 40 by the hand-eye transformation function performed by the controller 21a. The hand-eye function and other functions described herein are implemented in software executable by the controller 21a or any other suitable controller described herein. The posture of one of the hand controllers 38a can be implemented as a coordinate position and roll-pitch-yaw (RPY) orientation relative to a coordinate reference system fixed to the surgeon's console 30. The desired posture of the instrument 50 is relative to the fixed system on the robotic arm 40. The posture of the hand controller 38a is then scaled by the zoom function performed by the controller 21a. In an embodiment, the coordinate position can be reduced and the orientation can be enlarged by the zoom function. In addition, the controller 21a can also perform a clutch function for disengaging the hand controller 38a from the robotic arm 40. In particular, if certain movement limits or other thresholds are exceeded, the controller 21a stops transmitting movement commands from the hand controller 38a to the robotic arm 40 and essentially acts like a virtual clutch mechanism, eg, limiting mechanical input from affecting mechanical output.

[0062] The desired pose of the manipulator 40 is based on the pose of the hand controller 38a and then passed through the inverse kinematics function executed by the controller 21a. The inverse kinematics function calculates the angles of the joints 44a, 44b, 44c of the manipulator 40 that achieve the scaled and adjusted pose input by the hand controller 38a. The calculated angles are then passed to the manipulator controller 41c, which includes a joint axis controller with a proportional derivative (PD) controller, a friction estimator module, a gravity compensator module, and a double-sided saturation block, which is configured to limit the command torque of the motors of the joints 44a, 44b, 44c.

[0063] refer to Figure 5 , the surgical robotic system 10 is set up around an operating table 90. The system 10 includes movable carts 60a-d, which may be numbered "1" to "4". During set up, each of the carts 60a-d is positioned around the operating table 90. The position and orientation of the carts 60a-d depends on a number of factors, such as the placement of a plurality of access ports 55a-d, which in turn depends on the surgery being performed. Once the port placement is determined, the access ports 55a-d are inserted into the patient's body and the carts 60a-d are positioned to insert the instruments 50 and laparoscopic cameras 51 into the corresponding ports 55a-d.

[0064] During use, each of the mechanical arms 40a-d engages the latch 46c ( Figure 2 ) is attached to the entry port 55 ( Figure 3) is attached to one of the access ports 55a-d inserted into the patient's body. The IDU 52 is attached to the holder 46, and then the SIM 43 is attached to the distal portion of the IDU 52. Thereafter, the instrument 50 is attached to the SIM 43. The instrument 50 is then inserted through the access port 55 by moving the IDU 52 along the holder 46. The SIM 43 includes a plurality of drive shafts that are configured to transmit the rotation of each motor of the IDU 52 to the instrument 50, thereby actuating the instrument 50. In addition, the SIM 43 provides a sterile barrier between the instrument 50 and other components of the robotic arm 40 (including the IDU 52). The SIM 43 is also configured to fix a sterile sheet (not shown) to the IDU 52.

[0065] refer to Figure 6 The method for intraoperative fusion of different imaging modalities includes combining preoperative imaging and intraoperative imaging to provide a combined 3D image of an organ, tumor, or any other tissue and an overlay of different modalities. Preoperative imaging includes any suitable imaging modality, such as computed tomography (CT), magnetic resonance imaging (MRI), or a method capable of obtaining such as Fig. 8A Any other imaging modality that can be used to image the 3D image shown. Intraoperative imaging can be ultrasound imaging. CT imaging is well suited for preoperative use because CT provides high quality images. However, the use of intraoperative CT is not desirable due to the radiation exposure and supine positioning of the patient. Ultrasound imaging is well suited for intraoperative use because it is safe for frequent imaging regardless of the patient's position, even though ultrasound provides noisy images with a limited viewing angle. In addition to ultrasound, other imaging modalities can be used, such as gamma radiation, Raman spectroscopy, multispectral imaging, time-resolved fluorescence spectroscopy (ms-TRFS) probes, and autofluorescence.

[0066] At step 100, the image processing device 56 receives the preoperative image, which may be accomplished by obtaining a plurality of 2D images and reconstructing a 3D volume image therefrom. In an embodiment, the preoperative image may be provided to any other computing device (eg, outside the operating room) to perform the image processing steps described herein.

[0067] Figure 7 Provided Figure 6 Additional sub-steps to each main step of the method. Step 100 includes a plurality of segmentation steps 100a-d, i.e., segmentation of organ surfaces, vasculature, landmarks, and tumors. As used herein, the term "segmentation" means obtaining a plurality of 2D slices or segments of an object.

[0068] refer to Figure 6 At step 102, the image processing device 56 or another computing device generates Figure 8BThe 3D model shown may be a wire mesh model based on the preoperative image. The image processing device 56 may generate a 3D model including a plurality of points or vertices interconnected by line segments based on the segmentation, and including surface textures on the vertices and segments.

[0069] Steps 100 and 102 are performed preoperatively, wherein subsequent steps are performed once a surgical procedure has begun, the surgical procedure including setting up the robotic system 10, such as Figure 5 In an embodiment, the method of the present disclosure may use an independent imaging system 80 ( Fig.10 ), a laparoscopic camera 51, a laparoscopic ultrasound probe 70, and one or more screens 32 and 34 ( Figure 1 ), the independent imaging system, the laparoscopic camera and the laparoscopic ultrasound probe are connected to the image processing device 56. At step 104, the ultrasound probe 70 is inserted through one of the access ports 55a-d and can be controlled by one of the robotic arms 40a-d and the corresponding IDU 52. Fig. 9 and Fig.10 , the ultrasound probe 70 includes an ultrasound transducer 72 configured to output ultrasound waves. The laparoscopic camera 51 is positioned so that the surgical site is within its field of view, and the ultrasound probe 70 is then also moved into the field of view of the laparoscopic camera 51. At step 106, the image processing device 56 positions the ultrasound probe 70. The image processing device 56 may store (in a memory or storage device) the size of the ultrasound probe 70 and the positions and distances between key points or virtual reference markers 74. Reference Figure 8C , the image processing device 56 analyzes all stereo pairs of frames (left channel images and right channel images) from the video stream to identify a plurality of key points or virtual fiducial markers 74 ( Fig. 9 ), which then enables the image processing device 56 to determine the 3D dimensions in the video stream, such as a depth map.

[0070] The virtual fiducial markers 74 are generated by a machine learning image processing algorithm configured to recognize the geometry of the ultrasound probe 70. The image processing device 56 is configured to execute an image processing algorithm, which may include a deep learning model to estimate the 6 degrees of freedom (DoF) pose of a rigid object (i.e., the ultrasound probe 70) from a stereo or monocular laparoscopic camera image. Realistic training data for probe positioning of the deep learning model is provided by a custom synthetic data generation pipeline. A synthetic 3D anatomically accurate surgical site is developed based on actual data from a surgical procedure. The ultrasound probe can be presented on the surgical site using a 3D virtual (e.g., computer-aided drafting) model of the probe and the stereo laparoscopic camera geometry from camera calibration. The data may include multiple synthetic images (e.g., about 100,000) to develop a deep learning network to estimate the 6DoF pose of the ultrasound probe directly from the image without modifying the probe in any way, i.e., without physical fiducial markers on the probe. This deep learning model can be trained individually on each image in a pair of stereo images rather than in pairs. In further embodiments, the probe 70 may include physical fiducial markers in addition to the virtual fiducial markers 74, which may be formed by any visually distinctive (e.g., white, fluorescent, etc.) paint, dye, etching, and / or object (e.g., a dot, block, instrument component, marking, etc.).

[0071] The positioning may be based on image processing performed by the image processing device 56 as described above, or may additionally include kinematic data of the robotic arm 40 that moves the ultrasound probe 70. The kinematic data includes position, velocity, posture, orientation, joint angles, and other data based on movement commands provided to the robotic arm 40 and the execution of the commands by the robotic arm. In further embodiments, other tracking techniques may also be used, such as electromagnetic tracking.

[0072] During the image analysis for positioning the ultrasound probe 70, the image processing device 56 treats the ultrasound probe 70 as a rigid object and uses the rigid object to estimate the pose and position in real time based on the image from the laparoscopic camera 51. In some aspects, a machine learning image processing algorithm can be used to perform pose estimation. These algorithms can be trained on multiple (e.g., about 100,000) synthetic stereo images that have presented the ultrasound probe and surgical scene with blood, smoke, and other realistic artifacts to improve generalization capabilities. Machine learning can be implemented using a neural network in a two-stage process with a keypoint detector and a pose regressor.

[0073] In embodiments that also use kinematic data, the final pose of the ultrasound probe 70 is determined by first estimating a coarse positioning pose using kinematics, followed by a fine pose estimate through vision (ie, image processing by the image processing device 56).

[0074] In an embodiment, the ultrasound probe 70 may be held by the instrument 50, such as Figure 8C and Fig.10 At step 108 , the image processing device uses the same deep learning algorithm used to identify virtual fiducial markers as described above with respect to step 106 to locate the instrument 50 .

[0075] The image processing device 56 may be further configured to estimate the articulation posture and orientation of the instrument 50 holding the ultrasound probe 70. In particular, the image processing device 56 may be configured to estimate the posture and orientation of the ultrasound probe 70 by combining the posture and orientation of the probe 70 and the posture and orientation of the instrument 50 holding the probe 70.

[0076] At step 110, the image processing device 56 generates a depth map of the surgical site from the camera 51 to estimate the 3D positions of the instruments 50, the probe 70, and the anatomical structures in the laparoscopic camera reference frame. At step 112, the image processing device 56 positions the camera 51. This can be done using a deformable vision simultaneous localization and mapping (DV-SLAM) pipeline at each acquired image frame over time relative to a world coordinate system (WCS) ( Fig.10 ). The WCS may be associated with an access port 55 through which the camera 51 is inserted, in which case the location of the access port 55 on the patient's anatomy is estimated based on one or more external cameras (not shown) that may be mounted on the mobile cart 60a-d and / or the system tower 10 or other suitable location. In particular, localization of the camera includes depth mapping from a stereo pair followed by DV-SLAM of successive stereo image pairs over time. The depth map provides a single-frame snapshot of how far an object is from the camera, while the DV-SLAM provides localization of the camera in the WCS.

[0077] At step 114, the ultrasound probe 70 and the instrument 50 are positioned in the video feed of the image processing device 56. At step 116, the ultrasound probe 70 and the instrument 50 are positioned in the WCS. In particular, the WCS is either associated with an access port 55 from which the camera 51 is inserted, as described above. In another embodiment, the WCS may also be associated with an access port 55 from which one of the instruments 50, namely the grasper instrument 50 that manipulates the ultrasound probe 70, is inserted. In this case, the position of the access port 55 in the patient's body is estimated by dividing the axis of the instrument 50 into multiple images captured by the camera 51 and calculating the intersection between the lines fitted to the axis, thereby locating the remote center of motion (RCM) of the access port 55 of the instrument 50. The image processing device 56 is configured to set the position and orientation of the ultrasound probe 70 in each image relative to the WCS, so that all images are transmitted relative to a fixed reference frame.

[0078] After positioning the ultrasound probe 70, the ultrasound probe 70 is used to obtain an ultrasound image of the tissue at step 118. Figure 7 , the ultrasound probe 70 is used to perform segmentation of blood vessels, landmarks, and tumors at steps 118a, 118b, and 118c, respectively. The image processing device 56 may include an ultrasound image processor and other components for displaying ultrasound images next to the video images from the laparoscopic camera 51 in any suitable manner (e.g., on separate screens 32 and 34, picture-in-picture, overlay, etc.).

[0079] At step 120 , the image processing device 56 is further configured to construct a 3D ultrasound volume from the segmented ultrasound image. Fig.11 Exemplary segmented ultrasound image slices and volumes reconstructed based on sub-tissue landmarks (eg, critical structures, tumors, veins, arteries, etc.) are shown. The segmented landmarks may be displayed as different colored overlays on the ultrasound image displayed on screen 32 and / or screen 34.

[0080] An ultrasound volume can be generated using a method that relies on 6DoF probe pose estimation based on calibrated stereoscopic endoscopic images. The ultrasound probe is positioned in 3D space by performing 6DoF probe pose estimation in a stereoscopic endoscopic reference frame based on stereoscopic endoscopic images. The stereoscopic endoscopic camera itself can be located in 3D space in a reference WCS associated with the entry port 55 into which the camera 51 is inserted or in a reference WCS associated with the entry port 55 into which the grasper instrument 50 is inserted. Stereo reconstruction and DV-SLAM are used to update the position of the camera 51 based on the images provided by the camera itself. Once the ultrasound probe is positioned in the endoscopic image, its 3D position is tracked in a stationary reference WCS associated with a landmark (e.g., one or more trocars). For 6DoF ultrasound probe positioning, virtual key points on the probe are estimated at each time step based on the two images of the stereo pair. Then, the pose regression of the detected key points is run through a deep learning model. At runtime, a neural network based on a deep learning model is executed separately on each channel of stereo vision to estimate the DoF pose from each channel. Stereo calibration is used to combine the estimated pose from each channel (i.e., left or right image in an image pair). This positions the probe in 3D.

[0081] refer to Fig.14 The image processing device may also be configured to generate an ultrasound volume by calculating a value for each ultrasound voxel and interpolating between values ​​of ultrasound image slice pixels overlapping the corresponding voxel after placing each ultrasound image in a world coordinate system. The 2D ultrasound images are represented by their plane equations using 3 points and values ​​of 3D ultrasound voxels between slices calculated from distance-weighted orthogonal projections.

[0082] The image processing device 56 is configured to generate an ultrasound volume at step 122 by registering the individual ultrasound slices together based on the positioning of the ultrasound probe 70 (i.e., using kinematics for coarse positioning and rigid object pose estimation for fine pose estimation). The stitching of the ultrasound images can be completed by matching key points obtained at different orientations when the ultrasound probe 70 is moved in multiple directions on the tissue being subjected to ultrasound. In particular, the image registration process may include detecting invariant key points using scale-invariant feature transform (SIFT), accelerated robust features (SURF), robust independent basic features (BRIEF), oriented FAST and rotation BRIEF (ORB), or any other suitable image matching technology. Thereafter, the image processing device 56 extracts robust feature descriptors from the key points detected in each continuous ultrasound slice image, and calculates the distance between the descriptors of each key point in the continuous ultrasound images. Further, the image processing device 56 selects the top match for each descriptor of the continuous ultrasound images, estimates the homography matrix, and aligns the continuous images to create a stitched volume ultrasound panorama.

[0083] At step 124, the image processing device 56 enhances or refines the ultrasound volume from the ultrasound image segment by tracking across the ultrasound image and by matching a plurality of key points in each 2D ultrasound image in the plurality of ultrasound images. The image processing device may be further configured to generate a 3D ultrasound model based on the volumetric image of the tissue formed by the first modality image, and deform the 3D ultrasound model to conform to the ultrasound volume.

[0084] At step 126, the image processing device 56 compensates for the patient's respiratory motion, since respiration imparts movement to the tissue and the attached instrument 50 and camera 51. The image processing device 56 is configured to receive respiration data from the patient, which may come from a pulse oximetry device configured to measure respiration rate, pulse, etc.

[0085] At step 128, the 3D ultrasound model is deformed, which includes: segmenting the 2D / 3D anatomical target surface and tracking the instrument to isolate the motion of the instrument from the anatomical structure; compensating for patient motion, such as respiratory motion compensation, by tracking the target anatomy and estimating the movement of the anatomy while ignoring the movement of the instrument; and performing biomechanical modeling to estimate the physically realistic movement of the anatomical structure surrounding the organ of interest and the tracked tissue.

[0086] At step 130, in steps 1c-e ( Figure 7 ) shows an ultrasound slide projection overlay 92 on the endoscopic image ( Fig.10 and Fig.12 ). At step 132, the image processing device 56 registers the constructed ultrasound volume with the 3D model constructed in step 102. Fig.15 , the image processing device 56 utilizes cross-modal transfer learning for this purpose. In particular, a generative adversarial network (GAN) architecture can be used to migrate preoperative image or model (e.g., CT) slices to intraoperative image (e.g., ultrasound) slices. Since the ultrasound probe 70 can be located at any orientation and position, the CT volume is re-sliced ​​at any slice orientation. A transition data set that is a synthetic ultrasound data set constructed from CT data is used to pre-train the segmentation network. Thereafter, transfer learning can be used to fine-tune the neural network on a real ultrasound data set. The trained neural network can then be used to align the constructed ultrasound volume with the 3D model. Therefore, alignment is performed between a 3D model based on the preoperative CT volume and an intraoperative ultrasound volume generated by aligning multiple ultrasound images at arbitrary orientations and overlapping portions.

[0087] Prior to registering the 3D model to the ultrasound image / volume, common landmarks in the preoperative (e.g., CT / MRI) images and intraoperative images (e.g., ultrasound and laparoscopic video feeds) are identified. This involves ultrasound sub-tissue structure identification (vessels, tumors, etc.) by ultrasound segmentation, followed by matching and aligning corresponding landmarks in the preoperative imaging / model (i.e., same vessels, tumors, etc.). Preoperative model registration includes the following: identification of anatomical landmarks in preoperative imaging / model (organ surface, subsurface internal key structures (e.g., vessels, tumors)); identification of surface anatomical landmarks in stereoscopic endoscopic images; and segmentation of subsurface internal key structures (e.g., vessels, tumors).

[0088] At step 134, the image processing device 56 deforms the 3D model to align with the ultrasound model. For this step, the image processing device 56 can use another neural network trained on ultrasound and preoperative imaging / models of the same tissue to learn ultrasound to CT / MRI registration, i.e., by learning the appearance mapping between CT / MRI images of the tissue of interest (e.g., tumor) and the corresponding ultrasound images and by learning the appearance mapping between CT / MRI key structure landmarks (vessels, arteries, etc.) and ultrasound key structure landmarks. The image processing device 56 is further configured to deform the 3D model after registering the vessels, landmarks, and tumors between the preoperative (e.g., CT / MRI) images and the intraoperative (e.g., ultrasound) images.

[0089] At step 136, the image processing device 56 displays the deformed 3D model as an overlay on the endoscopic video stream of the laparoscopic camera 51, which may be shown on the first screen 32. The image processing device 56 may also display the deformed 3D model as an overlay on the ultrasound feed of the ultrasound probe 70, which may be shown on the second screen 34. In embodiments, the video stream and the ultrasound stream may be displayed side-by-side, in picture-in-picture, or in any other manner on the same screen 32 or 34. Fig.12 The video stream from the laparoscopic camera 51 is enhanced by a 3D model 91 of the tissue of interest (eg, a tumor) and an ultrasound slide projection overlay 92 extending from the ultrasound probe 70. The ultrasound slide projection overlay 92 is continuously generated along the imaging plane of the ultrasound probe 70.

[0090] refer to Fig.13A and Fig. 13B These overlays can be used to identify the margins of tissue of interest (e.g., tumor "T") in organ "O" by moving, pivoting, rotating, etc. the ultrasound probe 70 to the edge of the tumor and beginning to cut the tissue to resect the tumor. The systems and methods of the present disclosure advantageously allow for continuous and simultaneous positioning of margins during resection.

[0091] It will be understood that various modifications may be made to the embodiments disclosed herein. Therefore, the above description should not be interpreted as limiting, but merely as an illustration of various embodiments. Those skilled in the art will envision other modifications within the scope and spirit of the appended claims.

Claims

1. An imaging system, comprising: a laparoscopic ultrasound probe configured to be inserted through the access port and to obtain a plurality of 2D ultrasound images of tissue; a laparoscopic camera configured to capture a video stream of the tissue; An image processing device, the image processing device being configured to: generating a depth map of the surgical site based on the video stream to estimate the 3D position of the laparoscopic ultrasound probe in a reference frame of the laparoscopic camera; positioning the laparoscopic camera and the laparoscopic ultrasound probe in a world coordinate system based on the depth map; receiving a volumetric image of tissue formed from the first modality image; generating an ultrasound volume from the plurality of 2D ultrasound images; registering the ultrasound volume with the volumetric image of the tissue; and generating a superposition of a volumetric image of the tissue and a 2D ultrasound image from the plurality of 2D ultrasound images; as well as A screen is configured to display the video stream showing the laparoscopic ultrasound probe and the overlay extending from the laparoscopic ultrasound probe.

2. The imaging system according to claim 1, wherein: The image processing device is further configured to generate a plurality of virtual fiducial markers on the laparoscopic ultrasound probe using a deep learning image processing algorithm.

3. The imaging system according to claim 2, wherein: The image processing device is further configured to position the laparoscopic ultrasound probe in the video stream based on the virtual fiducial markers.

4. The imaging system according to claim 3, wherein: The image processing device is further configured to estimate the pose and orientation of the laparoscopic ultrasound probe from the video stream based on the virtual fiducial markers.

5. The imaging system according to claim 1, wherein: The laparoscopic camera is a stereoscopic camera, and the video stream comprises a pair of stereoscopic image streams.

6. The imaging system according to claim 5, wherein: The image processing device is further configured to locate the laparoscopic camera and the laparoscopic ultrasound probe in the world coordinate system using a Deformable Vision Simultaneous Localization and Mapping (DV-SLAM) pipeline based on the pair of stereo image streams.

7. The imaging system of claim 1, wherein: The image processing apparatus is further configured to compensate for deformation of the ultrasound volume in response to respiratory motion based on respiratory data from the pulse oximeter device.

8. The imaging system of claim 1, wherein: The image processing apparatus is further configured to generate the ultrasound volume by matching a plurality of key points in each of the plurality of 2D ultrasound images; and Provides visual guidance of negative resection margins encompassing and surrounding the tumor.

9. The imaging system of claim 1, wherein: The image processing device is further configured to generate a 3D model based on a volumetric image of the tissue formed from the first modality image; and The 3D model is deformed to conform to the ultrasound volume.

10. The imaging system of claim 9, wherein: The image processing device is further configured to transfer the slices of the 3D model to corresponding 2D ultrasound images of the ultrasound volume using a neural network.

11. The imaging system of claim 1, further comprising a laparoscopic imaging probe selected from the group consisting of a gamma radiation probe, a Raman spectroscopy probe, a multispectral probe, a time-resolved fluorescence spectroscopy (ms-TRFS) probe, and an autofluorescence probe.

12. A surgical robot system, comprising: a first robotic arm including a laparoscopic ultrasound probe configured to be inserted through the access port and to obtain a plurality of 2D ultrasound images of tissue during surgery; a second robotic arm comprising a laparoscopic camera configured to capture a video stream of the tissue; An image processing device, the image processing device being configured to: generating a depth map of the surgical site based on the video stream to estimate the 3D position of the laparoscopic ultrasound probe in a reference frame of the laparoscopic camera; positioning the laparoscopic camera and the laparoscopic ultrasound probe in a world coordinate system based on the depth map; receiving a 3D model of tissue formed from the first modality image; generating an ultrasound volume from the plurality of 2D ultrasound images; registering the ultrasound volume with the 3D model of the tissue; deforming at least one of the 3D model or the ultrasound volume based on at least one of pressure exerted on the tissue by the laparoscopic ultrasound probe or respiratory motion; and generating an overlay of the 3D model of the tissue and the 2D ultrasound images of the plurality of 2D ultrasound images; and A screen is configured to display the video stream showing the laparoscopic ultrasound probe and the overlay including the 3D model and the 2D ultrasound image extending along an imaging plane of the laparoscopic ultrasound probe.

13. The surgical robot system according to claim 12, wherein: The image processing device is further configured to position the laparoscopic ultrasound probe based on the kinematic data of the first robotic arm.

14. The surgical robot system according to claim 12, wherein: The image processing device is further configured to generate a plurality of virtual fiducial markers on the laparoscopic ultrasound probe using a deep learning image processing algorithm.

15. The surgical robot system according to claim 14, wherein: The image processing device is further configured to position the laparoscopic ultrasound probe in the video stream based on the virtual fiducial markers.

16. The surgical robot system according to claim 15, wherein: The image processing device is further configured to estimate the pose and orientation of the laparoscopic ultrasound probe from the video stream based on the virtual fiducial markers.

17. The surgical robot system according to claim 12, wherein: The laparoscopic camera is a stereoscopic camera, and the video stream comprises a pair of stereoscopic image streams.

18. The surgical robot system according to claim 17, wherein: The image processing device is further configured to locate the laparoscopic camera and the laparoscopic ultrasound probe in the world coordinate system using a Deformable Vision Simultaneous Localization and Mapping (DV-SLAM) pipeline based on the pair of stereo image streams.

19. A method for intraoperative imaging of tissue, the method comprising: Obtain multiple 2D ultrasound images from a laparoscopic ultrasound probe; Obtain video stream from laparoscopic camera; generating a depth map of the surgical site based on the video stream to estimate the 3D position of the laparoscopic ultrasound probe in a reference frame of the laparoscopic camera; positioning the laparoscopic camera and the laparoscopic ultrasound probe in a world coordinate system based on the depth map; receiving a 3D model of tissue formed from the first modality image; generating an ultrasound volume from the plurality of 2D ultrasound images; registering the ultrasound volume with the 3D model of the tissue; and generating an overlay of the 3D model of the tissue and the 2D ultrasound images of the plurality of 2D ultrasound images; as well as generating an overlay of the 3D model of the tissue and the 2D ultrasound images of the plurality of 2D ultrasound images; The video stream is displayed, the video stream including the laparoscopic ultrasound probe and the overlay including the 3D model and the 2D ultrasound image extending along an imaging plane of the laparoscopic ultrasound probe.

20. The method according to claim 19, further comprising: At least one of the 3D model or the ultrasound volume is deformed based on at least one of pressure exerted on the tissue by the laparoscopic ultrasound probe or respiratory motion to produce a deformed model.

21. The method according to claim 20, further comprising: Based on this deformed model registered with the ultrasound volume, a suggested resection path for the tumor is displayed along with a selected negative resection margin for the tumor.

22. An imaging system comprising: a laparoscopic camera configured to capture a video stream of tissue; an intraoperative imaging device configured to be inserted through the access port and to obtain a plurality of signals from tissue; An image processing device, the image processing device being configured to: generating a 3D reconstruction of the surgical site from a laparoscopic camera video stream to estimate a 3D position of the intraoperative imaging device in a reference frame of the laparoscopic camera; Positioning the laparoscopic camera and the intraoperative imaging device in a world coordinate system based on the 3D reconstruction of the surgical site; receiving a volumetric image of the tissue formed by a preoperative imaging modality; generating a plurality of frame representations from a plurality of signals from the intraoperative imaging device; registering the multi-frame representation with a volumetric image of the tissue; deforming a volumetric image of the tissue according to the multi-frame representation; and generating a volumetric image of the tissue and an overlay of the multiple frame representations; as well as A screen is configured to display the video stream showing data based on the plurality of signals from the intraoperative imaging device and the overlay extending from the intraoperative imaging device.

23. The imaging system of claim 22, wherein: The intraoperative imaging device is a laparoscopic ultrasound probe and the preoperative imaging modality is at least one of MRI or CT.

Images