Auxiliary port placement for minimally invasive or robot-assisted surgery

Abstract

A surgical robotic system is disclosed that includes a robotic arm that holds a laparoscopic camera inserted through an access port. The system also includes a controller configured to generate a port position of an access port on the 3D model of the patient and to generate a patient-specific installation guide for configuring the access port and the robotic arm. The system also includes an external camera configured to register the robotic arm and the patient. The system further includes a display configured to output the port position of the access port as a stack overlaid on an external image of the patient based on a registration of the robotic arm and the patient.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of and priority to U.S. Provisional Patent Application Serial No. 63 / 355,713, filed on June 27, 2022, and U.S. Provisional Patent Application Serial No. 63 / 440,985, filed on January 25, 2023. The entire disclosures of the foregoing applications are incorporated herein by reference. Background Art

[0003] Minimally invasive surgery (MIS) and robotic-assisted surgery (RAS) use single or multiple incisions on external human anatomy to access the surgical site. The location of these incisions determines the placement of the access port (or trocar). The surgeon and assistant insert instruments (including laparoscopic cameras) through these ports to access the surgical site. The current practice is for the surgeon to palpate the patient lying on the bed and visually locate their anatomical landmarks (e.g., thorax and umbilicus). Fixed port placement guidance templates are used to make incisions on the patient's external anatomy and place the port. Advanced users who are proficient in the surgical field can efficiently perform this procedure for the MIS or RAS platform for which they have been pre-trained. The needs of novice users of existing MIS or RAS platforms and all users of new RAS platforms for auxiliary port placement have not been met. The installation time of robotic surgical systems can be long and depends on the placement of the access port. Therefore, a system for virtual placement of the access port is needed, which is then used to determine the initial placement of the movable cart and robotic arm. Summary of the invention

[0004] According to one embodiment of the present disclosure, a surgical robotic system is disclosed. The surgical robotic system includes a robotic arm that holds a laparoscopic camera inserted through an access port. The system also includes a controller that is configured to generate a port position of an access port on a 3D model of a patient and to generate a patient-specific setup guide for configuring the access port and the robotic arm. The system also includes an external camera that is configured to register the robotic arm and the patient. The system further includes a display that is configured to output the port position of the access port as an overlay on an external image of the patient based on the registration of the robotic arm and the patient.

[0005] The implementation of the above embodiment may include one or more of the following features. According to one aspect of the above embodiment, the display may be a monitor or a head-mounted display. The external camera may be further configured to capture multiple external images of the patient and the image of the robotic arm. The laparoscopic camera may be configured to capture an internal image of the surgical site. The controller may be further configured to generate a depth map of the surgical site based on the internal image of the surgical site. The controller may be further configured to generate a registration of the patient's preoperative imaging data with the depth map. The controller may be additionally configured to track the position of the laparoscopic camera via the kinematics of the robotic arm and visual simultaneous localization and mapping (visual SLAM). The controller may be further configured to update the registration between the preoperative imaging data and the depth map via fully automatic registration using real-time robotic arm kinematics data and the visual SLAM.

[0006] According to another embodiment of the present disclosure, a method for assisting in the placement of an access port is disclosed. The method includes inserting a laparoscopic camera held by a robotic arm through an access port. The method also includes generating, at a controller, a port position of the access port on a 3D model of the patient, and generating a patient-specific setup guide for configuring the access port and the robotic arm. The method further includes registering the robotic arm and the patient at an external camera, and outputting, on a display, the port position of the access port as an overlay on an external image of the patient based on the registration of the robotic arm and the patient.

[0007] Implementations of the above embodiments may include one or more of the following features. According to one aspect of the above embodiments, the port position of the access port may be output as the stack on at least one of a monitor or a head mounted display. The method may also include capturing multiple external images of the patient and an image of the robotic arm by the external camera. The method may additionally include capturing an internal image of the surgical site by the laparoscopic camera. The method may also include generating a depth map of the surgical site at the controller based on the internal image of the surgical site. The method may further include generating a registration of the patient's preoperative imaging data and the depth map at the controller. In various aspects, the method may further include tracking the position of the laparoscopic camera via the kinematics of the robotic arm and visual simultaneous localization and mapping (visual SLAM). The method may further include tracking the registration between the preoperative imaging data and the depth map via fully automatic registration using real-time robotic arm kinematics data and the visual SLAM.

[0008] According to another embodiment of the present disclosure, a method for determining access port placement is disclosed. The method includes capturing multiple external images of a patient from an external visual system, and generating an external 3D model of the patient based on the multiple external images. The method also includes generating an internal 3D model of the patient based on preoperative imaging data, and generating a combined 3D model based on the external 3D model and the internal 3D model. The method further includes determining a port position of at least one access port based on the combined 3d model, and generating a placement guide for configuring at least one access port and a robotic arm. The method further includes outputting the port position of the at least one access port and the placement guide as an overlay on the external image of the patient on a display based on the registration of the robotic arm and the patient.

[0009] Implementations of the above embodiments may include one or more of the following features. According to one aspect of the above embodiments, the method may further include generating the external 3D model of the patient based on the depth map of the patient. The method may further include generating a skeleton model based on the multiple external images, and the skeleton model may include multiple key points of the patient. The method may further include generating the external 3D model based on the skeleton model. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Various embodiments of the present disclosure are described herein with reference to the accompanying drawings, in which:

[0011] Figure 1 is a schematic diagram of a surgical robotic system according to an embodiment of the present disclosure, the surgical robotic system comprising a control tower, a control console, and one or more surgical robotic arms each disposed on a movable cart;

[0012] Figure 2 According to the embodiments of the present disclosure Figure 1 A three-dimensional diagram of a surgical robot arm of a surgical robot system;

[0013] Figure 3 is a perspective view of a mobile cart having a mounting arm according to an embodiment of the present disclosure, the mounting arm having Figure 1 The surgical robot arm of the surgical robot system;

[0014] Figure 4 According to the embodiments of the present disclosure Figure 1 A schematic diagram of the computer architecture of a surgical robot system;

[0015] Figure 5 is positioned around an operating table according to an embodiment of the present disclosure Figure 1 A schematic plan view of a movable cart;

[0016] Figure 6is a flow chart of a method for generating installation guidelines for auxiliary port placement according to an embodiment of the present disclosure;

[0017] Figure 7 is a computed tomography image according to an embodiment of the present disclosure;

[0018] Figure 8 Patients and Figure 1 White light (ie, RGB) still images and corresponding depth map images of a surgical robotic system;

[0019] Fig. 9 is a schematic flow chart of posture estimation of a patient according to an embodiment of the present disclosure;

[0020] Fig.10 is a computer-generated 3D model of a patient including a skeletal model according to an embodiment of the present disclosure;

[0021] Fig.11 is an image of a graphical user interface (GUI) including a 3D model of a patient and suggested access port locations according to an embodiment of the present disclosure;

[0022] Fig.12 is an image of a GUI including port locations for instruments and a camera according to an embodiment of the present disclosure;

[0023] Fig.13 is an image of a GUI including a preview window of a camera at different orientations according to an embodiment of the present disclosure;

[0024] Fig.14 is a flow chart of a method for auxiliary port placement based on installation guidelines according to an embodiment of the present disclosure;

[0025] Fig.15 is a perspective view of a virtual reality system for assisting port placement according to an embodiment of the present disclosure;

[0026] Fig.16 is a projected image of internal organs and port locations according to an embodiment of the present disclosure; and

[0027] Fig.17 is a projected image of internal organs and access ports according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] Embodiments of the surgical robotic system disclosed herein are described in detail with reference to the drawings, in which like reference numerals designate like or corresponding elements in each of the several views.

[0029] As will be described in detail below, the present disclosure relates to a surgical robotic system comprising a surgeon's console, a control tower, and one or more movable carts having a surgical robotic arm coupled to a mounting arm. The surgeon's console receives user input via one or more interface devices. The input is processed by the control tower as movement commands for moving the surgical robotic arm and instruments and / or cameras coupled thereto. Thus, the surgeon's console enables remote operation of the surgical arm and attached instruments / cameras. The surgical robotic arm comprises a controller configured to process movement commands to control one or more actuators of the robotic arm, which in turn move the robotic arm and instruments in response to the movement commands.

[0030] refer to Figure 1 , the surgical robotic system 10 includes a control tower 20 that is connected to all components of the surgical robotic system 10, including a surgeon's console 30 and one or more mobile carts 60. Each mobile cart 60 includes a robotic arm 40 having a surgical instrument 50 coupled thereto. The robotic arm 40 is also coupled to the mobile cart 60. The robotic system 10 may include any number of mobile carts 60 and / or robotic arms 40.

[0031] The surgical instrument 50 is configured for use during minimally invasive surgical procedures. In an embodiment, the surgical instrument 50 can be configured for open surgical procedures. In other embodiments, the surgical instrument 50 can be an electrosurgical clamp configured to seal tissue by compressing the tissue between the jaw members and applying an electrosurgical current thereto. In yet other embodiments, the surgical instrument 50 can be a surgical stapler comprising a pair of jaws configured to grasp and clamp tissue, simultaneously deploy multiple tissue fasteners (e.g., staples) and cut stapled tissue. In yet other embodiments, the surgical instrument 50 can be a surgical clip applicator comprising a pair of jaws configured to apply a surgical clip to the tissue.

[0032] One of the robotic arms 40 may include a laparoscopic camera 51 configured to capture a video of the surgical site. The laparoscopic camera 51 may be a stereo endoscope configured to capture two side-by-side (i.e., left and right) images of the surgical site to produce a video stream of the surgical scene. The laparoscopic camera 51 is coupled to a video processing device 56, which may be disposed within the control tower 20. The video processing device 56 may be any computing device as described below, which is configured to receive a video feed from the laparoscopic camera 51 and output a processed video stream.

[0033] The surgeon console 30 includes a first display 32 that displays a video feed of the surgical site provided by a camera 51 disposed on the robotic arm 40 and a second display 34 that displays a user interface for controlling the surgical robotic system 10. The first display 32 and the second display 34 may be touch screens that allow for the display of various graphical user inputs.

[0034] The surgeon console 30 also includes a plurality of user interface devices, such as a foot pedal 36, and a pair of handle controllers 38a and 38b used by a user to remotely control the robotic arm 40. The surgeon console further includes armrests 33 for supporting the clinician's arms while operating the handle controllers 38a and 38b.

[0035] The control tower 20 includes a display 23 (which may be a touch screen) and outputs on a graphical user interface (GUI). The control tower 20 also serves as an interface between the surgeon console 30 and one or more robotic arms 40. In particular, the control tower 20 is configured to control the robotic arms 40 based on a set of programmable instructions and / or input commands from the surgeon console 30 so as to move the robotic arms 40 and corresponding surgical instruments 50 in such a way that the robotic arms 40 and surgical instruments 50 perform a desired sequence of movements in response to inputs from the foot pedals 36 and the handle controllers 38a and 38b. The foot pedals 36 may be used to enable and lock the hand controllers 38a and 38b, reposition camera movement, and electrosurgical activation / deactivation. In particular, the foot pedals 36 may be used to perform clutch actions on the hand controllers 38a and 38b. The clutch is activated by pressing one of the foot pedals 36, which disconnects the hand controllers 38a and / or 38b from the robotic arm 40 and the corresponding instrument 50 or camera 51 attached thereto (i.e., prevents movement input). This allows the user to reposition the hand controllers 38a and 38b without moving the robotic arm(s) 40 and instruments 50 and / or camera 51. This is useful when reaching the control boundaries of the surgical space.

[0036] Each of the control tower 20, surgeon console 30 and robotic arm 40 includes a corresponding computer 21, 31, 41. The computers 21, 31, 41 are interconnected with each other using any suitable communication network based on wired or wireless communication protocols. The term "network" as used herein, whether plural or singular, means a data network, including but not limited to the Internet, an intranet, a wide area network or a local area network, and is not limited to the full scope of the definition of communication networks covered by the present disclosure. Suitable protocols include but are not limited to the Transmission Control Protocol / Internet Protocol (TCP / IP), the Datagram Protocol / Internet Protocol (UDP / IP) and / or the Datagram Congestion Control Protocol (DCCP). Wireless communication can be achieved via one or more wireless configurations, such as radio frequency, optical, Wi-Fi, Bluetooth (an open wireless protocol for exchanging data from fixed and mobile devices over short distances using short-length radio waves to create a personal area network (PAN)), (Specification of a suite of advanced communications protocols using small, low-power digital radios based on the IEEE 122.15.4-1203 standard for Wireless Personal Area Networks (WPANs).

[0037] Computers 21, 31, 41 may include any suitable processor (not shown) operably connected to a memory (not shown), which may include one or more of volatile, nonvolatile, magnetic, optical or dielectric, such as read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM (EEPROM), nonvolatile RAM (NVRAM) or flash memory. The processor may be any suitable processor (e.g., control circuit) suitable for performing the operations, calculations and / or instruction sets described in the present disclosure, including but not limited to hardware processors, field programmable gate arrays (FPGAs), digital signal processors (DSPs), central processing units (CPUs), graphics processing units (GPUs), microprocessors and combinations thereof. Those skilled in the art will appreciate that the processor may be replaced by using any logical processor (e.g., control circuit) suitable for performing the algorithms, calculations and / or instruction sets described herein.

[0038] refer to Figure 2 , each robotic arm 40 may include a plurality of links 42a, 42b, 42c that are interconnected at joints 44a, 44b, 44c, respectively. Other configurations of links and joints may be used as known to those skilled in the art. Joint 44a is configured to secure the robotic arm 40 to the mobile cart 60 and defines a first longitudinal axis. Figure 3, the movable cart 60 includes a lift 67 and a mounting arm 61, which provides a base for mounting the robotic arm 40. The lift 67 allows the mounting arm 61 to move vertically. The movable cart 60 also includes a display 69, which is used to display information about the robotic arm 40. In an embodiment, the robotic arm 40 can include any type and / or number of joints.

[0039] The installation arm 61 includes a first link 62a, a second link 62b and a third link 62c, which provide lateral maneuverability of the robot arm 40. The links 62a, 62b, 62c are interconnected at joints 63a and 63b, and each joint may include an actuator (not shown) for rotating the links 62b and 62b relative to each other and relative to the link 62c. In particular, the links 62a, 62b, 62c can move in their corresponding lateral planes parallel to each other, thereby allowing the robot arm 40 to extend relative to the patient (e.g., an operating table). In an embodiment, the robot arm 40 can be connected to an operating table (not shown). The installation arm 61 includes a control device 65, which is used to adjust the movement of the links 62a, 62b, 62c and the elevator 67. In an embodiment, the installation arm 61 can include any type and / or number of joints.

[0040] The third link 62c may include a rotatable base 64 having two degrees of freedom. In particular, the rotatable base 64 includes a first actuator 64a and a second actuator 64b. The first actuator 64a can rotate around a first fixed arm axis perpendicular to the plane defined by the third link 62c, and the second actuator 64b can rotate around a second fixed arm axis transverse to the first fixed arm axis. The first actuator 64a and the second actuator 64b allow the robot 40 to be fully oriented in three dimensions.

[0041] The actuator 48b of the joint 44b is connected to the joint 44c via the belt 45a, and the joint 44c is further connected to the joint 46b via the belt 45b. The joint 44c can include a transfer case connecting the belts 45a and 45b, so that the actuator 48b is configured to rotate each of the links 42b, 42c and the holder 46 relative to each other. More specifically, the links 42b, 42c and the holder 46 are passively connected to the actuator 48b, which is forced to rotate around the pivot point "P", which is located at the intersection of the first axis defined by the link 42a and the second axis defined by the holder 46. In other words, the pivot point "P" is the remote center of motion (RCM) of the robot 40. Therefore, the actuator 48b controls the angle θ between the first axis and the second axis, thereby allowing the surgical instrument 50 to be oriented. Due to the interconnection of the links 42a, 42b, 42c and the retainer 46 via the straps 45a and 45b, the angle between the links 42a, 42b, 42c and the retainer 46 is also adjusted to achieve the desired angle θ. In an embodiment, some or all of the joints 44a, 44b, 44c may include actuators to eliminate the need for mechanical linkages.

[0042] The joints 44a and 44b include actuators 48a and 48b that are configured to drive the joints 44a, 44b, 44c relative to each other through a series of belts 45a and 45b or other mechanical linkages, such as drive rods, cables, levers, etc. In particular, the actuator 48a is configured to rotate the robotic arm 40 about a longitudinal axis defined by the link 42a.

[0043] refer to Figure 2 The holder 46 defines a second longitudinal axis and is configured to receive an instrument drive unit (IDU) 52 ( Figure 1 ). The IDU 52 is configured to be coupled to the actuation mechanisms of the surgical instrument 50 and the camera 51 and is configured to move (e.g., rotate) and actuate the instrument 50 and / or the camera 51. The IDU 52 transmits an actuation force from its actuator to the surgical instrument 50 to actuate components of the end effector of the surgical instrument 50. The holder 46 includes a sliding mechanism 46a configured to move the IDU 52 along a second longitudinal axis defined by the holder 46. The holder 46 also includes a joint 46b that rotates the holder 46 relative to the link 42c. During an endoscopic procedure, the instrument 50 can pass through an endoscope access port 55 ( Figure 3 ) is inserted. The holder 46 also includes a port lock 46c ( Figure 2 ).

[0044] The IDU 52 is attached to the holder 46, and then the sterile interface module (SIM) 43 is attached to the distal portion of the IDU 52. The SIM 43 is configured to secure a sterile drape (not shown) to the IDU 52. The instrument 50 is then attached to the SIM 43. The instrument 50 is then inserted through the access port 55 by moving the IDU 52 along the holder 46. The SIM 43 includes a plurality of drive shafts that are configured to transmit the rotation of the respective motors of the IDU 52 to the instrument 50, thereby actuating the instrument 50. In addition, the SIM 43 provides a sterile barrier between the instrument 50 and other components of the robotic arm 40 (including the IDU 52).

[0045] The robot arm 40 also includes a plurality of manual override buttons 53 ( Figure 1 ), the plurality of manual override buttons can be used in manual mode. The user can press one or more of these buttons 53 to move the component associated with the button 53.

[0046] refer to Figure 4 , each of the computers 21, 31, 41 of the surgical robot system 10 may include multiple controllers, which may be implemented in hardware and / or software. The computer 21 of the control tower 20 includes a controller 21a and a safety observer 21b. The controller 21a receives data about the current position and / or orientation of the handle controllers 38a and 38b and the status of the foot pedal 36 and other buttons from the computer 31 of the surgeon's console 30. The controller 21a processes these input positions to determine the desired drive commands for each joint of the robotic arm 40 and / or the IDU 52, and transmits these desired drive commands to the computer 41 of the robotic arm 40. The controller 21a also receives the actual joint angles measured by the encoders of the actuators 48a and 48b, and uses this information to determine force feedback commands, which are transmitted back to the computer 31 of the surgeon's console 30 to provide tactile feedback through the handle controllers 38a and 38b. The safety observer 21b performs validity checks on data entering and exiting the controller 21a and, if an error in data transmission is detected, notifies the system fault handler to place the computer 21 and / or surgical robot system 10 into a safe state.

[0047] The computer 41 includes a plurality of controllers, namely, a cart main controller 41a, a mounting arm controller 41b, a robot arm controller 41c, and an instrument drive unit (IDU) controller 41d. The cart main controller 41a receives and processes joint commands from the controller 21a of the computer 21, and transmits them to the mounting arm controller 41b, the robot arm controller 41c, and the IDU controller 41d. The cart main controller 41a also manages instrument exchange and the overall status of the movable cart 60, the robot arm 40, and the IDU 52. The cart main controller 41a also transmits the actual joint angles back to the controller 21a.

[0048] Each of the joints 63a and 63b and the rotatable base 64 of the mounting arm 61 is a passive joint (i.e., there is no actuator therein) that allows manual adjustment by the user. The joints 63a and 63b and the rotatable base 64 include brakes that are disengaged by the user to configure the mounting arm 61. When the brake is engaged, the mounting arm controller 41b monitors the sliding of each of the joints 63a and 63b and the rotatable base 64 of the mounting arm 61, or when the brake is disengaged, the mounting arm controller can be freely moved by the operator without affecting the control of other joints. The robot arm controller 41c controls each joint 44a and 44b of the robot arm 40 and calculates the desired motor torque required for gravity compensation, friction compensation, and closed-loop position control of the robot arm 40. The robot arm controller 41c calculates a movement command based on the calculated torque. The calculated motor command is then transmitted to one or more of the actuators 48a and 48b in the robot arm 40. The actual joint positions are then transmitted by actuators 48a and 48b back to the robot controller 41c.

[0049] The IDU controller 41d receives the desired joint angles (such as wrist and jaw angles) of the surgical instrument 50 and calculates the desired current for the motors in the IDU 52. The IDU controller 41d calculates the actual angles based on the motor positions and transmits the actual angles back to the cart master controller 41a.

[0050] The robotic arm 40 is controlled in response to the posture of the handle controller (e.g., handle controller 38a) controlling the robotic arm 40, which is transformed into the desired posture of the robotic arm 40 by the hand-eye transformation function performed by the controller 21a. The hand-eye function and other functions described herein are implemented in the software that can be executed by the controller 21a or any other suitable controller described herein. The posture of one of the handle controllers 38a can be implemented as a coordinate position and roll-pitch-yaw (RPY) orientation relative to the coordinate reference system fixed to the surgeon's console 30. The desired posture of the instrument 50 is relative to the fixed system on the robotic arm 40. The posture of the handle controller 38a is then scaled by the zoom function performed by the controller 21a. In an embodiment, the coordinate position can be reduced and the orientation can be enlarged by the zoom function. In addition, the controller 21a can also perform a clutch function for disengaging the handle controller 38a from the robotic arm 40. In particular, if certain movement limits or other thresholds are exceeded, the controller 21a stops transmitting movement commands from the handle controller 38a to the robotic arm 40 and essentially acts like a virtual clutch mechanism, eg, limiting mechanical input from affecting mechanical output.

[0051] The desired pose of the manipulator 40 is based on the pose of the handle controller 38a and then passed through the inverse kinematics function executed by the controller 21a. The inverse kinematics function calculates the angles of the joints 44a, 44b, 44c of the manipulator 40 to achieve the scaled and adjusted pose input by the handle controller 38a. The calculated angles are then passed to the manipulator controller 41c, which includes a joint axis controller with a proportional derivative (PD) controller, a friction estimator module, a gravity compensator module, and a double-sided saturation block, which is configured to limit the command torque of the motors of the joints 44a, 44b, 44c.

[0052] refer to Figure 5 , the surgical robotic system 10 is set up around an operating table 90. The system 10 includes movable carts 60a-d, which may be numbered "1" through "4". During set up, each of the carts 60a-d is positioned around the operating table 90. The position and orientation of the carts 60a-d depends on a number of factors, such as the placement of the plurality of access ports 55a-d, which in turn depends on the procedure being performed. Once the port placement is determined, the access ports 55a-d are inserted into the patient's body, and the carts 60a-d are positioned to insert the instruments 50 and laparoscopic camera 51 into the corresponding ports 55a-d. During use, the latch 46c ( Figure 2 ) is attached to the entry port 55 ( Figure 3) and each of the robotic arms 40a-d is attached to one of the access ports 55a-d inserted into the patient's body. The IDU 52 is attached to the holder 46, and then the SIM 43 is attached to the distal portion of the IDU 52. Thereafter, the instrument 50 is attached to the SIM 43. Then, the instrument 50 is calibrated by the robotic arm 40 and inserted through the access port 55 by moving the IDU 52 along the holder 46.

[0053] refer to Figure 6 , a method for assisting port placement can be implemented as software instructions executable by a processor (e.g., controller 21a) and a software application with a graphical user interface (GUI) for planning a surgical procedure to be performed by the robotic system 10. The software receives various 3D images and positional data related to the patient's anatomical structure as input and generates a 3D virtual model of the patient with suggested placements for the camera 51 and instruments 50. The software application also allows a user to manipulate the computer model of the camera 51 and instruments 50 in the model of the patient to view the endoscopic view of the camera 51.

[0054] At step 100, preoperative internal imaging of the patient is obtained, which can be performed using any suitable imaging modality, such as computed tomography (CT), magnetic resonance imaging (MRI), or any other imaging modality capable of obtaining 3D images. The preoperative images are then used to construct an internal tissue and organ model 200, such as Figure 7 The interior 3D model 200 may be constructed using any suitable image processing computer.

[0055] At step 102, an external image 202 of a patient's body is obtained, and a depth map 204 is generated from the image 202, such as Figure 8 As shown. An external visual system 70 ( Figure 5) to obtain the image 202, the external vision system 70 may be a passive stereo camera to provide depth imaging and generate a depth map 204. In an embodiment, the external vision system 70 may be an active stereo infrared (IR) camera with an IR module configured to project an array of invisible IR dots onto the patient. The external vision system 70 is configured to detect the dots and analyze the pattern to create a depth map 204, which can then be used to create an external 3D model 208 of the patient. In further embodiments, the external vision system 70 may be a time-of-flight camera with an IR laser module that paints the scene with a short burst of IR laser light and generates a dense depth map based on the time it takes for each laser shot to be received back at the detector. In further embodiments, the external vision system 70 may be a multi-camera subsystem of an augmented reality headset worn by clinical staff. The external vision system 70 may also be used as a user input system to monitor user gestures, which are then processed to determine desired user inputs, such as to manipulate Figures 11 to 13 GUI 300.

[0056] At step 104, the controller 21a uses a machine learning image processing algorithm to estimate the patient's posture 206, such as Fig. 9 In an embodiment, machine learning may include a convolutional neural network (CNN) and / or a visual transformer (ViT) backbone for feature extraction and a lightweight decoder for pose estimation. The CNN or ViT may be trained on prior data (e.g., synthetic and / or real images of patients in various poses).

[0057] At step 106, the controller 21a predicts the patient's skeleton. The prediction can be used to generate an external 3D model 208 of the patient, such as Fig.10 The external 3D model 208 includes a virtual skeleton model 210 having a plurality of key points 212 corresponding to joints of the patient's anatomical structure. The external 3D model 208, the skeleton model 210, and the key points 212 are based on the external image 202, the depth map 204, and the pose 206. The key points 212 can be generated using various machine learning algorithms trained on data sets that include synthetic and / or real images of patients in various poses.

[0058] In step 108, the external 3D model 208 is further refined into a patient-specific skeleton. The patient-specific key point refinement algorithm can rely on preoperative imaging (CT / MRI) scans. The preoperative imaging data can be segmented at different density levels so that: the first level of processing can generate the patient's low-density soft tissue external anatomical area from the preoperative imaging data; the second level of processing can generate the patient's high-density internal bone joints from the preoperative imaging data. Therefore, the preoperative imaging data (CT / MRI) is used to adjust the position of the key points 212. Then, the 3D model can be fitted around the refined skeleton model 210. The external 3D model 208 can also be based on the patient's body shape input into the software application. The controller 21a can receive user input for modifying the simulated patient shape and modify the simulated patient shape based on the user input. The user input can include a slider or other input to adjust the patient's body size, body position and / or leg / arm position. In addition, the initial state of the slider or other input can be automatically adjusted based on the refined skeleton model 210.

[0059] At step 110, the controller 21a is configured to determine the optimal access port (i.e., access port 55) position 302 based on the procedure being performed (e.g., the organ being operated on). For example, a partial nephrectomy involves a different port placement than a radical prostatectomy, etc. The port position 302 is also determined based on the patient's specific anatomical structure, which in turn is based on the patient's internal 3D model 200 and external 3D model 208. The port position is determined by an optimization algorithm based on an initial fixed port placement guide for a specific procedure, a patient-specific external model and internal model, a suggested instrument and its kinematics, and a robotic arm model. The optimization algorithm can start with an initial static port placement guide as a starting point. The optimization algorithm can include a reinforcement learning algorithm, in which the environment is in the form of a patient's anatomical structure to be operated on, the algorithm generates a set of port positions as actions, and the reward is measured in the form of the best approach to the patient's anatomical structure without collision. The optimization algorithm can present multiple equally optimal port position plans to the user, allowing the user to select one.

[0060] In step 112, if Figures 11 to 13As shown, the optimal port position 302 is displayed in the GUI 300 of the software application. The GUI 300 displays the combination of the external 3D model 208 and the internal 3D model 200, including the port position 302. The external 3D model 208 can be partially transparent to display the internal 3D model 200 including the patient's organs, which can be color-coded for easy identification. The GUI 300 is used to generate, view and modify the port position 302 based on the internal 3D model 200 and the external 3D model 208 of the patient. In addition, the GUI includes 3D geodesic distances between different ports calculated in real time on the patient's anatomical structure. The distances between the ports are displayed to clinical staff to ensure that the ports are placed at least a minimum distance from each other, thereby minimizing the possibility of arm collision. In addition, the GUI can display the distance between each of the port positions to the organ of interest based on the internal 3D model 200.

[0061] The GUI 300 allows the port locations 302 to be moved along the outer surface of the external 3D model 208. The user can select the port location 302 that needs to be modified by simulating the operation of the instrument 50 and / or camera 51 inserted at the port location 302 through the access port 55a-d. In addition, if the clinical staff moves the port so that the distance between any two ports is less than a threshold distance, the GUI can display a warning. In addition, the GUI can display a virtual boundary around the suggested port location so that moving the port within the virtual boundary will still meet all optimal port placement constraints.

[0062] The simulation process is performed at step 114 and Fig.12 and Fig.13 300, where the GUI 300 provides the user with an initial view of the internal 3D model 200. The controller 21a automatically identifies which of the port locations 302 is used by the camera 51 and the instrument 50, and can provide a corresponding icon (e.g., a camera icon) showing the device inserted at that port location 302. The GUI 300 provides the user with the option to select the icon corresponding to the port location 302 and insert a virtual instrument 50 or camera 51, which can be shown as an insertion model 304 and 306 of the instrument 50 and camera 51, respectively. The models 304 and 306 include outer parts 304a and 306a and inner parts 304b and 306b, respectively, which are visually distinguished from each other, such as by transparency, color, etc., to show the parts of the instrument 50 and camera 51 in the patient's body.

[0063] In embodiments where the insertion model 304 illustrates a device having an articulated joint (e.g., an instrument with a robotic wrist), the insertion model 304 may also include joints 304c and an end effector model 304d. To provide additional simulation detail, the controller 21a may also generate cones 305a and 305b to illustrate the degrees of freedom of the insertion model 304 and each joint 304c. The access ports 55a-d restrict movement of a device inserted through the ports around a center of motion that corresponds to the point at which the access port is inserted through the patient's body. Since the port locations 302 correspond to these points, the cone 305a having its vertex at the port locations 302 represents the limits of motion of the insertion model 304. Similarly, the cone 305b represents the limits of motion of the end effector model 304d around the joint 304c.

[0064] refer to Fig.13 , a user can place multiple models 306 representing different insertion trajectories of the camera 51 through a single port location 302. Each of the models 306 includes a preview 308 of the endoscopic view of the camera 51. The preview 308 includes a viewpoint of the internal 3D model 200 from the perspective of the model 306. The user can manipulate the model 306, such as rotating, advancing, retracting, etc., and the preview 308 is updated in real time as the model 306 is manipulated. User input can be through the GUI 300 using a variety of devices (e.g., a touch screen, a pointer, a keyboard, etc.).

[0065] Preview 308 shows the modeled organ, allowing the user to confirm whether the proposed optimal camera position is appropriate. Changes in the patient's anatomical structure and position deform the patient's organ. Therefore, the controller 21a is also configured to generate a deformed internal 3D model 200 based on the patient's position and skeleton model 210. The controller 21a can use a neural network trained in preoperative imaging / modeling of the same organ, which uses key structural landmarks (e.g., blood vessels, arteries, etc.) to learn the changes in the shape of the organ caused by shifting the patient's position and orientation. The user can select a model inserted based on the desired view in the model 306 and discard other models. In an embodiment, the user can shift the port position 302 to a different position and repeat the preview process to confirm the desired port position 302. The preview window 308 can be closed during the movement of the port position 302, and can automatically reappear after the movement stops.

[0066] GUI 300 may also provide playback (ie, animation) of insertion and internal manipulation of model 306 to simulate a surgical procedure. The playback may also include movement of the end effector trajectory to assess the workspace and collisions between the camera and instruments with the selected port location.

[0067] refer to Figure 6At step 116, the user confirms the port positions 302, including the port position 302 of the camera 51, and the controller 21a generates instructions for configuring and positioning the patient on the operating table 90 and configuring and positioning the robotic arm 40 around the patient (eg, Figure 5 The guide may include a plurality of text, image, and / or video instructions that are to be implemented by operating room staff to set up the surgical robotic system 10. The guide may include instructions for inserting the access ports 55a-d at the port locations 302, the instruments 50 to be used, the position and orientation of the movable carts 60a-d relative to the operating table 90, and the like.

[0068] Fig.14 Shown according to the use Figure 6 The setup guide generated by the method positions the patient on the operating table 90 and positions the robotic arm 40 around the patient (eg, Figure 5 Use Figure 6 The installation guide generated by the method can be used Fig.15 The augmented or virtual reality system 400 is implemented or displayed on any of the displays 23, 32, 33 of the system 10. In addition, an external vision system 70 is also used, and it may include a combination of multiple vision-based sensors 70a-d (RGB-D, LiDAR or time-of-flight cameras) mounted on the movable carts 60a-d and / or the robotic arms 40a-d, such as Figure 5 As shown. The data from these vision sensors 70a-d can be processed using a separate computer (not shown) with an embedded GPU installed with the vision sensors 70a-d. The output from the vision sensors 70a-d is transmitted from each movable cart 60a-d to the tower 20. This reduces the need to transmit raw data, thereby reducing the communication bandwidth bottleneck.

[0069] refer to Fig.15 , the virtual reality system 400 includes a head mounted display 402 that the user wears around his head and one or more optional handheld controllers 404. The head mounted display 402 and the handheld controllers 404 can generally enable the user to navigate and / or interact with the virtual robotic surgical environment to place access ports 55a-d and perform other installation steps. The head mounted display 402 and / or the handheld controller 404 can communicate with the controller 21a via a wired or wireless connection.

[0070] In some embodiments, head mounted display 402 and / or handheld controllers 404 may be modified versions of those included in any suitable virtual reality hardware system commercially available for use in applications including virtual and augmented reality environments, such as the Virtual Reality 3D system commercially available from Microsoft Corporation of Redmond, Washington. , META QUEST available from Meta of Menlo Park, California In embodiments where a closed virtual reality set is being used, a forward-facing camera may be provided on the head mounted display 402 to provide a forward-facing view (e.g., an external image of the patient) to be displayed on the head mounted display 402 along with the virtual or augmented projection. In embodiments where the head mounted display 402 is an augmented reality display with a projection window, only the projection is displayed on the screen.

[0071] The head mounted display 402 and / or the handheld controller 404 can be modified to enable a user to interact with the virtual robotic surgical environment (e.g., the handheld controller 404 can be modified to operate the surgeon console 30). In some embodiments, the virtual reality system 400 can further include one or more tracking transmitters as part of the external visual system 70 that emit infrared light into the workspace. The tracking transmitters can be mounted, for example, on a wall, ceiling, fixture, or other suitable mounting surface. Sensors, markers, or reflectors can be coupled to the outward-facing surface of the head mounted display 402 and / or the handheld controller 404 for detecting the emitted infrared light or reflecting light back to the external visual system 70. Based on the position of any sensor that detects the emitted light and when such a sensor detects the emitted light after emitting the light, the controller 21a can be configured to determine (e.g., by triangulation) the position and orientation of the head mounted display 402 and / or the handheld controller 404 within the workspace. In other embodiments, other suitable means (eg, other sensor technologies such as accelerometers or gyroscopes, other sensor arrangements, etc.) may be used to determine the position and orientation of head mounted display 402 and handheld controller 404 .

[0072] In some embodiments, the head mounted display 402 may include a strap (e.g., with buckles, elastic, snaps, etc.) that facilitates mounting the display 402 to the user's head. For example, the head mounted display 402 may be configured like goggles, a headband or headset, a cap, etc. The head mounted display 402 may include two eyepiece assemblies that provide a stereoscopic immersive display, although may alternatively include any suitable display.

[0073] The handheld controller 404 may include interactive features that the user can manipulate to interact with the virtual robotic surgery environment. For example, the handheld controller 404 may include one or more buttons, triggers, touch-sensitive features, scroll wheels, switches, and / or other suitable interactive features. In addition, the handheld controller 404 may have any of a variety of form factors, such as a stick, a gripper, a generally circular shape (e.g., a sphere or egg shape), etc. In some embodiments, the handheld controller may include a portable device (e.g., a stick, a remote device, etc.) and / or a garment worn on the user's hand (e.g., a glove, a ring, a wristband, etc.), and include sensors and / or be configured to cooperate with external sensors to provide tracking of the user's hand, a single finger, a wrist, etc. Other suitable controllers (e.g., a cuff configured to provide tracking of the user's arm) may be used in addition or alternatively.

[0074] Port placement guidance is provided in the form of a visual overlay superimposed on the real-time patient anatomy. The visualization system can be used with a variety of hardware installations, which can be used alone or in combination. In one configuration, any one of the displays 23, 32, 34 of the system 10 is used as a visualization device. In another configuration, a head-mounted display 402 is used for visualization. In yet another configuration, the display is used with the head-mounted display 402. The video stream from the external visual system 70 is displayed on the displays 23, 32, 34 and / or the head-mounted display 402 of the system 10 together with the guidance overlay. The combined video output (i.e., stream and overlay) provides the surgeon with enhanced guidance for port placement. The video stream from the head-mounted display 402 is also simultaneously displayed on the displays 23, 32, 34 of the system 10 to assist staff in obtaining situational awareness and for training purposes.

[0075] Port placement guidance is provided during initial setup of the system 10 and includes guidance for each of the movable carts 60a-d to be setup at a specific distance and angle relative to the operating table 90 according to the procedural guidance. The system 10 also provides visual feedback to the setup staff in the form of a visual overlay and guidance to move the movable carts 60a-d and their robotic arms 40a-d, including the setup arm 61.

[0076] The port placement guidance uses an external vision system 70 to automatically detect exposed human anatomical structures. The controller 21a then segments the surface of the human anatomical structure, estimates the human posture and predicts the internal skeleton position, as described above. The patient-specific preoperative CT model is then registered and visualized on the external human anatomical structure. Finally, a visual overlay is displayed to the surgeon on the display 23, 32, 34 and / or head-mounted display 402 of the system 10 to make an incision for port placement. The guidance takes into account the number of robotic arms 40a-d, their placement, angles, and the probability of collision between the arms based on the patient-specific preoperative model during the surgical procedure. After the laparoscopic access port 55a has been placed, during the "initial observation" phase, sub-tissue tumors and key structures are also superimposed on the image from the camera 51 to help the surgeon visualize the internal anatomical structure and change the port planning when necessary.

[0077] refer to Fig.14 After the patient is brought into the operating room and transferred and positioned on the operating table 90, the patient's anatomy is marked by placing fiducial markers at desired locations on the patient at step 502. The external vision system 70 uses the fiducial markers to detect the patient and determine the posture. Instead of fiducial markers, the external vision system 70 can identify specific features of the patient (e.g., joints).

[0078] At step 504, the registration device (e.g., external vision system 70) is configured to confirm that all of its components are communicating with each other and are aligned for registering the patient and system 10. At step 506, the external vision system 70 is used to scan and / or register the patient and estimate or predict the patient's posture. The patient's posture can be determined using machine learning, which can include using a CNN and / or ViT backbone to extract features of the posture and using a lightweight decoder for posture estimation. The CNN or ViT can be trained on prior data (e.g., synthetic and / or real images of the patient in various postures). The posture can be used to generate an external 3D model 208 of the patient, such as Fig.10 The external 3D model 208 includes a virtual skeleton model 210 having a plurality of key points 212 corresponding to the joints of the patient's anatomical structure. The external 3D model 208, the skeleton model 210, and the key points 212 may also be generated using various machine learning algorithms trained on data sets including images of the patient in various postures and / or real images.

[0079] At step 508, an internal organ (such as the liver) is visualized by projecting a 3D image of the liver 550 onto the display 23, 32, 34 and / or the head mounted display 402 of the system 10, such as Fig.16In step 510, the patient's abdomen and posture are again registered and / or scanned. In step 512, the patient's abdomen and posture are also projected and scanned. Fig.11 and Fig.12 The port position 302 corresponds to the position 552 to visualize the entry port planning of the installation guide.

[0080] At step 514, the patient is insufflated, and the first access port 50a of the camera 51 is inserted at the projected position, followed by the camera 51 being inserted through the access port 50a. Then at step 516, the camera 51 is used in an 'initial observation' phase, where the camera 51 images the surgical site, and the video feed is used to generate a dense depth map of the surgical site. The camera 51 can be a stereo endoscope, and the controller 21a is configured to generate a depth map using a machine learning algorithm based on the stereo video feed. The controller 21a can track the endoscope position via robotic arm kinematics and visual simultaneous localization and mapping (visual SLAM) to generate a real-time 3D pose of the endoscope. In addition, the controller 21a can generate an extended suture intraoperative 3D depth map of the surgical site from multiple laparoscopic camera poses using a combination of robotic arm kinematics, visual SLAM, point cloud generation, and sequential point cloud registration.

[0081] In step 518, the patient's organs and surgical sites are registered and / or scanned using the camera 51. The controller 21a can initialize the registration of the patient's preoperative internal 3D model with the depth map via point-to-point semi-automatic registration or via fully automatic registration. For semi-automatic registration, the user can mark multiple points on the preoperative 3D model and corresponding points on the 3D surface generated from the camera 51. The controller 21a can use a machine learning model in the form of a neural network to register the preoperative 3D model and the 3D surface generated from the depth map of the stereoscopic image from the camera 51. This neural network can be trained to take into account the movement of organs caused by abdominal insufflation. The controller 21a can also use real-time robotic arm kinematics data and the visual SLAM to update the registration between the patient's preoperative internal 3D model and the intraoperative 3D depth map via fully automatic registration. In step 520, the controller 21a superimposes the patient's internal 3D model 200 on the registered patient's organs. At step 522, the internal organ is visualized by projecting a 3D image of the organ 554 onto the display 23, 32, 34 and / or the head mounted display 402 of the system 10, such as Fig.17 shown.

[0082] At step 524, the final port position 552 is confirmed, and the remaining one or more access ports 55b-d can be inserted through the marked port position 552 under direct vision via the camera 51 to avoid puncturing any internal anatomical organs by the cannula needle. The process of generating a real-time sequentially registered intraoperative textured 3D model of the surgical site during the initial observation phase creates a non-occupied volume showing the distance between the abdominal wall and the internal organs. The controller 21a can predict the trajectory of the cannula needle inserted through the additional port, and the GUI can display an augmented reality overlay of the cannula needle inserted through the port position. If the projected trajectory or endpoint of any of the cannula needles inserted through any of the additional ports may collide with internal organs, the GUI can further display warnings and alarms, as shown in the non-occupied volume. The GUI can also allow clinical staff to use the 3D surgical site model to move the virtual viewpoint of the stereoscopic endoscope camera to more closely observe the projected cannula needle trajectory through any of the additional ports.

[0083] After the final port position 552 is confirmed, instructions are also provided for each of the movable carts 60a-d so that the carts 60a-d are positioned at a specific distance and angle relative to the operating table 90, such as Figure 5 The system 10 also provides visual feedback to the installation personnel in the form of a visual overlay and instructions to move the movable carts 60a-d and their robotic arms 40a-d, including the installation arm 61.

[0084] It should be understood that various modifications may be made to the embodiments disclosed herein. Therefore, the above description should not be interpreted as limiting, but merely as an illustration of various embodiments. Those skilled in the art will envision other modifications within the scope and spirit of the appended claims.

Claims

1. A surgical robot system, comprising: a robotic arm holding a laparoscopic camera inserted through the access port; A controller configured to: generating a port location for the access port on the 3D model of the patient; and generating a patient-specific setup guide for configuring the access port and the robotic arm; an external camera configured to register the robotic arm and the patient; as well as A display is configured to output the port position of the access port as an overlay over an external image of the patient based on registration of the robotic arm and the patient.

2. The surgical robot system according to claim 1, wherein: The display is at least one of a monitor or a head mounted display.

3. The surgical robot system according to claim 1, wherein: The external camera is further configured to capture a plurality of external images of the patient and images of the robotic arm.

4. The surgical robot system according to claim 3, wherein: The laparoscopic camera is configured to capture internal images of a surgical site.

5. The surgical robot system according to claim 4, wherein: The controller is further configured to generate a depth map of the surgical site based on the internal image of the surgical site.

6. The surgical robot system according to claim 5, wherein: The controller is further configured to generate a registration of the patient's preoperative imaging data with the depth map.

7. The surgical robot system according to claim 6, wherein: The controller is further configured to track the position of the laparoscopic camera via the kinematics of the robotic arm and visual simultaneous localization and mapping (visual SLAM).

8. The surgical robot system according to claim 7, wherein: The controller is further configured to update the registration between the pre-operative imaging data and the depth map via fully automatic registration using real-time robotic arm kinematics data and the visual SLAM.

9. A method for assisting access port placement, the method comprising: capturing images at a laparoscopic camera held by a robotic arm and inserted through an access port; generating, at a controller, a port location for the access port on the 3D model of the patient; generating, at the controller, patient-specific setup instructions for configuring the access port and the robotic arm; generating, at the controller, a registration of the robotic arm and the patient based on an external image received from an external camera; as well as The port position of the access port is output on a display as an overlay over an external image of the patient based on registration of the robotic arm and the patient.

10. The method according to claim 9, wherein: The port location of the access port is output as the overlay on at least one of a monitor or a head mounted display.

11. The method of claim 9, further comprising capturing, by the external camera, a plurality of external images of the patient and the image of the robotic arm.

12. The method of claim 11, further comprising capturing internal images of the surgical site with a laparoscopic camera inserted through the access port.

13. The method of claim 12, further comprising generating, at the controller, a depth map of the surgical site based on the internal image of the surgical site.

14. The method of claim 13, further comprising generating, at the controller, a registration of the patient's preoperative imaging data with the depth map.

15. The method of claim 14, further comprising tracking the position of the laparoscopic camera via kinematics of the robotic arm and visual simultaneous localization and mapping (visual SLAM). 16 . The method of claim 15 , further comprising tracking the registration between the pre-operative imaging data and the depth map via fully automatic registration using real-time robotic arm kinematics data and the visual SLAM.

17. A method for determining access port placement, the method comprising: receiving a plurality of external images of a patient captured by an external vision system; generating an external 3D model of the patient based on the plurality of external images; generating an internal 3D model of the patient based on preoperative imaging data; generating a combined 3D model based on the external 3D model and the internal 3D model; determining a port location of at least one access port based on the combined 3D model; generating a setup guide for configuring at least one access port and a robotic arm; The port position of the at least one access port and the placement guide are output as an overlay on an external image of the patient on a display based on registration of the robotic arm and the patient.

18. The method of claim 17, further comprising generating the external 3D model of the patient based on the depth map of the patient.

19. The method of claim 17, further comprising generating a skeletal model including a plurality of key points of the patient based on the plurality of external images.

20. The method of claim 19, further comprising generating the external 3D model based on the skeleton model.

Images