Pre-docking pose optimization

Abstract

A robotic medical system may recommend a pose of a robotic arm based on patient and medical procedure information, and communicate the recommended pose to a user of the robotic medical system. A robotic medical system may include a robotic arm and a processor. The robotic medical system may be configured to obtain an image of a patient, simulate a medical procedure to identify collisions, and select a pose based on a number of collisions. The robotic medical system may provide information indicative of the recommended pose.

Description

Technical Field

[0001] The systems and methods disclosed herein relate to robotic medical systems, and more particularly to a robotic control arm of a robotic medical system configured for a medical procedure. Background Art

[0002] The robotically enabled medical system is capable of performing a variety of medical procedures, including both minimally invasive procedures such as laparoscopy and non-invasive procedures such as endoscopy (eg, bronchoscopy, ureteroscopy, gastroscopy, etc.).

[0003] Such a robotic medical system may include a robotic arm configured to control the movement of a medical tool during a given medical procedure. To achieve the desired positioning of the medical tool, the robotic arm holding the medical tool may be placed in a specific pose during setup or during teleoperation of the robotic medical system. Summary of the invention

[0004] Surgical procedures can benefit from precise port placement and often have a small margin for error. For example, for a larger patient, port placement may need to be within about 1.5 centimeters from the optimal position, while for a smaller patient, port placement may need to be within about 1 centimeter from the optimal position. Achieving such precise port placement that meets small error tolerances can be challenging.

[0005] Furthermore, different surgical procedures may involve different cannula and / or port locations. While conventional port planning typically involves the physician's inherent knowledge of the type of procedure to be performed, such planning may not take into account the entire patient area and possible collisions between the arms and external objects that may occur based on the port settings.

[0006] Robotic medical systems including multiple robotic arms (e.g., four or more robotic arms) can provide surgeons with greater flexibility in performing medical procedures. They can also facilitate the development of novel surgical procedures and methods. However, such complex systems require careful positioning of the robotic arms during setup to avoid intraoperative collisions and still be able to reach the target anatomy during the surgical procedure. In particular, there is a need for a robotic medical system that can identify the precise location of port placement and the corresponding optimal starting position of the robotic arms.

[0007] As disclosed herein, a robotic medical system may be configured to select an initial robotic arm pose and corresponding port locations for a given patient and procedure. In some embodiments, the selected initial robotic arm pose and corresponding port locations are recommended or optimized initial robotic arm pose and corresponding port locations. Some embodiments disclosed herein combine recommended port placement data with information about (i) patient presentation, such as height, weight, description of abdomen (chest) size and shape, and surgical history (e.g., indicating internal organ changes or scars); and (ii) surgical procedure to generate a pre-docking pose for the robotic arm.

[0008] Some embodiments disclosed herein use a three-dimensional (3-D) scanner (e.g., a scanning unit) communicatively connected to a robotic medical system to determine a recommended initial robot arm pose (e.g., a pre-docking pose) and a corresponding port position. The 3-D scanner may be included in a robotic medical system (e.g., as a component of a robotic medical system, forming a part of a robotic medical system, etc.). In some configurations, the 3-D scanner may be a component of a robotic medical system located on a robotic arm, a patient support platform (e.g., a bed), or a pendant of a robotic medical system, as an integrated component or accessory. In some other configurations, the 3-D scanner may be a part of an instrument (e.g., a medical tool such as a laparoscope) held by a robotic arm of a robotic medical system. Alternatively, the 3-D scanner may be a separate accessory (e.g., a stand-alone scanner, a handheld scanner, etc.) communicatively connected to a robotic medical system, and attached to (e.g., mounted on) a robotic arm, a patient support platform (e.g., a bed), a pendant of a robotic medical system, or on a wall and / or ceiling of an operating room where the robotic medical system is located. The 3-D scanner can provide information such as patient views, patient characteristics, and / or locations of installed accessories, which can be used by the robotic medical system (e.g., in combination with other information such as the target procedure and / or anatomy) to determine a recommended initial robotic arm pose (e.g., a pre-docked pose) and corresponding port locations specifically tailored for the patient and / or the medical procedure to be performed.

[0009] As disclosed herein, a 3-D scanner can be used to position a patient relative to a robotic medical system. While the robotic medical system can determine the position and / or orientation of a patient support platform and / or a robotic arm of the robotic medical system (e.g., using sensors and / or encoders attached thereto), it can be challenging for the robotic medical system to register other objects (such as the patient and installed accessories, which may include, but are not limited to, liver retractors, leg stirrups, arm boards, and anesthesia accessories, etc.). Therefore, positioning a patient relative to a robotic medical system using a 3-D scanner can provide the robotic medical system with information related to the position of the patient and / or installed accessories.

[0010] As disclosed herein, data from a 3-D scanner can be combined with other information / inputs (such as information about a target procedure and / or target anatomy) to generate a recommended initial robotic arm pose and corresponding port location. For example, the scan information (e.g., imaging information, 3-D imaging information, etc.) can include real-time patient information, such as patient size, patient characteristics (e.g., whether the patient is an amputee), and associated objects (e.g., if there is a feeding tube extending from the patient, this can delineate a "keep-out zone"). The robotic medical system can partition the patient into different parts (e.g., head, torso, legs), and using the marked locations (e.g., the patient's abdomen) and / or the relevant organ locations for a particular surgical procedure, the robotic medical system (e.g., via an algorithm) can recommend port placement and starting poses that will avoid or minimize collisions, such as between the patient and the robotic arm, between the robotic arm and the robotic arm, and / or between instruments, while maximizing the available workspace during the procedure.

[0011] Thus, in one aspect, a robotic medical system includes a plurality of robotic arms; one or more processors; and a memory storing instructions that, when executed by the one or more processors, cause the one or more processors to: (i) obtain imaging data of a patient; (ii) obtain information about a medical procedure for the patient; (iii) for each of a plurality of starting states, simulate the medical procedure from the starting state to identify a number of collisions with an arm of a plurality of robotic arms, the simulation using the imaging data and the information about the medical procedure; (iv) select a first starting state of the plurality of starting states that has the least number of collisions, wherein the first starting state identifies a corresponding starting pose of each arm of the plurality of robotic arms; and (v) provide pose information of the first starting state to an operator of the robotic medical system.

[0012] In another aspect, a method for determining a starting state of a robotic system comprising a plurality of robotic arms comprises: (i) obtaining imaging data of a patient; (ii) obtaining information about a medical procedure for the patient; (iii) for each starting state in a plurality of starting states, simulating the medical procedure from the starting state to identify a number of collisions with arms in a plurality of robotic arms, the simulation using the imaging data and the information about the medical procedure; (iv) selecting a first starting state in the plurality of starting states having a minimum number of collisions, wherein the first starting state identifies a corresponding starting pose of each arm in the plurality of robotic arms; and (v) providing pose information of the first starting state to an operator of the robotic medical system.

[0013] The systems, methods and devices of the disclosure each have several innovative aspects, no single one of which is solely responsible for the desirable attributes disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The disclosed aspects will hereinafter be described in conjunction with the accompanying drawings, which are provided to illustrate and not to limit the disclosed aspects, wherein like reference numerals represent like elements.

[0015] Figure 1 Embodiments of a cart-based robotic system arranged for use in diagnostic and / or therapeutic bronchoscopy procedures are illustrated in accordance with some embodiments.

[0016] Figure 2 Depicted according to some embodiments Figure 1 Other aspects of the robotic system.

[0017] Figure 3 Illustrated is a device arranged for ureteroscopy according to some embodiments. Figure 1 Implementation scheme of a robotic system.

[0018] Figure 4 Illustrated is a schematic diagram of a device arranged for use in a vascular procedure according to some embodiments. Figure 1 Implementation scheme of a robotic system.

[0019] Figure 5 An embodiment of a table-based robotic system arranged for use in a bronchoscopy procedure is illustrated in accordance with some embodiments.

[0020] Figure 6 An example system configured to stow a robotic arm is illustrated in accordance with some embodiments.

[0021] Figure 7 An alternative implementation of a table-based robotic system according to some embodiments is illustrated.

[0022] Figure 8 Illustrated according to some embodiments Figure 7 End view of a stage-based robotic system.

[0023] Fig. 9 An end view of a table-based robotic system with a robotic arm attached thereto is illustrated according to some embodiments.

[0024] Fig.10 An example instrument driver is illustrated according to some embodiments.

[0025] Fig.11 An example medical device having a pair of instrument drivers is illustrated in accordance with some embodiments.

[0026] Fig.12 Alternative designs of instrument drivers and instruments are illustrated according to some embodiments, wherein the axis of the drive unit is parallel to the axis of the elongated shaft of the instrument.

[0027] Fig.13 An instrument having an instrument-based insertion architecture according to some embodiments is illustrated.

[0028] Fig.14 An example controller is illustrated according to some embodiments.

[0029] Fig.15 An example robotic system is illustrated in accordance with some embodiments.

[0030] Fig.16 Another view of an example robotic system is illustrated in accordance with some embodiments.

[0031] FIG. 17A to FIG. 17C Different views of an example robotic arm are illustrated, according to some embodiments.

[0032] 18A to 18C Example cannula placements for different surgical procedures are illustrated in accordance with some embodiments.

[0033] Fig.19 An example workflow for generating pose and port information according to some embodiments is illustrated.

[0034] FIG. 20A to FIG. 20B An example 3-D scanning and registration process for a robotic medical system, according to some embodiments, is illustrated.

[0035] FIG. 21A to FIG. 21B An example process for identifying a target anatomical location is illustrated in accordance with some embodiments.

[0036] Fig.22A An example set of port locations for medical procedures is illustrated in accordance with some embodiments.

[0037] FIG. 22B to FIG. 22D Illustrated according to some embodiments Fig.22A Example arm poses for a collection of port positions.

[0038] Fig.22E Another example set of port locations for medical procedures is illustrated in accordance with some embodiments.

[0039] FIG. 22F to FIG. 22H Illustrated according to some embodiments Fig.22E Example arm poses for a collection of port positions.

[0040] Fig.23A and Fig. 23B is a flow chart illustrating an example method for selecting a robot arm pose, according to some embodiments.

[0041] Fig.24 is a schematic diagram of electronic components of an example medical system, according to some embodiments. DETAILED DESCRIPTION

[0042] 1. Overview .

[0043] Aspects of the present disclosure may be integrated into a robotically enabled medical system that is capable of performing a variety of medical procedures, including both minimally invasive procedures (such as laparoscopy) and non-invasive procedures (such as endoscopy). Among endoscopy procedures, the system may be able to perform bronchoscopy, ureteroscopy, gastroscopy, etc.

[0044] In addition to performing a wide range of procedures, the system can provide additional benefits such as enhanced imaging and guidance to assist physicians. Additionally, the system can provide physicians with the ability to perform procedures with improved ease of use, such that one or more of the system's instruments can be controlled by a single user.

[0045] For illustrative purposes, various embodiments will be described below in conjunction with the accompanying drawings. It should be understood that many other implementations of the disclosed concepts are possible and that various advantages can be achieved using the disclosed implementations.

[0046] A. Robotic System-Cart .

[0047] Robot-enabled medical systems can be configured in a variety of ways, depending on the specific procedure. Figure 1 An embodiment of a cart-based, robotically enabled system 10 arranged for use in a diagnostic and / or therapeutic bronchoscopy procedure is illustrated. During a bronchoscopy procedure, the system 10 may include a cart 11 having one or more robotic arms 12 to deliver a medical instrument such as a steerable endoscope 13 (which may be a procedure-specific bronchoscope for bronchoscopy) to a natural orifice entry point (i.e., the oral cavity of a patient positioned on a table in this example) to deliver the diagnostic and / or therapeutic tool. As shown, the cart 11 may be positioned proximate to the patient's upper torso to provide access to the entry point. Similarly, the robotic arms 12 may be actuated to position the bronchoscope relative to the access point. The robotic arms 12 may also be utilized when performing a gastrointestinal (GI) procedure with a gastroscope. Figure 1 The arrangement in . Figure 2 An example embodiment of a cart is depicted in greater detail.

[0048] Continue to refer Figure 1, once the cart 11 is properly positioned, the robotic arm 12 can insert the steerable endoscope 13 into the patient robotically, manually, or a combination thereof. As shown, the steerable endoscope 13 may include at least two telescopic portions, such as an inner guide portion and an outer sheath portion, each portion coupled to a separate instrument driver from a collection of instrument drivers 28, each instrument driver coupled to a distal end of a separate robotic arm. This linear arrangement of instrument drivers 28 that facilitates coaxial alignment of the guide portion with the sheath portion creates a "virtual track" 29 that can be repositioned in space by manipulating one or more robotic arms 12 to different angles and / or positions. The virtual track described herein is illustrated in the accompanying drawings (such as in Figure 3 and Figure 4 ) are depicted using dashed lines, and thus the dashed lines do not depict any physical structure of the system. Translation of the instrument driver 28 along the virtual track 29 causes the inner guide portion to telescope relative to the outer sheath portion, or to advance or retract the endoscope 13 from the patient. The angle of the virtual track 29 can be adjusted, translated, and pivoted based on clinical application or physician preference. For example, in bronchoscopy, the angle and position of the virtual track 29 as shown represents a compromise between providing the physician with access to the endoscope 13 while minimizing the friction caused by bending the endoscope 13 into the patient's mouth.

[0049] After insertion, the endoscope 13 can be guided down the patient's trachea and lungs using precise commands from the robotic system until the target destination or surgical site is reached. To enhance navigation through the patient's pulmonary network and / or to reach the desired target, the endoscope 13 can be manipulated to telescopically extend the inner guide portion from the outer sheath portion to obtain enhanced articulation and a larger bending radius. The use of a separate instrument driver 28 also allows the guide portion and the sheath portion to be driven independently of each other.

[0050] For example, the endoscope 13 can be guided to deliver a biopsy needle to a target, such as, for example, a lesion or nodule in a patient's lung. The needle can be deployed downward along a working channel that extends the length of the endoscope to obtain a tissue sample to be analyzed by a pathologist. Depending on the pathological results, additional tools can be deployed downward along the working channel of the endoscope for additional biopsies. After identifying that the nodule is malignant, the endoscope 13 can deliver tools through the endoscope to remove the potential cancerous tissue. In some cases, diagnostic and therapeutic treatments can be delivered in separate procedures. In other cases, diagnostic and therapeutic treatments can be delivered during the same procedure.

[0051] The system 10 may also include a movable tower 30 that may be connected to the cart 11 via a support cable to provide control, electronic, fluid, optical, sensor, and / or electrical support to the cart 11 .

[0052] To support the robotic system described above, the tower 30 may include components of a computer-based control system that stores computer program instructions, for example, in a non-transitory computer-readable storage medium such as a permanent magnetic storage drive, a solid-state drive, or the like. Whether the execution occurs in the tower 30 or in the cart 11, the execution of these instructions may control the entire system or a subsystem thereof. For example, when executed by a processor of a computer system, the instructions may cause components of the robotic system to actuate associated brackets and arm mounts, actuate a robotic arm, and control a medical device. For example, in response to receiving a control signal, a motor in a joint of a robotic arm may position the arm into a particular posture.

[0053] The tower 30 may also include pumps, flow meters, valve controllers, and / or fluid pathways to provide controlled irrigation and aspiration capabilities to the system that may be deployed through the endoscope 13. These components may also be controlled using the computer system of the tower 30. In some embodiments, the irrigation and aspiration capabilities may be delivered directly to the endoscope 13 via separate cables.

[0054] The tower 30 may also include support equipment for sensors deployed throughout the robotic system 10. For example, the tower 30 may include optoelectronic equipment for detecting, receiving, and processing data received from optical sensors or cameras throughout the robotic system 10. In conjunction with the control system, such optoelectronic equipment may be used to generate real-time images for display in any number of consoles deployed throughout the system (including display in the tower 30). Similarly, the tower 30 may also include electronic subsystems for receiving and processing signals received from deployed electromagnetic (EM) sensors. The tower 30 may also be used to house and position EM field generators for detection by EM sensors in or on medical devices.

[0055] The tower 30 may also include a console 31 in addition to other consoles available in the rest of the system (e.g., a console mounted on top of a cart). The console 31 may include a user interface and display screen, such as a touch screen, for a physician operator. The console in the system 10 is typically designed to provide both preoperative information and real-time information for robotic control and procedures, such as navigation and positioning information for the endoscope 13. When the console 31 is not the only console available to the physician, it may be used by a second operator (such as a nurse) to monitor the patient's health or vital signs and the operation of the system 10, as well as to provide procedure-specific data, such as navigation and positioning information. In other embodiments, the console 31 is housed in a body separate from the tower 30.

[0056] Figure 2 Provided by Figure 1Detailed illustration of an embodiment of a cart 11 of a cart-based robotic-enabled system is shown. The cart 11 generally includes an elongated support structure 14 (commonly referred to as a "column"), a cart base 15, and a console 16 at the top of the column 14. The column 14 may include one or more brackets, such as for supporting one or more robotic arms 12 ( Figure 2 The cradle 17 (alternatively "arm support") may include a separately configurable arm mount that rotates along a vertical axis to adjust the base of the robotic arm 12 to better position it relative to the patient. The cradle 17 also includes a cradle interface 19 that allows the cradle 17 to translate vertically along the column 14.

[0057] The bracket interface 19 is connected to the column 14 through slots, such as slot 20, which are positioned on opposite sides of the column 14 to guide the vertical translation of the bracket 17. The slot 20 includes a vertical translation interface to position and maintain the bracket 17 at various vertical heights relative to the cart base 15. The vertical translation of the bracket 17 allows the cart 11 to adjust the reach of the robotic arm 12 to meet a variety of table heights, patient sizes, and physician preferences. Similarly, the individually configurable arm mounts on the bracket 17 allow the robotic arm base 21 of the robotic arm 12 to be angled in a variety of configurations.

[0058] Column 14 may internally include mechanisms such as gears and motors designed to mechanized translate carriage 17 using a vertically aligned lead screw in response to control signals generated in response to user input (eg, input from console 16 ).

[0059] The robotic arm 12 may generally include a robotic arm base 21 and an end effector 22 separated by a series of linkages 23, the series of linkages being connected by a series of joints 24, each joint including an independent actuator, each actuator including an independently controllable motor. Each independently controllable joint represents an independent degree of freedom available to the robotic arm 12. Each of the robotic arms 12 may have seven joints and thus provide seven degrees of freedom. Multiple joints result in multiple degrees of freedom, thereby allowing "redundant" degrees of freedom. The redundant degrees of freedom allow the robotic arm 12 to position its corresponding end effector 22 at a specific position, orientation, and trajectory in space using different linkage positions and joint angles. This allows the system to position and guide the medical device from a desired point in space, while allowing the physician to move the arm joint to a clinically advantageous position away from the patient to create greater access while avoiding arm collisions.

[0060] The cart base 15 balances the weight of the column 14, bracket 17, and robotic arm 12 on the floor. Thus, the cart base 15 houses heavier components such as electronics, motors, power supplies, and components that enable the cart 11 to be moved and / or secured. For example, the cart base 15 includes rollable wheel-shaped casters 25 that allow the cart 11 to be easily moved around the room prior to a procedure. After reaching the appropriate position, the casters 25 can be secured using wheel locks to keep the cart 11 in place during the procedure.

[0061] The console 16 positioned at the vertical end of the column 14 allows both a user interface and a display screen (or dual-purpose device, such as, for example, a touch screen 26) for receiving user input to provide both preoperative and intraoperative data to the physician user. Potential preoperative data on the touch screen 26 may include preoperative planning, navigation and mapping data derived from a preoperative computerized tomography (CT) scan, and / or records from a preoperative patient interview. Intraoperative data on the display may include optical information provided from tools, sensors, and coordinate information from sensors, as well as important patient statistics, such as respiration, heart rate, and / or pulse.

[0062] Figure 3 An embodiment of a robotically enabled system 10 arranged for ureteroscopy is illustrated. In a ureteroscopy procedure, a cart 11 can be positioned to deliver a ureteroscope 32 (a protocol-specific endoscope designed to traverse a patient's urethra and ureters) to the patient's lower abdominal region. As shown, the cart 11 can be aligned at the foot of the table to allow the robotic arm 12 to position the ureteroscope 32 for direct linear access to the patient's urethra. The robotic arm 12 can insert the ureteroscope 32 directly through the urethra into the patient's lower abdomen along a virtual track 33 from the foot of the table.

[0063] After insertion into the urethra, the ureteroscope 32 can be navigated into the bladder, ureters, and / or kidneys for diagnostic and / or therapeutic applications using similar control techniques as in bronchoscopy. For example, the ureteroscope 32 can be guided into the ureters and kidneys to break up accumulated kidney stones using a laser or ultrasonic lithotripsy device deployed down the working channel of the ureteroscope 32. After lithotripsy is complete, the resulting stone fragments can be removed using a basket deployed down the ureteroscope 32.

[0064] Figure 4An embodiment of a robotically enabled system 10 similarly arranged for a vascular procedure is illustrated. In a vascular procedure, the system 10 may be configured so that the cart 11 can deliver a medical device 34, such as a steerable catheter, to an access point in the femoral artery in the patient's leg. As in a ureteroscopy procedure, the cart 11 may be positioned toward the patient's leg and lower abdomen to allow the robotic arm 12 to provide a virtual track 35 for direct linear access to the femoral artery access point in the patient's thigh / hip region. After insertion into the artery, the medical device 34 may be guided and inserted by translating the instrument drive 28. Alternatively, the cart may be positioned around the patient's upper abdomen to reach alternative vascular access points, such as the carotid and brachial arteries near the shoulder and wrist.

[0065] B. Robotic system - unit .

[0066] Embodiments of the robotic-enabled medical system may also incorporate a patient table. Incorporating a table reduces the amount of capital equipment within the operating room by removing a cart, which allows for greater access to the patient. Figure 5 An embodiment of such a robotically enabled system is illustrated arranged for use in a bronchoscopy procedure. System 36 includes a support structure or column 37 for supporting a platform 38 (shown as a "table" or "bed") on a floor. Much like a cart-based system, the end effector of a robotic arm 39 of system 36 includes an instrument drive 42 designed to manipulate an elongated medical instrument, such as a slender medical instrument, through or along a virtual track 41 formed by the linear alignment of the instrument drive 42. Figure 5 In practice, a C-arm for providing fluoroscopic imaging can be positioned over the upper abdominal region of the patient by placing the emitter and detector around the table 38.

[0067] As shown, the column 37 may include one or more brackets 43, shown as rings in the system 36, on which one or more robotic arms 39 may be based. The brackets 43 may translate along a vertical column interface 44 extending along the length of the column 37 to provide different vantage points from which the robotic arms 39 may be positioned to reach the patient. The brackets 43 may rotate about the column 37 using a mechanical motor positioned within the column 37 to allow the robotic arms 39 to access multiple sides of the table 38, such as, for example, both sides of the patient.

[0068] The system 36 may also include a tower (not shown) that divides the functionality of the system 36 between the table and the tower to reduce the form factor and volume of the table. As in previously disclosed embodiments, the tower may provide a variety of support functions to the table, such as processing, computing and control capabilities, power, fluid and / or optical and sensor processing. The tower may also be movable to be positioned away from the patient, thereby improving the physician's access and eliminating clutter in the operating room. The tower may also include a main controller or console that provides a user interface such as a keyboard and / or suspension for user input, and a display screen (or touch screen) for preoperative and intraoperative information such as real-time imaging, navigation and tracking information. In some embodiments, the tower may also include a holder for a gas tank to be used for insufflation.

[0069] In some embodiments, the table base folds and stores the robotic arm when not in use. Figure 6 A system 47 for stowing a robotic arm in an embodiment of a table-based system is illustrated. In the system 47, a carriage 48 can be translated vertically into a base 49 to stow a robotic arm 50, an arm mount 51, and the carriage 48 within the base 49. A base cover 52 can be translated and retracted open to deploy the carriage 48, the arm mount 51, and the arm 50 around a column 53, and closed to stow the carriage, the arm mount, and the arm to protect them when not in use.

[0070] Figure 7 and Figure 8 An isometric view and an end view of an embodiment of a table-based surgical robotic system 100 are illustrated. The surgical robotic system 100 includes a table 101 that can be configured to support one or more robotic arms (see, e.g., Fig. 9 ) is provided with one or more adjustable arm supports 105 of the table 101. In the illustrated embodiment, a single adjustable arm support 105 is shown, but additional arm supports may be provided on opposite sides of the table 101. The adjustable arm support 105 may be configured so that it can be moved relative to the table 101 to adjust and / or change the orientation of the adjustable arm support 105 and / or any robotic arm mounted thereto relative to the table 101. For example, the adjustable arm support 105 may be adjusted in one or more degrees of freedom relative to the table 101. The adjustable arm support 105 provides high flexibility to the system 100, including the ability to easily stow the one or more adjustable arm supports 105 and any robotic arm attached thereto below the table 101. The adjustable arm support 105 may be raised from a stowed orientation to a position below the upper surface of the table 101. In other embodiments, the adjustable arm support 105 can be raised from the stowed position to a position above the upper surface of the table 101 .

[0071] The adjustable arm support 105 may provide several degrees of freedom, including lift, lateral translation, tilt, etc. Figure 7 and Figure 8 In the illustrated embodiment, the arm support 105 is configured to have four degrees of freedom, which are Figure 7 105 are shown by arrows in . A first degree of freedom allows adjustment of the adjustable arm support 105 in the z-direction ("Z lift"). For example, the adjustable arm support 105 may include a bracket 109 that is configured to move upward or downward along or relative to the column 102 of the support table 101. A second degree of freedom may allow the adjustable arm support 105 to tilt. For example, the adjustable arm support 105 may include a swivel joint that allows the adjustable arm support 105 to be aligned with the bed in a Trendelenburg position. A third degree of freedom may allow the adjustable arm support 105 to "pivot up," which may be used to adjust the distance between one side of the table 101 and the adjustable arm support 105. A fourth degree of freedom may allow the adjustable arm support 105 to translate along the longitudinal length of the table.

[0072] Figure 7 and Figure 8 The surgical robot system 100 in the embodiment of the present invention may include a table supported by a column 102 mounted to a base 103. The base 103 and the column 102 support the table 101 relative to the support surface. The floor axis 131 and the support axis 133 are located at Figure 8 Shown in.

[0073] An adjustable arm support 105 can be mounted to column 102. In other embodiments, arm support 105 can be mounted to table 101 or base 103. Adjustable arm support 105 can include bracket 109, rod or track connector 111, and rod or track 107. In some embodiments, one or more robotic arms mounted to track 107 can translate and move relative to each other.

[0074] Fig. 9 An end view of a surgical robotic system 140A having two adjustable arm supports 105A, 105B mounted on opposite sides of a table 101 is illustrated. A first robotic arm 142A is attached to a rod or track 107A of the first adjustable arm support 105B. The first robotic arm 142A includes a base 144A attached to the track 107A. The distal end of the first robotic arm 142A includes an instrument drive mechanism 146A that can be attached to one or more robotic medical instruments or tools. Similarly, the second robotic arm 142B includes a base 144B attached to the track 107B. The distal end of the second robotic arm 142B includes an instrument drive mechanism 146B. The instrument drive mechanism 146B can be configured to be attached to one or more robotic medical instruments or tools.

[0075] In some embodiments, one or more of the robotic arms 142A, 142B include an arm with seven or more degrees of freedom. In some embodiments, one or more of the robotic arms 142A, 142B may include eight degrees of freedom, including an insertion axis (including 1 degree of freedom for insertion), a wrist (including 3 degrees of freedom for wrist pitch, yaw, and roll), an elbow (including 1 degree of freedom for elbow pitch), a shoulder (including 2 degrees of freedom for shoulder pitch and yaw), and a base 144A, 144B (including 1 degree of freedom for translation). In some embodiments, the insertion degrees of freedom may be provided by the robotic arms 142A, 142B, while in other embodiments, the instrument itself provides insertion via an instrument-based insertion architecture.

[0076] C. Device Drivers and Interfaces .

[0077] The end effector of the system's robotic arm includes (i) an instrument driver (alternatively referred to as an "instrument drive mechanism" or "instrument device manipulator") that incorporates an electromechanical device for actuating a medical device; and (ii) a removable or detachable medical device that may not contain any electromechanical components, such as a motor. This dichotomy may be driven by the need to sterilize medical instruments used in medical procedures; and the inability to adequately sterilize expensive capital equipment due to its complex mechanical assemblies and sensitive electronics. Therefore, the medical device may be designed to be disassembled, removed, and interchanged from the instrument driver (and therefore from the system) for separate sterilization or disposal by a physician or physician's staff. In contrast, the instrument driver does not need to be changed or sterilized, and can be covered for protection.

[0078] Fig.10 An example instrument driver is illustrated. The instrument driver 162, positioned at the distal end of the robotic arm, includes one or more drive units 163 arranged in parallel axes to provide controlled torque to a medical device via a drive shaft 164. Each drive unit 163 includes a separate drive shaft 164 for interacting with the instrument, a gear head 165 for converting the motor shaft rotation into a desired torque, a motor 166 for generating the drive torque, an encoder 167 for measuring the speed of the motor shaft and providing feedback to the control circuit, and a control circuit 168 for receiving control signals and actuating the drive unit. Each drive unit 163 is independently controlled and motorized, and the instrument driver 162 can provide multiple (such as) to the medical device. Fig.11 In operation, the control circuit 168 will receive the control signal, transmit the motor signal to the motor 166, compare the resulting motor speed measured by the encoder 167 to the desired speed, and modulate the motor signal to generate the desired torque.

[0079] For the procedures requiring a sterile environment, the robotic system can be combined with a drive interface, such as a sterile adapter connected to a sterile cover, which is located between the instrument driver and the medical device. The main purpose of the sterile adapter is to transfer angular motion from the drive shaft of the instrument driver to the drive input of the instrument, while maintaining the physical separation between the drive shaft and the drive input and thus maintaining sterility. Therefore, the example sterile adapter may include a series of rotational inputs and rotational outputs intended to cooperate with the drive shaft of the instrument driver and the drive input on the instrument. The sterile cover formed by a thin flexible material (such as transparent or translucent plastic) connected to the sterile adapter is designed to cover capital equipment, such as instrument drivers, robot arms and carts (in cart-based systems) or tables (in tables-based systems). The use of the cover will allow capital equipment to be positioned near the patient while still being located in an area that does not require sterilization (i.e., non-sterile area). On the other side of the sterile cover, the medical device can be docked with the patient in an area that requires sterilization (i.e., sterile area).

[0080] D. Medical devices .

[0081] Fig.11 An example medical device with a paired instrument driver is illustrated. Similar to other instruments designed for use with a robotic system, the medical device 170 includes an elongated shaft 171 (or elongated body) and an instrument base 172. The instrument base 172, also referred to as an "instrument handle," is intentionally designed for manual interaction by a physician, and can typically include a rotatable drive input 173 (e.g., a socket, pulley, or reel) that is designed to mate with a drive output 174 of a drive interface on an instrument driver 175 extending through a distal end of a robotic arm 176. When physically connected, latched, and / or coupled, the mating drive input 173 of the instrument base 172 can share an axis of rotation with the drive output 174 in the instrument driver 175 to allow torque to be transferred from the drive output 174 to the drive input 173. In some embodiments, the drive output 174 can include splines that are designed to mate with a receptacle on the drive input 173.

[0082] The elongated shaft 171 is designed to be delivered through an anatomical opening or lumen (e.g., as in endoscopy) or through a minimally invasive incision (e.g., as in laparoscopy). The elongated shaft 171 can be flexible (e.g., having properties similar to an endoscope) or rigid (e.g., having properties similar to a laparoscope), or a customized combination of both flexible and rigid portions. When designed for laparoscopy, the distal end of the rigid elongated shaft can be connected to an end effector extending from an articulated wrist formed by a connecting fork having at least one degree of freedom and a surgical tool or medical instrument (such as, for example, a grasper or scissors) that can be actuated based on force from a tendon when the drive input rotates in response to a torque received from the drive output 174 of the instrument driver 175. When designed for endoscopy, the distal end of the flexible elongated shaft may include a steerable or controllable bending section to articulate and bend based on torque received from the drive output 174 of the instrument driver 175 .

[0083] The torque from the instrument driver 175 is transmitted down the shaft 171 to the elongated shaft 171 using tendons. These individual tendons (such as pull wires) can be individually anchored to individual drive inputs 173 within the instrument handle 172. From the handle 172, the tendons are guided downward along one or more pull lumens of the elongated shaft 171 and anchored at the distal portion of the elongated shaft 171, or anchored in a wrist at the distal portion of the elongated shaft. During surgical procedures such as laparoscopic, endoscopic, or hybrid procedures, these tendons can be connected to a distally mounted end effector, such as a wrist, grasper, or scissors. In such an arrangement, the torque applied to the drive input 173 transmits tension to the tendons, causing the end effector to actuate in some manner. In some embodiments, during a surgical procedure, the tendons can cause the joint to rotate about an axis, causing the end effector to move in one direction or another. Alternatively, the tendon may be connected to one or more jaws of a grasper at the distal end of the elongated shaft 171, wherein tension from the tendon causes the grasper to close.

[0084] In endoscopy, the tendon may be coupled to a bending or articulation segment located along the elongated shaft 171 (e.g., at the distal end) via an adhesive, a control ring, or other mechanical fastener. When fixedly attached to the distal end of the bending segment, torque applied to the drive input 173 will be transmitted downward along the tendon, causing the softer bending segment (sometimes referred to as the segment or region capable of articulation) to bend or articulate. Along the non-bending segment, it may be advantageous to spiral or swirl a separate traction lumen that guides the separate tendon along the wall (or inside) of the endoscope shaft to balance the radial forces caused by the tension in the pull wire. For specific purposes, the angle of the spiral and / or the spacing therebetween may be varied or designed, wherein a tighter spiral presents less axial compression under load force, while a lower spiral amount causes greater axial compression under load force, but also presents limited bending. In another case, the traction lumen may be guided parallel to the longitudinal axis of the elongated shaft 171 to allow controlled articulation in the desired bending or articulation segment.

[0085] In endoscopy, the elongated shaft 171 accommodates multiple components to assist in robotic procedures. The shaft may include a working channel for deploying surgical tools (or medical devices), irrigation pieces, and / or suction pieces to the operating area at the distal end of the shaft 171. The shaft 171 may also accommodate wires and / or optical fibers to transmit signals to / from an optical assembly at the distal end, which may include an optical camera. The shaft 171 may also accommodate an optical fiber to carry light from a proximally located light source (such as a light emitting diode) to the distal end of the shaft.

[0086] At the distal end of the instrument 170, the distal tip may also include an opening for a working channel for delivering tools for diagnosis and / or treatment, irrigation, and aspiration to the surgical site. The distal tip may also include a port for a camera, such as a fiberscope or a digital camera, to capture images of the internal anatomical space. Relatedly, the distal tip may also include a port for a light source for illuminating the anatomical space when the camera is in use. Fig.11 In the example of FIG. 5 , the axis of the drive shaft, and therefore the drive input axis, is orthogonal to the axis of the elongated shaft.

[0087] Fig.12An alternative design of an instrument driver and an instrument is illustrated, wherein the axis of the drive unit is parallel to the axis of the elongated shaft of the instrument. As shown, a circular instrument driver 180 includes four drive units, whose drive outputs 181 are aligned in parallel at the end of a robot arm 182. The drive units and their corresponding drive outputs 181 are accommodated in a rotating assembly 183 of an instrument driver 180 driven by one of the drive units in the assembly 183. In response to the torque provided by the rotating drive unit, the rotating assembly 183 rotates along a circular bearing, which connects the rotating assembly 183 to the non-rotating portion 184 of the instrument driver. In other embodiments, the rotating assembly 183 responds to a separate drive unit integrated into the non-rotating portion 184, and is therefore not parallel to other drive units. The rotating mechanism 183 allows the instrument driver 180 to allow the drive unit and its corresponding drive output 181 to rotate around the instrument driver axis 185 as a single unit.

[0088] Similar to previously disclosed embodiments, the instrument 186 may include an elongated shaft portion 188 and an instrument base 187 (shown as having a transparent outer surface for discussion purposes) that includes a plurality of drive inputs 189 (such as sockets, pulleys, and spools) configured to receive drive outputs 181 in the instrument driver 180. Unlike previously disclosed embodiments, the instrument shaft 188 extends from the center of the instrument base 187, with the axis of the instrument base being substantially parallel to the axis of the drive inputs 189, rather than being substantially parallel to the axis of the drive inputs 189. Fig.11 as orthogonal in the design.

[0089] When coupled to the rotation assembly 183 of the instrument driver 180, the medical instrument 186, which includes the instrument base 187 and the instrument shaft 188, rotates in combination with the rotation assembly 183 about the instrument driver axis 185. Since the instrument shaft 188 is positioned at the center of the instrument base 187, when attached, the instrument shaft 188 is coaxial with the instrument driver axis 185. Thus, rotation of the rotation assembly 183 causes the instrument shaft 188 to rotate about its own longitudinal axis.

[0090] Fig.13 An instrument having an instrument-based insertion architecture according to some embodiments is illustrated. The instrument 200 can be coupled to any of the instrument drivers described above. The instrument 200 includes an elongated shaft 202, an end effector 212 connected to the shaft 202, and a handle 220 coupled to the shaft 202. The elongated shaft 202 includes a tubular member having a proximal portion 204 and a distal portion 206. Manipulation of one or more cables 230a (e.g., via an instrument driver) causes actuation of the end effector 212.

[0091] The instrument handle 220 (also referred to as the instrument base) may generally include an attachment interface 222 having one or more mechanical inputs 224 (e.g., a socket, pulley, or spool) that are designed to reciprocately mate with one or more torque couplers on an attachment surface of the instrument driver.

[0092] In some embodiments, the instrument 200 includes a series of pulleys or cables that enable the elongated shaft 202 to translate relative to the handle 220. In other words, the instrument 200 itself includes an instrument-based insertion architecture that accommodates instrument insertion, thereby minimizing reliance on a robotic arm to provide insertion of the instrument 200. In other embodiments, the robotic arm may be largely responsible for instrument insertion.

[0093] E. Controller .

[0094] Any of the robotic systems described herein may include an input device or controller for manipulating an instrument attached to the robotic arm. In some embodiments, the controller may be coupled to the instrument (e.g., communicatively, electronically, electrically, wirelessly, and / or mechanically) such that manipulation of the controller causes corresponding manipulation of the instrument, e.g., via master-slave control.

[0095] Fig.14 24 is a perspective view of an embodiment of a controller 242. In the illustrated embodiment, the controller 242 is configured to allow manipulation of two medical devices and includes two handles 244. Each of the handles 244 is connected to a gimbal 246. Each gimbal 246 is connected to a positioning platform 248.

[0096] like Fig.14 As shown, each positioning platform 248 includes a SCARA arm (selective compliant assembly robot arm) 258 coupled to the column 254 via a prismatic joint 256. The prismatic joint 256 is configured to translate along the column 254 (e.g., along the track 257) to allow each of the handles 244 to translate in the z direction, thereby providing a first degree of freedom. The SCARA arm 258 is configured to allow the handles 244 to move in the xy plane, thereby providing two additional degrees of freedom.

[0097] F. Navigation and Control .

[0098] Conventional endoscopy may involve the use of fluoroscopy (e.g., as may be delivered by a C-arm) and other forms of radiation-based imaging modalities to provide intracavitary guidance to the operator physician. In contrast, the robotic system contemplated by the present disclosure may provide non-radiation-based navigation and positioning devices to reduce physician exposure to radiation and reduce the amount of equipment in the operating room. As used herein, the term "positioning" may refer to determining and / or monitoring the position of an object in a reference coordinate system. Technologies such as preoperative mapping, computer vision, real-time EM tracking, and robotic command data may be used alone or in combination to achieve a radiation-free operating environment. In other cases where radiation-based imaging modalities are still used, preoperative mapping, computer vision, real-time EM tracking, and robotic command data may be used alone or in combination to improve information obtained only by radiation-based imaging modalities.

[0099] As discussed above, the robotic systems discussed herein can be designed to incorporate a combination of one or more of the above techniques. A computer-based control system for a robotic system located in a tower, bed, and / or cart can store computer program instructions, for example, in a non-transitory computer-readable storage medium (such as a permanent magnetic storage drive, a solid-state drive, etc.), which when executed causes the system to receive and analyze sensor data and user commands, generate control signals for the entire system, and display navigation and positioning data, such as the orientation of an instrument in a global coordinate system, an anatomical map, etc.

[0100] 2. Posture optimization of the robot system .

[0101] Fig.15 A robotic medical system 300 is illustrated according to some embodiments. In some embodiments, the robotic medical system 300 is a robotic surgical system. Fig.15 In the example of FIG. 3 , the robotic medical system 300 includes a patient support platform 302 (e.g., a patient platform, table, bed, etc.). The two ends along the length of the patient support platform 302 are referred to as the "head" and "legs", respectively. The two sides of the patient support platform 302 are referred to as the "left side" and the "right side", respectively. The patient support platform 302 includes a support member 304 (e.g., a rigid frame) for the patient support platform 302.

[0102] The robotic medical system 300 also includes a base 306 for supporting the robotic medical system 300. The base 306 includes wheels 308 (castors) that allow the robotic medical system 300 to be easily moved or repositioned in a physical environment. In some embodiments, the wheels 308 are omitted from the robotic medical system 300 or the wheels are retractable and the base 306 rests directly on the ground or floor. In some embodiments, the wheels 308 are replaced with feet.

[0103] The robotic medical system 300 includes one or more robotic arms 310. The robotic arms 310 may be configured to perform the operations described above with reference to Figures 1 to 14 The robotic medical procedures described. Fig.15 Five robotic arms 310 (eg, arms 310 - 1 through 310 - 5 ) are shown, but it should be understood that the robotic medical system 300 may include any number of robotic arms, including less than five or six or more robotic arms.

[0104] According to some embodiments, the robotic medical system 300 also includes one or more arm supports 320 (e.g., rods or adjustable rods) supporting the robotic arms 310. Each of the robotic arms 310 is supported on the arm support 320 by a corresponding base joint of the robotic arm and is movably coupled to the arm support. In some embodiments, as previously described, the arm support 320 can provide several degrees of freedom, including lifting, lateral translation, tilting, etc. In some embodiments, each of the robotic arms 310 and / or the adjustable arm supports 320 is also referred to as a corresponding kinematic chain.

[0105] Fig.15 Three robotic arms 310 - 1 , 310 - 2 and 310 - 3 are shown supported by an arm support 320 within the field of view of the figure. The remaining two robotic arms 310 - 4 and 310 - 5 are supported by another pole located on another length of the patient support platform 302 .

[0106] In some embodiments, the adjustable arm support 320 can be configured to provide a base position for one or more of the robotic arms 310 to perform a robotic medical procedure. The robotic arm 310 can be adjusted by translating the robotic arm along the length of the arm support 320 below the robotic arm 310 and / or by adjusting the position and / or orientation of the robotic arm 310 via one or more joints and / or linkages (see, e.g., Fig.17A ), the robotic arm 310 can be positioned relative to the patient support platform 302. In some embodiments, the arm support 320 posture can be changed via manual manipulation, remote operation and / or power-assisted movement.

[0107] In some embodiments, the adjustable arm support 320 can translate along the length of the patient support platform 302. In some embodiments, translation of the arm support 320 along the length of the patient support platform 302 causes one or more of the robotic arms 310 supported by the arm support 320 to translate simultaneously with or relative to the arm support. In some embodiments, the arm support 320 can translate while keeping one or more of the robotic arms stationary relative to the base 306 of the robotic medical system 300.

[0108] exist Fig.15 In the example of , the adjustable arm support 320 is positioned along the length of the patient support platform 302. In some embodiments, the adjustable arm support 320 extends across part or all of the length of the patient support platform 302 and / or extends across part or all of the width of the patient support platform 302.

[0109] According to some embodiments, during a robotic medical procedure, one or more of the robotic arms 310 may also be configured to hold an instrument 312 (e.g., a robotically controlled medical instrument or tool, such as an endoscope and / or any other instrument that may be used during a surgical procedure (e.g., sensors, lighting instruments, cutting instruments, etc.)) and / or be coupled to one or more accessories, including one or more cannulas.

[0110] Fig.16 Illustrated according to some embodiments Fig.15 310 - 6. In this example, the robotic medical system 300 includes six robotic arms 310-1, 310-2, 310-3, 310-4, 310-5, and 310-6. The patient platform 302 is supported by a column 314 extending between a base 306 and the patient platform 302. In some embodiments, the patient platform 302 includes a tilt mechanism 316. The tilt mechanism 316 can be positioned between the column 314 and the patient platform 302 to allow the patient platform 302 to pivot, rotate, or tilt relative to the column 314. The tilt mechanism 316 can be configured to allow lateral tilting and / or longitudinal tilting of the patient platform 302. In some embodiments, the tilt mechanism 316 allows the patient platform 302 to be tilted laterally and longitudinally at the same time.

[0111] Fig.16 The patient platform 302 is shown in a non-tilted state or position. In some embodiments, the non-tilted state or position is the default position of the patient platform 302. In some embodiments, the default position of the patient platform 302 is as follows: Fig.16 As shown, in the non-tilted state, the patient platform 302 can be positioned horizontally or parallel to the surface (e.g., the ground or floor) supporting the robotic medical system 300. In some embodiments, the term "non-tilted" refers to a state in which the angle between the default position and the current position is less than a threshold angle (e.g., less than 5 degrees, less than an angle that would cause the patient to shift on the patient platform, etc.). In some embodiments, the term "non-tilted" refers to a state in which the patient platform is substantially perpendicular to the direction of gravity, regardless of the angle formed by the surface supporting the robotic medical system relative to gravity.

[0112] Continue to refer Fig.16In the illustrated example of the robotic medical system 300, the patient platform 302 includes a support 304. In some embodiments, the support 304 includes a rigid support structure or frame and can support one or more surfaces, pads, or cushions 322. An upper surface of the patient platform 302 can include a support surface 324. During a medical procedure, a patient can be placed on the support surface 324.

[0113] Fig.16 The robotic arm 310 and the adjustable arm support 320 are shown in an example deployed configuration, wherein the robotic arm 310 reaches above the patient platform 302. In some embodiments, due to the configuration of the robotic medical system 300 that enables various components to be stowed below the patient platform 302, the robotic arm 310 and the arm support 320 can occupy space below the patient platform 302. Thus, in some embodiments, the tilt mechanism 316 has a low profile and / or small volume to increase the space available for storage below.

[0114] Fig.16 Also illustrated are example x, y, and z coordinate systems that can be used to describe certain features of the embodiments disclosed herein. It should be understood that the coordinate system is provided only for the purpose of example and explanation, and other coordinate systems may be used. In the example shown, when the patient platform 302 is in an untilted state, the x-direction or x-axis extends across the patient platform 302 in a lateral direction. In some configurations, when the patient platform 302 is in an untilted state, the x-direction extends from one lateral side (e.g., right side) across the patient platform 302 to another lateral side (e.g., left side). When the patient platform 302 is in an untilted state, the y-direction or y-axis extends along the patient platform 302 in a longitudinal direction. That is, when the patient platform 302 is in an untilted state, the y-direction extends from one longitudinal end (e.g., head end) to another longitudinal end (e.g., leg end) along the patient platform 302. In an untilted state, the patient platform 302 may be located in or parallel to the xy plane, which may be parallel to the floor or ground. In the illustrated example, the z-direction or z-axis extends in a vertical direction along the column 314. In some embodiments, the tilt mechanism 316 is configured to enable the patient platform 302 to be tilted laterally by rotating the patient platform 302 about a lateral tilt axis parallel to the y-axis. The tilt mechanism 316 can further be configured to enable the patient platform 302 to be tilted longitudinally by rotating the patient platform 302 about a longitudinal tilt axis parallel to the x-axis.

[0115] A.Robot Arm .

[0116] FIG. 17A to FIG. 17C Different views of a robotic arm 210 are illustrated, according to some embodiments. Fig.17AThe illustrated robotic arm 310 includes a plurality of links 402 (eg, a linkage). The links 402 are connected by one or more joints 404. Each of the joints 404 includes one or more degrees of freedom (DoF). Fig.17A , the joint 404 includes a first joint 404-1 (e.g., a base joint or an A0 joint) located at or near the base 406 of the robotic arm 310. In some embodiments, the base joint 404-1 includes a prismatic joint that allows the robotic arm 310 to translate along the arm support 320 (e.g., along the y-axis). The joint 404 also includes a second joint 404-2. In some embodiments, the second joint 404-2 rotates relative to the base joint 404-1. The joint 404 also includes a third joint 404-3 connected to one end of the link 402-2. In some embodiments, the joint 404-3 includes multiple DoFs and facilitates tilting and rotation of the link 402-2 relative to the joint 404-3.

[0117] Fig.17A Also shown is a fourth joint 404-4 connected to the other end of the link 402-2. In some embodiments, the joint 404-4 includes an elbow joint connecting the link 402-2 and the link 402-3. The joint 404 also includes a pair of joints 404-5 (e.g., wrist roll joint) and 404-6 (e.g., wrist pitch joint), which are located on the distal portion of the robot arm 310. The proximal end of the robot arm 310 can be connected to the base 406, and the distal end of the robot arm 310 can be connected to an advanced device manipulator (ADM) 408 (e.g., a tool driver, an instrument driver, or a robot end effector, etc.). The ADM 408 can be configured to control the positioning and manipulation of a medical device 312 (e.g., a tool, a scope, etc.).

[0118] The robotic arm 310 may also include a cannula sensor 410 for detecting the presence of a cannula or the proximity of a cannula to the robotic arm 310. In some embodiments, when the cannula sensor 410 (e.g., via one or more processors of the robotic medical system 300) detects the presence of a cannula, the robotic arm 310 is placed in a docked state (e.g., a docked position). In some embodiments, when the robotic arm 310 is in the docked position, the robotic arm 310 may perform zero-space motion to maintain the position and / or orientation of the cannula, as discussed in further detail below. Conversely, when the cannula sensor 410 does not detect a cannula, the robotic arm 310 is placed in an undocked state (e.g., an undocked position).

[0119] In some embodiments, and as Fig.17AAs illustrated, the robotic arm 310 includes an input or button 412 (e.g., a donut-shaped button, or other type of control, etc.) that can be used to place the robotic arm 310 in an admittance mode (e.g., by pressing the button 412). The admittance mode is also referred to as an admittance scheme or admittance control. In the admittance mode, the robotic system 310 measures the force and / or torque (e.g., applied to the robotic arm 310) and outputs the corresponding speed and / or position. In some embodiments, the robotic arm 310 can be manually manipulated by the user in the admittance mode (e.g., during the setting of a procedure, or between procedures, etc.). In some cases, by using admittance control, the operator does not need to overcome all inertia in the robotic medical system 300 to move the robotic arm 310. For example, under admittance control, when the operator applies a force on the arm, the robotic medical system 300 can measure the force and help the operator move the robotic arm 310 by driving one or more motors associated with the robotic arm 310, thereby obtaining the desired speed and / or position of the robotic arm 310.

[0120] In some embodiments, the link 402 is removably coupled to the medical device 312 (e.g., to facilitate easy installation and removal of the medical device 312 from the robotic arm 310). The joint 404 provides multiple degrees of freedom (DoF) to the robotic arm 310, thereby facilitating control of the medical device 312 via the ADM 408. In embodiments including multiple robotic arms, each robotic arm can hold its corresponding medical tool and pivot the medical tool about a remote center of motion.

[0121] Fig. 17B A front view of the robotic arm 310 is illustrated. Fig. 17C A perspective view of a robotic arm 310 is illustrated. In some embodiments, the robotic arm 310 includes Fig.17A A second input portion or button 414 (e.g., a push button) different from the button 412 in the figure is used to place the robotic arm 310 in impedance mode (e.g., by a single press or continuous pressing and holding of the button 414). In this example, the button 414 is located between the joint 404-5 and the joint 404-6. The impedance mode is also referred to as an impedance scheme or impedance control. In the impedance mode, the robotic medical system 300 measures displacement (e.g., changes in position and velocity) and outputs force and / or torque to facilitate manual movement of the robotic arm. In some embodiments, the robotic arm 310 can be manually manipulated by a user in the impedance mode (e.g., during a setup procedure). In some embodiments, in the impedance mode, the operator's movement of a portion of the robotic arm 310 causes movement of one or more joints and / or links of the entire robotic arm 310.

[0122] In some embodiments, for admittance control, a force sensor or load sensor can measure the force that the operator is applying to the robotic arm 310 and move the robotic arm 310 in a way that feels light. Admittance control may feel lighter than impedance control because under admittance control, the perceived inertia of the robotic arm 310 can be hidden because the motors in the controller can help accelerate the mass. In contrast, according to some embodiments, with impedance control, the user is responsible for most, if not all, of the mass acceleration.

[0123] In some cases, depending on the position of the robot arm 310 relative to the operator, it may be inconvenient to reach the button 412 and / or the button 414 to activate the manual steering mode (e.g., the admittance mode and / or the impedance mode). Therefore, in these cases, the operator may conveniently trigger the manual steering mode instead of triggering the manual steering mode via a button.

[0124] In some embodiments, the robotic arm 310 includes a single button (e.g., button 412 or 414) that can be used to place the robotic arm 310 in an admittance mode and / or an impedance mode (e.g., by using different presses, such as a long press, a short press, a press and hold, etc.). In some embodiments, the robotic arm 310 is placed in an impedance mode by a user pushing an arm linkage (e.g., link 402) and / or a joint (e.g., joint 404) and overcoming a force threshold. In some embodiments, the admittance mode and the impedance mode are common because both modes allow a user to grab the robotic arm 310 and command movement by interacting directly with the robotic arm.

[0125] In some embodiments, the robotic arm 310 includes an input control for activating the arm following mode. For example, in some embodiments, the robotic arm 310 includes a designated contact point located on a link 402 or joint 404 of the robotic arm (e.g., a housing of the link 402 or a button 416). User interaction (e.g., user touch, contact, etc.) with the designated contact point activates the arm following mode. In some embodiments, the robotic arm 310 includes multiple contacts. User interaction with any (e.g., one or more) of the contacts activates the arm following mode.

[0126] During a medical procedure, it may be desirable for the ADM 408 of the robotic arm 310 and / or the remote center of motion (RCM) of the instrument 312 coupled thereto to remain in a static pose (e.g., position and / or orientation). The RCM may refer to a point in space at which a cannula or other access port inserted into the medical instrument 312 is restricted in motion. In some embodiments, the medical instrument 312 includes an end effector that is inserted through an incision or natural orifice of a patient while maintaining the RCM. In some embodiments, the medical instrument 312 includes an end effector that is in a retracted state during the setup process of the robotic medical system.

[0127] In some cases, the robotic medical system 300 may be configured to be able to move one or more links 402 of the robotic arm 310 within a "null space" to avoid collisions with nearby objects (e.g., other robotic arms) while the ADM 408 of the robotic arm 310 and / or RCM remains in its corresponding pose (e.g., position / orientation). The null space may be considered to be the set of joint states that the robotic arm 310 can move that does not cause the ADM 408 and / or RCM to move, thereby maintaining the position and / or orientation of the medical device 312 (e.g., within the patient's body). In some embodiments, the robotic arm 310 may have multiple positions and / or configurations that can be used for each pose of the ADM 408.

[0128] For the robotic arm 310 to move the instrument to a desired position in space, in some embodiments, the robotic arm 310 has at least six DoF, with three DoF for translation (e.g., X, Y, and Z position) and three DoF for rotation (e.g., yaw, pitch, and roll). In some embodiments, each joint 404 provides the robotic arm 310 with a single DoF, and thus the robotic arm 310 has at least six joints to achieve the degrees of freedom of motion to position the ADM 408 in any position in space. To further maintain the ADM 408 and / or remote center or motion of the robotic arm 310 at a desired position, the robotic arm 310 may also have at least one additional "redundant joint". Therefore, in some embodiments, the system includes a robotic arm 310 with at least seven joints 404, thereby providing a robotic arm 310 with at least seven DoF. In some embodiments, the robotic arm 310 includes a subset of joints 404, each joint having more than one degree of freedom, thereby achieving additional DoF for zero-space motion. However, depending on the embodiment, the robotic arm 310 may have more or fewer DoFs.

[0129] In addition, as regards Figure 7As described, the arm support 320 (e.g., an adjustable arm support) can provide several degrees of freedom, including lifting, lateral translation, tilting, etc. Thus, according to embodiments, the robotic medical system can have more degrees of freedom for robotic control than just in the robotic arm 310 to provide zero-space movement and collision avoidance. In corresponding embodiments of these embodiments, the end effector of one or more robotic arms (and any tools or instruments coupled thereto) and / or the remote center along the axis of the tool can advantageously maintain a posture and / or position within the patient.

[0130] The robotic arm 310 having at least one redundant DoF has at least one more DoF than the minimum number of DoFs required to perform a given task. For example, according to some embodiments, the robotic arm 310 may have at least seven DoFs, wherein one of the joints 404 of the robotic arm 310 may be considered a redundant joint. The one or more redundant joints may allow the robotic arm 310 to move in null space to maintain the pose of the ADM 408 and the position of the RCM while avoiding collisions with other robotic arms or objects.

[0131] In some embodiments, the robotic medical system 300 is configured to perform collision avoidance by utilizing movement of one or more redundant joints in null space to avoid, for example, collisions between adjacent robotic arms 310. For example, when a robotic arm 310 collides with or approaches (e.g., within a defined distance) another robotic arm 310, one or more processors of the robotic medical system 300 may be configured to detect (e.g., via kinematics) a collision or an impending collision. Thus, the robotic medical system 300 may control one or both of the robotic arms 310 to adjust their respective joints in null space to avoid the collision or the impending collision. In one embodiment including at least one pair of robotic arms, the base of one of the robotic arms and its end effector may remain in its pose while the link or joint between the two moves in null space to avoid collisions with the adjacent robotic arm.

[0132] B. Example Port Placement for Different Surgical Procedures .

[0133] As mentioned previously, the pose of the robotic arm relative to the target anatomy and the port placement are critical to the success of medical procedures, which often require precise port locations and often have a small margin for error.

[0134] 18A to 18C Example cannula placements for different surgical procedures are illustrated in accordance with some embodiments. Fig.18AAn example surgical environment 500 is illustrated that includes four cannulas 502-1, 502-2, 502-3, and 502-4 placed at multiple anatomical locations within a patient 504 to provide access to at least a portion of the patient's anatomy 506 (e.g., the abdomen). The cannulas 502 are capable of passing through a port location 508 within the patient's body (or within the patient's anatomy 506). As used herein, a port location (e.g., a port, an access port, an access point, a port area, a port region, or a port location, etc.) refers to a location on the patient's body through which a medical tool / instrument (e.g., held by a robotic arm) can be inserted and movement is constrained. In some embodiments, the port location corresponds to an incision point (or incision area) formed through the patient's skin to facilitate a medical procedure or procedure. In some embodiments, the port location corresponds to a natural orifice, such as a patient's mouth (e.g., for a bronchoscopy procedure). In Fig.18A In the example of FIG. 5 , the illustrated placement of cannula 502 may allow access to the upper right quadrant of the patient's abdomen.

[0135] In some embodiments, a camera is docked to one of the cannulas 502 (eg, cannula 502-3) to provide a view of the patient's anatomy (eg, Fig.18A ). A medical device can be docked to one or more of the cannulas 502 (e.g., 502-1, 502-2, and / or 502-4). Each of the cannulas 502 provides access to the patient's anatomical structure 506 from a unique position and / or orientation, thereby allowing medical procedures to be performed in a flexible manner. Arrows extending from the respective cannulas 502 illustrate from which angles a medical tool can enter the patient's anatomical structure 506 through the cannulas 502. As an example, in the initial stages of a medical procedure, a physician may use medical tools docked to cannulas 502-1 and 502-2, and in subsequent stages may use medical tools docked to cannulas 502-3 and 502-4. By using medical tools docked to different cannulas 502, a physician is able to access the patient's anatomical structure 506 from different angles, thereby having more options for directly accessing various portions of the patient's anatomical structure 506.

[0136] In some embodiments, such as Fig.18A As shown, cannulas 502-1 and 502-4 provide access to a patient's anatomical structure 506 from substantially opposite sides, thereby allowing a physician to access both sides of the anatomical structure 506 by simply selecting a medical tool that docks to cannulas 502-1 and 502-4.

[0137] Fig.18B An example surgical environment 510 is illustrated that includes five cannulas 502-5 through 502-9 placed at anatomical locations of a patient 505 to provide access to the upper left quadrant of the patient's abdomen.

[0138] Fig. 18C An example surgical environment 520 is illustrated that includes five cannulas 502 - 10 through 502 - 14 in anatomical locations of a patient 504 to provide access to multiple quadrants of the abdomen of the patient 504 . Fig. 18C Placement of cannula 502 in may be used to perform a colectomy as an example surgical procedure.

[0139] C. Example workflow for optimal robot arm pose

[0140] Fig.19 An example workflow 600 for generating optimal port positions and robotic arm poses for a medical procedure or procedures is illustrated in accordance with some embodiments. In some embodiments, the workflow 600 is implemented as a program executed by one or more processors (e.g., processor 380, Fig.24 ) for recommending port placement and starting posture to avoid collisions during a surgical procedure (e.g., between a patient and a robotic arm, between two or more robotic arms, between a robotic arm and a surgical tool, between two or more surgical tools, etc.).

[0141] According to some embodiments, the workflow 600 includes placing 602 a patient on a table of a robotic medical system (e.g., a patient support platform 302, a bed, etc.). In some embodiments, the patient is positioned on the table, which is the position in which the patient will be during the medical procedure or procedure.

[0142] According to some embodiments, workflow 600 includes obtaining (604) (e.g., creating, generating, receiving, determining, collecting, etc.) a 3-D scan (e.g., 3-D data, imaging data, scan data, etc.) including a view of the patient. Fig. 20A and Fig. 20B An example 3-D scanning and registration process of a patient relative to a robotic medical system in accordance with some embodiments is illustrated. Fig. 20A A side view 700 is illustrated in which a patient 702 is illustrated positioned on a patient support platform 302 of the robotic medical system 300. In this example, a 3-D scanner 704 is attached to the robotic arm 310 of the robotic medical system 300. Fig. 20B A top view 710 is illustrated that includes a portion of a patient 702 (eg, an anatomical structure such as an abdomen) and the positions of the robotic arms 310 - 1 to 310 - 6 and the adjustable arm supports 320 - 1 and 320 - 2 of the robotic medical system 300 .

[0143] In some embodiments, the 3-D scanner 704 is included in the robotic medical system 300 (e.g., as a component of the robotic medical system, forming a part of the robotic medical system, etc.). For example, the 3-D scanner 704 can be a component of, integrated into, or attached to the robotic medical system 300. For example, the 3-D scanner 704 can be located on (e.g., positioned on) the robotic arm 310, the patient support platform 302 (e.g., a bed), the adjustable arm support 320, the pendant, and / or any portion of the robotic medical system 300 that includes a patient view.

[0144] In some embodiments, the 3-D scanner 704 is part of an instrument (e.g., a medical tool such as a laparoscope) held by the robotic arm 310 of the robotic medical system 300. In some embodiments, the 3-D scanner 704 is a separate accessory (e.g., a stand-alone scanner, a handheld scanner, etc.) that is communicatively connected to the robotic medical system 300 and is attached to (e.g., mounted on) the robotic arm 310, the patient support platform 302 (e.g., a bed), the adjustable arm support 320, the pendant, and / or any portion of the robotic medical system 300 that includes a patient view. In some embodiments, the 3-D scanner 704 is mounted on the wall and / or ceiling of the operating room where the robotic medical system 300 is located.

[0145] In some embodiments, the 3-D scanner 704 uses time of flight (e.g., laser imaging, detection and ranging (LIDAR)), point pattern recognition, stereo photography (e.g., using a stereo camera), and / or other established scanning technologies to perform 3-D scanning. It should be noted that these scanning technologies are examples and are not intended to be limiting.

[0146] The 3-D scanner 704 creates (e.g., generates, obtains, etc.) data (e.g., scan data, 3-D scan data, etc.) including a view of the patient 702. In some embodiments, the 3-D scan data includes visual data (e.g., imaging data) of at least a portion of the patient 702 and the environment (e.g., space and / or other objects) surrounding the patient 702. The scan data can capture size (e.g., dimensions), shape, and / or depth information of at least a portion of the patient 702 and one or more objects (e.g., a patient support platform, a medical tool, etc.) surrounding the patient 702. In some embodiments, the scan data includes dimensional information (e.g., distance, angle, etc. information) that can be used to determine the relative spacing (e.g., spatial spacing, angular spacing) between the patient 702 and other objects.

[0147] According to some embodiments, the workflow 600 includes positioning (606) a patient on a table (e.g., a patient support platform 302) based on the position of the robotic arm 310. For example, the 3-D scan data may include a view of the patient and a portion of the robotic medical system (such as the robotic arm, the patient support platform, and / or the adjustable arm support (or a portion thereof)). Because the robotic medical system 300 can determine the position and / or orientation of the robotic arm, the patient support platform, and / or the adjustable arm support (e.g., via data obtained by sensors and / or encoders throughout the robotic medical system), the robotic medical system 300 can position (e.g., determine one or more positions of the patient) the patient relative to components of the robotic medical system 300 (e.g., relative to the robotic arm 310).

[0148] In some embodiments, after positioning the patient based on the position of the robotic arm 310, the robotic medical system proceeds to place the patient on a coordinate system of the robotic medical system 300 and uses the coordinate system to determine coordinates corresponding to different parts of the patient (e.g., head, toes, navel, etc.). For example, in some embodiments, the robotic medical system 300 includes a coordinate system (e.g., a robot coordinate system, a coordinate frame, a system frame, etc.), and the corresponding positions of the patient support platform 302, the robotic arm 310, and / or the adjustable arm support 320 can be represented as coordinates on the coordinate system (e.g., x, y, and z coordinates). The robotic medical system 300 can use the scan data obtained by the 3-D scanner 704 (e.g., imaging data including views of a portion of the patient and the robotic arm 310) to "match" the patient frame with the system frame (e.g., the robotic arm frame) and "register" the patient to the coordinate system.

[0149] According to some embodiments, the workflow 600 includes registering (608) (e.g., determining) anatomical structures (e.g., external anatomical structures and / or internal anatomical structures) during a 3-D scan by the robotic medical system 300. In some embodiments, the robotic medical system 300 compares (e.g., matches) the 3-D scan data with previous generalized imaging data (e.g., heuristic data, previous imaging data, imaging data from a database, etc.) (such as MRI imaging data and / or CT scan data) (e.g., automatically or manually based on user input). The robotic medical system 300 identifies one or more anatomical structures based on the comparison (e.g., the robotic medical system 300 identifies one or more anatomical structures based on a determination that the anatomical structures shown in the 3-D scan data match the anatomical structures in the database of imaging data).

[0150] In some embodiments, the robotic medical system 300 obtains preoperative imaging data corresponding to the patient (e.g., MRI imaging data, CT scan data, etc.). The robotic medical system 300 compares the 3-D scan data to the preoperative imaging data (e.g., automatically or manually based on user input, compares side by side, overlays the two sets of data, etc.), and identifies (e.g., registers) one or more anatomical structures based on the comparison.

[0151] In some embodiments, the robotic medical system 300 registers (e.g., automatically or manually based on user input) one or more anatomical landmarks (e.g., anatomical landmarks, anatomical structures) of the patient from the 3-D scan data. For example, the one or more anatomical landmarks may include external anatomical landmarks (such as the umbilicus) or internal anatomical landmarks (e.g., the xiphoid process or the anterior superior iliac spine (ASIS)), which can be inferred based on the 3-D scan data. In some configurations, the robotic medical system 300 uses image processing software or pattern recognition software to identify the location of the one or more external anatomical landmarks. In some configurations, the robotic medical system 300 determines the location of the one or more internal anatomical landmarks based on the location of the one or more external anatomical landmarks (e.g., the internal anatomical landmark is located below a location that is a certain distance from two or more external anatomical landmarks).

[0152] In some embodiments, the robotic medical system registers (e.g., automatically or manually based on user input) one or more anatomical markers based on a fiducial (e.g., anatomical side marker) placed on the patient prior to collecting the 3-D scan. In some embodiments, the position of the one or more fiducials is considered to correspond to the position of the one or more anatomical markers. In some embodiments, the position of the one or more anatomical markers is determined based on the position of the one or more fiducials. For example, the position of the one or more anatomical markers is considered to correspond to a predefined distance below the one or more fiducials. In another example, the position of the anatomical marker is considered to correspond to a position determined based on two or more fiducials (e.g., having an equal distance from the two or more fiducials).

[0153] According to some embodiments, the workflow 600 includes receiving 610 user input from a user (such as a surgeon, nurse, or surgeon's assistant) of the robotic medical system 300. The input may include identification of a medical procedure to be performed on a patient, a target anatomy (or target anatomy), and / or a number of robotic arms (and / or a selection of robotic arms) to be used for the procedure. Fig.21A User input of a target anatomy or procedure is illustrated according to some embodiments. Fig.21AAs shown, the operator can provide user input through the user interface. In some embodiments, the robotic medical system includes a touch-sensitive display (e.g., a touch screen), and the operator can provide user input through the touch-sensitive display. Alternatively, the operator can provide user input through any other input device (e.g., a mouse, a keyboard, one or more buttons or switches, etc.). Fig.21B The robotic medical system 300 is illustrated estimating the location of a target anatomical structure (e.g., stomach). In some embodiments, the robotic medical system 300 estimates the location of the target anatomical structure based on the method described in operation 608 of the workflow 600 and / or in conjunction with user input in operation 610.

[0154] Workflow 600 includes combining 3-D scan data (e.g., operation 604) with registered anatomy (e.g., obtained from operation 608) and an anticipated (or selected) procedure and / or target anatomy (e.g., obtained from operation 610) (612) to identify candidate patient-specific robotic arm poses and port locations for a medical procedure according to some embodiments. In some embodiments, the combining includes simulating a medical procedure based on the combined data (e.g., plotting a trajectory of a robotic arm during the medical procedure).

[0155] The workflow 600 includes generating (614) one or more recommended (e.g., optimized) port locations and robotic arm poses specific to a patient and a procedure (medical procedure). For example, the robotic medical system identifies a plurality of locations within a first predefined distance from a target anatomical structure (e.g., a target organ), excludes locations adjacent to one or more exclusion zones (also referred to as "exclusion zones," such as zones where other critical anatomical structures (e.g., organs) are located, zones where other tools or other accessories are located), identifies the number of ports associated with an intended (or selected) procedure and the orientation of the ports relative to the target anatomical structure (e.g., storing the number of ports and orientation from a memory of the robotic medical system), and selects a location within a predefined distance from the target anatomical structure at the identified orientation. In addition, the robotic medical system also identifies a starting pose of a robotic arm participating in the procedure. The robotic medical system simulates the procedure and the movement of the robotic arm during the procedure to identify any collisions that may occur (e.g., collisions between the robotic arms and / or between the robotic arms and a patient, other equipment, or personnel).

[0156] In some embodiments, the robotic medical system identifies one or more potential starting postures (e.g., each starting posture corresponds to a recommended set of port placement) and simulates procedures and movements for each potential starting posture. For example, the robotic medical system can select multiple potential starting postures from a predefined set of potential starting postures. In another example, the robotic medical system can generate multiple potential starting postures by changing the joint angles of one or more joints (e.g., generating multiple potential starting postures corresponding to specific joint angles of specific joints). In some embodiments, the robotic medical system selects the potential starting posture that causes the least number of collisions (e.g., zero) as the recommended starting posture. For example, the robotic medical system (e.g., based on simulation) can record the number of collisions of the corresponding starting postures in multiple potential starting postures, and select the starting posture with the least number of collisions in multiple potential starting postures. In another example, the robotic medical system can select a starting posture with less than a predefined number of collisions. In some embodiments, select port positions separated by at least a second predefined distance (e.g., to provide a sufficiently long distance between tools). In some embodiments, the port locations are optimized using a value function (during the optimization process) based on the distance of each port location to the target anatomical structure, the distance of each port location to other port locations, the distance of each port location to one or more exclusion regions, and / or the orientation of the target anatomical structure provided by each port location.

[0157] Workflow 600 includes displaying (616) (or causing to be displayed) optimized port positions and robot arm poses according to some embodiments. Fig.22A A top view 800 of a patient 702 is illustrated, including an optimal port location 802 displayed (eg, projected) on the abdomen of the patient 702 . Fig. 22B Optimized port positions and robotic arm poses are illustrated as displayed on a user interface 820 of the robotic medical system 300. In some embodiments, the user interface 820 is located on a surgeon viewer (e.g., a tower viewer, a pendant, etc.), or on any display device communicatively connected to the robotic medical system 300. In some embodiments, the robotic medical system 300 uses augmented reality (AR) glasses communicatively connected to the robotic medical system 300 to display recommended arm poses and port positions. For example, a user of the robotic medical system 300 can wear the AR glasses during the setup phase of a medical procedure.

[0158] In some embodiments, the robotic medical system 300 prints the recommended robotic arm poses and port locations onto a medium. For example, in some embodiments, the recommended port locations are printed onto a sterile temporary tattoo that can be applied to a patient. In some embodiments, the printed medium includes sterilized paper, film, and / or fabric on which a patient view including the recommended port locations is printed and attached (e.g., taped) to the patient.

[0159] As mentioned above, Fig.22A A set of recommended port locations 802 for a medical procedure is illustrated in accordance with some embodiments. Fig.22A In the example of FIG. 8 , the recommended port locations 802 include four port locations 802 - 1 , 802 - 2 , 802 - 3 , and 802 - 4 in the patient's stomach region. Fig. 22B An example set of initial (eg, pre-docked) poses of the robotic arm 804 corresponding to the port location 802 is illustrated. Fig. 22B As shown, a subset of all available robotic arms can be used for corresponding port positions (e.g., robotic arms 804-1, 804-2, 804-3, and 804-4). For example, four robotic arms are used in a system with up to six robotic arms. In some embodiments, each robotic arm has multiple joints (e.g., as described above with reference to FIG. 17A to FIG. 17C Discussed) and identifying an initial arm pose includes identifying an initial position of each of a plurality of joints.

[0160] Fig. 22C and Fig.22D Example arm posture movements during a procedure according to some embodiments are illustrated, the example arm posture movements corresponding to Fig. 22B The set of port positions 802 and the initial arm pose are shown in . Fig. 22C The user interface 820 is shown in which the robotic arm 804-4 moves from an initial position 805-a to a new position 805-b and the robotic arm 804-3 moves from an initial position 807-a to a new position 807-b. Fig.22D The user interface 820 is shown in which the robotic arm 804-1 moves from an initial position 809-a to a new position 809-b and the robotic arm 804-2 moves from an initial position 811-a to a new position 811-b. Fig.22D Also shown is a collision between robot arm 804-1 and robot arm 804-2 caused by the movement. According to some embodiments, Fig. 22C and Fig.22D The movement shown in corresponds to the Fig. 22BThe set of port locations 802 and the initial arm pose shown in FIG. 8 are used to simulate the movement of the robotic arm 804 during the medical procedure (e.g., a simulation performed by the system 279). In some embodiments, the robotic medical system abandons the selection due to the identification of the collision. Fig. 22B The port position 802 and initial pose shown in 802 are used as recommended port positions and recommended initial poses (eg, unless there is no other combination of port position and initial pose that results in less collision).

[0161] Fig.22E Another set of port locations 812 for medical procedures is illustrated in accordance with some embodiments. Fig.22E In the example of , the recommended port locations 812 include four port locations 812-1, 812-2, 812-3, and 812-4 in the patient's stomach region. In some embodiments, the port locations 812 correspond to feasible alternatives to the port locations 802 for a given medical procedure. Fig.22F An example set of initial (eg, pre-docked) poses of the robotic arm 804 corresponding to the port location 812 is illustrated.

[0162] Figure 22G and Fig.22H Example arm posture movements during a procedure according to some embodiments are illustrated, the example arm posture movements corresponding to Fig.22F The set of port positions 812 and the initial arm pose are shown in . Figure 22G The user interface 820 is shown in which the robotic arm 804-4 moves from an initial position 813-a to a new position 813-b and the robotic arm 804-3 moves from an initial position 815-a to a new position 815-b. Fig.22H A user interface 820 is shown in which the robotic arm 804-1 moves from an initial position 819-a to a new position 819-b and the robotic arm 804-2 moves from an initial position 817-a to a new position 817-b. According to some embodiments, Figure 22G and Fig.22H The movement shown in corresponds to the Fig.22F The set of port locations 812 and initial arm poses shown in FIG. 8 are used to simulate the movement of the robotic arm 804 during a medical procedure (e.g., a simulation performed by the system 279). Figure 22G and Fig.22H Select the move shown in Fig.22F The port position 812 and initial position shown in FIG. 8 are used as the recommended port position and position so that no collision occurs (or compared to Fig. 22C and Fig.22D The alternative shown in has fewer collisions).

[0163] Despite FIG. 22A to FIG. 22H Not illustrated in the examples, but in some embodiments, one or more additional robotic arms that do not correspond to port placement are used during a procedure (e.g., to hold lights or equipment that do not require a corresponding port). In embodiments where additional robotic arms are present, the robotic medical system takes these robotic arms into account when determining the initial pose and simulating collisions during a medical operation or procedure.

[0164] Fig.23A and Fig. 23B is a flow chart illustrating a method 900 for determining a starting state of a robotic medical system according to some embodiments. The method 900 is performed at a robotic medical system (e.g., medical system 36 or robotic medical system 300) having one or more processors (e.g., processor 280), a memory (e.g., memory 282), and a plurality of robotic arms. In some embodiments, the memory (e.g., memory 282) stores instructions executed by the one or more processors (e.g., processor 280). In some embodiments, at least a subset of the robotic arms are coupled to respective medical devices.

[0165] The robotic medical system obtains (902) imaging data of the patient. For example, the robotic medical system obtains imaging data, as described above with respect to operation 604. In some embodiments, the imaging data includes a 3D surface scan of the patient. In some embodiments, the robotic medical system includes an imager (e.g., 3-D scanner 704). In some embodiments, the imager is not part of the robotic medical system, but communicates with the robotic medical system. In some embodiments, the imager incorporates at least one of time of flight or point pattern recognition. In some embodiments, the robotic medical system uses the imaging information to identify one or more skeletal landmarks (e.g., rib cage or hip bone) of the patient.

[0166] In some embodiments, the imaging data of the patient includes (904) an image of the patient after the patient is prepared for a medical procedure. In some embodiments, the imaging data of the patient includes an image after the patient's body cavity is inflated. In some embodiments, the system determines (e.g., in real time) one or more physical characteristics of the patient (e.g., size and / or other characteristics, such as unique anatomical structures). In some embodiments, the imaging data of the patient includes an image of the patient after the patient is positioned in position for the medical procedure (e.g., positioned on the patient support platform 302). In some embodiments, the imaging data includes a portion of a robotic arm and is used to position the patient relative to the robotic arm (e.g., as described above with respect to operation 606).

[0167] The robotic medical system obtains (906) information about the medical procedure for the patient. For example, the robotic medical system obtains information about the medical procedure, as described above with respect to operation 610. As another example, the physician or assistant notifies the system of the type of medical procedure being performed (e.g., enters the medical procedure details in the console 31). In some embodiments, the information about the medical procedure includes the type of procedure to be performed. In some embodiments, the information about the medical procedure includes information about additional equipment or tools that will be used during the medical procedure (e.g., a foot pedal, a liver retractor, an arm board, and / or an anesthesia accessory).

[0168] The robotic medical system simulates (908) a medical procedure from the starting state for each of the plurality of starting states to identify a number of collisions with arms of the plurality of robotic arms. The simulation uses the imaging data and information about the medical procedure. For example, the simulation includes simulating information about the medical procedure. FIG. 22B to FIG. 22H The robot arm moves. In some embodiments, the medical procedure involves a plurality of sequential steps, and at least a subset of the steps has a corresponding robot arm position requirement. In some embodiments, simulating the medical procedure includes simulating the movement of the robot arm between the required robot arm positions. In some embodiments, the simulation includes simulating alternative movements between the required robot arm positions to identify which alternative method causes the least collision. In some embodiments, the simulation is further based on the type of robotic medical system (e.g., how many arms the system has and how these arms are mounted). In some embodiments, the simulation is further based on the type of robotic arm (e.g., how many joints the robot has and the available degrees of freedom). For example, in some embodiments, the robot arm support has four degrees of freedom, the robot arm has five degrees of freedom (e.g., rolling and twisting), and the medical device attached to the robot arm has four degrees of freedom (e.g., rolling, pitching, yaw, and insertion). In some embodiments, the simulation includes determining the number of robotic arms of the system that will be used in the medical procedure (e.g., determining whether a subset of the robotic arms will not be used and can be stowed during the medical procedure).

[0169] In some embodiments, the simulation includes: (i) identifying a first set of port placements, (ii) determining a starting pose based on the first set of port placements, and (iii) plotting a trajectory of the robotic arm during the medical procedure as the robotic arm moves from the starting pose to positions required for different steps of the procedure. In some embodiments, the simulation includes: (i) repeating the identifying, determining, and plotting for multiple sets of port placements and multiple starting poses; and (ii) selecting the set of port placements and starting poses that results in the least number of collisions.

[0170] In some embodiments, the robotic medical system obtains (910) an anatomical target for the medical procedure, and the simulation further uses the anatomical target. For example, the robotic medical system obtains the anatomical target as described above with respect to operation 610. In some embodiments, the user inputs the medical procedure and the target anatomical structure.

[0171] In some embodiments, the robotic medical system uses the patient's imaging data to identify (912) anatomical targets. For example, the robotic medical system identifies the anatomical targets as described above with respect to operation 608. In some embodiments, the robotic medical system estimates the position of one or more key anatomical structures (such as related organs) based on the imaging data. In some embodiments, the anatomical targets include external anatomical landmarks (such as the navel) or internal anatomical landmarks (e.g., the xiphoid process or the anterior superior iliac spine), which can be inferred based on the imaging data. In some embodiments, the system compares the patient's imaging data with generalized imaging data (e.g., heuristic data, previous imaging data, imaging data from a database, etc.), and identifies the analysis target based on the comparison. In some embodiments, the system estimates the position of the internal organs based on the anatomical targets.

[0172] In some embodiments, the robotic medical system obtains (914) imaging data for the environment surrounding the robotic medical system, and the simulation further uses the imaging data for the environment surrounding. For example, the robotic medical system determines one or more boundary conditions (e.g., spatial limitations) of the robotic arm. In some embodiments, the data includes visual data (e.g., imaging data) of at least a portion of the patient and the environment surrounding the patient (e.g., space and / or objects). In some embodiments, the environment surrounding includes non-robotic components, such as bedside accessories clamped to the system (e.g., liver retractors and anesthesia equipment that can be clamped to the bed and covered on the patient's head). In some embodiments, the environment surrounding includes movable and removable (optional) portions of the patient support platform, such as movable or removable leg supports. In some embodiments, based on the visual data of the environment surrounding, the robotic medical system identifies one or more boundary conditions or "forbidden zones" of the robotic arm (e.g., to prevent the robotic arm from colliding with objects in the environment). In some embodiments, the robotic medical system uses one or more machine learning techniques to identify non-robotic components and / or movable and removable portions of the patient support platform (e.g., object recognition algorithms).

[0173] In some embodiments, the robotic medical system obtains (918) position information of one or more objects (or entities) near the robotic medical system, and the simulation further uses the position information. For example, the position information captures size (e.g., dimensions), shape, and / or depth information of at least a portion of the patient and one or more objects (e.g., lamps, oxygen tanks, carts, displays, and trays) around the patient. The position information includes dimensional information (e.g., distance, angle, etc.) that can be used to determine the relative spacing (e.g., spatial spacing, angular spacing) between the patient and other objects. In some embodiments, the objects include accessories (e.g., mounted on a bedside or robotic arm, etc.), such as stirrups, liver retractors, arm boards, anesthesia accessories, etc.

[0174] In some embodiments, the robotic medical system maps (920) a trajectory of each of the plurality of robotic arms during the medical procedure and identifies collisions based on the trajectory. For example, the robotic medical system maps the trajectory to identify collisions between a patient and a robotic arm, between two robotic arms, between two instruments, between a robotic arm and an instrument, and / or between a robotic arm and another nearby object.

[0175] The robotic medical system selects (922) a first starting state from a plurality of starting states, the first starting state having the least number of collisions. The first starting state identifies a respective starting pose for each of the plurality of robotic arms. For example, the robotic medical system generates optimized pose information and port placement, as described above with respect to operation 614. In some embodiments, selecting the first starting state includes identifying how many robotic arms will be used for the medical procedure (e.g., by simulating the medical procedure using different numbers of robotic arms). In some embodiments, the robotic medical system selects the first starting state based on which the least number of robotic arms are required.

[0176] In some embodiments, each arm of the plurality of robotic arms has (924) a plurality of joints, and for each arm of the plurality of robotic arms, the first starting state includes positioning information for each joint of the plurality of joints. In some embodiments, the first starting state includes positioning information for each robotic arm and each robotic arm support (e.g., arm support 105 or 320). In some embodiments, the plurality of joints includes a subset of joints that are locked in place during a medical procedure, and the positioning information includes the locked positions of the subset of joints. For example, the robotic arm may include: (i) a setup arm for positioning a surgical end of the arm (e.g., a proximal joint of the arm, such as joints 404-1 to 404-6), and (ii) a distal end of the surgical end of the arm. In some embodiments, the setup joint remains in a locked position (identified in the positioning information) during the medical procedure.

[0177] In some embodiments, the medical procedure requires (926) a medical device, and the pose information includes information about which arm of a plurality of robotic arms the medical device should be mounted to. In some embodiments, the medical device has different size options, and the pose information includes information about the recommended size option for the medical device. In some embodiments, the robotic medical system identifies an assigned task for each robotic arm (e.g., holding a cannula, a light, a camera, etc.). For example, a medical procedure requires three different medical devices, and the system identifies which arm holds each device (e.g., to minimize the number of collisions during the procedure).

[0178] In some embodiments, the robotic medical system selects (928) a recommended port location based on the simulation (e.g., Fig.22A and Fig.22E ). A recommended port location is a location on a patient where a medical tool / instrument is inserted and where its movement is restricted (e.g., positioned). In some embodiments, a port location corresponds to an incision point (or incision area) made through the patient's skin to facilitate a medical procedure. In some embodiments, a port location corresponds to a natural orifice, such as a patient's mouth (e.g., for a bronchoscopy procedure).

[0179] The robotic medical system provides (930) posture information of the first starting state to an operator of the robotic medical system. For example, the robotic medical system provides the posture information as described above with respect to operation 616. In some embodiments, the system notifies the user of the optimal pre-docking posture of one or more arms, and provides the user (e.g., a hospital assistant) with the option of moving the one or more arms to the corresponding optimal posture. In some embodiments, the robotic medical system displays one or more graphical representations indicating the posture information on a display (e.g., touch screen 26), or provides an electrical signal to the display for displaying the posture information (e.g., in VR or AR display).

[0180] In some embodiments, the robotic medical system moves (932) at least one of the plurality of robotic arms to be in a respective starting position. In some embodiments, the arm is moved from a hanging position to be in a starting position. In some embodiments, the robotic medical system is automatically driven to a first starting state. In some embodiments, the robotic medical system automatically docks to the cannula via an integrated sensing mechanism. In some embodiments, a user manually completes docking after the plurality of robotic arms are moved to be in respective starting positions.

[0181] In some embodiments, the arm is moved from a stowed position (e.g., under a table used for a medical procedure) to a starting position. In some embodiments, the robotic arm is initially in a stowed position (e.g., under a table used for a medical procedure) of the robotic medical system. Figure 6In some embodiments, the robotic arm is initially in a stowed position adjacent to the patient support platform (e.g., as shown in FIG. Fig.16 In some embodiments, the arm is deployed in an initial dangling position so that it can be dangling to maintain sterile boundaries. In some embodiments, the system moves the robotic arm from the dangling position to a corresponding starting position (e.g., based on the 3-D scan data and surgical procedure information).

[0182] In some embodiments, a robotic medical system includes: (i) multiple robotic arms; (ii) one or more processors; and (iii) a memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the following operations: (a) obtain imaging data of a patient; (b) obtain information about a medical procedure for the patient; (c) for each of a plurality of starting states, simulate the medical procedure from the starting state to identify a number of collisions with an arm of a plurality of robotic arms, the simulation using the imaging data and the information about the medical procedure; (d) select a first starting state of the plurality of starting states that has the least number of collisions, wherein the first starting state identifies a corresponding starting pose of each of the plurality of robotic arms; and (e) provide pose information of the first starting state to an operator of the robotic medical system.

[0183] In some embodiments, the robotic medical system includes an adjustable rod to which at least a subset of the arms are coupled. In some embodiments, each robotic arm is coupled to an instrument for performing a medical procedure on a patient. In some embodiments, the robotic medical system includes multiple robotic arms (e.g., four, five, or six arms) to provide greater flexibility to the surgeon (e.g., requiring careful setup and positioning to avoid intraoperative collisions while still being able to reach target anatomical structures).

[0184] In some embodiments, the robotic medical system displays one or more graphical representations indicating posture information on a display, or provides electrical signals to a display for displaying posture information (e.g., in VR or AR display).

[0185] In some embodiments, the memory also includes instructions that, when executed by one or more processors, cause the one or more processors to obtain an anatomical target for a medical procedure. The simulation further uses the anatomical target. In some embodiments, the memory also includes instructions that, when executed by one or more processors, cause the one or more processors to use the patient's image data to identify the anatomical target. In some embodiments, the memory also includes instructions that, when executed by one or more processors, cause the one or more processors to obtain imaging data for the environment surrounding the robotic medical system, wherein the simulation further uses the imaging data for the environment. In some embodiments, the memory also includes instructions that, when executed by one or more processors, cause the one or more processors to obtain position information of one or more objects near the robotic medical system, wherein the simulation further uses the position information.

[0186] In some embodiments, the memory further comprises instructions that, when executed by the one or more processors, cause the one or more processors to select a recommended port location based on the simulation. In some embodiments, each of the plurality of robotic arms has a plurality of joints, and for each of the plurality of robotic arms, the first starting state comprises positioning information for each of the plurality of joints. In some embodiments, the memory further comprises instructions that, when executed by the one or more processors, cause the one or more processors to move at least one of the plurality of robotic arms to a corresponding starting pose.

[0187] 3. Implement systems and terminology .

[0188] Fig.24 2 is a schematic diagram of electronic components of an example system 279 (e.g., a robotic medical system) according to some embodiments. In some embodiments, system 279 is an example of one of systems 10, 36, 47, 100, and 140A. System 279 includes one or more processors 380, which store memory for performing any of the methods described herein (e.g., regarding Fig.19 , Fig.23A and Fig. 23B The computer-readable storage medium (e.g., a computer memory device such as a random access memory, a read-only memory, a static random access memory, and a non-volatile memory, as well as other storage devices such as a hard drive, an optical disk, a magnetic tape record, or any combination thereof) of the memory 382 that stores instructions for performing the operations described herein.

[0189] The one or more processors 380 also communicate with an input / output controller 384 (via a system bus or any suitable circuitry). The input / output controller 384 receives sensor data from one or more sensors 388-1, 388-2, etc., and relays the sensor data to the one or more processors 380. The input / output controller 384 also receives instructions and / or data from the one or more processors 380, and relays the instructions and / or data to one or more actuators, such as the first motors 387-1 and 387-2, etc. In some embodiments, the input / output controller 384 is coupled to one or more actuator controllers 386, and provides instructions and / or data to at least a subset of the one or more actuator controllers 386, which in turn provides control signals to selected actuators. In some embodiments, the one or more actuator controllers 386 are integrated with the input / output controller 384, and the input / output controller 384 provides control signals directly to the one or more actuators 387 (in the absence of a separate actuator controller). Although Fig.24 There is shown to be one actuator controller 386 (e.g., one actuator controller for the entire mobile medical platform), but in other embodiments, additional actuator controllers are used (e.g., one actuator controller for each actuator, etc.). In some embodiments, the one or more processors 380 communicate with the one or more displays 381 to display information as described herein (e.g., recommended robot arm poses and port locations).

[0190] It should be noted that, as used herein, the term "coupled" or other variations of the word coupled may indicate an indirect connection or a direct connection. For example, if a first component is "coupled" to a second component, the first component may be indirectly connected to the second component via another component or directly connected to the second component.

[0191] The term "computer-readable medium" refers to any available medium that can be accessed by a computer or processor. By way of example and not limitation, such a medium may include a random access memory (RAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a compact disk read-only memory (CD-ROM) or other optical disk storage device, a magnetic disk storage device or other magnetic storage device, or any other medium that can be used to store the desired program code in the form of an instruction or data structure and can be accessed by a computer. It should be noted that a computer-readable medium may be tangible and non-transient. As used herein, the term "code" may refer to software, instructions, code or data that can be executed by a computing device or processor.

[0192] The method disclosed herein includes one or more steps or actions for implementing the method. The method steps and / or actions may be interchangeable with each other without departing from the scope of the claims. In other words, unless the correct operation of the method being described requires a specific order of steps or actions, the order and / or use of specific steps and / or actions may be modified without departing from the scope of the claims.

[0193] As used herein, the term "plurality" means two or more. For example, a plurality of components indicates two or more components. The term "determine" encompasses a variety of actions, and thus, "determine" may include calculating, computing, processing, deriving, investigating, searching (e.g., searching in a table, a database, or another data structure), ascertaining, etc. Additionally, "determine" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), etc. Additionally, "determine" may include parsing, selecting, choosing, establishing, etc.

[0194] Unless explicitly stated otherwise, the phrase “based on” does not mean “based only on.” In other words, the phrase “based on” describes both “based only on” and “based at least on.”

[0195] The foregoing description of the disclosed specific implementations is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these specific implementations will be apparent to those skilled in the art, and the general principles defined herein may be applied to other specific implementations without departing from the scope of the present invention. For example, it should be understood that one of ordinary skill in the art will be able to adopt a plurality of corresponding alternatives and equivalent structural details, such as equivalent ways of fastening, mounting, coupling or engaging tool parts, equivalent mechanisms for producing specific actuating motions, and equivalent mechanisms for delivering electrical energy. Therefore, the present invention is not intended to be limited to the specific implementations shown herein, but is given the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A robotic medical system, include: multiple robotic arms; one or more processors; and a memory storing instructions that, when executed by the one or more processors, cause the one or more processors to: Obtaining imaging data of the patient; obtaining information regarding medical procedures for said patient; for each starting state of a plurality of starting states, simulating the medical protocol from the starting state to identify a number of collisions with arms of the plurality of robotic arms, the simulating using the imaging data and the information about the medical protocol; selecting a first starting state among the plurality of starting states, the first starting state having a minimum number of collisions, wherein the first starting state identifies a respective starting pose of each of the plurality of robot arms; and The posture information of the first starting state is provided to an operator of the robotic medical system.

2. The robotic medical system of claim 1, wherein the memory further comprises instructions which, when executed by the one or more processors, cause the one or more processors to obtain an anatomical target of the medical procedure, wherein the simulation further uses the anatomical target.

3. The robotic medical system of claim 2, wherein the memory further comprises instructions that, when executed by the one or more processors, cause the one or more processors to use the image data of the patient to identify the anatomical target.

4. The robotic medical system according to any one of claims 1 to 3, wherein the simulation include: plotting a trajectory of each of the plurality of robotic arms during the medical procedure; as well as A collision is identified based on the trajectory.

5. The robotic medical system according to any one of claims 1 to 4, wherein the memory further comprises instructions which, when executed by the one or more processors, cause the one or more processors to select a recommended port location based on the simulation.

6. The robotic medical system according to any one of claims 1 to 5, wherein each of the plurality of robotic arms has a plurality of joints, and wherein for each of the plurality of robotic arms, the first starting state includes positioning information of each of the plurality of joints.

7. The robotic medical system according to any one of claims 1 to 6, wherein the memory further comprises instructions which, when executed by the one or more processors, cause the one or more processors to obtain imaging data for a surrounding environment of the robotic medical system, wherein the simulation further uses the imaging data for the environment.

8. The robotic medical system according to any one of claims 1 to 7, wherein the memory further comprises instructions which, when executed by the one or more processors, cause the one or more processors to obtain position information of one or more objects near the robotic medical system, wherein the simulation further uses the position information.

9. The robotic medical system according to any one of claims 1 to 8, wherein the medical procedure requires a medical instrument, and wherein the pose information includes information about which arm of the plurality of robotic arms the medical instrument should be mounted to.

10. The robotic medical system according to any one of claims 1 to 9, wherein the memory further comprises instructions which, when executed by the one or more processors, cause the one or more processors to move at least one of the plurality of robotic arms to be in the corresponding starting position.

11. The robotic medical system according to any one of claims 1 to 10, wherein the imaging data of the patient comprises an image of the patient after the patient has been prepared for the medical procedure.

12. A method for determining a starting state of a robot system comprising a plurality of robot arms, the method include: Obtaining imaging data of the patient; obtain information about the patient's medical procedures; for each starting state of a plurality of starting states, simulating the medical protocol from the starting state to identify a number of collisions with arms of the plurality of robotic arms, the simulating using the imaging data and the information about the medical protocol; selecting a first starting state among the plurality of starting states, the first starting state having a minimum number of collisions, wherein the first starting state identifies a respective starting pose of each of the plurality of robot arms; and The pose information of the first starting state is provided to an operator of the robot system.

13. The method of claim 12, further comprising obtaining an anatomical target for the medical procedure, wherein the simulation further uses the anatomical target.

14. The method according to any one of claims 12 to 13, wherein the simulation include: plotting a trajectory of each of the plurality of robotic arms during the medical procedure; as well as A collision is identified based on the trajectory.

15. The method of any one of claims 12 to 14, further comprising selecting a recommended port location based on the simulation.

16. The method of any one of claims 12 to 15, further comprising obtaining imaging data for a surrounding environment of the robotic system, wherein the simulation further uses the imaging data for the surrounding environment.

17. The method according to any one of claims 12 to 16, further comprising obtaining position information of one or more objects in the vicinity of the robotic system, wherein the simulation further uses the position information.

18. The method of any one of claims 12 to 17, wherein the medical procedure requires a medical device, and wherein the pose information includes information regarding to which arm of the plurality of robotic arms the medical device should be mounted.

19. The method of any one of claims 12 to 18, further comprising moving at least one of the plurality of robotic arms into the respective starting pose.

20. The method of any one of claims 12 to 19, wherein the imaging data of the patient comprises images of the patient after the patient has been prepared for the medical procedure.

Images