System and method for generating workspace geometry of instrument

By generating three-dimensional models of internal and external workspaces, the internal geometry is determined to define the accessible volume, solving the problem of contact between the instrument and the manipulator arm in minimally invasive medical operations, improving operational safety and efficiency.

CN119014982APending Publication Date: 2024-11-26INTUITIVE SURGICAL OPERATIONS INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410592924.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-05-23
Filing Date
2024-05-14
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

In minimally invasive medical operations, undesired contact between the device and the manipulator arm leads to problems with extended medical procedures and inaccurate insertion positions, making it difficult to determine whether the device is in contact with a patient or an obstacle, affecting the visibility and safety of the operation .

Method used

By generating three-dimensional models of internal and external workspaces, internal geometry is determined to define the reachable volume and provide output related to medical protocol execution, avoiding undesired contact and optimizing port placement to increase the reachable volume of internal workspaces.

Benefits of technology

It effectively avoids contact between the instrument and the manipulator arm, improves the safety and efficiency of medical operations, and ensures the correct positioning and operating range of the instrument in the patient's body.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119014982A_ABST
    Figure CN119014982A_ABST
Patent Text Reader

Abstract

Systems and methods for generating a workspace geometry of an instrument. Systems described herein may include an instrument configured to be inserted at least partially from an outer workspace through a body wall into an inner workspace within a body; and a controller configured to generate a first three-dimensional model of the internal workspace, a first portion of the instrument being inserted into the internal workspace during execution of a medical procedure using the instrument, generate a second three-dimensional model of the external workspace, where a second portion of the instrument is located in the external workspace during execution of the medical procedure, and generate a second three-dimensional model of the external workspace. The method further includes determining, based on the first three-dimensional model and the second three-dimensional model, an internal geometry within the internal workspace, the internal geometry defining a reachable volume within the internal workspace within which the instrument is positionable, and providing an output related to execution of the medical procedure based on the determined internal geometry.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 63 / 503,803, filed on May 23, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to systems and methods for performing procedures using instruments without direct line of sight to one or more workspaces in which instruments or manipulators are deployed to perform the procedures. More specifically, aspects of the present disclosure relate to systems, methods, and devices (such as instruments for performing minimally invasive medical procedures) for determining and / or generating workspace geometry to aid in planning such procedures. Background Art

[0004] The benefits of minimally invasive medical procedures are well known, and when compared with traditional open incision surgery, they include less patient trauma, less blood loss and faster recovery time. In minimally invasive medical procedures, one or more incisions can be made in the patient's body wall, wherein one or more instruments are inserted through these incisions directly or via one or more other guide devices (such as ports, cannulas, etc.). In some cases, all instruments used pass through a single incision in the patient's body wall, or in some cases through a single natural orifice of the patient's body. In other cases, multiple incisions are made in the patient's body wall, multiple natural orifices are used, or a combination thereof, wherein different instruments are inserted in different incisions / orifices. In addition, minimally invasive medical procedures can be performed using instruments controlled at least in part by a computer-assisted system (sometimes referred to as a robotic system and its arrangement) employing robotics, instruments driven manually, or a combination of the two. In any of the above situations, it is possible for the various instruments themselves and / or components of the manipulator systems to which they are operably coupled in a computer-assisted system to contact, collide with, or otherwise interfere with each other, such as at an interior space where instruments are inserted through a body wall and / or from the outside due to the need to move instruments and / or components of the manipulator system to adjust the positioning and orientation of the instruments internally during a medical procedure. Such potential interferences limit the range of internal reach of the instruments available to the surgeon. In addition, it may be difficult for the surgeon (or other personnel controlling the instruments to perform the medical procedure) to know whether one or more instruments and / or multiple manipulators (in the case of a computer-assisted procedure) are at risk of contact with (such as collision with or other undesirable contact with) the patient or other obstacles in the patient's surroundings.

[0005] In addition, depending on the medical procedures being performed and the patient's body size, there are changes in the placement of one or more incisions and / or natural orifices for inserting one or more instruments, and therefore there are changes in the placement of one or more ports placed at those incisions. If the placement of the position where the instrument will be inserted on the patient's body is not correctly planned, the medical procedures may be affected. For example, placing two ports at specific positions relative to each other may increase the possibility of undesirable contact (between each other, between the instrument and the patient, and / or between the instrument and an external object) of the instruments inserted at these positions, and / or may reduce the accessible volume of the instrument in the internal working space where the procedures of interest are performed. In the case of using a robotic medical system (having multiple manipulator arms connected to the instrument and providing drive input to control the movement and other functions of the instrument) to perform medical procedures, the manipulator arms may be prone to contact (e.g., collision) in a similar manner. Undesirable or unintentional contact can extend the time of the medical procedure and / or cause the reinsertion of the incision / port placement and the instrument to be re-performed.

[0006] Improvements related to the planning and execution of medical procedures are needed to address one or more of the problems set forth above, as well as other problems that will be apparent from the description that follows. Summary of the invention

[0007] Various aspects of the present disclosure may address one or more of the above problems and / or may exhibit one or more of the above desirable features.Other features and / or advantages may become apparent from the following description.

[0008] In one aspect of the present disclosure, a computer-assisted system is provided, comprising: an instrument configured to be inserted at least partially through a body wall from an external working space into an internal working space within a body; and a controller configured to: generate a first three-dimensional model of the internal working space, a first portion of the instrument being inserted into the internal working space during use of the instrument to perform a medical procedure, generate a second three-dimensional model of the external working space, wherein during the execution of the medical procedure, a second portion of the instrument is located in the external working space, determine an internal geometry within the internal working space based on the first three-dimensional model and the second three-dimensional model, the internal geometry defining a reachable volume within the internal working space, the instrument being positionable within the reachable volume, and provide output related to the execution of the medical procedure based on the determined internal geometry.

[0009] In another aspect of the present disclosure, a computer-implemented method is provided, comprising: generating a first three-dimensional model of an internal working space of a body, a first portion of an instrument operably coupled to a manipulator arm being inserted into the internal working space during use of the instrument to perform a procedure, the manipulator arm being configured to transmit a driving force to the instrument; generating a second three-dimensional model of an external working space, wherein a second portion of the instrument is located in the external working space during the performance of the procedure; determining an internal geometry within the internal working space based on the first three-dimensional model and the second three-dimensional model, the internal geometry defining a reachable volume within the internal working space, the instrument being positionable within the reachable volume by the manipulator arm; and providing output related to the performance of the procedure based on the determined internal geometry. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The present disclosure may be understood by the following detailed description alone or in conjunction with the accompanying drawings. The accompanying drawings are included to provide a further understanding of the present disclosure and are incorporated into and constitute a part of this specification. The accompanying drawings illustrate one or more aspects of the present teachings and together with the specification explain certain principles and operations. In the drawings:

[0011] Figure 1 shows a diagram of an environment according to various embodiments;

[0012] FIG. 2A to FIG. 2B views showing components of a robotic surgical system according to various embodiments, respectively;

[0013] Figure 3 shows a block diagram of an exemplary surgical system according to various embodiments;

[0014] Figure 4 shows an exemplary process flow according to various embodiments;

[0015] FIG. 5A to FIG. 5B each showing a view of an exemplary surgical system configuration according to various embodiments;

[0016] Figure 6 Another exemplary process flow according to various embodiments is shown;

[0017] Figure 7 illustrates exemplary image views according to various embodiments;

[0018] Figure 8 shows yet another exemplary process flow according to various embodiments; and

[0019] Fig. 9 Exemplary user interface display screens are shown in accordance with various embodiments. DETAILED DESCRIPTION

[0020] As described above, in minimally invasive medical procedures, whether performed manually (e.g., laparoscopically), via a computer-assisted surgical system utilizing robotics, or a combination of the two, the use of instruments inserted through one or more ports may result in the risk of undesired contact, such as, but not limited to, between multiple instruments, between manipulator arms coupled to such instruments, and / or between instruments and another object (including, for example, a patient or a person) outside the body into which the instrument is inserted. For medical personnel such as surgeons or other auxiliary personnel, it may be difficult to know whether the risk of undesired contact as described above is occurring or is about to occur. Even in the case where feedback (such as tactile feedback or other feedback) is provided to notify of undesired contact, such feedback is typically provided via a channel that also transmits other information unrelated to such contact, which may make the feedback unclear. In addition, particularly in the case of a computer-assisted surgical system, external contact with the body, a person, or other objects that can provide softer contact may not be sufficient to be perceived by a person controlling the manipulation of one or more instruments to perform a medical procedure. In addition, the reduced visibility caused by surgical drapes, external objects that impair vision, body bending, etc. may make it difficult to observe (such as via auxiliary personnel) the occurrence or potential occurrence of contact between various objects outside the patient's body.

[0021] Thus, various embodiments herein provide the surgeon with the ability to understand the location of boundaries within an internal working space that may result in external collisions or other defined contacts that may be desirable to avoid if the surgeon attempts to reach an area beyond the boundaries.

[0022] Furthermore, where multiple ports are used, there is variation in how the ports are placed. In some cases, port placement may impact the medical procedure itself, such as causing contact with the arm and / or failure to reach certain portions of the medical workspace. Therefore, there is a need for systems, devices, and methods for determining internal reach geometry that will result in external contact and / or for determining and generating port placement guidance outputs.

[0023] Therefore, various embodiments herein also provide the surgeon with the ability to place ports in a configuration that increases the size of the accessible volume within the internal working space while reducing the possibility of potentially undesirable external contact. Ports can be planned and placed in an iterative manner to ensure that the accessible volume is sufficient for the procedure when using the minimum number of ports. Although the following description is presented with reference to a single cart (manipulator system) multi-port system, this is only for the purpose of explanation and not for the purpose of limitation. The methods, operations, and embodiments described herein may also be applied to a single cart single port system, a multi-cart multi-port system, a single port system mounted on an operating table, a multi-port system mounted on an operating table, a single port system mounted on a wall, a multi-port system mounted on a wall, a single port system mounted on a ceiling, a multi-port system mounted on a ceiling, and the like.

[0024] Various embodiments may also provide the ability to update the internal reach geometry based on changing conditions during a medical procedure, such that the external workspace and internal workspace are dynamically updated and mapped to provide updated internal reach geometry throughout the medical procedure.

[0025] In the present context, the terms "inside" and "outside" are used relative to the body, such that the external working space may at least partially surround the body and the internal working space, and may correspond to the surrounding environment from which instruments are inserted into the internal working space and the instruments are controlled. In the context of medical procedures, the boundary between the internal working space and the external working space may correspond to or include an outer surface portion of the patient's body, such as a body wall through which instruments are inserted, for example, through a port or a natural orifice of the body. In a non-medical context, the boundary between the internal working space and the external working space may correspond to the body of the object, such as but not limited to an opening of a conduit, an opening of a well, etc., through which instruments are inserted to perform non-medical procedures (such as inspection, maintenance, etc.), and may be used under the same operating principles as described herein in the context of medical procedures. As used herein, a "medical procedure" may refer to a diagnostic procedure, a surgical procedure, a therapeutic procedure, etc.

[0026] Reference now Figure 1 , a schematic diagram of a system for performing minimally invasive medical procedures deployed within a medical procedure environment 100 is shown and will be described. Figure 1In the present invention, a system includes an external image sensing system including one or more external image sensors 102 (which may be mounted to, for example, a manipulator assembly, a ceiling, a wall, or any one or more of a plurality of locations in an external environment in which a patient is located, for performing a medical procedure) positioned to provide an image of an external working space 122, and an internal image sensing system (e.g., including one or more endoscopic instruments 114) positioned to provide an image of an internal working space 124. The external image sensors 102 are associated with the external image sensor reference frame 104 and have a field of view 106. The external image sensing system including the one or more external image sensors 102 may provide an external image of at least a portion of a patient's body through which at least a portion of a medical instrument 112 and an endoscopic instrument 114 are inserted to perform a medical procedure in the internal working space 124, wherein uninserted portions of the instruments 112, 114 (or the cannulas through which they are inserted) are visible in the external working space image. The instrument 112 includes an end effector that can be controlled to perform the desired function of the instrument 112 (e.g., cutting, grasping, suturing, sensing, applying electrical energy, etc.) and a wrist that can be operated to control the movement and / or orientation of the end effector. In embodiments where the external image sensor 102 is located on the arm of the manipulator assembly, the external image provides a self-centered view from the manipulator assembly and can provide an effective view for tracking various external objects (including external portions of the medical instrument 112 and / or the endoscopic instrument 114). Figure 1 As shown, an internal image sensor 116 disposed at the distal end of an endoscopic instrument 114 is positioned to provide an image of an internal working space 124 using an internal image sensor reference frame 118 and has a field of view 120. The internal image sensor 116 can provide an internal image of the interior of a patient's body at least partially surrounded by the patient's body wall 110 through which the instruments 112, 114 are inserted to perform a medical procedure at the working site.

[0027] As described above, one or more medical instruments 112 are inserted through an incision in the patient's body wall 110 to access the internal working space 124 in which it is desired to perform one or more medical procedures. In some procedures, the instrument 112 is inserted through a guide device (such as a cannula, a port body, a trocar, or a combination thereof) that is first inserted into the incision and provides a pathway for inserting the instrument from the external working space 122 that at least partially surrounds the body wall 110 into the internal working space 124. In order to provide visualization of the internal working space 124 to the medical personnel performing the procedure, an internal image sensor 116 (which may be part of an instrument (such as an endoscope 114)) is also inserted into the internal working space 124 through the same or another incision in the body wall 110, and an image of the internal working space 124 is relayed to a display 132 (see also FIG. 1 ). Figure 2A and Figure 2B , described in more detail below), which is visible to medical personnel and is operably coupled to the endoscope 114 to receive internal image data. Throughout the disclosure, the term "port" and variations thereof are used to broadly describe a location on the body wall 110 or a natural orifice of the patient's body through which an instrument 112 / 114 can be inserted, which may include directly through an incision or natural orifice, or through a guide device inserted into the incision or natural orifice, wherein the instrument is inserted through the guide device. A control system 134 is also provided (which may be described below with reference to Figure 3 102 , 116 ), the control system includes a processor and communicates with an imaging sensing system including image sensors 102 , 116 to receive image data transmitted by those systems.

[0028] Figure 1 The system can be implemented for use in a manual medical system, a computer-assisted (robotic) medical system, or a combination of both types of systems. For example, in a manual medical system, the medical instrument 112 and the endoscopic instrument 114 (and Figure 1 One or both of one or more additional instruments (not shown) such as by a posterior end ( Figure 1 In a computer-assisted medical system that relies in part on robotics, medical instrument 112 and endoscopic instrument 114 (and Figure 1 One or both of the one or more additional instruments (not shown) may be operated using robotics, wherein at the rear end ( Figure 1 A driving input force (i.e., actuating force) is provided at a force transmission mechanism having various inputs at a proximal end portion (not shown in the figure), the force transmission mechanism being configured to be operably coupled to a manipulator interface of a manipulation system and to receive a driving force output therefrom.

[0029] refer to Figure 2A and Figure 2B The schematic diagram shows an example and implementation of a computer-assisted robotic medical system.

[0030] like Figure 2A and Figure 2B As shown, a computer-assisted robotic medical system 200 according to various embodiments may include a manipulation system 202, a user control system 204, and an auxiliary system 206 that are communicatively coupled to each other. The medical system 200 may be used by a medical team to perform a computer-assisted medical procedure ( Figure 2A). As shown, the medical team may include one or more of a surgeon 210-1, an assistant 210-2, a nurse 210-3, and an anesthesiologist 210-4, all of which may be collectively referred to as "medical team members 210." Additional or alternative medical team members may be present during a medical procedure.

[0031] As used herein, a medical procedure may include any of a surgical procedure, a diagnostic procedure, a therapeutic procedure, a sensing procedure, etc., and a medical procedure includes not only an intraoperative phase of a procedure, but may also include a preoperative phase (which may include the setup of a medical system), a postoperative phase, and / or other suitable phases of a medical procedure. In some embodiments, one or more of the preoperative, intraoperative, postoperative, and / or other phases of a medical procedure may include one or more segments. For example, different segments of the intraoperative phase of a medical procedure may be associated with the use of different instruments.

[0032] like Figure 2A As shown, the manipulation system 202 may include a plurality of manipulator arms 212 (e.g., manipulator arms 212-1 to 212-4) to which a plurality of medical instruments 112 may be operably coupled. Each instrument 112 may be configured with a specific function (e.g., a tool having a tissue interaction function, such as but not limited to cutting, suturing, ablation, sealing, etc.), an imaging device (e.g., an endoscope, an ultrasound tool, etc.), a sensing instrument (e.g., force sensing, pressure sensing, etc.), a diagnostic instrument, etc., which may be used for computer-assisted medical procedures on a patient 208 (e.g., by being at least partially inserted into a patient 208 and being manipulated by the manipulation system 202 to perform a computer-assisted medical procedure on the patient 208). Although the manipulation system 202 is depicted and described herein as including four manipulator arms 212, it should be appreciated that the manipulation system 202 may include only a single manipulator arm 212 or any other number of manipulator arms that may serve a particular embodiment. Furthermore, in some embodiments, a combination of instruments 112 operated by manipulator arms 212 and manually operated instruments may be utilized. For example, Figure 2A It is shown that a patient is being operated on by four medical instruments 112 operably connected to four manipulator arms 212-1 to 212-4, while an assistant 210-2 manually operates an endoscope. In other examples, the endoscope can be operated by one of the manipulator arms 212, while the assistant 210-2 (or multiple assistants) operates one or more medical instruments. In still other examples, as described above, all medical instruments and endoscopes can be manually operated.

[0033] The manipulator arm 212 and / or an instrument attached to the manipulator arm 212 may include one or more displacement transducers, orientation sensors, and / or position sensors for generating raw (i.e., uncorrected) kinematic information. One or more components of the system 200 may be configured to use the kinematic information to track the instrument (e.g., determine its posture) and / or control the instrument and any object connected to the instrument and / or manipulator arm. The manipulator arm 212 may include a kinematic series of links connected by joints, wherein the instrument is coupled to a distal link in the kinematic series. The individual joints and kinematic links may be powered or non-powered, and collectively may provide a variety of different motions to orient the instrument in an internal workspace, wherein portions of the kinematic structure of the manipulator and a proximal portion of the instrument (including a force transmission mechanism operably coupled to receive drive input from an interface of the manipulator arm) move in an external workspace. Although Figure 2A and Figure 2B The manipulator arm 212 is shown attached to the center column 222 of the manipulator system 202, but in other embodiments, the manipulator arm 212 may be attached to the ceiling or wall of the operating room, to an operating table that also supports the patient 208, to another mounting element, or other suitable attachment location.

[0034] exist Figure 2B In the detailed view shown, the operating system 202 is shown with external image sensors 220 (image sensors 220-1 to 220-4) attached to components of the operating system 202. The external image sensors 220 may be connected to Figure 1 202 and may have the same or similar structure. As shown, image sensor 220-1 may be attached to orientation platform 224 of manipulator system 202, image sensor 220-2 may be attached to manipulator arm 212-1 of manipulator system 202, image sensor 220-3 may be attached to manipulator arm 212-4 of manipulator system 202, and image sensor 220-4 may be attached to base 226 of manipulator system 202. In embodiments where manipulator system 202 is positioned close to a patient (e.g., as a patient-side cart), placing image sensors 220 at strategic locations on manipulator system 202 provides different imaging viewpoints close to the patient and the medical procedure being performed on the patient.

[0035] In an embodiment (e.g., using Figure 2BIn embodiments where the external image sensor 220 is a single-cart, multi-port system or one of the other system architectures described above, the one or more external image sensors 220 may be mounted at other locations, such as to a stand, wall, ceiling, etc. In embodiments, the one or more external image sensors 220 may be removable devices that can be temporarily attached (e.g., to a cart, to other locations, or to a manipulator arm 212). In other embodiments, the one or more external image sensors 220 may be handheld. In such embodiments, the external image sensor 220 may be installed and / or operated only when a user desires to scan or rescan a patient, the external workspace 122, or both. Additionally, in a single-port system, embodiments may be somewhat less complex in terms of the boundaries to be considered and displayed, as well as the overall range of motion of the instrument and / or manipulator in the external workspace.

[0036] The user control system 204 may be configured to facilitate the surgeon 210-1 to control the manipulator arms 212 and the medical instruments 112 attached to the manipulator arms 212. For example, the surgeon 210-1 may interact with the user control system 204 to remotely move or manipulate the manipulator arms 212 and, thereby, remotely move or manipulate the instruments 112 operably coupled to the drive outputs of the manipulator arms 212. The user control system 204 may also provide the surgeon 210-1 with an internal workspace associated with the patient 208 (e.g., an endoscope 114 that may be mounted to one of the manipulator arms or manually held) as captured by an internal imaging system (e.g., an endoscope 114 that may be mounted to one of the manipulator arms or manually held) Figure 1 In some examples, the user control system 204 may include a stereoscopic display (eg, a 3D image) having two displays. Figure 2B 240), where surgeon 210-1 can view a stereoscopic image of a surgical site associated with patient 208 and generated by a stereoscopic imaging system. Surgeon 210-1 can utilize the image displayed by user control system 204 to perform one or more procedures with one or more medical instruments 112 attached to manipulator arm 212.

[0037] To facilitate control of the mounted instrument 112, the user control system 204 may include a set of input controls 242 (eg, a left input control 242-1 and a right input control 242-2), such as Figure 2B1. These input controls 242 can be manipulated by the surgeon 210-1 to control the movement of the instrument 112 (e.g., by utilizing robotic and / or teleoperation technology). The input controls 242 can be configured to detect various hand, wrist, and finger movements of the surgeon 210-1. In this way, the surgeon 210-1 can use one or more surgical instruments to intuitively perform procedures. The user control system 204 also includes feedback mechanisms 244 (e.g., left feedback mechanism 244-1 and right feedback mechanism 244-2) that can be equipped to provide tactile feedback to the surgeon 210-1.

[0038] The auxiliary system 206 may include one or more computing devices configured to perform processing operations of the medical system 200. In such a configuration, the one or more computing devices included in the auxiliary system 206 may control and / or coordinate operations performed by various other components of the surgical system (e.g., the manipulation system 202 and the user control system 204). For example, a computing device included in the user control system 204 may transmit instructions to the manipulation system 202 through the one or more computing devices included in the auxiliary system 206 based on input at the input control 242 of the user control system 204. As another example, the auxiliary system 206 may receive and process image data representing images captured by one or more external and / or internal image sensors.

[0039] In some examples, the assistance system 206 may be configured to present visual content, such as for access by a medical team member 210 who may not have access to an image provided to the surgeon 210-1 at the display 240 of the user control system 204. To this end, the assistance system 206 may include a display monitor 214 configured to display one or more user interfaces, such as an image of an internal workspace, information associated with the patient 208 and / or a surgical procedure, and / or any other visual content that may serve a particular embodiment. For example, the display monitor 214 may display an image of the internal workspace and additional content (e.g., graphical content, contextual information, etc.) displayed simultaneously with the image. In some embodiments, the display monitor 214 is implemented by a touch screen display with which the team member 210 may interact (e.g., via touch gestures) to provide user input to the medical system 200.

[0040] like Figure 2BAs shown, the auxiliary system 206 also includes a central electronic data processing unit 232 and a visual device 234. The central electronic data processing unit 232 includes circuits, devices and / or other components to implement some or all of the data processing operations used to operate the surgical system, including but not limited to those operations described in more detail below. However, in other embodiments, at least a portion of the data processing operations may be distributed in the user control system 204 and / or the manipulation system 202. The visual device 234 may include a stereo endoscope 114 (see Figure 1 ) for the left image capture function and the right image capture function. Vision equipment 234 may also include lighting equipment (e.g., a xenon lamp) that provides illumination for imaging the internal working space and / or external image capture equipment (e.g., an image sensor similar to or the same as image sensor 220) that captures images of the external working space where the surgical site is located. Figure 2B As shown, auxiliary system 206 includes a monitor 214, however, in actual implementations, monitor 214 may be mounted elsewhere, such as on manipulator system 202 or on a wall or ceiling of an operating room.

[0041] The operating system 202, the user control system 204, and the auxiliary system 206 may be communicatively coupled to each other in any suitable manner. Figure 2A and Figure 2B As shown, the operating system 202, the user control system 204, and the auxiliary system 206 can be communicatively coupled by means of a control line 216, which can represent any wired or wireless communication link that can serve a particular implementation. To this end, the operating system 202, the user control system 204, and the auxiliary system 206 can each include one or more wired or wireless communication interfaces, such as one or more local area network interfaces, Wi-Fi network interfaces, cellular interfaces, etc.

[0042] In some examples, an imaging device (such as an external image sensor 102) may be attached to other components of the surgical system and / or components of the surgical facility in which the surgical system is disposed. For example, the image sensor 220 may be attached to a component other than the manipulation system 202. Thus, after a one-time calibration has been performed, the system may use the kinematic information of the components of the manipulation system 202 to derive the kinematic information of the attached image sensor to identify the relationship between the tracking kinematics of the components of the manipulation system 202 and the image sensor attached to the components of the manipulation system 202. Although the above description relates to an example in which the internal image sensor 116 is a stereo image sensor and is part of the endoscope 114, in embodiments, the internal image sensor 116 may take any form capable of determining a three-dimensional image. For example, the internal image sensor 116 may be, but is not limited to, a stereo image sensor, a parallax image sensor, a time of flight (TOF) sensor, a light detection and ranging (LiDAR) sensor, a structured light camera, and the like. Additionally or alternatively, in some embodiments, external scanning or imaging equipment (including but not limited to a 3D x-ray imaging system) may be used to obtain internal images. In addition, although the above description shows the external image sensors 102 and 220 as a single sensor for capturing a two-dimensional image, in embodiments, the external image sensors 102 and / or 220 may take any form or combination of forms that can collectively determine a three-dimensional image. For example, any of the external image sensors 102 / 220 may be, but are not limited to, one or more separate complementary metal oxide semiconductor (CMOS) sensors, charge coupled devices (CCDs), stereo image sensors, parallax image sensors, TOF sensors, LiDAR sensors, structured light sensors, infrared (IR) sensors, sonar sensors, radar sensors, touch probes (e.g., touch sensors mounted on manipulator arms), etc.

[0043] The medical system 200 may also include components configured to sense and / or detect various signals. For example, the medical system 200 may include one or more position or orientation sensors to determine the position, orientation, speed, acceleration, etc. of various components of the medical system 200 (such as the arm 212, the instrument, and / or the patient). The position or orientation sensor may include an accelerometer, a gravimeter, a gyroscope, a Hall sensor, a magnetometer, and the like. In some embodiments, an image sensor 102 / 220 may be used to perform position or orientation sensing, such as by detecting a mark on the instrument (e.g., a QR code), which may help to assist in aligning the external workspace with the internal workspace through vectorization of the instrument for use in various embodiments according to the present disclosure as described below. In an embodiment, the medical system 200 may use the systems and methods discussed in International Patent Application No. PCT / US2021 / 065444 filed on December 29, 2021 and now published as WO2022 / 147074 to perform alignment between the external workspace and the internal workspace, the entire contents of which are incorporated herein by reference.

[0044] Figure 3 Shown include Figure 1 components and, in the case of computer-assisted medical systems using robotics, Figure 2A and Figure 2B A block diagram of an embodiment of a medical system of components. Figure 3 The system shown can be used to perform various aspects of medical procedures related to utilizing three-dimensional representations and registration of internal and external workspaces, such as, but not limited to, port placement planning, defining internal workspace reach volumes, controlling instrument and / or manipulator system movement to avoid contact between objects, providing feedback to medical personnel to assist in completing one or more of the above, or various combinations thereof. As shown, system 300 includes an external imaging system 302 (which may, for example, correspond to external image sensor 102 and / or image sensor 220), an internal imaging system 304 (which may, for example, correspond to internal image sensor 116), at least one instrument 306, a control system 308, and an output system 310. Although Figure 3 Various systems and devices are shown as separate components, but this is for illustration purposes only, and in actual implementations, the various components may be integrated with each other. For example, the control system 308 and the output system 310 may be components of the same device.

[0045] The control system 308 is illustrated as including a processor 312 and a memory 314. The processor 312 may include components (e.g., circuits and circuit systems) configured to control other elements of the medical system, process and / or execute instructions received from the memory 314 or other sources, perform various method operations (including but not limited to those described herein), apply algorithms to analyze data, perform calculations and / or predictions, etc. In some examples, the processor 312 may be or include one or more electronic processing units, such as a central processing unit (CPU), an arithmetic logic unit (ALU), a floating point unit (FPU), and / or other microcontrollers. The memory 314 may include components (e.g., circuits and circuit systems) configured to store and / or receive information, including but not limited to computer-readable instructions that cause the control system 308 to perform various operations when executed by the processor 312. The memory 314 may be or include one or more storage elements, such as a random access memory (RAM), a read-only memory (ROM), an optical storage drive and / or disk, a magnetic storage drive and / or tape, a hard disk, a flash memory, a removable storage medium, etc. The control system 308 may include additional components, such as communication circuits configured to allow communication (i.e., transmission and reception) with other components and devices. The communication circuits may provide physical and / or virtual interfaces and communication ports for performing wired communications, wireless communications via radio transmissions, optical communications via optical fibers or free electromagnetic radiation, and the like. The communication circuits may also provide connections to peripheral devices, such as Universal Serial Bus (USB) devices. In embodiments, the output system 310 may be an input / output (I / O) system capable of receiving input from a user. Thus, the output system 310 may include components configured to allow interaction with a user, including, but not limited to, presenting information to the user (e.g., via a display for providing visual feedback, such as a display). Figure 1 The display 132 and / or FIG. 2A to FIG. 2B The output system 310 may include a monitor 214 for providing audio feedback, a speaker for providing tactile feedback, a tactile feedback device for providing tactile feedback, etc.) and / or receive information from a user (e.g., via a touch screen, microphone, camera or other gesture detection device, physical button, soft button, etc.). The output system 310 may include a user interface, which may include or generate one or more graphical user interfaces (GUIs) and associated elements (such as icons, menus, images, etc.). The user interface may be configured to report to the user the results of the calculations and / or predictions generated by the controller.

[0046] The processor 312 is programmed with instructions to generate various 3D models (e.g., three-dimensional models of the internal workspace 124 and the external workspace 122), determine various geometric structures based thereon, and provide outputs (e.g., via the output system 310). The processor 312 may also be configured to control and / or plan various aspects of a medical procedure based on the models and geometric structures, as will be discussed further below with respect to various embodiments.

[0047] In the corresponding Figure 1 In one example of environment 100 of FIG. 1 , processor 312 may be configured to generate a first three-dimensional model of internal workspace 124 (in which a first portion of instrument 112 is inserted during use of instrument 112 to perform a medical procedure) and to generate a second three-dimensional model of external workspace 122 (in which a second portion of instrument 112 is located during use of instrument 112 to perform a medical procedure). Processor 312 may also be configured to determine an internal geometry within the internal workspace based on the first three-dimensional model and the second three-dimensional model, the internal geometry defining an accessible volume within the internal workspace within which instrument 112 may be positioned, and to provide output related to the performance of the medical procedure based on the determined internal geometry.

[0048] Although various embodiments contemplate Figure 1 and FIG. 2A to FIG. 2B The systems and related components and related methods performed by these systems and as further described below may be implemented in the context of manually actuated instruments (e.g., in laparoscopic or endoscopic medical procedures using handheld and steerable instruments), but various embodiments herein contemplate Figure 1 and FIG. 2A to FIG. 2B The medical procedure and one or more medical instruments, imaging systems, and other components are implemented as part of a computer-assisted medical system using robotics.

[0049] Figure 4 An exemplary process flow that can be used, for example, to generate a reachable workspace geometry is shown. The reachable workspace geometry can refer to the total volume of the internal workspace that is reachable by one or more inserted instruments. In various embodiments, the reachable workspace geometry can be determined by applying different criteria, such as, but not limited to, one or more physical constraints on the ability of an inserted instrument to reach a given position within the internal workspace based on its insertion through the external workspace and the port location of the body wall, preventing one or more inserted instruments and / or manipulator arms from contacting each other or other objects in the external workspace (e.g., the patient's body, medical personnel, other medical equipment, sterile surgical drapes, and / or other environmental objects), and preventing one or more inserted instruments from contacting objects in the internal workspace (e.g., another inserted instrument, an implant, an internal organ, a bone structure, and / or a cartilage structure). Figure 4 The process flow may be executed by, for example, the control system 308. In some embodiments, Figure 4 The process flow of the embodiment of the present invention may be embodied in the form of instructions stored in a non-transitory computer-readable medium (e.g., memory 314), so that the control system 308 performs the following operations when executing the instructions: Figure 4 In other embodiments, Figure 4 The process flow can be embodied in the form of a carrier signal (for example, provided to the processor 312), so that the control system 308 executes when receiving the instruction Figure 4 For the purpose of explanation, Figure 4 The operation will be described as Figure 1 The environment is executed in / above 100.

[0050] Figure 4 The process flow of the invention includes an operation 402 of generating a first three-dimensional (3D) model of an internal working space of a body into which a first portion of an instrument is inserted during a procedure performed in the internal working space using the instrument. The instrument may be associated with Figure 1 The instrument 112 and / or endoscope 114 shown are the same or similar. Thus, the first portion of the instrument may correspond to the distal end portion of the instrument, which may include an end effector, a wrist, and a first portion of the shaft of the instrument located in the internal working space 124. The first 3D model may be generated based on image data (e.g., 3D image data) of the internal working space 124 corresponding to the reference frame 118 received by the control system 308. The image data of the internal working space may originate from a first imaging system located within the internal working space, such as from the image sensor 116 of the endoscope 114.

[0051] The process flow also includes an operation 404 of generating a second 3D model of the external workspace in which the second portion of the instrument is located during the execution of the procedure. The second portion of the instrument may correspond to a proximal portion of the instrument 112 and / or endoscope 114 extending from the body wall 110 into the external workspace 122, which may include a proximal portion of a shaft and a transmission mechanism, which in various embodiments may be manually actuated or configured to be operably coupled to a manipulator arm 212 in the computer-assisted medical system 200. In some embodiments, the instrument 112 and / or endoscope 114 may be inserted through a cannula, as is familiar to those of ordinary skill in the art, but for simplicity purposes, the ... Figure 1 The sleeve is omitted in the illustration and the following description.

[0052] The second 3D model may be generated based on image data (e.g., 3D image data) of the external workspace 122 received by the control system 308. Operation 404 may include identifying an outer boundary trajectory based on the image data of the external workspace 122, which may correspond to an outer portion of the body wall 110. The image data of the external workspace 122 may originate from a second imaging system (e.g., image sensors 102 and / or 220) that is located in the room (e.g., mounted to a wall or ceiling of an operating room, mounted on a manipulator system (such as manipulator system 202), mounted on an auxiliary system (such as auxiliary system 206), or located at any other location that can capture an image of a portion of the body wall 110 in which one or more instruments are inserted and a sufficiently large area of ​​interest surrounding the body wall 110 outside the patient). Those skilled in the art will appreciate that multiple instruments may be used in a procedure, and therefore, multiple first portions of multiple instruments may be located in the internal workspace, and multiple second portions of multiple instruments may be located in the external workspace.

[0053] The process flow includes an operation 406 of determining an internal geometry within the internal working space 124 that defines an accessible volume within the internal working space 124 within which the instrument 112 and / or the endoscope 114 can be positioned. The accessible volume can refer to the total volume within the internal working space 124, which is defined as within the reach of any one or more of the inserted instruments without causing contact or collision, and in some cases can be based on one or more criteria. The internal geometry can be determined based on at least the first and second 3D models. In some embodiments, the internal geometry can be further determined based on the type of medical procedure being performed. In some embodiments, the internal geometry can be further determined based on segmentation of the medical procedure (e.g., where different segments use different instruments that may have different sizes, working volumes, reachability requirements, and other factors that may affect the determination of the internal geometry). The internal geometry may include (or be at least partially defined by) an internal reach boundary that is determined by a trajectory of locations that an end effector of the instrument 112 can reach within the internal working space 124, but at which contact between objects in the external working space 122 may occur, which may include any defined contact as further explained below. In other words, the trajectory of locations that define the internal reach boundary (where the reachable volume is the volume enclosed from the insertion orientation of the instrument and surrounding portions of the body wall 110 to the internal reach boundary) may correspond to a position at which a defined contact in the external working space 122 will occur if the portion of the instrument and / or manipulator arm holding the instrument in the external working space 122 is oriented or moved in such a manner as is necessary to allow the end effector of the instrument 112 to reach the position. The defined contact may include, but is not limited to, contact between any two or more manipulator arms (e.g., manipulator arms 212) (in the case of a computer-assisted medical procedure utilizing robotics), and / or contact between a manipulator arm and / or instrument and an object in the external workspace 122 (e.g., another instrument, another manipulator arm (if any), a portion of a patient's body, a portion of a manipulator system (if any), an operating table supporting a patient, a person, other medical equipment supporting a medical procedure, or a combination thereof). Operation 406 may include identifying an internal boundary track based on, for example, the external boundary track (identified in operation 404), the internal 3D image data, and the external 3D image data.

[0054] As an illustration, refer to Figure 5A and Figure 5B , which schematically illustrates a situation in which the possibility of a defined contact is a criterion for notifying the determination of an internal reaching boundary. Figure 5A and Figure 5B In the figure, it is shown that Figure 2B202 to perform a robotic medical procedure; however, those skilled in the art will appreciate that this is for illustrative purposes only and that if manually operated Figure 5A and Figure 5B Similar situations may occur if some or all of the devices or components shown in Figure 5A and Figure 5B In Figure 1 As in the embodiment of the present invention, the body wall 110 separates the external working space 122 from the internal working space 124. A first port 502-1 and a second port 502-2 (collectively referred to as "ports 502") provide access to the internal working space 124. The medical instrument 112 is mounted to the manipulator arm 212-1, which is in turn mounted to the central column 222; and the endoscope 114 is mounted to the manipulator arm 212-2, which is in turn also mounted to the central column 222. The instrument 112 and the endoscope 114 extend from the external working space 122 to the internal working space 124 via the port 502. Although FIG. 5A to FIG. 5B An example is shown in which the manipulator arms 212 are mounted to the central column 222, but as described above, one or both of the manipulator arms 212 may be mounted to a wall, ceiling, operating table, etc.

[0055] exist Figure 5A In FIG. 1 , manipulator arm 212-1 is oriented in a first position such that the distal end portion of instrument 112 is disposed at a particular location within internal working space 124. Because there is no contact between manipulator arms 212, the system can determine that the area where the distal end of instrument 112 is located forms part of the reachable volume. However, in FIG. Figure 5B 124, the manipulator arm 212-1 would need to be oriented in the second position in order for the distal end of the instrument 112 to reach another position within the internal working space 124. However, in such a position, contact between the manipulator arm 212-1 and the center column 222 would occur. Therefore, the system can determine that the area where the distal end of the instrument 112 will be located is not part of the reachable volume (is excluded), and can define a portion of the inner boundary trajectory 504 corresponding to the farthest range that the instrument 112 can be placed without undesired contact (e.g., determined by a portion of the end effector of the instrument or other distal position). The inner boundary trajectory 504 can be located at the actual reach boundary itself, or can be determined by a proximity test (e.g., by adding a predetermined distance as a buffer) to incorporate a safety threshold.

[0056] Those skilled in the art will appreciate that although Figure 5BThe effect of contact or collision on the reachable internal geometry is shown, but in reality, the reachable range of one or more instruments 112 may be limited due to kinematic singularities, range of motion restrictions in the manipulator arm 212, and other factors. For example, if the manipulator arm 212 approaches a singularity, the motor speed may increase and some parts of the manipulator may move in a way that the bedside staff cannot anticipate. In such a scenario, the fast-moving parts of the manipulator arm 212 may present a greater likelihood of contact with objects in the external workspace.

[0057] Operation 408 may include providing output related to the execution of the medical procedure based on the determined internal geometry. For example, operation 408 may include outputting information related to the defined internal boundary trajectory and / or displaying the internal geometry of the reachable volume to an operator of the instrument (e.g., a surgeon at the user control system 204). The output may include feedback, such as visual feedback (e.g., causing the system 300 to display the internal geometry, a representation of the instrument portion within the internal working space 124, a representation of a contact or potential contact within the external working space 122, or a combination thereof to the operator of the instrument), tactile feedback, audio feedback, or a combination thereof. The feedback may include features for distinguishing various types of situations; for example, the visual feedback may include different indicators to distinguish different types of contact and / or different risk levels. As non-limiting examples, the indicators may include: visual feedback, such as using colors (such as red for patient contact, yellow for robot self-collision, etc.); audio feedback (such as different tones or verbal identifiers) for distinguishing different types of contact and / or different risk levels; and / or tactile feedback, where different sensations are associated with different types of contact and / or risk levels. In some implementations, one or more types of feedback may be selected by a user.

[0058] Figure 4 The process flow can be repeated at predetermined intervals and / or on demand, with the system being configured to be used in an automatic mode, a manual mode, or a mode that allows switching between the two. For example, in response to identifying an accessible volume that does not include an orientation in which an instrument is desired to be placed in the internal workspace, after changes to the external workspace have been made, a manual user request can be made to the control system in a manual mode to recapture internal and / or external 3D image data (e.g., to undock one or more manipulator arms located in a potential defined contact area). The images are then recaptured and the internal geometry model can then be updated, such as by performing some or all of operations 402, 404, 406, and 408 using the new image data. Thereafter, the accessible volume can be determined and output regarding the accessible volume can be provided so that various planning and actions of the medical procedure can be performed.

[0059] In some embodiments, for example, Figure 4 The output provided at operation 408 in may include various forms of output useful to a person performing the medical procedure (such as a surgeon) (such as feedback and / or control of an instrument, manipulator arm, or other portion of the medical system) and / or output for controlling or assisting in controlling one or more instruments and / or arms. Figure 6 An exemplary process flow is shown that may be based on achievable workspace geometry and provide such feedback and / or control. Figure 6 The process flow may be executed by, for example, the control system 308. In some embodiments, Figure 6 The process flow of the embodiment of the present invention may be embodied in the form of instructions stored in a non-transitory computer-readable medium (e.g., memory 314), so that the control system 308 performs the following operations when executing the instructions: Figure 6 In other embodiments, Figure 6 The process flow can be embodied in the form of a carrier signal (for example, provided to the processor 312), so that the control system 308 executes when receiving the instruction Figure 6 operation.

[0060] Figure 6 The process flow may take as input: orientation, position, or other relevant information received from sensors disposed on one or more of the arm 212, instrument 112, and / or endoscope 114; orientation, position, or other relevant information received from sensors associated with the system 300; internal geometry within the internal working space 124 (e.g., as generated in operation 406); information about system capabilities (e.g., the presence of a tactile feedback device, etc.); information about the system mode (e.g., whether the system is operating in a manual override mode); information about system settings (e.g., user-defined tolerances); a first generated 3D model; a second generated 3D model; information about the instrument type; information about the procedure type; and combinations thereof. Figure 6 The process flow includes an operation 602 of determining a position of one or more instruments (eg, 112 and / or 114 ) within the internal working volume 124 (eg, within the internal geometry) and / or within the external working volume 122 .

[0061] Figure 6 The process flow of the embodiment also includes an operation 604 of providing feedback to an operator of the system (e.g., a surgeon, an assistant, etc.). The feedback may include tactile feedback, such as via a tactile feedback device (e.g., a tactile feedback device) associated with a force transmission mechanism of an instrument and / or an input device at a user control system 204 used by the surgeon to control movement of the instrument. Figure 2BThe feedback may include audio feedback, such as via one or more speakers, such as at the manipulation system 202, the user control system 204, and / or the auxiliary system 206, or anywhere within the hearing range of the person, for which reason such feedback may be useful during the medical procedure. The feedback may additionally or alternatively include visual feedback, such as via the stereoscopic image display system 240, Figure 2B The visual feedback may take the form of an icon displayed as part of the GUI, a warning or information message, a request (e.g., a reposition request to reposition the manipulator arm 212), a light, etc.

[0062] In some embodiments, the visual feedback may include displaying an image of a portion of the instrument within the internal working space, either alone or simultaneously with visualization of the determined internal geometry within the internal working space. In some embodiments, the visual feedback may include display of a visual indication of a portion (or all) of the boundary of the determined internal geometry. Such representations may include a graphical element corresponding to the boundary, which is overlaid with a real-time image corresponding to the output of the endoscope 114 and / or a visualization (e.g., an image or rendering) of the external working space 122. For example, when the distal tip of the instrument end effector or other distal end of the instrument is within a defined proximity of the determined internal reach boundary, the visual feedback may visually highlight the boundary (e.g., by using a graphical outline overlaid on a stereoscopic image captured by the endoscopic image device and displayed at one or more displays of the user control system 204 or the auxiliary system 206). Figure 7 One example of visual feedback is shown, where the control system 308 is configured to display a live video 702 (which includes an image of the internal workspace 124 in which the instrument 112 is located) with a graphic outline 704 overlaid thereon showing the boundaries of the determined internal geometry. Depending on certain system settings, operation 604 may be omitted. For example, a user may temporarily or permanently disable warning graphics, boundary surface overlays, etc.

[0063] Figure 6The process flow of optionally includes an operation 606 to control actuation (or positioning) of an instrument or arm or to assist in controlling an instrument or arm based on the orientation determined in operation 602. Operation 606 may include controlling one or more instruments 112 and / or manipulator arms 212 to modify (e.g., prevent or slow) a particular movement and / or actuation in at least one direction (e.g., actuation of an end effector of an instrument), even if the control system 308 receives a command to perform such movement and / or actuation. For example, even if the control system receives a command from an input control 242 at the user control system 204 and / or other input device operably coupled to control movement and / or actuation of a portion of the instrument and / or manipulator system 202 (such as at a manually actuated force transmission mechanism of the instrument) (e.g., a foot pedal or other input device configured to provide a command to deliver energy or other types of flux from the instrument), the control system 308 may ignore the command, for example, if it is determined that the instrument is near, at, or beyond the boundary of the internal working space 124. Alternatively, operation 606 may include providing modified controls to provide actuation different from that specified by input at the user control system 204 or a manually actuated force transmission mechanism (not shown) of the instrument (e.g., actuation of either end effector function and / or instrument movement is performed at a slower rate than commanded or with a delay and feedback to allow the user to back off actuation). Depending on certain system settings, operation 606 may be omitted. For example, the surgeon may disable the actuation controls and / or may override the controls.

[0064] Figure 6The operation (which can generally be classified as allowing the modification of the desired reach) can adopt one of several different embodiments. In one embodiment, the control system 308 can provide visual feedback prompts or guidance to reposition one or more of the instruments 112 and / or manipulator arms 212 (e.g., by displaying a repositioning prompt at a display or via an indicator associated with the arm corresponding to the prompt). After such a repositioning prompt, the repositioning of the relevant (one or more) instruments and / or arms can be performed via automatic operation through the control system 308, thereby automatically moving the arm 212 to the desired position, or via manual operation by medical personnel, where the medical personnel physically repositions the arm 212, for example to increase the size of the accessible internal geometry. In the case where the repositioning is automatic, the control system 308 can wait to receive a confirmation to perform the repositioning operation and / or some other response input (e.g., pressing a button, actuating a switch (e.g., a kill switch), etc.). If multiple instruments 112 and / or arms 212 are to be repositioned, the arms 212 can be moved one at a time. In this case, if one instrument 112 and / or arm 212 obstructs the repositioning path of another instrument 112 and / or arm 212, the obstructing instrument 112 and / or arm 212 may be moved first. If the final desired repositioning state cannot be reached (e.g., due to approaching obstacles or patients), the repositioning operation may be terminated.

[0065] In another embodiment, the control system 308 may provide visual cues to one or more members of the medical team to semi-manually reposition one or more arms 212, such as via any of the display devices, for example. Figure 2B As shown. The control system 308 may illuminate one or more light-emitting elements at the appropriate touch points of the arm 212 to be moved first. The assistant or other personnel then manually moves the arm 212. When the arm 212 approaches the final repositioning state, the control system 308 may automatically position the arm 212 (e.g., by applying an arm brake) and provide a confirmation notification (such as an audible tone). As described above, if the final repositioning state cannot be reached (e.g., due to an obstruction or the proximity of the patient), the assistant or other personnel may terminate the repositioning operation. If an additional arm 212 is to be repositioned, the sequence may be repeated. These embodiments are presented by way of example only, and other embodiments may be utilized instead.

[0066] As part of operation 606, the control system 308 may also be configured to implement a manual boundary correction mode. In such a mode, if the user determines that the predicted internal arrival boundary is incorrect (e.g., Figure 4406), and then may execute the process flow to recalculate the body model in the vicinity of the new (manually taught) boundary. Figure 6 The remaining operations.

[0067] In various other embodiments, Figure 4 The operations may additionally or alternatively be used to assist the port planning procedure. Figure 8 The process flow of FIG. 1 shows an exemplary port planning process flow according to the present disclosure, which may begin after an initial port (such as a port for an endoscope) has been placed. The initial port placement may be based on parameters including, but not limited to, patient characteristics (e.g., body mass index or BMI), the type of procedure being performed, the size of the instruments used for the procedure, previously acquired images such as CT / MRI images (if available), the orientation of the intended ports for manual laparoscopic instruments, and / or various other factors familiar to those of ordinary skill in the art. Figure 8 An operation 802 of generating an initial internal geometry is shown, as described above, for example with reference to Figure 4 As described. Figure 8 During the process flow, operation 802 may include Figure 4Operations 402-408 of the present invention are described herein. Thus, operation 802 may include the following sub-operations: generating a first 3D model of an internal working space 124 of the body into which a first portion of the instrument 112 will be inserted during use of the instrument to perform a procedure; and generating a second 3D model of an external working space 122 in which a second portion of the instrument 112 is located during performance of the procedure. Operation 802 may also include the following sub-operations: receiving imaging data of the internal working space 124 from an image sensor (e.g., endoscope 114) and / or receiving imaging data of the external working space 122 from another image sensor (e.g., image sensor 102) if the imaging data is not available or if otherwise updating the imaging data is preferred. For example, a surgeon or other personnel may insert an endoscope 114 into the body via an initial port and sweep the internal working space 124 to collect internal imaging data. As a result of the initial performance of the operation, operation 802 will generate an initial internal geometry; that is, an internal geometry corresponding only to the presence of the initial port.

[0068] The initial internal geometry may be used at operation 804 to determine the port positioning of one or more additional ports (i.e., ports in addition to the initial port). Operation 804 may be based on a comparison between the initial internal geometry and the target internal geometry (i.e., the internal geometry including the desired positions within the internal working space 124 to perform the medical procedure), based on the type of procedure being performed, and / or based on environmental (external) parameters, for example, to ensure that the reachable volume includes all desired positions in the internal working space. Operation 804 may also be based on external image data, such as aligning the external port placement positions with the internal working space. In general, operation 804 may be based on a volume defined by an internal boundary trajectory, a probability of defined contact between an instrument and a body part within the internal boundary trajectory, a probability of defined contact between an instrument 112 and / or an associated manipulator arm 212 and an object in the external working space 122, or a combination thereof. Operation 804 may include providing port planning information. The information may include the planned port locations themselves and / or information related to possible configurations of the arms 212 and / or instruments 112 in the internal or external working spaces based on the planned port locations (e.g., a cone of possible or allowable angles of the instruments 112 relative to the corresponding ports). In one example, the information takes the form of a port planning map that includes one or more orientations for placing a port configured for insertion of one or more instruments 112 to extend from the external working space 122 to the internal working space 124. The port planning map may be displayed to the operator on a display and / or projected directly onto the patient's body using a projector.

[0069] Operation 806 includes updating the internal geometry based on the port placement guidance generated in operation 804. Operation 806 may occur immediately after operation 804, or alternatively may occur after one or more additional incisions are made and one or more corresponding additional ports are placed based on the guidance generated in operation 804. To update the internal geometry, existing internal image data (e.g., image data of the internal working space 124 obtained via the endoscope 114 using the initial port) may be used; however, in some embodiments, external and / or internal image data (e.g., image data of the internal working space 124 obtained via the endoscope 114 using the newly placed port) may be used in addition or alternatively. The updated image data may be obtained at predetermined intervals, on demand, or in real time. Operations 804 and 806 may be repeated in an iterative or continuous manner until a final port layout is generated that includes positioning information for a sufficient number of ports so that the desired accessible volume of one or more instruments 112 is achieved. In such embodiments, each iteration of operation 804 may generate one (or, in some examples, more than one) planned location of an additional port based on the updated internal geometry information from the previous iteration of operation 806. The updated port layout map may then be used in subsequent iterations of operation 806 to generate newly updated internal geometry information, which may then be used in subsequent operations 804, and so on, until a final port layout map is obtained. In examples, these iterative operations may be performed in real time or near real time. For example, a user may move a pointer (e.g., a finger, an obturator, or other device) around the body wall 110 near the planned port locations, and operation 806 may be continuously performed to update the internal reach boundary in real time.

[0070] Although Figure 4 , Figure 6 and Figure 8The operations are shown in a particular order, but in actual embodiments, the operations may be performed in any order, as long as the operation of obtaining input is performed after the operation of generating input (e.g., operations 402 and 404 are performed before operation 406). In addition, some operations may be performed continuously and / or continuously, such as by continuously performing operations 402-408 to update the 3D model of the internal workspace 124 and / or the 3D model of the external workspace 122, and continuously (dynamically) adjusting the reachable volume of a given instrument during the medical procedure, and continuously providing output based on the reachable volume. Dynamic adjustments may be performed in response to one or more triggering events, including but not limited to instrument changes, environmental changes, configuration changes (e.g., posture changes) of the manipulator system or instrument, changes in the orientation and / or configuration of the operating table, introduction of additional medical equipment (such as a mobile x-ray machine), movement of the operating table, changes in the stage of the medical procedure (which itself may include one or more of the changes listed herein or is related to one or more of the changes listed herein), etc. In addition, in some embodiments, one or more operations may be performed in parallel. For example, in Figure 4 In some embodiments, the control system 308 may use the same or different processing resources to simultaneously generate a 3D model of the internal workspace 124 and a 3D model of the external workspace 122. In some embodiments, one or more operations may be performed multiple times. Figure 4 In the example embodiment, an initial iteration of operation 402 may be performed to generate an initial 3D model of internal workspace 124 , and subsequent iterations of operation 402 may be performed to update the initial 3D model of internal workspace 124 and / or generate a new, updated 3D model of internal workspace 124 .

[0071] Figure 4 , Figure 6 or Figure 8 One or more of the operations shown may be performed manually (e.g., by a surgeon, assistant, and / or other personnel), automatically or computer-assisted (e.g., by manipulation system 202, user control system 204, auxiliary system 206, components thereof, and / or other equipment), or a combination thereof. In one specific example, Figure 4 and Figure 8 The operation may be performed in a combined manual and computer-assisted manner as follows. In this example, operation 404 may first be performed to use a depth sensing camera ( Figure 1 The external image sensor 102 and / or Figure 2BThe depth sensing camera may scan the patient and reference points set with markers (e.g., the patient's navel). Based on the 3D model of the external workspace, the system performing the operation may provide an indication of a proposed first entry point (i.e., initial port placement), which may be based on factors including, but not limited to, the procedure being performed, a segment of the procedure, or patient data. A medical professional (such as a surgeon or an assistant) performing the procedure may manually palpate the anatomical structure to verify the proposed initial port placement. In an example of a partial performance of operation 408, these operations may be guided by real-time video based on image data from the depth sensing camera. If the medical professional confirms that the proposed initial port placement is correct or feasible, the medical professional (or another medical professional) may make an incision at the first entry point and insert an imaging device (such as a trocar and a laparoscope). Figure 1 An example of an internal image sensor 116).

[0072] By manually operating the laparoscope, the medical staff may perform operation 402 to scan the inside of the patient and generate a 3D model of the internal working space. The medical staff may manipulate the laparoscope to obtain image data of one or more desired features of the internal working space (such as the target of the procedure, the patient's organ or bone area). In an example of partial execution of operation 408, these operations may be guided by real-time video based on image data from the laparoscope. Based on the image data of the external working space and the internal working space, operation 406 may be performed. In this example, the depth sensing camera may track the laparoscope, for example, by detecting a mark (e.g., a QR code) on the handle of the laparoscope. Therefore, operation 406 may include international patent application number PCT / US2021 / 065444 filed on December 29, 2021 and now disclosed as WO2022 / 147074 The system and method discussed in the application performs registration between the external working space and the internal working space, which is incorporated herein by reference. These operations may therefore correspond to the sub-operations of operation 802 as described above.

[0073] Then in this example, the medical staff can view the internal geometry and select the target workspace. In a specific example, the medical staff can draw the target boundary on a touch pad associated with the system, thereby defining the target workspace. Thereafter, the system can continue to determine the port location (example of operation 804), such as by automatically testing the compatibility of all possible port arrangements with the external workspace (e.g., to avoid undesirable or unintentional contact) and / or the compatibility of the internal workspace (e.g., to achieve the target workspace with the least number of incisions). The system can generate a port arrangement diagram and can display it to the medical staff. The display can be performed on a display device, such as a touch pad used by the medical staff to select the target workspace, or by projecting a diagram onto the patient. The medical staff can then mark and / or create additional incisions while viewing a real-time image of the external workspace using a depth sensing camera.

[0074] Figure 4 , Figure 6 or Figure 8 One or more operations shown in the may include a sub-operation of tracking an object (e.g., an instrument, a manipulator arm, a body part, and / or another internal or external object) using images of the body's exterior and interior. The control system 308 may receive one or more images from one or more external image sensors 102 / 220 and one or more images from the internal image sensor 116. The control system 308 may perform registration to determine the correspondence between the images provided by these image sensors (e.g., an alignment relationship including some or all degrees of freedom) and transform the image data to a common reference system. In some examples, such registration may involve determining a three-degree-of-freedom translation and a three-degree-of-freedom transformation (or a subset of these degrees of freedom) between the field of view of the image sensor and the common reference system. Using the transformed image data from the external image sensor 102 / 220 and / or the internal image sensor 116, the control system 308 may track the movement of the object and generate a tracking result to indicate the movement state of the object. In various embodiments, the tracking result may indicate the tracked object, the direction of movement of the object, the total number of sub-parts of the object that have been moved, the orientation of the object relative to another object, etc.

[0075] In an example, the control system 308 may determine a tracking configuration based on the object to be tracked and the expected or ongoing operation, and use the tracking configuration to perform tracking. For example, the control system 308 may determine the tracking configuration based on the type of object (e.g., an instrument, a manipulator arm, a body part, and / or another internal or external object). As another example, the tracked operation may include an object entering or leaving the body (e.g., an instrument entering and / or leaving the body; removing and / or transplanting a body part; removing, placing, and / or replacing an implant, etc.).

[0076] In a tracking operation, the control system 308 may receive sensor data including external image data from one or more external image sensors 102 / 220 and / or internal image data including internal image data from the internal image sensor 116. The external or internal image data may include an object to be tracked (e.g., an instrument, an instrument tip, a manipulator arm, a part of the body, a room feature, etc.) or a representation / model of the object to be tracked. The object to be tracked may be identified in the internal image data by a marker, color, shape, size / dimensions, any other suitable feature, an association with equipment that can interact with the object, an attribute or feature identified by a machine learning model, and / or a combination thereof. In some embodiments, the control system 308 may use the same one or more features to identify the object to be tracked in the external image data and the internal image data. Alternatively, in some embodiments, the control system 308 may use different one or more features to identify the object to be tracked in the external image data and the internal image data, for example based on different image attributes (e.g., imaging conditions, image resolution, image metadata, etc.) of the first image data and the second image data. In some embodiments, the views of the external image sensor 102 / 220 and the internal image sensor 116 may be mutually exclusive (e.g., one sees inside the body and the other sees outside the body), and the tracked object may not be included in the simultaneously captured first image data and second image data. In such embodiments, a synthetic (e.g., based on a model of the object to be tracked) overlay of the object to be tracked (e.g., based on information from one image data) may be provided in the other image data where the object to be tracked is not directly visible.

[0077] The tracking operation may also include determining the registration between the external image data and the internal image data. In such embodiments, the control system 308 may determine the registration between the external image sensor(s) 102 / 220 and the internal image sensor 116. In some embodiments, the registration may be performed by registering the image sensor to the manipulator arm coupled to the image sensor (e.g., where the external image sensor 102 / 220 is mounted on the manipulator arm 212 or the internal image sensor 116 is manipulated by the manipulator arm 212), and registering the manipulator arms to each other. The control system may also use various image registration methods to determine such an alignment relationship using the external image data and the internal image data. In some embodiments, the registration is further performed using additional image data (e.g., preoperative and intraoperative image data, a calculated patient grid, etc.), and the registered image set includes the external image data and the internal image data from the external image sensor and the internal image sensor and those registered additional image data. The control system 308 may transform the external and / or internal image data according to the alignment relationship with the common reference frame. In some embodiments, the common reference frame may be a 3D coordinate system that is consistent with a reference frame of either image sensor, a 3D coordinate system of a manipulator assembly, or a 2D image plane of either image sensor.

[0078] The control system 308 may also track an object relative to one or more other objects (e.g., an instrument, an instrument tip, a manipulator arm, a part of the body, a room feature, etc.) based on the external and / or internal image data and registration to generate tracking results, thereby indicating whether there is a possibility of undesired contact or near contact between the tracked object and one or more other objects. In one example, the control system 308 may generate one or more movement paths of the tracked object and / or other (one or more) objects. The movement paths may be represented using multiple images or video images. In some embodiments, when the object to be tracked is not directly present in the external or internal image data, or when the object to be tracked is present in the external or internal image data but is occluded, an estimate of its position and movement may be generated based on its past recorded position and / or movement in either image data.

[0079] Fig. 9 An exemplary GUI 900 is shown that may be presented during the execution of a medical procedure and / or during the preparation of a medical procedure. In some examples, the GUI 900 presents an operator with Figure 4 , Figure 6 and / or Figure 8 GUI 900 can be used to display information generated by one or more of the process flows. Figure 2BGUI 900 includes a menu portion 902 and a video display portion 904. Menu portion 902 can present a list of menu items to the operator, thereby allowing the operator to switch between various views and screens. Based on the selected menu item, video display portion 904 displays appropriate image data. For example, menu portion 902 may allow selection between: a patient information screen, in which patient information may be entered and / or displayed; an instrument screen, in which information related to the type of instrument 112 used for a medical procedure may be entered and / or displayed; an equipment and room screen, in which information related to the type of equipment used and / or the operating room may be entered and / or displayed; an orientation screen, in which information related to the position and orientation of the patient, operating table, etc. may be entered and / or displayed; a port placement screen, in which information related to the location and / or planned location of a surgical port may be entered and / or displayed; a procedure screen, in which information related to a procedure may be entered and / or displayed; a settings screen, in which information related to various settings, operations, and / or preferences may be entered and / or displayed; and an analysis screen, in which various performance indicators may be entered and / or displayed. Various portions of GUI 900 may be used independently of other portions of GUI 900. For example, an operator may use menu portion 902 to select which screen to display, which may cause GUI 900 to update only video display portion 904 while preserving other portions of GUI 900.

[0080] exist Fig. 9 , the port placement screen has been selected so that port placement information is presented on the video display portion 904. In this example, the video display portion 904 presents an actual video feed of a portion of a patient's body wall in an external workspace that can be accessed for port placement to perform a medical procedure (left side of portion 904) and a 3D rendering of a portion of the patient's body wall (right side of portion 904). A series of port locations 906 are overlaid on both the actual video feed and the 3D rendering, for example as an overlay, which in some cases may be partially transparent so as not to obscure the video image of the patient's body. If desired, some or all of the port locations 906 may be associated with an identifier (e.g., a different color, shape, shading, etc.), for example to identify an endoscope port. The video display portion 904 also includes an input field 908 that an operator can use to provide input, for example to initiate various operations, including but not limited to the above with respect to Figure 4 , Figure 6 and Figure 8 In other examples, video display portion 904 may present only a single video and / or may show information about an internal workspace.

[0081] Various aspects of the present disclosure as described herein may be well suited for use in any of a variety of medical procedures for which obtaining and / or using the accessible volume of the internal working space may be desirable and / or advantageous, as described above. Such procedures may be performed on, for example, human patients, animal patients, human cadavers, animal cadavers, and portions of human or animal anatomical structures. The medical procedures contemplated herein include any of those described herein and for non-surgical diagnostics, including cosmetic procedures, imaging of human or animal anatomical structures, collecting data from human or animal anatomical structures, training medical or non-medical personnel, and procedures on tissues removed from human or animal anatomical structures (without returning to human or animal anatomical structures). Even if applicable to such medical procedures, various aspects may also be used for benchtop procedures on non-living materials and configurations that are not part of human or animal anatomical structures. In addition, some aspects are also suitable for use in non-medical applications, such as industrial robot uses, and sensing, inspection, and / or manipulation of non-tissue artifacts. In non-limiting aspects, the techniques, methods, and apparatus described herein may be used in or may be a part of a computer-assisted surgical system that employs robotics, such as the various surgical systems commercialized by Intuitive Surgical, Inc. of Sunnyvale, California. However, those skilled in the art will appreciate that the various aspects disclosed herein may be implemented and realized in a variety of ways and systems in medical and non-medical applications, including manually operated minimally invasive medical systems and computer-assisted teleoperated systems. References to surgical systems are illustrative and should not be considered limiting the scope of this disclosure.

[0082] As used herein and in the claims, terms such as computer-assisted manipulator systems, manipulator systems, or variations thereof should be understood to broadly refer to any system comprising one or more controllable kinematic structures ("manipulators") comprising one or more links coupled together by one or more joints, which can be operated to cause the kinematic structures to move. Such systems are sometimes referred to as robot-assisted systems or robotic systems in the art and in common use. The manipulator may have an instrument permanently or removably mounted thereon and may move and operate the instrument. The joints may be driven by drive elements that may utilize any convenient form of power, such as, but not limited to, electric motors, hydraulic actuators, servo motors, and the like. The operation of the manipulator may be controlled by a user (e.g., by remote operation), automatically controlled by a computer (so-called autonomous control), or controlled by some combination of these. In examples where the user controls at least some of the operations of the manipulator, an electronic controller (e.g., a computer) may facilitate or assist the operation. For example, the electronic controller may "assist" the user-controlled operation by converting control inputs received from the user into electrical signals that actuate the drive elements to operate the manipulator, providing feedback to the user, implementing safety restrictions, and the like. The term "computer," as used in "computer-assisted manipulator system," refers broadly to any electronic control device used to control or assist a user in controlling the operation of a manipulator, and is not intended to be limited to objects formally defined or colloquially referred to as "computers." For example, the electronic control devices in a computer-assisted manipulator system may range from a traditional "computer" (e.g., a general-purpose processor plus a memory that stores instructions for execution by the processor) to low-level, special-purpose hardware devices (analog or digital) such as discrete logic circuits or application-specific integrated circuits (ASICs), or anything in between. In addition, the manipulator system may be implemented in a variety of contexts to perform a variety of procedures, both medical and non-medical. Thus, although some of the examples described in more detail herein may focus on a medical context, the devices and principles described herein are also applicable to other contexts, such as industrial manipulator systems.

[0083] It should be understood that both the general description and the detailed description provide exemplary aspects that are illustrative in nature and are intended to provide an understanding of the present disclosure without limiting the scope of the present disclosure. Various mechanical, compositional, structural, electrical and operational changes may be made without departing from the spirit and scope of the present specification and claims. In some cases, well-known circuits, structures and techniques are not shown or described in detail to avoid obscuring these aspects.

[0084] In addition, the terms used in this article to describe various aspects of the present disclosure (such as spatial and relational terms) are selected to help the reader understand the various aspects of the present disclosure, but are not intended to limit the present disclosure. For example, spatial terms—such as "below", "below", "lower", "above", "above", "near side", "distal side", "up", "down", etc.—may be used in this article to describe the direction or spatial relationship of one element or feature to another element or feature (as shown in the accompanying drawings). These spatial terms are used relative to the accompanying drawings and are not limited to a specific reference system in the real world. Therefore, for example, the direction "up" in the accompanying drawings does not necessarily correspond to "up" in the world reference system (e.g., away from the surface of the earth). In addition, if a reference system different from the reference system shown in the accompanying drawings is considered, the spatial terms used in this article may need to be interpreted differently in the different reference system. For example, a direction referred to as "up" relative to one of the accompanying drawings may correspond to a direction referred to as "down" relative to a different reference system rotated 180 degrees from the reference system of the accompanying drawings. As another example, if the device is flipped 180 degrees in the world reference system compared to the way the device is shown in the figure, an item described herein as being "above" or "on top of" a second item relative to the figure would be "below" or "below" the second item relative to the world reference system. Thus, depending on which reference system is being considered, different spatial terms may be used to describe the same spatial relationship or direction. In addition, the postures of the items shown in the figures are selected for ease of illustration and description, but in practical implementations, the items may assume different postures. In general, "distal" as used herein refers to the direction toward the end effector or other free working end of the kinematic chain of the instrument, and "proximal" refers to the opposite direction.

[0085] In addition, the singular forms "a", "an", and "the" are also intended to include the plural forms, unless the context indicates otherwise. In addition, the terms "including", "comprising", etc. specify the presence of stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups. Components described as coupled may be directly coupled electrically or mechanically, or they may be indirectly coupled via one or more intermediate components, unless otherwise specifically stated. Mathematical and geometric terms are not necessarily intended to be used in accordance with their strict definitions unless the context of the specification indicates otherwise, as one of ordinary skill in the art will understand that, for example, substantially similar elements that operate in a substantially similar manner are readily within the scope of a descriptive term, even if the term also has a strict definition.

[0086] Whenever feasible, elements and their associated aspects described in detail with reference to one aspect may be included in other aspects that are not specifically shown or described. For example, if an element is described in detail with reference to one aspect and not described with reference to a second aspect, the element may still be claimed to be included in the second aspect.

[0087] Unless otherwise specified herein or implied by the context, when approximate terms such as "substantially", "approximately", "about", "roughly", etc. are used in conjunction with a specified value, attribute, or relationship, such as the endpoints of a range or a geometric attribute / relationship (e.g., parallel, perpendicular, straight, etc.), this should be understood to mean that mathematical precision is not required for the value, attribute, or relationship, but rather refers to a range of variation that includes but is not strictly limited to the stated value, attribute, or relationship. Specifically, the range of variation around the stated value, attribute, or relationship includes at least any unreasonable variation from the stated value, attribute, or relationship, such as a variation that is equivalent to the stated value, attribute, or relationship. The range of variation around the stated value, attribute, or relationship also includes at least those variations that are typical in the relevant field of the type of item in question due to manufacturing or other tolerances. In addition, the range of variation also includes at least variations within ±5% of the stated value, attribute, or relationship. Thus, for example, a line or surface may be considered to be "approximately parallel" to a reference line or surface if any of the following are true: the minimum angle between the line / surface and the reference is less than or equal to 4.5° (i.e., 5% of 90°), the angle is less than or equal to manufacturing or other tolerances typical in the art, or the line / surface would constitute a surface that is functionally equivalent to the line / surface if they were perfectly parallel.

[0088] In view of the disclosure herein, further modifications and alternative aspects will be apparent to those of ordinary skill in the art. For example, the apparatus and method may include additional components or steps omitted from the diagram and description for operational clarity. Therefore, the description is interpreted as illustrative only, and is intended to teach those skilled in the art the general manner of performing this teaching. It should be understood that the various aspects shown and described herein should be considered exemplary. Elements and materials and the arrangement of these elements and materials may replace those shown and described herein, components and processes may be reversed, and certain features of this teaching may be utilized independently, all of which are apparent to those skilled in the art after benefiting from the description herein. The elements described herein may be changed without departing from the spirit and scope of this teaching and the appended claims.

[0089] It is to be understood that the specific examples and aspects set forth herein are non-limiting and that modifications in structure, dimensions, materials, and methods may be made without departing from the scope of the present teachings.

[0090] Other aspects of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the present disclosure disclosed herein. The specification and examples are intended to be considered exemplary only, with the appended claims to be given their fullest breadth, including equivalents, under applicable law.

Claims

1. A computer-aided system comprising: an instrument configured to be inserted at least partially through a body wall from an external working space to an internal working space within the body; and A controller, the controller being configured to: generating a first three-dimensional model of the internal working space into which a first portion of the instrument is inserted during use of the instrument to perform a medical procedure, generating a second three-dimensional model of an external workspace in which a second portion of the instrument is located during said performance of the medical procedure, determining an internal geometry within the internal working space based on the first three-dimensional model and the second three-dimensional model, the internal geometry defining an accessible volume within the internal working space within which the instrument is positionable, and Output related to performance of the medical procedure is provided based on the determined internal geometry.

2. The system of claim 1 further comprising a manipulator arm operably coupled to the instrument.

3. A system according to claim 2, wherein the internal geometry includes an internal reach boundary, which is a boundary that defines the orientation of the first portion of the instrument within the internal working space, at which orientation one or both of the instrument or manipulator arm contacts another object within the external working space.

4. The system of claim 2, wherein the controller is further configured to transmit a driving force via the manipulator arm operably coupled to the instrument and positioned in the external workspace to control movement of the instrument during the execution of the medical procedure.

5. The system of claim 2, wherein the controller is further configured to: determining the position of the instrument within the internal working space; and A reposition request is displayed, the reposition request including a visual indication to reposition the manipulator arm within the internal working space and operably coupled to actuate movement of the instrument.

6. The system of claim 2, wherein the controller is further configured to generate guidance for adjusting the manipulator arm to increase a size of the internal geometry.

7. The system of claim 1, wherein the internal geometry is further determined based on a type of the medical procedure using the instrument.

8. The system of claim 1, wherein the internal geometry is further determined based on a stage of the medical procedure.

9. The system of claim 1, wherein the controller is further configured to cause the system to display the internal geometry, a representation of the first portion of the instrument within the internal working space, or both to an operator of the instrument.

10. The system of claim 1, wherein the controller is further configured to cause the system to: determining the position of the instrument within the internal working space; and Feedback is provided to an operator of the instrument based on the orientation of the instrument.

11. The system of claim 10, wherein the feedback comprises tactile feedback, visual feedback, audio feedback, or a combination thereof.

12. The system of claim 1, wherein the controller is further configured to cause the system to: determining a relative orientation of the instrument relative to a boundary of the internal working space, and Feedback is provided to an operator of the instrument based on the relative orientation of the instrument.

13. The system of claim 1, wherein the controller is further configured to cause the system to: determining the position of the instrument within the internal working space; and Actuation of the instrument is controlled based on the position of the instrument within the internal working space.

14. The system of claim 1, wherein the controller is further configured to cause the system to: determining the position of the instrument within the internal working space; and In response to determining that the position of the instrument is proximate, at, or beyond a boundary of the internal working space, movement of the instrument in at least one direction is modified based on the position of the instrument within the internal working space.

15. The system of claim 1, wherein the controller is further configured to cause the system to: Based on the first three-dimensional model and the second three-dimensional model, one or more locations on the outer surface of the body for inserting the instrument to access the internal working space during the procedure are identified.

16. The system of claim 15, wherein the one or more locations include multiple locations identified in a continuous manner based on procedure type, body type, instrument configuration, arm configuration, achievable target geometry, or a combination thereof.

17. The system of claim 1, wherein the controller is further configured to: receiving internal three-dimensional image data of the internal working space, receiving external three-dimensional image data of the external workspace, wherein a boundary is defined between the internal workspace and the external workspace and includes an outer surface portion of the body, identifying an outer boundary trajectory based on the outer three-dimensional image data, defining an inner boundary trajectory based at least on the identified outer boundary trajectory and the inner three-dimensional image data, and Output information related to the defined internal boundary trajectory.

18. The system of claim 17, wherein the controller is further configured to cause the system to display the information related to the defined inner boundary trajectory, and the information includes a partially transparent graphical element corresponding to the inner boundary trajectory overlaid with a real-time image.

19. The system of claim 17, wherein the controller is further configured to cause the system to display the information related to the defined internal boundary trajectory simultaneously with the external three-dimensional image data.

20. The system of claim 17, wherein the controller is further configured to cause the system to: determining the position of the instrument within the internal working space; and In response to determining that the instrument is within a predetermined distance of the defined inner boundary trajectory, at least a portion of the defined inner boundary trajectory is visually indicated on a display.

21. The system of claim 17, wherein the controller is further configured to: A location on the body for placement of a surgical port is determined based at least on the external three-dimensional image data.

22. The system of claim 21, wherein the internal boundary trajectory is defined by registering the position of the surgical port with the internal three-dimensional image data.

23. The system of claim 17, wherein the controller is further configured to generate a port layout map based on the outer boundary trajectory and the inner boundary trajectory, wherein the port layout map includes one or more positions for placing ports configured for inserting one or more instruments to extend from the outer working space to the inner working space.

24. The system of claim 23, wherein the port placement map is based on a volume defined by the inner boundary trajectory, a probability of collision between the instrument and the body portion within the inner boundary trajectory, a probability of collision between the instrument and the body portion outside the inner boundary trajectory, or a combination thereof.

25. The system of claim 17, wherein receiving the internal three-dimensional image data from a first image sensor located within the internal working space, and The external three-dimensional image data is received from a second image sensor positioned in a room in which the body is located.

26. The system of claim 1, wherein the controller is further configured to: providing information relevant to planning positioning of the instrument during the procedure based on the first three-dimensional model and the second three-dimensional model, and The first three-dimensional model is updated based on the updated output of the image sensor.

27. The system of claim 26, wherein the first three-dimensional model is updated in real time.

28. The system of claim 26, wherein the first three-dimensional model is updated at predetermined intervals.

29. The system of claim 26, wherein the controller is further configured to cause the system to display at least a portion of the first three-dimensional model, at least a portion of the second three-dimensional model, or both to an operator of the instrument.

30. A computer-implemented method comprising: generating a first three-dimensional model of an internal working space into which a first portion of an instrument operably coupled to a manipulator arm is inserted during use of the instrument to perform a procedure, the manipulator arm being configured to transmit actuation forces to the instrument; generating a second three-dimensional model of an external workspace in which a second portion of the instrument is located during the performance of the procedure; determining, based on the first three-dimensional model and the second three-dimensional model, an internal geometry within the internal working space, the internal geometry defining a reachable volume within the internal working space, the instrument being positionable within the reachable volume by the manipulator arm; as well as Output related to execution of the procedure is provided based on the determined internal geometry.

31. A method according to claim 30, wherein the internal geometry includes an internal reach boundary, which is a boundary that defines the orientation of the first portion of the instrument within the internal working space, at which orientation one or both of the instrument or manipulator arm contacts another object within the external working space.

32. The method of claim 30, wherein the internal geometry is further determined based on a type of procedure using the instrument.

33. The method of claim 30, wherein the internal geometry is further determined based on a stage of the procedure.

34. The method of claim 30, further comprising displaying the internal geometry, a representation of the first portion of the instrument within the internal working space, or both to an operator of the instrument.

35. The method of claim 30, further comprising: determining the position of the instrument within the internal working space, and Feedback is provided to an operator of the instrument based on the orientation of the instrument.

36. The method of claim 35, wherein the feedback comprises tactile feedback, visual feedback, audio feedback, or a combination thereof.

37. The method of claim 30, further comprising: determining a relative orientation of the instrument relative to a boundary of the internal working space, and Feedback is provided to an operator of the instrument based on the relative orientation of the instrument.

38. The method of claim 30, further comprising: determining the position of the instrument within the internal working space, and Actuation of the instrument is controlled based on the position of the instrument within the internal working space.

39. The method of claim 30, further comprising: determining the position of the instrument within the internal working space, and In response to determining that the position of the instrument is proximate, at, or beyond a boundary of the internal working space, movement of the instrument in at least one direction is modified based on the position of the instrument within the internal working space.

40. The method of claim 30, further comprising: Based on the first three-dimensional model and the second three-dimensional model, one or more locations on the exterior surface for inserting the instrument to access the internal working space during the procedure are identified.

41. The method of claim 40, wherein the one or more locations comprise a plurality of locations identified in a serial manner based on procedure type, body type, instrument configuration, arm configuration, achievable target geometry, or a combination thereof.

42. The method of claim 30 further comprising transmitting the drive force via the manipulator arm operably coupled to the instrument and positioned in the external workspace to control movement of the instrument during the performance of the procedure.

43. The method of claim 30, further comprising: receiving internal three-dimensional image data of the internal working space, receiving external three-dimensional image data of the external workspace, wherein a boundary is defined between the internal workspace and the external workspace and includes an exterior surface portion, identifying an outer boundary trajectory based on the outer three-dimensional image data, defining an inner boundary trajectory based at least on the identified outer boundary trajectory and the inner three-dimensional image data, and Information related to the defined internal boundary trajectory is output.

44. The method of claim 43, further comprising displaying the information associated with the defined inner boundary trajectory, and the information comprises a partially transparent graphical element corresponding to the inner boundary trajectory overlaid with a real-time image.

45. The method of claim 43, further comprising displaying the information associated with the defined interior boundary trajectory simultaneously with the exterior three-dimensional image data.

46. ​​The method of claim 43, further comprising: determining the position of the instrument within the internal working space, and In response to determining that the instrument is within a predetermined distance of the defined inner boundary trajectory, at least a portion of the defined inner boundary trajectory is visually indicated on a display.

47. The method of claim 43, further comprising determining a location for placement of a surgical port based at least on the external three-dimensional image data.

48. The method of claim 47, wherein the internal boundary trajectory is defined by registering the position of the surgical port with the internal three-dimensional image data.

49. The method of claim 43, further comprising generating a port layout map based on the outer boundary trajectory and the inner boundary trajectory, wherein the port layout map comprises one or more positions for placing a port configured to insert one or more instruments to extend from the outer working space to the inner working space.

50. The method of claim 49, wherein the port layout map is based on a volume defined by the inner boundary trajectory, a probability of collision between the instrument and an area within the inner boundary trajectory, a probability of collision between the instrument and an area outside the inner boundary trajectory, or a combination thereof.

51. The method of claim 43, wherein receiving the internal three-dimensional image data from a first image sensor located within the internal working space, and The exterior three-dimensional image data is received from a second image sensor positioned in a room in which the interior workspace is located.

52. The method of claim 30, further comprising: determining the position of the device within the internal working space; as well as A reposition request is displayed, the reposition request including a visual instruction to reposition the manipulator arm within the internal working space and operably coupled to actuate movement of the instrument.

53. The method of claim 30, further comprising: providing information relevant to planning positioning of the instrument during the procedure based on the first three-dimensional model and the second three-dimensional model, and The first three-dimensional model is updated based on the updated output of the image sensor.

54. The method of claim 53, wherein the first three-dimensional model is updated in real time.

55. The method of claim 54, wherein the first three-dimensional model is updated at predetermined intervals.

56. The method of claim 54, further comprising displaying at least a portion of the first three-dimensional model, at least a portion of the second three-dimensional model, or both to an operator of the instrument.

57. The method of claim 30, further comprising generating guidance for adjusting the manipulator arm to increase a size of the internal geometry.

58. A non-transitory computer readable medium comprising instructions which, when executed by a computer system, cause the computer system to perform the method of any one of claims 30 to 57.

59. A computer system comprising: at least one processor; and At least one memory comprising instructions which, when executed by the at least one processor, cause the computer system to perform a method according to any one of claims 30-57.