Computer-assisted distance measurement in surgical space

Abstract

The invention relates to computer-assisted distance measurement in a surgical space. The present disclosure describes a system for measuring a distance between two points in a surgical space. The system moves the imaging device to a first pose to capture first imaging data showing a first region of a surface of the anatomical structure, receives the first imaging data, determines a first distance between the imaging device in the first pose and a first point on the surface, and determines first coordinates of the first point. The system moves the imaging device to a second pose to capture second imaging data showing a second area of the surface, receives the second imaging data, determines a second distance between the imaging device in the second pose and a second point on the surface, and determines second coordinates of the second point. The system determines a third distance between the first point and the second point based on the first coordinate and the second coordinate.

Description

[0001] Related Applications

[0002] This application claims the benefit of U.S. Provisional Patent Application Serial No. 63 / 623,437, filed on January 22, 2024, which is incorporated herein by reference. Technical Field

[0003] The present disclosure generally relates to medical systems. Specifically, the present disclosure relates to using a medical system to measure distances in a surgical space. Background Art

[0004] Various surgical instruments are introduced into a surgical space to perform operations on a patient. Such surgical instruments may include cameras. In one example, a stereo camera is used. A stereo camera includes two cameras (e.g., a left camera and a right camera). Given a point in a 3D space and its projections onto images from the left and right cameras, triangulation techniques can be employed to determine the point, which provides a depth measurement for the point. Triangulation techniques rely on the baseline (center point) distance between the left and right cameras to determine the depth of the point. The baseline distance between the left and right cameras may be small, which results in the depth determined being less accurate the farther the point is from the stereo camera. Summary of the Invention

[0005] The present disclosure describes systems and methods for measuring the distance between two points in a surgical space. According to one embodiment, a system for measuring the distance between two points in a surgical space includes a memory and a controller communicatively coupled to the memory. The system moves an imaging device to a first pose to capture first imaging data showing a first region of the surface of an anatomical structure, determines a first distance between the imaging device in the first pose and a first point on the surface, and determines first coordinates of the first point on the surface based on the first pose and the first distance. The system also moves the imaging device to a second pose to capture second imaging data showing a second region of the surface of the anatomical structure, determines a second distance between the imaging device in the second pose and a second point on the surface, and determines second coordinates of the second point on the surface based on the second pose and the second distance. The system determines a third distance between the first point and the second point based on the first coordinates and the second coordinates, based on each of the first distance and the second distance to the first point and the second point on the surface being within a threshold.

[0006] According to another embodiment, a method for measuring the distance between two points in a surgical space includes moving an imaging device to a first pose to capture first imaging data showing a first region of the surface of an anatomical structure, determining a first distance between the imaging device in the first pose and a first point on the surface, and determining first coordinates of the first point on the surface based on the first pose and the first distance. The method further includes moving the imaging device to a second pose to capture second imaging data showing a second region of the surface of the anatomical structure, determining a second distance between the imaging device in the second pose and a second point on the surface, and determining second coordinates of the second point on the surface based on the second pose and the second distance. The method further includes determining a third distance between the first point and the second point based on the first coordinates and the second coordinates, based on each of the first distance and the second distance to the first point and the second point on the surface being within a threshold. Other embodiments include a non-transitory machine-readable medium storing instructions that, when executed by a processor, cause the processor to perform the method.

[0007] The foregoing general description and the following detailed description are exemplary and explanatory in nature and are intended to provide an understanding of the present disclosure without limiting the scope of the present disclosure. In this regard, additional aspects, features, and advantages of the present disclosure will be apparent to those skilled in the art from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1A Shows an example medical system.

[0009] Figure 1B and Figure 1C Shows Figure 1A example components in the system of.

[0010] Figure 2A Shows an example medical system.

[0011] Figure 2B and Figure 2C Shows Figure 2A example medical device systems in the system of.

[0012] Figures 3A to 3C Shows example operations performed by Figure 1A or Figure 2A the system of.

[0013] Figure 4 Is a flowchart of an example method performed by Figure 1A or Figure 2A the system of.

[0014] Figure 5 Shows example operations performed by Figure 1A or Figure 2A the system of.

[0015] Figures 6A to 6C Illustrate example operations performed by the Figure 1A or Figure 2A system.

[0016] Figure 7 is a flowchart of an example method performed by the Figure 1A or Figure 2A system.

[0017] Figure 8 is a flowchart of an example method performed by the Figure 1A or Figure 2A system.

[0018] Figure 9 Illustrate example operations performed by the Figure 1A or Figure 2A system.

[0019] Figure 10 Illustrate example operations performed by the Figure 1A or Figure 2A system.

[0020] Figure 11 is a flowchart of an example method performed by the Figure 1A or Figure 2A system.

[0021] Figure 12 Illustrate example operations performed by the Figure 1A or Figure 2A system.

[0022] Figure 13 Illustrate example operations performed by the Figure 1A or Figure 2A system. Detailed Description

[0023] Endoscopy is a medical procedure used to visualize the internal organs or natural cavities of the human body. An endoscope is a rigid or flexible tubular device that allows direct visualization of the interior of the body. An endoscope system can be constructed as a purely optical device containing lenses, transparent rods, or optical fibers, or combined with an integrated or attached camera. The tip of the endoscope can be inserted through a small channel created by an incision or through a natural lumen of the body. A light source on the endoscope provides sufficient illumination for the cavity being examined. A camera on the endoscope provides a field of view that can be visualized and recorded on a screen for later diagnosis or archiving. Two cameras (e.g., a left camera and a right camera) can be arranged side by side on the endoscope to form a stereoscopic camera, which provides a stereoscopic view and allows depth perception. For example, the two cameras can capture images from slightly different perspectives (similar to the way the human eye perceives depth). These dual images are then processed to create a 3D image or video, providing improved spatial awareness for surgeons and medical professionals during the procedure. For example, triangulation techniques can be used with the images from the two cameras to determine the distances between different points in the field of view and the cameras (e.g., the depth of the points) and the distances between these points, which allows the surgeon to better understand the relative distances between different structures or instruments in the field of view. This enhanced spatial awareness can be beneficial for performing delicate and precise maneuvers.

[0024] The triangulation technique uses the baseline (center point) distance between the left camera and the right camera and the difference in the azimuth of a first point between the images from the left camera and the right camera to determine the azimuth of the first point and the distance between the first point and the endoscope. This process can be repeated for a second point in the field of view to determine the azimuth of the second point and the distance between the second point and the endoscope. Then, the distances between the first point and the second point can be determined using the azimuths of the first point and the second point and the distances between the first point and the second point and the endoscope.

[0025] The baseline distance between the left camera and the right camera can be small (e.g., 4 mm), which results in: the farther the first point and the second point are from the stereoscopic camera in the endoscope, the less accurate the determined distances between the first point and the second point and the endoscope become. Moving the stereoscopic camera closer to the first point or the second point can improve the accuracy of the distance measurement; however, it may also cause the other point to disappear from the field of view, which may prevent the triangulation technique from being used for that other point. Therefore, the farther apart two points are from each other, the more difficult it becomes to accurately determine the distance between them.

[0026] The present disclosure describes a medical system that measures the distance between two points in a surgical space by utilizing kinematic data that describes the pose (e.g., orientation and position) of an endoscope. Generally, the system adjusts the pose of the endoscope such that the endoscope is close to the first point to triangulate the distance between the first point and the endoscope. The kinematic data indicates the pose of the endoscope, and thus the kinematic data can be used together with the determined distance between the first point and the endoscope to determine the global coordinates of the first point. Then, the system adjusts the pose of the endoscope such that the endoscope is close to the second point to triangulate the distance between the second point and the endoscope. The kinematic data indicates the new pose of the endoscope, and thus the kinematic data can be used together with the determined distance between the second point and the endoscope to determine the global coordinates of the second point. Adjusting the pose of the endoscope such that the endoscope is close to the second point may cause the first point to disappear from the field of view. However, since the kinematic data indicates the pose of the endoscope, it is possible to use the kinematic data to determine the global coordinates of the first point and the second point. Then these global coordinates are used to determine the distance between the first point and the second point.

[0027] In some embodiments, the system provides several technical advantages. For example, the system can provide a more accurate distance measurement between two points in a surgical space. Specifically, the system can bring the endoscope close to the two points to triangulate the depth and / or orientation of the two points, which improves the accuracy of triangulation. The system uses kinematic data to track the pose of the endoscope, which allows the system to use the determined depth and / or orientation of the two points to determine the distance between the two points, even though bringing the endoscope close to one of the points causes the other point to disappear from the field of view. A more accurate measurement of the distance between two points allows the surgeon to precisely locate anatomical structures, lesions, or target areas within the patient's body. This precision is important for performing delicate and targeted surgical procedures that minimize errors. The surgeon can also rely on the information to navigate between complex anatomical structures and avoid unintentional damage to surrounding tissue. The surgeon can use this information to plan and perform procedures with a high degree of confidence, ensuring that critical structures are identified and appropriately treated. More accurate spatial information also helps to guide the placement of instruments and allows for effective navigation within narrow and confined spaces, which helps to efficiently utilize surgical resources (including time and equipment). Thus, the system can improve patient health and safety and reduce recovery time and postoperative complications.

[0028] In some examples, one or more components of the medical system can be implemented as a computer-assisted surgical system. However, it should be understood that the medical system can be implemented in any type of medical system (e.g., a digital reference system, an anatomical body detection system, and a clinical guidance system). Figure 1AFIG. 0 illustrates an example computer-assisted surgical system 100 that can implement some of the features described herein.

[0029] The surgical system 100 includes a manipulator assembly 102, a user control device 104, and an assistance device 106, all of which are communicatively coupled to each other. A medical team uses the surgical system 100 to perform a computer-assisted medical procedure or other similar operation on the body of a patient 108 or any other body that may be used for a particular implementation. The medical team includes a first user 110-1 (e.g., a surgeon for a surgical procedure), a second user 110-2 (e.g., a patient-side assistant), a third user 110-3 (e.g., another assistant, nurse, intern, etc.), and a fourth user 110-4 (e.g., an anesthesiologist for a surgical procedure), all of these users being collectively referred to as users 110, and each user can control the surgical system 100, interact with the surgical system 100, or otherwise be a user of the surgical system 100. During a medical procedure, there may be more, fewer, or alternative users, which may be used for a particular implementation. For example, the team members for different medical procedures or non-medical procedures may be different and include users with different roles.

[0030] Although Figure 1A FIG. 1 shows a minimally invasive medical procedure (e.g., a minimally invasive surgical procedure) in progress, it should be understood that the surgical system 100 can be similarly used to perform an open medical procedure or other types of operations. For example, operations such as exploratory imaging operations, simulated medical procedures for training purposes, and / or other operations can also be performed.

[0031] The manipulator assembly 102 includes one or more manipulator arms 112 (e.g., manipulator arms 112-1 to 112-4), to which one or more instruments can be coupled. These instruments are used to perform a computer-assisted surgical procedure on the patient 108 (e.g., by at least partially inserting into the patient 108's body and manipulating within the patient 108's body). Although the manipulator assembly 102 is depicted and described herein as including four manipulator arms 112, the manipulator assembly 101 can include a single manipulator arm 122 or any other number of manipulator arms, which can be used for a particular implementation. Although the example of FIG. 1 shows the manipulator arms 112 as robotic manipulator arms, one or more instruments can be partially or fully manually controlled (e.g., by being held and manually controlled by a person). These partially or fully manually controlled instruments are used in combination with, or as an alternative to, the computer-assisted instruments coupled to the manipulator arms 112.

[0032] During a medical procedure, the user control device 104 facilitates remote operation control by user 110-1 of the manipulator arm 112 and an instrument attached to the manipulator arm 112. To this end, the user control device 104 provides user 110-1 with an image of the operative area associated with patient 108 captured by the imaging device. The manipulator arm 112 or any instrument coupled to the manipulator arm 122 mimics the dexterity of the hand, wrist, and fingers of user 110-1 in multiple degrees of freedom of movement. In this way, user 110-1 intuitively performs procedures (e.g., cutting procedures, suturing procedures, etc.) using one or more manipulator arms 112 or any instrument coupled to the manipulator arm 112.

[0033] The assistance device 106 includes one or more computing devices that perform assistance functions to support the procedure (e.g., provide insufflation, electrocautery energy, illumination, or other energy for the imaging device, image processing, or coordinating components of the surgical system 100). The assistance device 106 includes a display monitor 114 that displays one or more user interfaces or graphical or textual information to support the procedure. In some cases, the display monitor 114 is a touchscreen display that provides user input functionality. The enhanced content provided by the region-based enhancement system may be similar to or different from the content associated with the display monitor 114 or one or more display devices (not shown) in the operative area.

[0034] The manipulator assembly 102, the user control device 104, and the assistance device 106 are communicatively coupled to each other in any suitable manner. The manipulator assembly 102, the user control device 104, and the assistance device 106 may be communicatively coupled via a control line 116, which represents any wired or wireless communication link that may be used for a particular implementation. To this end, the manipulator assembly 102, the user control device 104, and the assistance device 106 may each include one or more wired or wireless communication interfaces (e.g., one or more local area network interfaces, Wi-Fi network interfaces, cellular interfaces, etc.).

[0035] Figure 1B An exemplary manipulator assembly 102 is shown. As Figure 2A shown, the manipulator assembly 102 includes a base 118, manipulator arms 112-1, 112-2, 112-3, and 112-4. Each manipulator arm 112-1, 112-2, 112-3, and 112-4 is pivotally coupled to the base 118. Although the base 118 may include casters to allow for easy movement, in some embodiments, the manipulator assembly 102 is fixedly mounted to the floor, ceiling, operating table, structural frame, etc.

[0036] In a typical procedure, two of the manipulator arms 112-1, 112-2, 112-3, or 112-4 hold the surgical instrument, and the third manipulator arm holds the stereoscopic endoscope. The remaining manipulator arm can be used to enable other instruments to be introduced at the work site. Alternatively, the remaining manipulator arm can be used to introduce another endoscope or another image capture device (such as an ultrasound transducer) into the work site.

[0037] Each of the manipulator arms 112-1, 112-2, 112-3, and 112-4 is formed by linkages that are coupled together and manipulated by actuatable joints. Each of the manipulator arms 112-1, 112-2, 112-3, and 112-4 can include a setting arm and a device manipulator. The setting arm positions the device it holds such that a pivot point occurs at the entry hole where it enters the patient's body. The device manipulator can then manipulate the device it holds such that the held device can pivot about the pivot point, be inserted into and retracted from the entry hole, and rotate about its shaft axis. Each of the manipulator arms 112-1, 112-2, 112-3, and 112-4 can include sensors (such as joint sensors, orientation sensors, accelerometers, etc.) that detect or track the movement of the manipulator arms 112-1, 112-2, 112-3, and 112-4. For example, these sensors can detect how far or how fast the manipulator arms 112-1, 112-2, 112-3, or 112-4 move in a certain direction.

[0038] Figure 1C An exemplary user control device 104 is shown. The user control device 104 includes a stereoscopic vision display 120 such that the user can view the surgical work site in stereoscopic vision from the images captured by the stereoscopic cameras of the manipulator assembly 102. A left eyepiece 122 and a right eyepiece 124 are provided in the stereoscopic vision display 120 such that the user can view the left display screen and the right display screen within the display 120 with the user's left eye and right eye, respectively. When viewing images of the surgical site on a suitable viewer or display, the surgeon performs a surgical procedure on the patient by manipulating the master control input device, which in turn controls the movement of the robotic instruments.

[0039] The user control device 104 also includes a left input device 126 and a right input device 128 that are grasped by the user with his / her left and right hands, respectively, to preferably manipulate a device (e.g., a surgical instrument) held by the manipulator arms 112-1, 112-2, 112-3, and 112-3 of the manipulator assembly 102 with six or more degrees of freedom (“DOF”). A foot pedal 130 with toe and heel controls is provided on the user control device 104 so that the user can control the movement and / or actuation of a device associated with the foot pedal.

[0040] A processing device 132 is provided in the user control device 104 for control and other purposes. The processing device 132 performs various functions in the surgical system 100. One function performed by the processing device 132 is to translate and transfer the mechanical motion of the input devices 126 and 128 to actuate its corresponding joints in its associated manipulator arms 112-1, 112-2, 112-3, and 112-4 so that a surgeon can effectively manipulate a device (e.g., a surgical instrument). Another function of the processing device 132 is to implement the methods, cross-coupling control logic, and controllers or processors described herein. The assist device 106 may include the processing device 132 that performs the functions or actions described herein. The processing device 132 includes a controller and a memory that perform the functions described herein. The controller may include one or more processors.

[0041] The controller may include any electronic circuitry, including but not limited to one or a combination of a microprocessor, a microcontroller, an application specific integrated circuit (ASIC), an application specific instruction set processor (ASIP), and / or a state machine, which is communicatively coupled to a memory and controls the operation of the user control device 104 and / or the assistive device 106. The controller may be 8-bit, 16-bit, 32-bit, 64-bit, or have any other suitable architecture. The controller may include an arithmetic logic unit (ALU) for performing arithmetic and logical operations, registers for providing operands to the ALU and storing the results of the ALU operations, and a control unit for fetching instructions from the memory and executing the instructions by directing the coordinated operation of the ALU, registers, and other components. The controller may include other hardware for operating software to control and process information. The controller executes software stored on the memory to perform any of the functions described herein. The controller controls the operation and management of the user control device 104 or the assistive device 106 by processing information (e.g., information received from the user control device 104, the manipulator assembly 102, the assistive device 106, and / or the memory). The controller is not limited to a single processing device, but may encompass multiple processing devices included in the same device or computer or distributed across multiple devices or computers. If multiple processing devices jointly perform a set of functions or actions, the controller is considered to have performed the set of functions or actions even if different processing devices perform different functions or actions within the set of functions or actions.

[0042] Figure 2A FIG. 200 illustrates an example computer-assisted surgical system that implements some of the features described herein. The surgical system 200 may be used for, for example, surgery, diagnosis, treatment, biopsy, or non-medical procedures. As Figure 2AAs shown, a surgical system 200 (which can be a robotic-assisted surgical system) includes one or more manipulator components 202 for operating one or more medical device systems 204 when performing various procedures on a patient P located on an operating table T in a medical environment. For example, the manipulator component 202 can drive the movement of a catheter or an end effector, can apply treatment to a target tissue, and / or can manipulate control members. The manipulator component 202 can be a remotely operated, non-remotely operated, or hybrid remotely operated and non-remotely operated component, having degrees of freedom of movement that can be electrified and / or remotely operated and degrees of freedom of movement that can be non-electrified and / or non-remotely operated. An operator input system 206 (which can be inside or outside the medical environment) generally includes one or more control devices for controlling the manipulator component 202. The manipulator component 202 supports the medical device system 204 and can optionally include a plurality of actuators or motors that drive inputs on the medical device system 204 in response to commands from a control system 212. The actuators can optionally include drive systems that, when coupled to the medical device system 204, can advance the medical device system 204 into natural or surgically created anatomical openings. Other drive systems can move the distal end of the medical device in multiple degrees of freedom, which can include three linear degrees of movement (e.g., linear movement along the x, y, and z Cartesian axes) and three rotational degrees of movement (e.g., rotation about the x, y, and z Cartesian axes). The manipulator component 202 can support various other systems for irrigation, treatment, or other purposes. Such systems can include fluid systems (e.g., reservoirs, heating / cooling elements, pumps, and valves), generators, lasers, interrogators, and ablation components.

[0043] The surgical system 200 also includes a display system 210 for displaying images or representations of the surgical site and the medical device system 204. The images or representations are generated by an imaging system 209, which can include an endoscopic imaging system. The display system 210 and the operator input system 206 can be oriented such that an operator O can control the medical device system 204 and the operator input system 206 through the perception of telepresence. A graphical user interface can be displayable on the display system 210 and / or on the display system of a separate planning workstation.

[0044] In some examples, the imaging system 209 includes an endoscopic imaging system having components that are integrally or removably coupled to the medical device system 204. However, in some examples, an independent imaging device (e.g., an endoscope) attached to an independent manipulator assembly can be used with the medical device system 204 to image a surgical site. The imaging system 209 can be implemented as hardware, firmware, software, or a combination thereof that interacts with or is otherwise executed by one or more computer processors, which can include a controller 214 of the control system 212.

[0045] The surgical system 200 also includes a sensor system 208. The sensor system 208 can include an orientation / position sensor system (e.g., an actuator encoder or an electromagnetic (EM) sensor system) and / or a shape sensor system (e.g., an optical fiber shape sensor) for determining the orientation, alignment, speed, velocity, pose, and / or shape of the medical device system 204. These sensors can also detect the orientation, alignment, or pose of the patient P on the operating table T. For example, the sensors can detect whether the patient P is face down or face up. As another example, the sensors can detect the direction in which the patient P's head is pointed. The sensor system 208 can also include temperature sensors, pressure sensors, force sensors, or contact sensors, etc.

[0046] The surgical system 200 can also include a control system 212 that includes at least one memory 216 and at least one controller 214 (which can include a processor) for implementing control between the medical device system 204, the operator input system 206, the sensor system 208, and the display system 210. The control system 212 includes programming instructions (e.g., a non-transitory machine-readable medium storing the instructions) to implement procedures using the surgical system 200, which include navigation, steering, imaging, deployment or retraction of engagement features, applying treatment to target tissue (e.g., via application of energy), etc.

[0047] The control system 212 may also include a virtual visualization system to provide navigation assistance to the operator O when controlling the medical device system 204 during an image-guided surgical procedure. Virtual navigation using the virtual visualization system may be based on a reference to a preoperative or intraoperative data set of the acquired anatomical passage. The virtual visualization system processes images of the surgical site imaged using imaging techniques such as computed tomography (CT), magnetic resonance imaging (MRI), fluoroscopy, thermography, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, and the like. The control system 212 uses preoperative images to locate the target tissue (using visual imaging techniques and / or by receiving user input) and create a preoperative plan (including the optimal first position for performing the treatment). The preoperative plan may include, for example, planning the size of an expandable device to be expanded, treatment duration, treatment temperature, and / or multiple deployment positions.

[0048] The controller 214 is any electronic circuitry, including but not limited to one or a combination of a microprocessor, a microcontroller, an application-specific integrated circuit (ASIC), an application-specific instruction set processor (ASIP), and / or a state machine, which is communicatively coupled to the memory 216 and controls the operation of the control system 212. The controller 214 may be 8-bit, 16-bit, 32-bit, 64-bit, or have any other suitable architecture. The controller 214 may include an arithmetic logic unit (ALU) for performing arithmetic and logical operations, processor registers for providing operands to the ALU and storing the results of ALU operations, and a control unit for fetching instructions from the memory and executing the instructions by directing the coordinated operation of the ALU, registers, and other components. The controller 214 may include other hardware for operating software to control and process information. The controller 214 executes software stored on the memory 216 to perform any of the functions described herein. The controller 214 controls the operation and management of the control system 212 by processing information (e.g., information received from the manipulator assembly 202, the operator input system 206, and the memory 216). The controller 214 is not limited to a single processing device and may encompass multiple processing devices included in the same device or computer or distributed across multiple devices or computers. If multiple processing devices jointly perform a set of functions or actions, the controller 214 is considered to perform the set of functions or actions even if different processing devices perform different functions or actions within the set of functions or actions.

[0049] The memory 216 may store data, operating software, or other information of the controller 214 permanently or temporarily. The memory 216 may include any one or a combination of volatile or non-volatile local or remote devices suitable for storing information. For example, the memory 216 may include random access memory (RAM), read-only memory (ROM), magnetic storage devices, optical storage devices, or any other suitable information storage device or a combination of these devices. The software represents any suitable set of instructions, logic, or code embodied in a computer-readable storage medium. For example, the software may be embodied in the memory 216, a disk, a CD, or a flash drive. In a particular embodiment, the software may include an application program executable by the controller 214 to perform one or more functions described herein. The memory 216 is not limited to a single memory and may encompass multiple memories included in the same device or computer or distributed across multiple devices or computers. If multiple memories jointly store a set of data, operating software, or information, the memory 216 is considered to store the set of data, operating software, or information even if different memories store different portions of the data, operating software, or information in the set.

[0050] Figure 2B An example medical device system 204 in the surgical system 200 is shown. In some embodiments, the medical device system 204 is used for image-guided medical procedures. For example, the medical device system 204 may be used for non-remotely operated exploratory procedures or procedures involving traditional manually operated medical devices (such as endoscopes).

[0051] The medical device system 204 includes an elongate flexible device 220 (such as a flexible catheter or an endoscope (such as a gastroscope, bronchoscope)) coupled to a drive unit 222. The elongate flexible device 220 includes a flexible body 224 having a proximal end 226 and a distal or tip portion 228. In some embodiments, the flexible body 224 has an outer diameter of approximately 14 millimeters - 20 millimeters. Other flexible body outer diameters may be larger or smaller. The flexible body 224 has an appropriate length so that when the flexible body 224 is inserted into a patient's oral or nasal cavity, it can reach certain parts of the anatomical body (such as the lungs, sinuses, throat, or upper or lower digestive tract regions).

[0052] The medical device system 204 includes a tracking system 230 that is configured to determine the position, orientation, velocity, speed, pose, and / or shape of the distal end 228 and / or one or more segments 232 along the flexible body 224 using one or more sensors and / or imaging devices. The flexible body 224 is effectively divided into segments 232 over its entire length between the distal end 228 and the proximal end 226. The tracking system 230 is implemented as hardware, firmware, software, or a combination thereof that interacts with or is otherwise executed by one or more computer processors, which may include the controller 214 of the control system 212.

[0053] The tracking system 230 uses a shape sensor 234 to track the distal end 228 and / or one or more segments 232. In some embodiments, the tracking system 230 uses an orientation sensor system 236 (such as an electromagnetic (EM) sensor system) to track the distal end 228. In some examples, the orientation sensor system 236 measures six degrees of freedom (e.g., three position coordinates x, y, and z and three orientation angles indicating pitch, yaw, and roll about a reference point) or five degrees of freedom (e.g., three position coordinates x, y, and z and two orientation angles indicating pitch and yaw about a reference point).

[0054] The flexible body 224 includes one or more channels 238 sized and shaped to receive one or more medical devices 240. In some embodiments, the flexible body 224 includes two channels 238 for individual devices 240, although a different number of channels 238 may be provided. Figure 2C Shown Figure 2B is an example portion of the medical device system 204. As Figure 2CAs shown, the medical device 240 extends through the flexible body 224. In some embodiments, the medical device 240 can be used for some procedures and aspects of procedures, such as surgery, biopsy, ablation, mapping, imaging, illumination, irrigation, or aspiration. The medical device 240 is deployed through the channel 238 of the flexible body 224 and used at a target location within the anatomy. The medical device 240 includes, for example, an image capture device, a biopsy instrument, an ablation instrument, a catheter, a laser ablation fiber, and / or other surgical, diagnostic, or therapeutic tools. The medical tool includes an end effector having a single working member, such as a scalpel, a blunt blade, a lens, an optical fiber, an electrode, etc. Other end effectors include, for example, forceps, grippers, balloons, needles, scissors, clip appliers, etc. Other end effectors also include electrically activated end effectors, such as electrosurgical electrodes, transducers, sensors, imaging devices, etc. The medical device 240 is advanced from the opening of the channel 238 to perform a procedure and then retracted into the channel when the procedure is complete. The medical device 240 is removed from the proximal end 226 of the flexible body 224 or from another optional instrument port (not shown) along the flexible body 224. The medical device 240 can be used in conjunction with an image capture device (such as an endoscopic camera) that is also within the elongated flexible device 220. Alternatively, the medical device 240 itself can be an image capture device.

[0055] The medical device 240 additionally houses cables, linkages, or other actuation controls (not shown) that extend between the proximal and distal ends to controllably bend the distal end of the medical device 240. The flexible body 224 also houses cables, linkages, or other steering controls (not shown) that extend between the drive unit 222 and the distal end 228 to controllably bend the distal end 228, as depicted by the dashed lines 242 of the distal end 228. In some examples, at least four cables are used to provide independent "up and down" steering to control the pitch movement of the distal end 228 and "left and right" steering to control the yaw movement of the distal end 228. In embodiments where the medical device system 204 is actuated by a robotic assist component, the drive unit 222 can include a drive input that removably couples to and receives power from a drive element (such as an actuator) of a remote operation component. In some embodiments, the medical device system 204 includes gripping features, manual actuators, or other components for manually controlling the movement of the medical device system 104. Information from the tracking system 230 can be sent to the navigation system 244, where the information is combined with information from the visualization system 246 and / or a preoperatively obtained model to provide real-time orientation information to a doctor or other operator.

[0056] Figures 3A to 13 shown by a medical system (e.g., Figure 1A surgical system 100 orFigure 2A Example operations performed by a computer system in a surgical system 200). Generally, the computer system (which may be implemented using a processing device 132 in a user control device 104 and / or an auxiliary device 106 of a surgical system 100, and / or using a controller 214 and a memory 216 in a control system 212 of a surgical system 200) measures the distance between two points in a surgical space by utilizing kinematic data and depth data.

[0057] Figure 3A Example operation 300A performed by the computer system is shown. The computer system receives videos 306A and 306B captured by cameras 304A and 304B (e.g., the left and right cameras of a stereo camera) of an imaging device 302 (e.g., an endoscope). Cameras 304A and 304B are positioned in front of the imaging device 302. Videos 306A and 306B can be captured from a perspective in front of the imaging device 302. The imaging device 302 can be set in a first pose (e.g., azimuth and orientation) such that the videos 306A and 306B show the surface of a first region of an anatomical structure (e.g., at a surgical site). Information from the videos 306A and 306B can form imaging data 330 of the first region of the anatomical structure.

[0058] The computer system monitors and tracks kinematic data 310 indicating the first pose of the imaging device 302. The kinematic data 310 can include three degrees of freedom of azimuth and three degrees of freedom of orientation. The three degrees of freedom of azimuth provide translational information of the imaging device 302 and indicate the azimuth 312 of the imaging device 302. The three degrees of freedom of orientation provide rotational information of the imaging device 302 and indicate the orientation 314 of the imaging device 302.

[0059] The computer system makes a distance measurement 320 from the imaging device 302 in the first pose to a first point on the first region of the anatomical structure. The distance measurement 320 can indicate a first distance between the imaging device 302 in the first pose and the first point. In some embodiments, the computer system uses triangulation to determine the distance measurement 320. For example, the distance between camera 304A and camera 304B (which can be referred to as a baseline or center point distance) can be provided to the computer system. Then, the computer system uses the imaging data 330 (which shows the first point) and the distance between cameras 304A and 304B to triangulate the distance from the imaging device 302 to the first point.

[0060] In some embodiments, the computer system uses video 306A from camera 304A to determine the distance between camera 304A and a first point. Then, the computer system can use video 306B from camera 304B to determine the distance between camera 304B and the first point. Then, the computer system uses the distance between camera 304A and the first point, the distance between camera 304B and the first point, and the distance between camera 304A and camera 304B to calculate the distance between imaging device 302 and the first point. For example, the computer system can calculate the distance between imaging devices 302 as the height of a triangle, where the side lengths of the triangle are equal to the distance between camera 304A and the first point, the distance between camera 304B and the first point, and the distance between camera 304A and camera 304B.

[0061] Then, the computer system uses kinematic data 310 indicating the orientation 312 and the orientation 314 of imaging device 302 together with distance measurements 320 to determine the coordinates 340 of the first point in the first region. For example, the orientation 312 can indicate the global coordinates of imaging device 302. The orientation 314 can indicate the direction in which imaging device 302 is pointing. Then, the computer system can determine the global coordinates of the first point by calculating the coordinates that are the distance measurements 320 away from the orientation 312 in the direction indicated by the orientation 314. These global coordinates of the first point can be the coordinates 340. The coordinates 340 are displayed on the display and can be represented as Cartesian coordinates (x1, y1, z1).

[0062] Figure 3B Shows an example operation 300B performed by the computer system. As Figure 3B shown, imaging device 302 has been moved to a second pose. For example, imaging device 302 may have been moved to a different orientation and / or rotated to a different orientation. In addition, imaging device 302 can be moved from the first pose to the second pose by a relocatable structure (such as Figure 1A the manipulator arms 112-1 to 112-4 described in). When imaging device 302 is in the second pose, cameras 304A and 304B capture videos 306C and 306D of the second region of the surface of the anatomical structure. Videos 306C and 306D form imaging data 370 of the second region of the anatomical structure.

[0063] The computer system monitors and tracks kinematic data 350 indicative of a second pose of the imaging device 302. The kinematic data 350 includes three degrees of freedom of position and three degrees of freedom of orientation. The three degrees of freedom of position provide translational information of the imaging device 302 and indicate the position 352 of the imaging device 302. The three degrees of freedom of orientation provide rotational information of the imaging device 302 and indicate the orientation 354 of the imaging device 302. Since the imaging device 302 moves from the first pose to the second pose, the position 352 and the orientation 354 may be different from the position 312 and the orientation 314 of the imaging device 302.

[0064] The computer system performs a distance measurement 360 from the imaging device 302 in the second pose to a second point on a second region of the anatomical structure. The distance measurement 360 may indicate a second distance between the imaging device 302 in the second pose and the second point. Similar to the first pose, the computer system may perform the distance measurement 360 by triangulating the distance between the imaging device 302 and the second point.

[0065] Then, the computer system uses the kinematic data 350 indicative of the position 352 and the orientation 354 of the imaging device 302 together with the distance measurement 360 to determine coordinates 380 of the second point of the second region. Similar to the first pose, the computer system may determine the global coordinates of the second point as the coordinates 380 by determining the coordinates that are the distance measurement 360 away from the position 352 in the direction indicated by the orientation 354. The coordinates 380 are displayed on a display and may be represented as Cartesian coordinates (x2, y2, z2).

[0066] Figure 3C An example operation 300C performed by the computer system is shown. In operation 300C, the computer system uses the coordinates 340 of the first point and the coordinates 380 of the second point to calculate a distance 390 between the first point and the second point. In one example, the distance 390 is a straight-line distance 392, which may be calculated as the square root of the sum of the squares of the differences between the coordinate pairs of the coordinates 340 and the coordinates 380 (e.g., sqrt((x2 - x1)^2 + (y2 - y1)^2 + (z2 - z1)^2)). In another example, the distance 390 is a curved distance 392 on the surface of the anatomical structure. The curved distance 392 may be measured along the surface of the anatomical structure from the first point (coordinates 340) to the second point (coordinates 380). The curved distance 392 may also be referred to as a surface distance.

[0067] In some embodiments, an operator of a computer system uses an imaging device 302 to indicate a first point and a second point to the computer system. For example, the operator can adjust the pose of the imaging device 302 to point the imaging device 302 at the first point or the second point. In this way, the operator uses the imaging device 302 to mark the first point or the second point for the computer system. Marking the first point or the second point can be separate from and independent of the system 100, the system 200, or other control features provided by the computer system. For example, marking the first point or the second point can be separate from and independent of a virtual control mode, in which the computer system allows the operator to move a virtual cursor to interact with the displayed user interface elements. As another example, marking the first point or the second point can be separate from and independent of moving a surgical instrument using the computer system.

[0068] Figure 4 is performed by Figure 1A system 100 or Figure 2A FIG. 400 is a flowchart of an example method 400 performed by system 200. In a particular embodiment, the computer system performs method 400. By performing method 400, the computer system implements certain features that assist a user in measuring the distance between any two points in a 3D space. These features improve the accuracy of distance measurement, which can enhance the safety and efficacy of a medical procedure.

[0069] In block 402, the computer system moves the imaging device 302 to a first pose. The imaging device 302 can include a camera 304. For example, the imaging device 302 can include a left camera and a right camera that form a stereo camera. These cameras 304 are positioned to capture images or video from slightly different perspectives.

[0070] In block 404, the computer system receives first imaging data 330 captured using the imaging device 302 in the first pose. The first imaging data 330 can include video 306 captured by the camera 304 when the imaging device 302 is in the first pose. The computer system monitors or tracks kinematic data 310 indicating the first pose (e.g., azimuth 312 and orientation 314) of the imaging device 302. Generally, the computer system sets the first pose of the imaging device 302 such that the imaging device 302 is positioned close to a first point (e.g., within ten centimeters or five centimeters of the first point) on a first region of an anatomical structure to capture the imaging data 330.

[0071] In block 406, the computer system determines a first distance between the imaging device 302 in the first pose and a first point. As an example, the computer system can use the imaging data 330 and the distance between the cameras 304 on the imaging device 302 to triangulate the distance between the imaging device 302 and the first point.

[0072] In block 408, the computer system determines the coordinates 340 of the first point. The computer system can use the first pose and the first distance to determine the coordinates 340, which can be represented as Cartesian coordinates (x1, y1, z1). For example, the computer system can determine the coordinates that are a first distance away from the orientation 312 in the direction indicated by the orientation 314.

[0073] In block 410, the computer system moves the imaging device 302 to a second pose. The second pose is different from the first pose, and when the imaging device 302 is moved to the second pose, the first point may disappear from the field of view of the camera 304. The imaging device 302 can be moved from the first pose to the second pose by a relocatable structure (such as Figure 1A the manipulator arms 112-1 to 112-4 described in

[0074] In block 412, the computer system receives second imaging data 370 captured using the imaging device 302 in the second pose. The second imaging data 370 can include video 306 captured by the camera 304 when the imaging device 302 is in the second pose. The computer system monitors or tracks the kinematic data 350 indicating the second pose of the imaging device 302 (such as the orientation 352 and the orientation 354). Generally, the computer system sets the second pose of the imaging device 302 such that the imaging device 302 is close to a second point on a second region of the anatomical structure (e.g., within ten centimeters or five centimeters of the second point).

[0075] In block 414, the computer system determines a second distance between the imaging device 302 in the second pose and the second point. The computer system can use the imaging data 370 and the distance between the cameras 304 on the imaging device 302 to triangulate the distance between the imaging device 302 and the second point.

[0076] In block 416, the computer system determines the coordinates 380 of the second point. The computer system can use the second pose and the second distance to determine the coordinates 380, which can be represented as Cartesian coordinates (x2, y2, z2). For example, the computer system can determine the coordinates that are a second distance away from the orientation 352 in the direction indicated by the orientation 354.

[0077] In block 418, the computer system determines a third distance between a first point and a second point. The computer system can use coordinate 340 and coordinate 380 to determine the third distance. In one example, the computer system calculates the third distance as a straight-line distance. In another example, the computer system determines the third distance as a distance on the surface of the anatomical structure (e.g., a curved distance). The computer system can display the third distance on the surface of the anatomical structure in video 306, and the computer system can update the third distance based on various movements of imaging device 302 and / or the anatomical structure itself.

[0078] Figure 5 Illustrates example operation 500 performed by a computer system. Generally, the computer system performs operation 500 to determine the distance on the surface of an anatomical structure between two points on the surface of the anatomical structure. The computer system uses a Simultaneous Localization and Mapping (SLAM) process to generate a model of the anatomical structure. The computer system then uses the model to determine distance 394 along the surface of the anatomical structure.

[0079] The computer system receives video 502 of the anatomical structure. Video 502 can be captured by one or more cameras 304 of imaging device 302. The computer system uses SLAM process 512 to generate a model 510 of the anatomical structure from video 502. Model 510 effectively serves as a virtual map of the anatomical structure. SLAM allows the computer system to map the environment in video 502 while determining the orientation of camera 304 that captured video 502 within that environment. The computer system can use kinematic data 310 and 350 to position camera 304 within the environment. By positioning camera 304, the computer system can determine the orientation and / or pose of camera 304 within the environment. The computer system can then present the map along with the orientation and / or pose of camera 304 on a display, which provides a broader view of imaging device 302 relative to the surface of the anatomical structure.

[0080] During SLAM process 512, the computer system can stitch together individual frames of video 502 to form a virtual map of the anatomical structure. The computer system can also add virtual geometry to the virtual map. When movement occurs, the computer system can compare the changing views in the video to the virtual map of the anatomical structure to determine the movement or updated pose of the camera. Thus, model 510 maps the surface of the anatomical structure.

[0081] When determining the distance 394 along the surface of the anatomical structure between the first point and the second point, the computer system may use the model 510. For example, the computer system may have determined the coordinates 340 and 380 of the first point and the second point. Then, the computer system may determine the curve along the surface of the anatomical structure in the model 510 between the two coordinates 340 and 380. Then, the computer system may determine the length of the curve as the distance 394 along the surface of the anatomical structure between the first point and the second point.

[0082] Figures 6A to 6C Illustrate example operations 600A, 600B, and 600C performed by a computer system. Generally, the computer system moves the imaging device 302 by moving the imaging device along the surface of the anatomical structure to determine the coordinates of the first point of the first region and the coordinates of the second point of the second region, thereby determining a third distance between the first point and the second point.

[0083] Figure 6A Illustrate the imaging device 302 in a first pose P1, which may include the orientation and orientation of the imaging device 302. In the first pose P1, the imaging device 302 captures first imaging data on the surface 612 of the anatomical structure 610. The computer system uses the first imaging data to triangulate the first distance d1 between the imaging device 302 and the first point 614 on the surface 612. The first distance d1 may be less than ten centimeters or five centimeters. The first distance d1 may be determined based on the axis of the imaging device 302. For example, the first distance d1 may be along the optical axis of one of the cameras of the imaging device 302 in the first pose P1. As another example, the distance d1 may be along a combination of the axes of the cameras of the imaging device 302 in the first pose P1 (e.g., a synthetic sum). Based on the first pose P1 and the first distance d1, the computer system determines the first coordinates of the first point 614 on the surface 612. The first coordinates may be represented as Cartesian coordinates T1(x1, y1, z1), which may be the global coordinates of the first point 614.

[0084] Figure 6BIt is shown that the imaging device 302 moves from the first pose P1 to the second pose P2. In the second pose P2, the imaging device 302 captures second imaging data on the surface 612 of the anatomical structure 610. When the imaging device 302 is moved to the second pose P2, the first point 614 may disappear from the field of view. The computer system uses the second imaging data to triangulate a second distance d2 between the imaging device 302 and a second point 616 on the surface 612. The second distance d2 can be less than ten centimeters or five centimeters. The second distance d2 can be determined based on the axis of the imaging device 302. For example, the first distance d2 can be along the optical axis of one of the cameras of the imaging device 302 in the second pose P2. As another example, the distance d2 can be along a combination of the axes of the cameras of the imaging device 302 in the second pose P2 (e.g., synthesized sum). Based on the second pose P2 and the second distance d2, the computer system determines the second coordinates of the second point 616 on the surface 612. The second coordinates can be represented as Cartesian coordinates T2(x2, y2, z2), which can be the global coordinates of the second point 616.

[0085] Figure 6C It is shown that the computer system determines a distance 620 between the first point 614 and the second point 616. The distance 620 can be a straight-line distance calculated directly from the coordinates T1 and T2. Alternatively or additionally, the distance 620 can be the length of the curve along the surface 612 between the first point 614 and the second point 616.

[0086] Figure 7 is in Figure 1A system 100 of Figure 2A The flowchart of an example method 700 executed in system 200. In a particular embodiment, the computer system executes method 700. By executing method 700, the computer system implements certain features to assist a user in measuring the distance between two points. For example, the computer system determines whether the triangulated distance between the imaging device 302 and a point is below a specific threshold, which can improve the accuracy of these distance measurements and the accuracy of the determined distance between the two points.

[0087] In block 702, the computer system determines a first distance between the imaging device 302 in the first pose and a first point on the surface. For example, the computer system can use the video captured by the imaging device 302 to triangulate the first distance.

[0088] In block 704, the computer system determines whether the first distance is less than a predetermined threshold (e.g., ten centimeters or five centimeters). If the first distance exceeds the threshold, then in block 705, the computer system adjusts the pose of the imaging device 302 such that the imaging device 302 moves closer to the first point. Then, the computer system returns to block 702 to determine the first distance.

[0089] If the first distance does not exceed the threshold, then in block 706, the computer system moves the imaging device 302 to a different region on the surface of the anatomical structure such that the imaging device 302 is in a second pose. Moving the imaging device 302 may cause the first point to disappear from the field of view. The imaging device 302 can be moved from the first pose to the second pose by a relocatable structure (e.g., Figure 1A the manipulator arms 112-1 to 112-4 described in

[0090] In block 708, the computer system determines a second distance between the imaging device 302 in the second pose and a second point on the surface. The computer system can triangulate the second distance using the video captured by the imaging device 302.

[0091] In block 710, the computer system determines whether the second distance is less than a predetermined threshold (e.g., ten centimeters or five centimeters). If the second distance exceeds the threshold, then in block 711, the computer system adjusts the pose of the imaging device 302 such that the imaging device 302 is moved closer to the second point. Then, the computer system returns to block 708 to determine the second distance.

[0092] If the second distance does not exceed the threshold, then in block 712, the computer system determines a third distance between the first point and the second point.

[0093] For example, the computer system can determine the global coordinates of the first point by determining the coordinates that are the first distance away from the coordinates of the imaging device 302 in the first pose in the direction indicated by the first pose. The computer system can also determine the global coordinates of the second point by determining the coordinates that are the second distance away from the coordinates of the imaging device 302 in the second pose in the direction indicated by the second pose. Then, the computer system uses the global coordinates of the first point and the second point to determine the third distance between the first point and the second point.

[0094] In one example, the computer system determines a threshold based on at least one of a depth measurement at a first point, a depth measurement at a second point, or a kinematic error. Triangulation techniques use the baseline (center point) distance between the left and right cameras of an imaging device and the azimuth difference of a first point between the images from the left and right cameras to determine the azimuth of the first point and the distance between the first point and the imaging device. Similarly, triangulation techniques use the baseline (center point) distance between the left and right cameras of an endoscope and the azimuth difference of a second point between the images from the left and right cameras to determine the azimuth of the second point and the distance between the second point and the imaging device. The baseline distance between the left and right cameras may be small, which results in the farther the first point and the second point are from the imaging device, the less accurate the determined distances between the first point and the second point and the imaging device are. Moving the imaging device closer to the first point or the second point can improve the accuracy of the distance measurement.

[0095] The computer system may set a threshold based on kinematic error. Kinematic error may be related to inaccuracies or deviations in the movement of the imaging device 302 during a medical procedure. Kinematic error may occur if there is a difference between the input or instructions of the surgeon and the movement of the imaging device 302. In one example, kinematic error may be related to calibration issues. If the threshold is set to ten centimeters and the computer system detects a calibration issue that affects the accuracy of the distance measurement to the first point or the second point, a notification or warning may be provided to the user to move the imaging device 302 closer to the surface of the anatomical structure by less than ten centimeters (e.g., set the threshold to five centimeters) to avoid the detected calibration issue. Thus, the detection of kinematic error may cause the user to move the imaging device 302 closer to the first point and / or the second point.

[0096] The computer system may set a threshold such that the distances between the first point and the second point and the imaging device 302 are accurate. The threshold indicates to the user how far the imaging device can be positioned from the first point and the second point. The threshold indication allows the user to adjust the positioning of the imaging device 302 (e.g., move the imaging device 302 closer to or farther from the first point and the second point) to obtain a more accurate distance measurement. The threshold may be based on the distance measurement. The computer system measures the distance from the first point to the left camera and measures the distance from the first point to the right camera. If the difference between the two measured distances is too large (e.g., exceeds a difference threshold), this difference may indicate an inaccuracy in the distance measurement. In response, the computer system may lower the threshold to indicate that the imaging device 302 should be moved closer to the first point to improve the accuracy of the distance measurement.

[0097] In another example, the computer system uses an error tolerance and a look-up table to set the threshold (as described below with reference to Figure 8 ), rather than using triangulation techniques.

[0098] Figure 8 Illustrates example operation 800 performed by a computer system. Generally, a computer system may use error tolerances and a look-up table (LUT) to assist in setting a first threshold and a second threshold.

[0099] The computer system captures video 306 from two cameras of the imaging device 302 and receives an error tolerance 810 from the user. The error tolerance 810 indicates the amount of error in distance measurements that the user or surgical procedure can tolerate (e.g., for safety reasons). The computer system then refers to a look-up table (LUT) 812 to determine the thresholds. The LUT 812 may indicate various distances between the imaging device 302 and a point and the error that may be introduced in distance measurements at those distances. By referring to the LUT 812 using the error tolerance 810, the computer system can determine the maximum distance in the LUT 812 that results in an error within the error tolerance 810. For example, the computer system may determine the error in the LUT 812 that is closest to the error tolerance 810 without exceeding the error tolerance 810. The computer system then determines the distance in the LUT 812 that maps to that error. The computer system sets the threshold to that distance at block 814. In this way, the computer system uses the error tolerance 810 and the LUT 812 to set the threshold for the distance between the imaging device 302 and a point.

[0100] Figure 9 Illustrates example operation 900 performed by a computer system. Generally, the computer system displays virtual markers to indicate a first point and a second point.

[0101] The computer system determines the coordinates 340 of the first point and the coordinates 380 of the second point. The computer system then generates an overlay 910 that includes virtual markers 920 and 922. The virtual marker 920 may indicate the first point, while the virtual marker 922 may indicate the second point. The virtual markers 920 and 922 may be arranged in the overlay 910 according to the coordinates 340 and 380 such that when the computer system positions the overlay 910 over a video of the anatomy on the display 930, the virtual markers 920, 922 are positioned over the first point and the second point.

[0102] The first virtual marker 920 and the second virtual marker 922 can have various rendering attributes, including for example shape, color, size, transparency, surface pattern, text, any other rendering attribute, and / or combinations thereof. Such rendering attributes of the virtual markers 920 and 922 can be used to indicate direction (e.g., front or back) and orientation. In one example, the first virtual marker 920 and the second virtual marker 922 can be shaped as cylinders and may not indicate direction or orientation. In another example, the first virtual marker 920 and the second virtual marker 922 can be shaped as cones, and color changes, patterns, symbols, text, and / or combinations thereof can be used to indicate the direction and / or orientation associated with the first virtual marker 920 and the second virtual marker 922. The first virtual marker 920 and the second virtual marker 922 can include any shape. For example, the first virtual marker 920 and the second virtual marker 922 can include one-dimensional shapes (e.g., straight lines), two-dimensional shapes (e.g., triangles, squares, rectangles, circles, ellipses), and / or three-dimensional shapes (e.g., cylinders, pyramids, prisms, cubes, rectangular prisms).

[0103] Figure 10 An example operation 1000 performed by a computer system is shown. Generally, the computer system displays a determined distance 390 between a first point and a second point on a display 1010.

[0104] The distance 390 can be determined according to Figure 3A 、 Figure 3B and Figure 3C the operations 300A, 300B, and 300C shown in. When there is movement (e.g., movement of the imaging device 302 or movement of the anatomical structure), the computer system also updates the distance 390.

[0105] The computer system detects one or more of the movement 1020 of the imaging device 302 or the movement 1022 of the anatomical structure. The computer system can detect the movement 1020 and / or the movement 1022 by analyzing the frames of the video of the anatomical structure. The difference in frames can indicate movement. When all pixels move between subsequent frames, the computer system can detect the movement 1020 of the imaging device 302. When a subset of pixels moves between subsequent frames, the computer system can detect the movement 1022 of the anatomical structure. In some embodiments, the computer system detects the movement 1020 of the imaging device 302 by monitoring or tracking the kinematic data of the imaging device 302. When the kinematic data indicates a change in the pose of the imaging device 302, the computer system can analyze the change in the kinematic data to determine the movement 1020 of the imaging device 302.

[0106] When movement 1020 of the imaging device 302 occurs, the computer system can update distance 390 to produce an updated distance 1030. For example, movement 1020 of the imaging device 302 can change the pose of the imaging device 302, but still keep the imaging device 302 close to the first point or the second point. Then, the computer system can re-triangulate the distance between the imaging device 302 and the first point or the second point, and use the re-triangulated distance and the new pose of the imaging device 302 to re-determine the global coordinates of the first point or the second point. Then, the computer system uses the re-determined global coordinates of the first point or the second point when updating distance 390. In some embodiments, the computer system can combine (e.g., average) the determined global coordinates of the first point or the second point, and use the combined global coordinates as the global coordinates of the first point or the second point. Then, the computer system uses the combined global coordinates to update distance 390.

[0107] When movement 1022 of an anatomical structure occurs, the computer system can update distance 390 to produce an updated distance 1030. For example, as part of normal physiological functions, the orientation or shape of an anatomical structure may change. Digestive organs (e.g., the stomach) undergo coordinated contractions called peristalsis. During breathing, other organs may move due to pressure changes within the thoracic cavity. These changes in orientation or shape may change the distance between a first point and a second point along the surface of the anatomical structure. When the computer system detects movement 1022, the computer system can use a SLAM process to update the model of the anatomical structure that serves as a virtual map of the anatomical structure. Then, the computer system can use the updated model to update the distance 390 between the first point and the second point along the surface of the anatomical structure.

[0108] Figure 11 is in Figure 1A system 100 or Figure 2A is a flowchart of an example method 1100 executed in system 200. In a particular embodiment, the computer system executes method 1100. By executing method 1100, the computer system uses the distance between a first point and a second point to determine whether there is sufficient space to safely perform a medical procedure.

[0109] In block 1102, the computer system determines first coordinates 340 of a first point on the surface of the anatomical structure. Coordinates 340 can be determined according to Figure 3A operation 300A shown in

[0110] In block 1104, the computer system determines second coordinates 380 of a second point on the surface of the anatomical structure. Coordinates 380 can be determined according to Figure 3B operation 300B shown in

[0111] In block 1106, the distance 390 between the first point and the second point is determined based on the coordinates 340 of the first point and the coordinates 380 of the second point. The distance 390 can be determined according to Figure 3C the operation 300C shown therein.

[0112] In block 1108, the computer system determines whether the distance 390 falls below a target distance. The target distance can ensure that there is sufficient space between the first point and the second point to safely perform a medical procedure (such as a surgical operation or an incision). If the distance 390 falls below the target distance, the computer system generates an alert in block 1110 to indicate that there is not enough space to perform the medical procedure. If the distance exceeds the target distance, the computer system outputs the distance 390 (e.g., displays the distance 390) to assist with the medical procedure.

[0113] Figure 12 An example operation 1200 performed by the computer system is shown. Generally, the computer system determines the distance between the first point 614 and the second point 616, and if the distance falls below a threshold, an alert is generated.

[0114] In a first step 1210, the computer system sets the imaging device 302 in a first pose P1. In the first pose P1, the imaging device 302 is close to the surface 612 of the anatomical structure 610. The computer system (e.g., using triangulation) determines a first distance d1 between the imaging device 302 in the first pose and the first point 614 on the surface 612. Based on the first pose and the first distance, the computer system determines a first coordinate T1 of the first point 614. In one example, the first distance d1 can be less than ten centimeters. In other examples, the distance d1 can be less than five centimeters.

[0115] In a second step 1220, the computer system moves the imaging device 302 to a second pose P2 to capture second imaging data showing a second region of a surface 612 of an anatomical structure 610. Moving the imaging device 302 may cause a first point 614 to disappear from the field of view of the imaging device 302. The computer system determines a second distance d2 between the imaging device 302 in the second pose and a second point 616 on the surface 612. Based on the second pose P2 and the second distance d2, the computer system determines second coordinates T2 of the second point 616. In one example, the second distance d2 may be less than ten centimeters. In other examples, the distance d2 may be less than five centimeters. Moving the stereo cameras of the imaging device 302 closer to the first point 614 or the second point 616 can improve the accuracy of distance measurement. Determining the correct distances between the first point 614 and the second point 616 and the imaging device 302 can be done by setting distance thresholds. The computer system may set these thresholds such that the distances between the first point 614 and the second point 616 and the imaging device 302 are accurate. These thresholds indicate to the user how far the imaging device 302 may be from the first point 614 and the second point 616. These thresholds allow the user to adjust the positioning of the imaging device 302 (e.g., move the imaging device 302 closer to or farther from the first point 614 and the second point 616) to obtain more accurate distance measurements.

[0116] The computer system uses the first coordinates T1 and the second coordinates T2 to determine a third distance d3 between the first point 614 and the second point 616. Then, the computer system determines whether the third distance d3 falls below a target distance. If the third distance d3 falls below the target distance, the computer system generates an alert to indicate that the third distance d3 falls below the target distance. Then, the operator of the computer system can select a different first point 614 and / or a different second point 616, which can increase the third distance d3 above the threshold.

[0117] Figure 13 Illustrates an example operation 1300 performed by a computer system. Generally, the computer system may use machine learning (e.g., neural networks) to assist in determining the distance between two points.

[0118] The imaging device 302 includes a left camera 1304 and a right camera 1306. When the imaging device 302 is in a first pose, video 1308 is captured by the left camera 1304 and the right camera 1306. A machine learning model 1310 (e.g., neural network) may be used to generate a first depth map 1312. This data may be used to determine a first distance 1314 to the first point. Then, the imaging device 302 is moved to a second pose. Video 1308 is captured in the second pose. A machine learning model 1310 may be used to generate a second depth map 1320. This data may be used to determine a second distance 1322 to the second point.

[0119] In one example, the machine learning model 1310 may include one or more neural networks. The neural networks may have been trained by viewing videos of various surgical procedures. These videos may show the regions of the organs or anatomical structures on which the operations have been performed. These videos may also show the procedures performed on these regions and the results. By analyzing these videos, the neural networks can learn how to determine the distances to points in the videos, which is used when generating depth maps. Then, during subsequent medical procedures, the neural networks can be used to guide or assist the procedures (by analyzing the video 1308 to generate depth maps 1312 and 1320, which indicate the depth of the points shown in the video 1308 or the distances to the points shown in the video 1308).

[0120] In summary, the present disclosure describes a medical system that measures the distance between two points in a surgical space by utilizing kinematic data that describes the pose (e.g., orientation and alignment) of an endoscope. Generally, the system adjusts the pose of the endoscope such that the endoscope is close to the first point to triangulate the distance between the first point and the endoscope. The kinematic data indicates the pose of the endoscope, and thus the kinematic data can be used together with the determined distance between the first point and the endoscope to determine the global coordinates of the first point. Then, the system adjusts the pose of the endoscope such that the endoscope is close to the second point to triangulate the distance between the second point and the endoscope. The kinematic data indicates the new pose of the endoscope, and thus the kinematic data can be used together with the determined distance between the second point and the endoscope to determine the global coordinates of the second point. Adjusting the pose of the endoscope such that the endoscope is close to the second point may cause the first point to disappear from the field of view. However, since the kinematic data indicates the pose of the endoscope, the kinematic data can be used to determine the global coordinates of the first point and the second point. Then these global coordinates are used to determine the distance between the first point and the second point.

[0121] The present specification and the drawings showing some aspects, embodiments or modules should not be considered restrictive. Various mechanical, component, structural, electrical and operational changes can be made without departing from the spirit and scope of the present specification and the claims. In some cases, well-known circuits, structures or techniques are not shown or described in detail so as not to obscure other features. Similar numbers in two or more figures represent the same or similar elements.

[0122] In this specification, specific details of some embodiments consistent with the present disclosure are set forth. Numerous specific details are set forth in order to provide a thorough understanding of the embodiments. However, it will be apparent to those skilled in the art that some or all of these specific details may be practiced without some or all of these specific details. The specific embodiments disclosed herein are intended to be illustrative and not restrictive. Those skilled in the art may implement other elements, which are not specifically described herein, but which are within the scope and spirit of the present disclosure. Additionally, to avoid unnecessary repetition, one or more features shown and described in connection with one embodiment may be incorporated into other embodiments, unless specifically described otherwise, or if one or more features would render the embodiment inoperative.

[0123] Furthermore, the terminology used in this specification is not intended to be limiting. For example, spatial relative terms (such as "beneath", "below", "lower", "above", "upper", "proximal", "distal", etc.) may be used to describe the relationship of one element or feature to another element or feature, as shown in the figures. These spatial relative terms are intended to cover different orientations (i.e., positions) and orientations (i.e., rotational placements) of the element or its operation, in addition to the orientations and orientations shown in the figures. For example, if the contents of one figure are flipped, an element described as "beneath" or "below" other elements or features will be "above" or "over" other elements or features. Thus, the exemplary term "beneath" can cover both the orientation and orientation of "above" and "beneath". The device may be oriented in other ways (rotated 90 degrees or in other orientations), and the spatial relative descriptors used herein are to be interpreted accordingly. Similarly, descriptions of movement along and around various axes include various specific element orientations and orientations. Additionally, unless the context indicates otherwise, the singular forms "a", "an", and "the" are intended to also include the plural forms. Also, the terms "comprises", "comprising", "includes", etc. specify the presence of the stated features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups. Components described as being coupled may be directly electrically or mechanically coupled, or they may be indirectly coupled via one or more intermediate components.

[0124] Elements described in detail with reference to one embodiment or module, where practicable, may be included in other embodiments or modules (where they are not specifically shown or described). For example, if an element is described in detail with reference to one embodiment and not described with reference to a second embodiment, the element may still be claimed as being included in the second embodiment. Thus, to avoid unnecessary repetition in the following description, unless specifically described otherwise, one or more elements shown and described in connection with one embodiment or application may be incorporated into other embodiments or aspects, unless one or more of the elements would render one or more of the embodiments inoperative, or unless two or more of the elements provide conflicting functions.

[0125] In some instances, well-known methods, procedures, components, and circuits are not described in detail so as not to unnecessarily obscure some aspects of the embodiments.

[0126] The present disclosure describes various devices, elements, and portions of computer-assisted devices and elements according to their state in three-dimensional space. As used herein, the term "orientation" refers to the position of an element or a portion of an element in three-dimensional space (e.g., three translational degrees of freedom along the Cartesian x, y, and z coordinates). As used herein, the term "orientation" refers to the rotational placement of an element or a portion of an element (three rotational degrees of freedom, e.g., roll, pitch, and yaw). As used herein, the term "shape" refers to the set orientation or orientation measured along an element. As used herein, and for devices having a repositionable arm, the term "proximal" refers to the direction along its kinematic chain toward the base of the computer-assisted device, and "distal" refers to the direction along the kinematic chain away from the base.

[0127] Some aspects of the present disclosure are described with reference to computer - aided systems and devices, which may include systems and devices for remote operation, remote control, autonomous, semi - autonomous, robotic, etc. Additionally, some aspects of the present disclosure are described in accordance with embodiments using a medical system (such as the DAVINCI SURGICAL SYSTEM or ION SYSTEM commercialized by Intuitive Surgical, Inc. of Sunnyvale, California). However, those skilled in the art will understand that some aspects disclosed herein can be embodied and implemented in various ways (including robotic and non - robotic embodiments, if applicable). The techniques described with reference to surgical instruments and surgical methods can be used in other contexts. Thus, the instruments, systems, and methods described herein can be used for humans, animals, parts of human or animal anatomical structures, industrial systems, general - purpose robots, or remote - operating systems. As a further example, the instruments, systems, and methods described herein can be used for non - medical purposes, which include industrial uses, general - purpose robotic uses, sensing or manipulating non - tissue workpieces, cosmetic improvements, imaging of human or animal anatomical structures, collecting data from human or animal anatomical structures, installing or removing systems, training medical or non - medical personnel, etc. Additional example applications include procedures for tissue removed from a human or animal anatomical structure (whether or not returned to the human or animal anatomical structure), and procedures for human or animal cadavers. Further, these techniques can also be used for medical treatment or diagnostic procedures with or without surgical aspects.

[0128] Although illustrative embodiments have been shown and described, numerous modifications, changes, and substitutions are contemplated in the foregoing disclosure, and in some instances, some features of the embodiments may be employed without the corresponding use of other features. Many variations, alternatives, and modifications will be recognized by those of ordinary skill in the art. Accordingly, the scope of the present disclosure should be limited only by the appended claims, and appropriately: the claims should be construed broadly and in a manner consistent with the scope of the embodiments disclosed herein.

[0129] Embodiments are disclosed herein as described in the following clauses:

[0130] Clause 1. A system for measuring the distance between two points in a surgical space, the system comprising:

[0131] a memory; and

[0132] a controller communicatively coupled to the memory, the controller being configured to:

[0133] move an imaging device to a first pose to capture first imaging data showing a first region of a surface of an anatomical structure;

[0134] Determine a first distance between the imaging device in the first pose and a first point on the surface;

[0135] Determine first coordinates of the first point on the surface based on the first pose and the first distance;

[0136] Move the imaging device to a second pose to capture second imaging data showing a second region of the surface of the anatomical structure;

[0137] Determine a second distance between the imaging device in the second pose and a second point on the surface;

[0138] Determine second coordinates of the second point on the surface based on the second pose and the second distance; and

[0139] Based on each of the first distance and the second distance to the first point and the second point on the surface being within a threshold, determine a third distance between the first point and the second point based on the first coordinates and the second coordinates.

[0140] Clause 2. The system according to Clause 1, wherein the third distance is a straight-line distance.

[0141] Clause 3. The system according to Clause 1, wherein the third distance is a distance on the surface.

[0142] Clause 4. The system according to Clause 3, wherein the controller is further configured to use a Simultaneous Localization and Mapping (SLAM) process to generate a model of the anatomical structure, and wherein the third distance is further based on the model.

[0143] Clause 5. The system according to Clause 1, wherein the controller is further configured to update the third distance based on movement of the surface of the anatomical structure.

[0144] Clause 6. The system according to Clause 1, wherein determining the first distance includes:

[0145] Determine the distance between the left camera of the imaging device and the first point;

[0146] Determine the distance between the right camera of the imaging device and the first point; and

[0147] Determine the first distance based on the distance between the left camera and the first point and the distance between the right camera and the first point.

[0148] Clause 7. The system according to Clause 1, wherein the controller is further configured to determine whether the first distance and the second distance to the first point and the second point on the surface are each within a threshold.

[0149] Clause 8. The system according to Clause 7, wherein the threshold is ten centimeters.

[0150] Clause 9. The system according to Clause 7, wherein the threshold is determined based on at least one of a depth measurement at the first point, a depth measurement at the second point, or a kinematic error.

[0151] Clause 10. The system according to Clause 7, wherein the threshold is determined based on a received error tolerance and a look-up table.

[0152] Clause 11. The system according to Clause 1, wherein the controller is further configured to determine the first pose and the second pose based on kinematic data associated with the imaging device.

[0153] Clause 12. The system according to Clause 11, wherein the kinematic data includes three degrees of freedom of orientation indicating translational information of the imaging device and three degrees of freedom of orientation indicating rotational information of the imaging device.

[0154] Clause 13. The system according to Clause 1, wherein the controller is further configured to control a relocatable structure to move the imaging device to the first pose and the second pose.

[0155] Clause 14. The system according to Clause 1, wherein the controller is further configured to:

[0156] present on a display a video from the imaging device showing the surface of the anatomical structure; and

[0157] add a first virtual marker over the first point to the video; and

[0158] add a second virtual marker over the second point to the video.

[0159] Clause 15. The system according to Clause 1, wherein the first distance is determined based on the axis of the imaging device.

[0160] Clause 16. The system according to Clause 1, wherein when viewing the second region, the first region is outside the field of view of the imaging device.

[0161] Clause 17. The system according to Clause 1, wherein the controller is further configured to present the third distance on a display.

[0162] Clause 18. The system according to Clause 1, wherein the controller is further configured to update the third distance based on movement of the imaging device or movement of the anatomical structure.

[0163] Clause 19. The system according to Clause 1, wherein the controller is further configured to convey an alert in response to determining that the third distance reaches or falls below a target distance.

[0164] Clause 20. The system according to Clause 1, wherein indication of the first point and the second point on the surface of the anatomical structure is independent of entering a virtual mode and moving a virtual cursor.

[0165] Clause 21. The system according to Clause 1, wherein indication of the first point and the second point on the surface of the anatomical structure is independent of moving a surgical instrument.

[0166] Clause 22. The system according to Clause 22, wherein determining the first distance includes generating a first depth map of the surface of the anatomical structure based on the first imaging data, and wherein the first distance is determined based on the first depth map.

[0167] Clause 23. The system according to Clause 22, wherein determining the second distance includes generating a second depth map of the surface of the anatomical structure based on the second imaging data, and wherein the second distance is determined based on the second depth map.

[0168] Clause 24. The system according to Clause 22, wherein generating the first depth map includes predicting the depth of the surface using a machine learning model based on the first imaging data and the first pose.

[0169] Clause 25. A method for measuring a distance between two points in a surgical space, the method comprising:

[0170] Moving an imaging device to a first pose to capture first imaging data showing a first region of the surface of an anatomical structure;

[0171] Determining a first distance between the imaging device in the first pose and a first point on the surface;

[0172] Determining first coordinates of the first point on the surface based on the first pose and the first distance;

[0173] Moving the imaging device to a second pose to capture second imaging data showing a second region of the surface of the anatomical structure;

[0174] Determine a second distance between the imaging device in the second pose and a second point on the surface;

[0175] Determine second coordinates of the second point on the surface based on the second pose and the second distance; and

[0176] Based on each of the first distance and the second distance to the first point and the second point on the surface being within a threshold, determine a third distance between the first point and the second point based on the first coordinates and the second coordinates.

[0177] Clause 26. The method according to clause 25, wherein the third distance is a straight-line distance.

[0178] Clause 27. The method according to clause 25, wherein the third distance is a distance on the surface.

[0179] Clause 28. The method according to clause 27, wherein the controller is further configured to generate a model of the anatomical structure using a Simultaneous Localization and Mapping (SLAM) process, and wherein the third distance is further based on the model.

[0180] Clause 29. The method according to clause 25, wherein the controller is further configured to update the third distance based on movement of the surface of the anatomical structure.

[0181] Clause 30. The method according to clause 25, wherein determining the first distance includes:

[0182] Determine the distance between the left camera of the imaging device and the first point;

[0183] Determine the distance between the right camera of the imaging device and the first point; and

[0184] Determine the first distance based on the distance between the left camera and the first point and the distance between the right camera and the first point.

[0185] Clause 31. The method according to clause 25, wherein the controller is further configured to determine whether the first distance and the second distance to the first point and the second point on the surface are each within a threshold.

[0186] Clause 32. The method according to clause 31, wherein the threshold is ten centimeters.

[0187] Clause 33. The method according to clause 31, wherein the threshold is determined based on at least one of a depth measurement at the first point, a depth measurement at the second point, or a kinematic error.

[0188] Clause 34. The method according to Clause 31, wherein the threshold is determined based on the received error tolerance and a look-up table.

[0189] Clause 35. The method according to Clause 25, wherein the controller is further configured to determine the first pose and the second pose based on kinematic data associated with the imaging device.

[0190] Clause 36. The method according to Clause 35, wherein the kinematic data includes three degrees of freedom of orientation indicating translational information of the imaging device and three degrees of freedom of orientation configured to provide translation and indicating rotational information of the imaging device.

[0191] Clause 37. The method according to Clause 25, wherein the controller is further configured to control a relocatable structure to move the imaging device to the first pose and the second pose.

[0192] Clause 38. The method according to Clause 25, wherein the controller is further configured to:

[0193] present on a display a video from the imaging device showing the surface of the anatomical structure; and

[0194] add a first virtual marker covering the first point to the video; and

[0195] add a second virtual marker covering the second point to the video.

[0196] Clause 39. The method according to Clause 25, wherein the first distance is determined based on an axis of the imaging device.

[0197] Clause 40. The method according to Clause 25, wherein when viewing the second region, the first region is outside the field of view of the imaging device.

[0198] Clause 41. The method according to Clause 25, wherein the controller is further configured to present the third distance on a display.

[0199] Clause 42. The method according to Clause 25, wherein the controller is further configured to update the third distance based on movement of the imaging device or movement of the anatomical structure.

[0200] Clause 43. The method according to Clause 25, wherein the controller is further configured to convey an alert in response to determining that the third distance has reached or fallen below a target distance.

[0201] Clause 44. The method according to Clause 25, wherein the indication of the first point and the second point on the surface of the anatomical structure is independent of entering the virtual mode and moving the virtual cursor.

[0202] Clause 45. The method according to Clause 25, wherein the indication of the first point and the second point on the surface of the anatomical structure is independent of moving the surgical instrument.

[0203] Clause 46. The method according to Clause 25, wherein determining the first distance includes generating a first depth map of the surface of the anatomical structure based on the first imaging data, and determining the first distance based on the first depth map.

[0204] Clause 47. The method according to Clause 46, wherein determining the second distance includes generating a second depth map of the surface of the anatomical structure based on the second imaging data, and determining the second distance based on the second depth map.

[0205] Clause 48. The method according to Clause 46, wherein generating the first depth map includes using a machine learning model to predict the depth of the surface based on the first imaging data and the first pose.

[0206] Clause 49. A non-transitory machine-readable medium storing instructions for measuring the distance between two points in a surgical space, the instructions causing the processor to:

[0207] Execute the method according to any one of Clauses 25 to 48.

Claims

1. A system for measuring the distance between two points in a surgical space, the system comprising: a memory; and a controller communicatively coupled to the memory, the controller being configured to: move an imaging device to a first pose to capture first imaging data showing a first region of a surface of an anatomical structure; determine a first distance between the imaging device in the first pose and a first point on the surface; determine first coordinates of the first point on the surface based on the first pose and the first distance; move the imaging device to a second pose to capture second imaging data showing a second region of the surface of the anatomical structure; determine a second distance between the imaging device in the second pose and a second point on the surface; determine second coordinates of the second point on the surface based on the second pose and the second distance; and determine a third distance between the first point and the second point based on the first coordinates and the second coordinates, based on each of the first distance and the second distance to the first point and the second point on the surface being within a threshold.

2. The system according to claim 1, wherein the third distance is a straight-line distance.

3. The system according to claim 1, wherein the third distance is a distance on the surface.

4. The system according to claim 1, wherein the controller is further configured to update the third distance based on movement of the surface of the anatomical structure.

5. The system according to claim 1, wherein determining the first distance includes: determining a distance between a left camera of the imaging device and the first point; determining a distance between a right camera of the imaging device and the first point; and determining the first distance based on the distance between the left camera and the first point and the distance between the right camera and the first point.

6. The system according to claim 1, wherein the threshold is determined based on at least one of a depth measurement at the first point, a depth measurement at the second point, or a kinematic error.

7. The system according to claim 1, wherein the threshold is determined based on a received error tolerance and a look-up table.

8. The system according to claim 1, wherein the controller is further configured to determine the first pose and the second pose based on kinematic data associated with the imaging device.

9. The system according to claim 1, wherein the controller is further configured to control a relocatable structure to move the imaging device to the first pose and the second pose.

10. The system according to claim 1, wherein the controller is further configured to: Present a video from the imaging device on a display showing the surface of the anatomical structure; and add a first virtual marker covering the first point to the video; and add a second virtual marker covering the second point to the video.

Images