Systems and methods for performing operations associated with a computer-assisted surgical system based on a depth sensor and an auxiliary sensor

Abstract

An example operating management system is configured to obtain, from a depth sensor included in an imaging device, depth data representing a depth map of an internal space of a patient, to obtain, from an auxiliary sensor not included in the imaging device, auxiliary sensor data, and to perform, based on the depth data and the auxiliary sensor data, an operation associated with a computer-assisted surgical system configured to perform a procedure within the internal space of the patient.

Description

[0001] Related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 888,115, filed August 16, 2019, entitled “SYSTEMS AND METHODS FOR PERFORMANCE OF DEPTH SENSOR AND AUXILIARYSENSOR-BASED OPERATIONS ASSOCIATED WITH A COMPUTER-ASSISTED SURGICAL SYSTEM”, the contents of which are incorporated herein by reference in their entirety. Background Technology

[0003] During procedures performed within a patient's internal space by a computer-assisted surgical system, it may be expected that the surgeon or other user will guide the system to perform various actions in response to user input. For example, the system may be expected to measure precise distances between user-selected points within an image captured by an imaging device controlled by the system. It may also be expected that the system will perform various actions automatically without user input. For instance, it may be expected to automatically perform efficiency-related actions, such as preventing surgical instruments from unintentionally contacting and damaging tissues within the patient's body. Accuracy, precision, and timely responsiveness are desired when the computer-assisted surgical system performs these and other actions. Summary of the Invention

[0004] The following description presents a simplified summary of one or more aspects of the systems and methods described herein. This summary is not a comprehensive overview of all aspects considered, and is neither intended to identify major or key elements of all aspects nor to indicate the scope of any or all aspects. Its sole purpose is to present one or more aspects of the systems and methods described herein as an introduction to the detailed description presented below.

[0005] An exemplary system includes a memory storing instructions and a processor communicatively coupled to the memory, the processor being configured to execute instructions to acquire depth data representing a depth map of the patient’s internal space from a depth sensor included in an imaging device, acquire auxiliary sensor data from an auxiliary sensor not included in the imaging device, and perform operations associated with a computer-assisted surgical system based on the depth data and the auxiliary sensor data, the computer-assisted surgical system being configured to execute procedures within the patient’s internal space.

[0006] An exemplary method includes: acquiring depth data representing a depth map of the patient's internal space from a depth sensor included in an imaging device by an operation management system; acquiring auxiliary sensor data from an auxiliary sensor not included in the imaging device by the operation management system; and performing operations associated with a computer-assisted surgical system based on the depth data and the auxiliary sensor data, the computer-assisted surgical system being configured to perform procedures within the patient's internal space by the operation management system.

[0007] An exemplary non-transitory computer-readable medium storage instruction, when executed, directs a processor of a computing device to acquire depth data representing a depth map of a patient’s internal space from a depth sensor included in an imaging device, acquire auxiliary sensor data from an auxiliary sensor not included in the imaging device, and perform operations associated with a computer-assisted surgical system configured to perform procedures within the patient’s internal space based on the depth data and the auxiliary sensor data. Attached Figure Description

[0008] The accompanying drawings illustrate various embodiments and are part of the specification. The illustrated embodiments are merely examples and do not limit the scope of this disclosure. In all the drawings, the same or similar reference numerals denote the same or similar elements.

[0009] Figure 1 An exemplary operation management system based on the principles described herein is illustrated.

[0010] Figure 2 An exemplary configuration based on the principles described herein is illustrated, wherein Figure 1 The operation management system performs operations associated with the computer-assisted surgical system based on depth data and auxiliary sensor data.

[0011] Figure 3 An exemplary implementation based on the principles described herein is illustrated, wherein the depth sensor is implemented by a time-of-flight sensor included in the imaging apparatus.

[0012] Figure 4 An exemplary implementation based on the principles described herein is shown, wherein the lighting system is implemented by a single light source.

[0013] Figure 5 An exemplary implementation based on the principles described herein is illustrated, wherein the lighting system is implemented by separate lighting sources.

[0014] Figure 6 An exemplary implementation based on the principles described herein is illustrated, wherein an illumination source is integrated into a time-of-flight sensor.

[0015] Figure 7An exemplary structural implementation of an imaging device based on the principles described herein is illustrated.

[0016] Figure 8 A cross-sectional view of the axis of an imaging device based on the principles described herein is depicted.

[0017] Figure 9 An exemplary implementation based on the principles described herein is illustrated, wherein the depth sensor is implemented by a visible light camera included in an imaging device.

[0018] Figure 10 An exemplary two-dimensional image is illustrated based on the principles described herein.

[0019] Figure 11 An exemplary three-dimensional endpoint is illustrated according to the principles described herein, between which one or more distances can be determined.

[0020] Figure 12 An exemplary side view of the patient's internal space is shown, based on the principles described herein.

[0021] Figure 13 An exemplary configuration based on the principles described herein is shown, in which surgical instruments are used within an internal space imaged by an imaging device.

[0022] Figure 14 A surgical instrument 1402 is shown prior to its contact with the target location, based on the principles described herein.

[0023] Figure 15 This illustrates the contact between a surgical instrument and a target location, based on the principles described herein. Figure 14 Surgical instruments.

[0024] Figure 16 An exemplary method based on the principles described herein is illustrated.

[0025] Figure 17 An exemplary computer-assisted surgical system based on the principles described herein is illustrated.

[0026] Figure 18 An exemplary computing device based on the principles described herein is illustrated. Detailed Implementation

[0027] This document describes systems and methods for performing operations associated with a computer-assisted surgical system based on depth sensors and auxiliary sensors. For example, an exemplary operations management system may acquire depth data representing a depth map of a patient's internal space from a depth sensor included in an imaging device, acquire auxiliary sensor data from an auxiliary sensor not included in the imaging device, and perform operations associated with a computer-assisted surgical system configured to perform procedures within the patient's internal space based on the depth data and the auxiliary sensor data.

[0028] The systems and methods described herein advantageously use both depth data and auxiliary sensor data to perform operations associated with a computer-assisted surgical system. This can result in operations that are more precise, accurate, and responsive than those performed by conventional computer-assisted surgical systems that cannot access both data types simultaneously. These and other advantages and benefits of the systems and methods described herein will become apparent here.

[0029] Figure 1 An exemplary operation management system 100 (“System 100”) is described, configured to perform operations associated with a computer-assisted surgical system based on depth sensors and auxiliary sensors. As shown, System 100 may include, but is not limited to, storage facility 102 and processing facility 104 selectively and communicatively coupled to each other. Facility 102 and facility 104 may each include or be implemented by hardware and / or software components (e.g., processor, memory, communication interface, instructions stored in memory for execution by the processor, etc.). For example, facility 102 and / or facility 104 may be implemented by any component within the computer-assisted surgical system itself. As another example, facility 102 and / or facility 104 may be implemented by a computing device that is separate from and communicatively coupled to the computer-assisted surgical system. In some examples, facility 102 and facility 104 may be distributed among multiple devices and / or multiple locations to meet a particular implementation.

[0030] Storage facility 102 may maintain (e.g., store) executable data used by processing facility 104 to perform one or more operations described herein. For example, storage facility 102 may store instructions 106 that can be executed by processing facility 104 to perform one or more operations described herein. Instructions 106 may be implemented by any suitable application, software, code, and / or other instance of executable data. Storage facility 102 may also maintain any data received, generated, managed, used, and / or transmitted by processing facility 104.

[0031] Processing facility 104 may be configured to perform (e.g., execute instructions 106 stored in storage facility 102 to perform) various operations described herein. For example, processing facility 104 may be configured to acquire depth data representing a depth map of the patient's internal space from depth sensors included in an imaging apparatus. Processing facility 104 may also be configured to acquire auxiliary sensor data from auxiliary sensors not included in (or otherwise part of) the imaging apparatus. Processing facility 104 may also be configured to perform operations associated with a computer-assisted surgical system based on depth data and auxiliary sensor data, the computer-assisted surgical system being configured to perform procedures within the patient's internal space. These and other operations that may be performed by system 100 (e.g., processing facility 104) are described herein.

[0032] Figure 2 An exemplary configuration is illustrated, in which system 100 performs operations 202 associated with computer-assisted surgical system 204 based on depth data 206 and auxiliary sensor data 208. As shown, system 100 acquires depth data 206 from depth sensor 210 included in imaging device 212. System 100 acquires auxiliary sensor data 208 from auxiliary sensor 214 not included in imaging device 212.

[0033] The computer-assisted surgical system 204 can be implemented by any suitable surgical system that uses robotics and / or teleoperation technology to perform procedures (e.g., minimally invasive surgical procedures) within the patient's internal space (e.g., the surgical area). An exemplary computer-assisted surgical system is described herein.

[0034] Operation 202 may include any suitable operation performed by the computer-assisted surgical system 204. Where system 100 is implemented by the computer-assisted surgical system 204 itself, operation 202 may be performed by the computer-assisted surgical system 204. Examples of operation 202 are described herein.

[0035] Imaging device 212 may be implemented by an endoscope or other camera device configured to capture images of the scene. In some examples, imaging device 212 may be configured to be attached to and controlled by a computer-assisted surgical system 204. In alternative examples, imaging device 212 may be handheld and manually operated by an operator (e.g., a surgeon).

[0036] In some examples, the scene captured by imaging device 212 may include a surgical area associated with the patient. In some examples, the surgical area may be entirely located within the patient's body and may include the area within or near where a surgical procedure is planned, is being performed, or has been performed within the patient's body. For example, for a minimally invasive surgical procedure performed on tissue within the patient's body, the surgical area may include the tissue, the anatomical structures beneath the tissue, and the space surrounding the tissue (e.g., where surgical instruments used to perform the surgical procedure are located). In some example embodiments, a surgical area entirely located within the patient's body may be referred to as an "internal space." As described herein, any internal anatomical structures of the patient (e.g., blood vessels, organs, and / or tissues) and / or surgical instruments located within the internal space may be referred to as objects and / or structures.

[0037] The depth sensor 210 included in the imaging device 212 can be implemented by any suitable sensor configured to generate depth data 206 representing a depth map of the patient's internal space. For example, as described herein, the depth sensor 210 can be implemented by a time-of-flight sensor, a structured light sensor, an interferometer, a stereo camera, and / or any other suitable component that may satisfy a particular implementation.

[0038] The auxiliary sensor 214 can be implemented by any suitable sensor that is not included in or is otherwise part of the imaging device 212. For example, as described herein, the auxiliary sensor 214 can be implemented by a user input sensor configured to generate user input data, a force sensor integrated into a surgical instrument controlled by the computer-assisted surgical system 204 and configured to output force sensing data representing the amount of external force applied to the surgical instrument, and / or any other type of sensor configured to output any other type of auxiliary sensing data 208 to satisfy a particular embodiment.

[0039] Various exemplary ways in which system 100 can acquire depth data 206 from depth sensor 210 in imaging device 212 will now be described.

[0040] Figure 3 An exemplary embodiment 300 is described, in which the depth sensor 210 is implemented by a time-of-flight sensor 302 included in the imaging device 212. Although the time-of-flight sensor 302 is... Figure 3As shown and mentioned in the examples provided herein, any other type of depth sensor separate from (i.e., physically different from) the visible light camera also included in the imaging device 212 may be used additionally or alternatively to implement depth sensor 210. For example, depth sensor 210 may alternatively be implemented by a structured light sensor, an interferometer, and / or any other suitable sensor configured to acquire depth data to satisfy a particular implementation.

[0041] In implementation 300, system 100 can acquire depth data 206 by guiding time-of-flight sensor 302 to acquire depth data 206 and receiving depth data 206 from time-of-flight sensor 302.

[0042] Therefore, in embodiment 300, system 100 is communicatively coupled to imaging device 212 via bidirectional communication link 304 and communicatively coupled to illumination system 306 via communication link 308. Communication links 304 and 308 can each be implemented using any suitable wired and / or wireless communication medium to satisfy a particular embodiment. System 100 can use communication links 304 and 308 to guide time-of-flight sensor 302 to acquire and receive depth data 206, as will be described herein.

[0043] As shown in the figure, the imaging device 212 includes a time-of-flight sensor 302 and a visible light camera 310 (“camera 310”), which is configured to generate image data 312 representing a two-dimensional visible light image of the scene. The time-of-flight sensor 302 may be implemented by one or more photodetectors (e.g., one or more single-photon avalanche diode (“SPAD”) detectors), a CCD sensor, a CMOS sensor, and / or any other suitable configuration configured to acquire depth data of the scene. The camera 310 may be implemented by any suitable image sensor, such as a charge-coupled device (“CCD”) image sensor, a complementary metal-oxide-semiconductor (“CMOS”) image sensor, etc.

[0044] In some examples, system 100 may be configured to control the operation of imaging device 212 (e.g., by controlling the operation of camera 310 and time-of-flight sensor 302). For example, system 100 may include one or more camera control units (“CCUs”) configured to control various parameters of camera 310 and / or time-of-flight sensor 302 (e.g., activation time, auto exposure, etc.).

[0045] System 100 may be additionally or alternatively configured to provide operating power for components included in imaging device 212. For example, when imaging device 212 is communicatively coupled to system 100, system 100 may transmit operating power to camera 310 and time-of-flight sensor 302 in the form of one or more electrical signals.

[0046] System 100 can be configured to use imaging device 212 and illumination system 306 to acquire depth data 206 and image data 312. In some examples, depth data 206 and image data 312 can be used to generate stereoscopic images of the scene. This will be described in more detail below.

[0047] The lighting system 306 can be configured to emit light 314 (e.g., in the direction of system 100) to illuminate a scene to be imaged by the imaging device 212. The light 314 emitted by the lighting system 306 can include visible and / or invisible light (e.g., infrared light). As shown, the light 314 can travel through the imaging device 212 (e.g., through an illumination channel within the imaging device 212, which may be implemented by one or more optical fibers, light guides, lenses, etc.) to the scene. Various embodiments and configurations of the lighting system 306 are described herein.

[0048] As shown, light 314 emitted by the illumination system 306 can be reflected onto a surface 316 within the scene imaged by the imaging device 212. A visible light camera 310 and a time-of-flight sensor 302 can each detect the reflected light 314. The visible light camera 310 can be configured to generate image data 312 representing a two-dimensional visible light image of the scene including the surface 316 based on the detected light. The time-of-flight sensor 302 can be configured to generate depth data 206 based on the detected light. The image data 312 and the depth data 206 can each have any suitable format.

[0049] To generate a stereoscopic image of the scene, system 100 can direct the lighting system 306 to emit light 314. System 100 can also activate (e.g., turn on) a visible light camera 310 and a time-of-flight sensor 302. The light 314 travels into the scene and is reflected from a surface 316 (and, in some examples, one or more other surfaces in the scene). Both the camera 310 and the time-of-flight sensor 302 detect the reflected light 314.

[0050] Based on the detected light 314, camera 310 (and / or other circuitry included in imaging device 212) can generate image data 312 representing a two-dimensional visible light image of the scene. This can be performed in any suitable manner. Visible light camera 310 (and / or other circuitry included in imaging device 212) can transmit image data 312 to system 100. This can also be performed in any suitable manner.

[0051] Based on the detected light 314, the time-of-flight sensor 302 can generate depth data 206 representing a depth map of the scene (e.g., a depth map of surface 316). This can be performed in any suitable manner. For example, the time-of-flight sensor 302 can measure the amount of time it takes for photons of light 314 to travel from the illumination system 306 to the time-of-flight sensor 302. Based on this amount of time, the time-of-flight sensor 302 can determine the depth of the position of surface 316 relative to the time-of-flight sensor 302. The data representing this depth can be represented in the depth data 206 in any suitable manner. For example, the depth map represented by the depth data 206 can include an array of depth values ​​(e.g., Z-buffer values) corresponding to each pixel in the image.

[0052] The time-of-flight sensor 302 (and / or other circuitry included in the imaging device 212) can transmit depth data 206 to the system 100. This can be done in any suitable manner.

[0053] System 100 can receive image data 312 and depth data 206 and perform one or more processing operations on the image data 312 and depth data 206. For example, based on the image data 312 and depth data 206, system 100 can generate a right-side perspective image of the scene and a left-side perspective image representing the scene. This can be performed in any suitable manner. System 100 can then instruct a display device to simultaneously display the right-side and left-side perspective images in a manner that forms a stereoscopic image of the scene. In some examples, the display device is included and / or communicatively coupled to computer-assisted surgical system 204.

[0054] System 100 can perform operation 202 based on depth data 206. In some examples, operation 202 may additionally be based on image data 312. Examples of operation 202 are described herein.

[0055] Figure 4 An exemplary embodiment 400 is shown, in which the lighting system 306 is implemented by a single light source 402. The light source 402 may be configured to emit visible light 314-1.

[0056] Visible light 314-1 may include one or more color components. For example, visible light 314-1 may include white light that includes the full spectrum of color components (e.g., red, green, and blue components). The red component has a wavelength between approximately 935 and 700 nanometers (“nm”). The green component has a wavelength between approximately 720 and 760 nm. The blue component has a wavelength between approximately 650 and 690 nm.

[0057] In some examples, visible light 314-1 is biased to include a color component that is more than the other color component. For example, visible light 314-1 can be biased to include a blue component that is more than the red and green components.

[0058] In embodiment 400, the time-of-flight sensor 302 is configured to also detect visible light 314-1. Therefore, the same illumination source 402 can be used for both the camera 310 and the time-of-flight sensor 302.

[0059] Figure 5 An exemplary embodiment 500 is described, in which the illumination system 306 is implemented by separate illumination sources 402-1 and 402-2. In embodiment 500, illumination source 402-1 is configured to emit visible light 314-1 detected by camera 310. Illumination source 402-2 is configured to emit light 314-2 reflected from surface 316 and detected by time-of-flight sensor 302. In some examples, light 314-2 is invisible light, such as infrared light. Because camera 310 and time-of-flight sensor 302 have separate illumination sources 402, camera 310 and time-of-flight sensor 302 can be configured to operate independently.

[0060] Figure 6 An exemplary implementation 600 is described, in which the illumination source 402-2 is integrated into the time-of-flight sensor 302. In implementation 600, the system 100 controls (e.g., activates) the illumination source 402-2 by transmitting instructions to the time-of-flight sensor 302.

[0061] Figure 7 An exemplary structural embodiment of the imaging device 212 is described. As shown, the imaging device 212 includes a camera head 702 and a shaft 704 coupled to and extending from the camera head 702. The camera head 702 and the shaft 704 together form the housing of the imaging device 212. The imaging device 212 can be manually operated and controlled (e.g., by a surgeon performing a surgical procedure on a patient). Alternatively, the camera head 702 can be coupled to a manipulator arm of a computer-assisted surgical system 204. In this configuration, the imaging device 212 can be controlled by the computer-assisted surgical system 204 using robotics and / or remote operation technologies.

[0062] As shown in the figure, the illumination channel 706 can pass through the camera head 702 and the axis 704. The illumination channel 706 is configured to provide a conduit for the light emitted by the illumination system 306 to travel to the scene imaged by the imaging device 212.

[0063] The distal end 708 of axis 704 can be positioned at or near the scene to be imaged by imaging device 212. For example, the distal end 708 of axis 704 can be inserted into a patient. In this configuration, imaging device 212 can be used to capture images of anatomical structures and / or other objects within the patient's body.

[0064] The camera 310 and the time-of-flight sensor 302 can be located anywhere along the axis 704 of the imaging device 212. Figure 7 In the example shown, camera 310 and time-of-flight sensor 302 are located at the distal end 708 of axis 704. This configuration may be referred to as a “chip-on-tip” configuration. Alternatively, camera 310 and / or time-of-flight sensor 302 may be located further toward camera head 702 and / or within camera head 702. In these alternative configurations, optics (e.g., lenses, optical fibers, etc.) included in axis 704 and / or camera head 206 can transmit light from the scene to camera 310 and / or time-of-flight sensor 302.

[0065] In some examples, the camera 310 and the time-of-flight sensor 302 may be staggered at different distances from the distal end 708 of the axis 704. By staggering the distances of the camera 310 and the time-of-flight sensor 302 from the distal end 708 of the axis 704, the imaging device 212 may be in a tapered configuration with a reduced size (e.g., diameter) toward the distal end 708 of the axis 704, which may facilitate insertion of the imaging device 212 into the patient's internal space.

[0066] Figure 8 The axis 704 of the imaging device 212 is depicted along... Figure 7 The figure shows a cross-sectional view along line 9-8. As shown, axis 704 includes a relatively flat bottom surface 802. Referring to this bottom surface 802, time-of-flight sensor 302 is positioned above camera 310. This positioning allows for a narrower axis 704 compared to the axis of a conventional imaging device with two cameras side-by-side for acquiring stereoscopic images. It will be appreciated that camera 310 and time-of-flight sensor 302 can have any suitable relative position within axis 704 to satisfy a particular implementation.

[0067] Figure 9 An exemplary embodiment 900 is described, wherein the depth sensor 210 is implemented by visible light cameras 310-1 and 310-2 included in the imaging device 212. In embodiment 900, the system 100 can acquire depth data 206 by guiding camera 310-1 to acquire a first image (e.g., a first two-dimensional image) of the patient's internal space, guiding camera 310-2 to acquire a second image (e.g., a second two-dimensional image) of the patient's internal space, and generating a depth map represented by depth data 206 based on the first and second images.

[0068] exist Figure 9 In the diagram, the first image acquired by camera 310-1 is represented by image data 312-1, and the second image acquired by camera 310-2 is represented by image data 312-2. As shown, image data 312-1 and image data 312-2 are transmitted to a depth data generator 902 implemented by system 100. The depth data generator 902 can determine depth data 206 based on image data 312-1 and image data 312-2 using any image-based visualization technique.

[0069] Various examples of operations 202 involving the computer-aided surgical system 204 performed by system 100 based on depth data 206 will now be provided. These examples are merely illustrations of many different types of operations that system 300 can perform based on depth data 206 according to the systems and methods described herein.

[0070] In some examples, system 100 may perform operation 202 by determining the distance between endpoints within an image of the patient's internal space acquired by imaging device 212. In some examples, the image may be a two-dimensional image acquired by visible light camera 310 and represented by image data 312. Additionally or alternatively, the image may be a stereoscopic image generated based on image data 312 and depth data 206 in any manner described herein.

[0071] In an example where system 100 performs operation 202 by determining the distance between endpoints within an image of the patient's internal space, auxiliary sensor 214 may be implemented by a user input sensor. The user input sensor may be configured to sense user input, such as user input provided via a keyboard, touchscreen, etc. In these examples, system 100 may be configured to obtain auxiliary sensor data 208 by receiving user input data from the user input sensor instructing the user to select a first two-dimensional endpoint within an image of the internal space. As described herein, the first two-dimensional endpoint corresponds to a first feature within the internal space depicted by the image. System 100 may perform operation 202 by defining a first three-dimensional endpoint corresponding to the first feature based on depth data 206 acquired by time-of-flight sensor 302 and user input data, and by determining the distance within the internal space from the first three-dimensional endpoint corresponding to the first feature to a second three-dimensional endpoint corresponding to the second feature based on the depth data 206 acquired by time-of-flight sensor 302. Because the depth data 206 acquired by time-of-flight sensor 302 is relatively accurate, system 100 may determine relatively accurate distance measurements according to the systems and methods described herein.

[0072] Figure 10An exemplary two-dimensional image 1002 of the patient's internal space, captured by a camera 310 included in imaging device 212 and represented by image data 312, is shown. As illustrated, image 1002 depicts a representation of the hernia 1004 to be measured so that, for example, a mesh patch can be cut to an appropriate size to properly repair the hernia 1004.

[0073] Image 1002 can be displayed or presented by system 600 in any suitable manner and / or on any suitable display screen. For example, image 1002 can be displayed on a display screen of a display device included in and / or communicatively coupled to computer-assisted surgical system 204.

[0074] To measure the distance between features displayed in image 1002 (e.g., the distance between the edges of hernia 1004), or the distance between a feature displayed in image 1002 and an additional feature not currently displayed in image 1002, a user can interact with image 1002 to provide user input specifying one or more user-selected two-dimensional endpoints corresponding to one or more features(s). Auxiliary sensor 214 can detect the user input.

[0075] To illustrate, Figure 10 An exemplary user-selected two-dimensional endpoint 1006 is illustrated in image 1002, specified by the user. The user-selected two-dimensional endpoint 1006 may correspond to a specific feature. As used herein, the “feature” corresponding to an endpoint (e.g., a two-dimensional or three-dimensional endpoint) may refer to a specific part, point, or other feature of a patient’s anatomical structure or other object within the internal space. For example, the feature corresponding to the user-selected two-dimensional endpoint 1006 is a specific portion of the edge of the hernia 1004 (in... Figure 10 (The right side is zoomed in to illustrate the feature in more detail). The user-selected two-dimensional endpoint 1006 may include multiple pixels that jointly depict the corresponding feature.

[0076] The auxiliary sensor 214 (implemented in this case by the user input sensor) can detect, receive, and / or sense any appropriate type of user input specifying the two-dimensional endpoint 1006 selected by the user. For example, the user can provide user input by touching a location on the touchscreen corresponding to a desired feature displayed within image 1002 (e.g., the edge of hernia 1004). The auxiliary sensor 214 can detect the user touching the touchscreen and output auxiliary sensor data 208 representing that location.

[0077] In some examples, the user-selected two-dimensional endpoint 1006 can be specified by a surgeon or another user of the computer-assisted surgical system 204 using a stereoscopic viewer. For example, a right-angled perspective image and a left-angled perspective image generated and forming a stereoscopic image together in any of the ways described herein can be displayed on the right-angled image display and the left-angled image display of the stereoscopic viewer, respectively. In this example, a pointer object can be made to appear to float above the tissue depicted in the scene until the user-selected two-dimensional endpoint 1006 is selected, at which point, when the corresponding three-dimensional endpoint is defined, the cursor can appear to "land" on the tissue at the appropriate depth (as described below). Alternatively, as yet another example, a user can use a three-dimensional cursor to select a point in the stereoscopic image. The projection of this selected point onto a two-dimensional image (e.g., image 1002) can then be used as the user-selected two-dimensional endpoint 1006.

[0078] In some examples, the designation of a user-selected two-dimensional endpoint (e.g., user-selected two-dimensional endpoint 1006) can be performed as a discrete event, such as a touch gesture, button press, mouse click, button release (e.g., to end a dragging motion from one user-selected two-dimensional endpoint to another). In other examples, the user selection of a user-selected two-dimensional endpoint can be performed dynamically as a pointer object (e.g., a cursor) moves within the display screen, without any additional user selection action (i.e., without actions such as button presses, mouse clicks, etc.). In other words, in some implementations, as the user moves a pointer object on the display, many user-selected two-dimensional endpoints for many different potential measurements can be automatically selected.

[0079] Once the user has specified a user-selected two-dimensional endpoint 1006 corresponding to a feature within image 1002 in any of these or other suitable ways, system 100 can define a first three-dimensional endpoint corresponding to the feature based on depth data 206 acquired by time-of-flight sensor 302 and the user input specifying the two-dimensional endpoint 1006. This can be performed in any suitable manner. For example, depth data 206 can indicate the depth values ​​of the pixels constituting two-dimensional endpoint 1006. Therefore, these depth values ​​can be used to define a three-dimensional endpoint representing the three-dimensional location of the feature.

[0080] System 100 can determine the distance between a first three-dimensional endpoint corresponding to a first feature and a second three-dimensional endpoint corresponding to a second feature in any suitable manner. The second three-dimensional endpoint can be selected or defined in the same manner as described above (e.g., the user can select a second two-dimensional endpoint representing a second feature within image 1002, and system 100 can define the second three-dimensional endpoint based on depth data 206). Alternatively, the second three-dimensional endpoint can be associated with the location of a known location (e.g., the location of a surgical instrument, the location of an imaging device 212, the origin of a coordinate system, etc.). In these examples, the user may therefore not need to specify a second user-selected two-dimensional endpoint based on features presented in the two-dimensional image. Instead, the user can indicate, in any suitable manner, that the additional three-dimensional endpoint corresponds to a specific surgical instrument, endoscope, origin, or other known location.

[0081] To illustrate, Figure 11 An exemplary three-dimensional endpoint 1102 (e.g., three-dimensional endpoints 1102-1 to 1102-6) is illustrated, in which one or more distances can be determined between them. Figure 11 As shown, a first three-dimensional endpoint 1102-1, defined based on a user-selected two-dimensional endpoint 1006 within image 1002, can be located at the first edge of hernia 1004. Once this first three-dimensional endpoint 1102-1 is defined based on the user-selected two-dimensional endpoint 1006, the distance from the three-dimensional endpoint 1102-1 to any additional three-dimensional endpoints represented by three-dimensional endpoints 1102-2 to 1102-6 can be determined. For example, the distance from one edge of hernia 1004 to another edge can be determined by determining the distance between three-dimensional endpoint 1102-1 and one of the three-dimensional endpoints 1102-2 and 1102-3.

[0082] In other examples, the additional features corresponding to the additional three-dimensional endpoint 1102 may be non-anatomical features (i.e., features included within the surgical area that are not part of the patient's anatomy). For example, three-dimensional endpoint 1102-4 may be associated with the tip of surgical instrument 1104-1, which is not currently represented in image 1002 but may still be within the patient's internal space. As another example, three-dimensional endpoint 1102-5 may be associated with the tip of surgical instrument 1104-2, which is represented in image 1002. As yet another example, three-dimensional endpoint 1102-6 may be associated with the tip of imaging device 212 (depicted as a dashed line around image 1002). In examples where the non-anatomical features corresponding to the three-dimensional endpoints are not represented in image 1002 (e.g., features corresponding to three-dimensional endpoints 1102-4 and 1102-6), kinematic data may be used to determine the coordinates of the three-dimensional endpoint 1102. Kinematic and / or image data 312 can be used to determine the coordinates of non-anatomical features represented within image 1002, such as the coordinates of three-dimensional endpoints 1102-5.

[0083] In examples where the distance to be determined is from an anatomical feature to a non-anatomical feature (e.g., a surgical instrument 1104 hovering above an anatomical feature), measuring the direct point-to-point distance between two three-dimensional endpoints may be useful. However, in various other examples, such as those involving three-dimensional endpoints associated with two anatomical features, it may be desirable to measure the contour distance from one three-dimensional endpoint to another along the contour of the tissue (i.e., not directly through the tissue). For example, in the example described above, to measure the distance across hernia 1004 so that the mesh patch can be cut to the appropriate size, it may be desirable to determine the distance across hernia 1004 along the surface of hernia 1004, rather than determining the distance directly through hernia 1004.

[0084] To illustrate, Figure 12 An exemplary side view 1200 of an interior space is shown, which includes an imaging device 212 for imaging a hernia 1004. Figure 12 An exemplary aspect of how the contour distance from one three-dimensional endpoint to an additional three-dimensional endpoint is shown. Specifically, as shown in side view 1200, three-dimensional endpoints 1202-1 and 1202-2 (collectively referred to as "three-dimensional endpoints 1202") may be located at opposite edges of hernia 1004. Figure 12 In this diagram, a direct point-to-point distance 1204 is drawn between the three-dimensional endpoints 1202, which passes directly through the tissue of the hernia 1004. While the point-to-point distance 1204 may be useful in some cases, it may not be the ideal distance for determining the mesh patch size to cover the hernia 1004, as the mesh patch will need to cover the surface of the hernia 1004. Therefore, it may be desirable to determine a distance above the surface of the hernia 1004, which may be referred to herein as the “contour distance” above the tissue.

[0085] To determine the contour distance, system 100 can automatically identify one or more three-dimensional midpoints 1206 (e.g., three-dimensional midpoints 1206-1 to 1206-3 and / or additional three-dimensional midpoints not explicitly shown) on a three-dimensional contour that connects the three-dimensional endpoints to the additional three-dimensional endpoints and extends along a physical surface (i.e., the outer surface of the hernia 1004) on which both the three-dimensional endpoints and the additional three-dimensional endpoints are located. Then, system 100 can determine intermediate distances 1208 (e.g., intermediate distances 1208-1 to 1208-4) for each segment of a linear segmented route from three-dimensional endpoint 1202-1 to three-dimensional endpoint 1202-2, each segment passing through each adjacent three-dimensional midpoint 1206, to substantially attach to the three-dimensional contour between the three-dimensional endpoints 1202. Based on the intermediate distance 1208, system 100 can calculate the contour distance from three-dimensional endpoint 1202-1 to three-dimensional endpoint 1202-2 as the sum of intermediate distances 1208. The sum of intermediate distances 1208 can provide an estimate of the accurate contour distance, which becomes more accurate as more three-dimensional midpoints 1206 and more intermediate distances 1208 are defined.

[0086] In some examples, the user can manually define the three-dimensional midpoint 1206 (e.g., by selecting two-dimensional midpoints point by point), or a two-dimensional line can be defined along which the three-dimensional midpoint 1206 is defined. For example, a touchscreen can be used to draw lines along an anatomical structure presented on the touchscreen (e.g., from one side of a hernia 1004 to the other) to specify the two-dimensional endpoints selected by the user and the midpoint between them. An auxiliary sensor 214 can detect this user input. Based on the user input, system 100 can estimate the contour distance between the endpoints by estimating the distance along the contour of the tissue connecting the endpoints through the midpoint. In other examples, similar results can be achieved using other types of user interfaces (e.g., pointer-based interfaces).

[0087] Additionally or alternatively, system 100 may perform operation 202 by executing efficiency-related operations associated with computer-assisted surgical system 204. For example, system 100 may perform operation 202 by controlling the movement of surgical instruments connected to computer-assisted surgical system 204 in a manner that prevents harm to the patient.

[0088] For illustration, the auxiliary sensor 214 can be implemented by a force sensor integrated into a surgical instrument controlled by a computer-assisted surgical system 204. The force sensor can be configured to generate force-sensing data representing the amount of external force applied to the surgical instrument. Based on the force-sensing data and depth data 206, system 100 can determine that the surgical instrument begins to move toward a structure (e.g., an organ or other tissue) located within the patient's internal space in response to an external force applied to it. In response, system 100 can instruct the computer-assisted surgical system 204 to stop the movement of the surgical instrument toward the structure before it damages the structure (e.g., by becoming trapped in the structure).

[0089] To illustrate, Figure 13 An exemplary configuration 1300 is shown, in which a surgical instrument 1302 is used within an internal space imaged by an imaging device 212. As shown, the surgical instrument 1302 includes a force sensor 1304. The force sensor 1304 can be integrated into the surgical instrument 1302 in any suitable manner. For example, as shown, the force sensor 1304 can be disposed on the outer surface of the surgical instrument 1302. Additionally or alternatively, the force sensor 1304 can be housed within the surgical instrument 1302. Although Figure 13 A force sensor 1304 is shown positioned distally toward the surgical instrument 1302, but it will be appreciated that the force sensor 1304 may be located anywhere on or within the surgical instrument 1302 to satisfy a particular implementation. Furthermore, in some examples, the surgical instrument 1302 may include multiple force sensors located at different locations on and / or within the surgical instrument 1302.

[0090] In some examples, a user (e.g., a surgeon) can remotely control the movement of surgical instrument 1302 by interacting with a set of master controls using his or her hand. These master controls are configured to detect various hand, wrist, and / or finger movements of the user. In response to the user's manipulation of the master controls, the computer-assisted surgical system 204 can move and / or otherwise manipulate the surgical instrument 1302 accordingly.

[0091] However, in certain situations, unintended forces may be applied to the surgical instrument 1302, causing the user's hand to disengage from the main control. In these cases, the surgical instrument 1302 may unintentionally move in an uncontrolled manner and / or at an uncontrolled speed. This may cause the surgical instrument 1302 to unintentionally come into contact with and / or become embedded in structures (e.g., organs or other tissues) within the patient's body. Such contact may result in damage to the structure.

[0092] To illustrate, while the surgeon remotely controls the movement of surgical instrument 1302, an assistant in the operating room can perform laparoscopic procedures inside the patient using laparoscopic tool 1306. In doing so, the assistant may inadvertently cause laparoscopic tool 1306 to collide with surgical instrument 1302. This collision could cause the surgeon's hand to detach from the master control used to control surgical instrument 1302. This, in turn, could cause surgical instrument 1302 to move toward structure 1308 in the direction indicated by arrow 1310.

[0093] Therefore, according to the system and method described herein, the depth data 206 generated by the depth sensor 210 within the imaging device 212, combined with the force sensing data generated by the force sensor 1304, can be used to determine that the surgical instrument 1302 begins to move toward the structure 1308 in response to an external force applied by the laparoscopic tool 1306. The force sensing data indicates the intensity and direction of the force. Based on this data, the system 100 can determine the approximate direction and velocity of the movement of the surgical instrument 1302. In some examples, kinematic data associated with the surgical instrument 1302 and maintained by the computer-aided surgical system 204 can also be used to determine the approximate direction and velocity of the movement of the surgical instrument 1302. The depth data 206 can indicate the depth of both the surgical instrument 1302 and the structure 1308, and this information can be used to determine how close the surgical instrument 1302 is to the structure 1308. Therefore, system 100 can be configured to instruct computer-assisted surgical system 204 to stop the movement of surgical instrument 1302 toward structure 1308 before the surgical instrument 1302 comes into contact with structure 1308.

[0094] Additionally or alternatively, system 100 can perform operation 202 by determining the tissue deformation that occurs when a surgical instrument comes into contact with tissue. In this example, the auxiliary sensor 214 can be implemented by a force sensor integrated into the surgical instrument controlled by the computer-assisted surgical system 204. As described above, the force sensor can be configured to generate force-sensing data representing the amount of external force applied to the surgical instrument. Based on the force-sensing data, system 100 can determine that the surgical instrument is in contact with tissue. In response to this determination, system 100 can use depth data 206 to determine the tissue deformation caused by the surgical instrument in contact with tissue.

[0095] To illustrate, Figures 14-15An exemplary configuration 1400 is shown, in which a surgical instrument 1402 is used to perform tissue deformation analysis within the internal space of a patient imaged by an imaging device 212. The surgical instrument 1402 may be similar to surgical instrument 1302 because it includes a force sensor 1404. As shown, the force sensor 1404 is positioned toward the distal end of the surgical instrument 1402.

[0096] exist Figures 14-15 In the example, the user can interact with tissue 1406 using surgical instrument 1402. Specifically, in order to perform tissue deformation analysis operations involving target location 1408, the user can guide surgical instrument 1402 toward target location 1408 of tissue 1406. Figure 14 The surgical instrument 1402 is shown prior to contact with the target location 1408, while Figure 15 The surgical instrument 1402 is shown in contact with the target location 1408.

[0097] In some examples, system 100 may allow surgical instruments 1402 to not contact tissue 1406 (e.g., in...). Figure 14 When the surgical instrument 1402 contacts the target location 1408 of the tissue 1406 (e.g., in the position shown), a first depth dataset is obtained (e.g., the first dataset included in depth data 206). System 100 can also be used when the surgical instrument 1402 contacts the target location 1408 of the tissue 1406 (e.g., in the position shown). Figure 15 When the surgical instrument 1402 contacts the target location 1408 of the tissue 1406, a second depth dataset is obtained (e.g., a second dataset included in depth data 206). Based on the first and second datasets, the system 100 can determine the deformation of the tissue 1406 at the target location 1408 caused by the contact between the surgical instrument 1402 and the target location 1408 of the tissue 1406. For example, the system 100 can determine the difference between the first and second depth datasets to determine the deformation of the tissue 1406.

[0098] Based on defined organizational characteristics, system 100 can perform one or more operations. For example, system 100 can determine the characteristics of organization 1406 based on organizational characteristics.

[0099] For example, based on the determined tissue deformation, system 100 can determine the presence of a structure beneath tissue 1406. To illustrate, by measuring the amount of force applied to a target location 1408 of tissue 1406 and determining the resulting amount of tissue deformation, system 100 can determine that a specific type of mass (e.g., a tumor) is beneath tissue 1406. This determination can be performed in any suitable manner.

[0100] As another example, based on determined tissue deformation, system 100 can determine the differences between tissue 1408 and other tissues within the patient's internal space. For instance, if target location 1406 deforms more in response to the same amount of force than different locations within tissue 1406, system 100 can determine that tissue 1406 at target location 1406 is diseased, damaged, or otherwise different from tissue 1406 at different locations.

[0101] Figure 16 An exemplary method 1600 is described that can be performed by an operations management system (e.g., system 100 and / or any implementation thereof). Although Figure 16 The exemplary operation according to one embodiment has been described, but other embodiments may omit, add, reorder, and / or modify it. Figure 16 Any operation shown.

[0102] In operation 1602, the operation management system obtains depth data representing a depth map of the patient's internal space from a depth sensor included in the imaging device. Operation 1602 can be performed in any of the manner described herein.

[0103] In operation 1604, the operation management system obtains auxiliary sensor data from auxiliary sensors not included in the imaging device. Operation 1604 can be performed in any of the manner described herein.

[0104] In operation 1606, the operation management system performs operations associated with a computer-assisted surgical system based on depth data and auxiliary sensor data, the computer-assisted surgical system being configured to perform procedures within the patient's internal space. Operation 1606 can be performed in any of the manner described herein.

[0105] Figure 17 An exemplary computer-assisted surgical system 1700 (“Surgical System 1700”) is described. Surgical System 1700 can be an implementation of Computer-Assisted Surgical System 204. As shown, Surgical System 1700 may include a control system 1702, a user control system 1704, and an auxiliary system 1706 communicatively coupled to each other. A surgical team can utilize Surgical System 1700 to perform computer-assisted surgical procedures on patient 1708. As shown, the surgical team may include surgeon 1710-1, assistant 1710-2, nurse 1710-3, and anesthesiologist 1710-4, all of whom may be collectively referred to as “surgical team member 1710”. Additional or alternative surgical team members may appear in the surgical session to meet specific implementations.

[0106] Although Figure 17The description illustrates an ongoing minimally invasive surgical procedure; however, it will be understood that the Surgical System 1700 can be similarly used to perform open surgical procedures or other types of surgical procedures that can similarly benefit from the accuracy and convenience of the Surgical System 1700. Furthermore, it will be understood that the entire surgical session employing the Surgical System 1700 may include not only the surgical phase of the surgical procedure (such as...) Figure 17 (As shown), and may also include preoperative, postoperative, and / or other appropriate surgical procedures. Surgical procedures may include any procedure that uses manual and / or instrumental techniques to examine or treat the patient's physical condition.

[0107] like Figure 17 As shown, the manipulation system 1702 may include a plurality of manipulator arms 1712 (e.g., manipulator arms 1712-1 to 1712-4), to which a plurality of surgical instruments may be coupled. Each surgical instrument may be implemented by any suitable surgical tool (e.g., a tool with tissue interaction capabilities), medical instrument, imaging device (e.g., an endoscope), sensor device (e.g., a force-sensing surgical instrument), diagnostic instrument, etc., which may be used to perform computer-assisted surgical procedures on the patient 1708 (e.g., by at least partially inserting into and being manipulated to perform computer-assisted surgical procedures on the patient 1708). Although the manipulation system 1702 is depicted and described herein as including four manipulator arms 1712, it will be appreciated that the manipulation system 1702 may include only a single manipulator arm 1712 or any other number of manipulator arms to meet a particular implementation.

[0108] The manipulator arm 1712 and / or the surgical instruments attached to the manipulator arm 1712 may include one or more displacement sensors, orientation sensors, and / or positioning sensors for generating raw (i.e., uncorrected) kinematic information. One or more components of the surgical system 1700 may be configured to use kinematic information to track (e.g., determine their positioning) and / or control the surgical instruments.

[0109] User control system 1704 can be configured to facilitate surgeon 1710-1's control of manipulator arm 1712 and surgical instruments attached to manipulator arm 1712. For example, surgeon 1710-1 can interact with user control system 1704 to remotely move or manipulate manipulator arm 1712 and surgical instruments. To this end, user control system 1704 can provide surgeon 1710-1 with images (e.g., high-resolution 3D images) of the surgical area associated with patient 1708 captured by an imaging system (e.g., any medical imaging system described herein). In some examples, user control system 1704 may include a stereoscopic viewer with two displays, wherein a stereoscopic image of the surgical area associated with patient 1708 and generated by a stereoscopic imaging system can be viewed by surgeon 1710-1. Surgeon 1710-1 can utilize the images to perform one or more procedures using one or more surgical instruments attached to manipulator arm 1712.

[0110] To facilitate control of surgical instruments, the user control system 1704 may include a set of master controls. These master controls can be manipulated by the surgeon 1710-1 to control the movement of surgical instruments (e.g., by utilizing robotic and / or teleoperation technologies). The master controls can be configured to detect various hand, wrist, and finger movements of the surgeon 1710-1. In this way, the surgeon 1710-1 can intuitively perform procedures using one or more surgical instruments.

[0111] The auxiliary system 1706 may include one or more computing devices configured to perform primary processing operations of the surgical system 1700. In such a configuration, the one or more computing devices included in the auxiliary system 1706 may control and / or coordinate operations performed by various other components of the surgical system 1700, such as the manipulation system 1702 and the user control system 1704. For example, the computing devices included in the user control system 1704 may transmit instructions to the manipulation system 1702 via the one or more computing devices included in the auxiliary system 1706. As another example, the auxiliary system 1706 may receive and process image data representing images captured by an imaging device attached to one of the manipulator arms 1712 from the manipulation system 1702.

[0112] In some examples, the assistive system 1706 may be configured to present visual content to a surgical team member 1710 who may not have access to the images provided to the surgeon 1710-1 at the user control system 1704. To this end, the assistive system 1706 may include a display monitor 1714 configured to display one or more user interfaces, such as images of the surgical area (e.g., 2D images), information associated with the patient 1708 and / or the surgical procedure, and / or any other visual content, to suit a particular implementation. For example, the display monitor 1714 may display an image of the surgical area along with additional content displayed concurrently with the image (e.g., graphical content, contextual information, etc.). In some embodiments, the display monitor 1714 is implemented via a touchscreen display that the surgical team member 1710 can interact with (e.g., via touch gestures) to provide user input to the surgical system 1700.

[0113] The operating system 1702, the user control system 1704, and the auxiliary system 1706 can be communicatively coupled to each other in any suitable manner. For example, as... Figure 17 As shown, the operating system 1702, user control system 1704, and auxiliary system 1706 can be communicatively coupled via control line 1716, which can represent any wired or wireless communication link to satisfy a specific implementation. Therefore, the operating system 1702, user control system 1704, and auxiliary system 1706 can each include one or more wired or wireless communication interfaces, such as one or more local area network interfaces, Wi-Fi network interfaces, cellular interfaces, etc.

[0114] In some examples, a non-transitory computer-readable medium may be provided for storing computer-readable instructions in accordance with the principles described herein. When executed by a processor of a computing device, the instructions may direct the processor and / or computing device to perform one or more operations, including one or more of the operations described herein. Such instructions may be stored and / or transmitted using any of a variety of known computer-readable media.

[0115] The term "non-transitory computer-readable medium" as used herein can include any non-transitory storage medium that contributes to providing data (e.g., instructions) that can be read and / or executed by a computing device (e.g., by the processor of the computing device). For example, a non-transitory computer-readable medium can include, but is not limited to, any combination of non-volatile storage media and / or volatile storage media. Exemplary non-volatile storage media include, but are not limited to, read-only memory, flash memory, solid-state drives, magnetic storage devices (e.g., hard disks, floppy disks, magnetic tapes, etc.), ferroelectric random access memory (“RAM”), and optical discs (e.g., compact discs, digital video discs, Blu-ray discs, etc.). Exemplary volatile storage media include, but are not limited to, RAM (e.g., dynamic RAM).

[0116] Figure 18 An exemplary computing device 1800 is described, which can be specifically configured to perform one or more processes described herein. Any of the systems, computing devices, and / or other components described herein can be implemented by computing device 1800.

[0117] like Figure 18 As shown, computing device 1800 may include communication interface 1802, processor 1804, storage device 1806, and input / output (“I / O”) module 1808, which are communicatively connected to each other via communication infrastructure 1810. Although in Figure 18 An exemplary computing device 1800 is shown, but Figure 18 The components described herein are not intended to be limiting. Additional or alternative components may be used in other embodiments. A more detailed description will now follow. Figure 18 Components of the computing device 1800 shown.

[0118] Communication interface 1802 can be configured to communicate with one or more computing devices. Examples of communication interface 1802 include, but are not limited to, wired network interfaces (e.g., network interface cards), wireless network interfaces (e.g., wireless network interface cards), modems, audio / video connections, and any other suitable interfaces.

[0119] Processor 1804 generally refers to any type or form of processing unit capable of processing data and / or interpreting, executing, and / or directing the execution of one or more of the instructions, procedures, and / or operations described herein. Processor 1804 can perform operations by executing computer-executable instructions 1812 (e.g., application programs, software, code, and / or other executable data instances) stored in storage device 1806.

[0120] Storage device 1806 may include one or more data storage media, devices, or configurations, and may take any type, form, and combination of data storage media and / or devices. For example, storage device 1806 may include, but is not limited to, any combination of non-volatile media and / or volatile media described herein. Electronic data, including the data described herein, may be stored temporarily and / or permanently in storage device 1806. For example, data representing computer-executable instructions 1812 configured to boot processor 1804 to perform any of the operations described herein may be stored within storage device 1806. In some examples, data may be arranged in one or more databases residing within storage device 1806.

[0121] I / O module 1808 may include one or more I / O modules configured to receive user input and provide user output. I / O module 1808 may include any hardware, firmware, software, or a combination thereof that supports input and output capabilities. For example, I / O module 1808 may include hardware and / or software for capturing user input, including but not limited to a keyboard or keypad, a touchscreen component (e.g., a touchscreen display), a receiver (e.g., an RF or infrared receiver), a motion sensor, and / or one or more input buttons.

[0122] I / O module 1808 may include one or more means for presenting output to a user, including but not limited to a graphics engine, a display (e.g., a screen), one or more output drivers (e.g., display drivers), one or more audio speakers, and one or more audio drivers. In some embodiments, I / O module 1808 is configured to provide graphical data to the display for presentation to the user. The graphical data may represent one or more graphical user interfaces and / or any other graphical content that may satisfy a particular implementation.

[0123] Various exemplary embodiments have been described in the foregoing description with reference to the accompanying drawings. However, it will be apparent that various modifications and changes can be made thereto, and additional embodiments can be implemented without departing from the scope of the invention as set forth in the appended claims. For example, certain features of one embodiment described herein may be combined with or substituted for features of another embodiment described herein. Therefore, the specification and drawings are to be considered illustrative rather than restrictive.

Claims

1. A system comprising: a memory storing instructions; and a processor communicatively coupled to the memory and configured to execute the instructions to: obtain, from a depth sensor included in an imaging device, depth data representing a depth map of an interior space of a patient; obtain, from a force sensor integrated into a surgical instrument controlled by a computer-assisted surgical system, force sensing data representing an amount of an external force applied to the surgical instrument; and based on the depth data and the force sensing data, perform an operation associated with the computer-assisted surgical system, the computer-assisted surgical system configured to perform a procedure within the interior space of the patient, wherein the performing of the operation associated with the computer-assisted surgical system comprises: based on the depth data and the force sensing data, determining, in response to the external force applied to the surgical instrument, that the surgical instrument begins to move toward a structure located in the interior space; and in response to determining that the surgical instrument begins to move toward the structure, instructing the computer-assisted surgical system to stop the movement of the surgical instrument toward the structure.

2. The system of claim 1, wherein: the depth sensor comprises a time-of-flight sensor in the imaging device; and the obtaining of the depth data representing the depth map comprises: directing the time-of-flight sensor to obtain the depth data; and receiving the depth data from the time-of-flight sensor.

3. The system of claim 2, wherein, the imaging device further comprises a visible light camera configured to obtain a two- dimensional visible light image of the interior space.

4. The system of claim 2, wherein: the processor is further configured to execute the instructions to receive user input data indicating a first two-dimensional endpoint within the image of the interior space is selected by a user, the first two-dimensional endpoint corresponding to a first feature within the interior space depicted by the image; and the performing of the operation associated with the computer-assisted surgical system further comprises: based on the depth data and the user input data, defining a first three-dimensional endpoint corresponding to the first feature; and based on the depth data, determining a distance within the interior space from the first three-dimensional endpoint corresponding to the first feature to a second three-dimensional endpoint corresponding to a second feature.

5. The system of claim 4, wherein: the processor is further configured to execute the instructions to receive additional user input data, the additional user input data indicating a second two-dimensional endpoint within the image is selected, the second two-dimensional endpoint corresponding to the second feature; and the performing of the operation associated with the computer-assisted surgical system further comprises defining the second three-dimensional endpoint corresponding to the second feature based on the depth data and the additional user input data.

6. The system of claim 1, wherein: the depth sensor comprises a first visible light camera in the imaging device and a second visible light camera in the imaging device; and the obtaining of the depth data representing the depth map comprises: directing the first visible light camera and the second visible light camera to obtain the depth data; and receiving the depth data from the first visible light camera and the second visible light camera. The acquiring of the depth data representative of the depth map comprises: directing the first visible light camera to acquire a first image of the interior space, directing the second visible light camera to acquire a second image of the interior space, and generating the depth map based on the first image and the second image.

7. The system of claim 1, wherein: the acquiring of the depth data comprises acquiring a first depth data set when a surgical instrument controlled by the computer-assisted surgical system is not in contact with tissue in the interior space of the patient and acquiring a second depth data set when the surgical instrument is in contact with the tissue; and the performance of the operation associated with the computer-assisted surgical system comprises: determining, based on the force sensor data, that the surgical instrument is in contact with the tissue, and determining, in response to determining that the surgical instrument is in contact with the tissue and based on the first depth data set and the second depth data set, an amount of deformation of the tissue caused by the surgical instrument being in contact with the tissue.

8. The system of claim 7, wherein, the processor is further configured to execute the instructions to determine a property of the tissue based on the amount of deformation.

9. The system of claim 1, wherein, the external force is applied to the surgical instrument by an object that is not being controlled by the computer-assisted surgical system.

10. A non-transitory computer-readable medium storing instructions that, when executed, direct a processor of a computing device to perform a method comprising: acquiring, from a depth sensor included in an imaging device, depth data representative of a depth map of an interior space of a patient; acquiring, from a force sensor integrated into a surgical instrument controlled by a computer-assisted surgical system, force sensor data representative of an amount of an external force applied to the surgical instrument; and performing, based on the depth data and the force sensor data, an operation associated with the computer-assisted surgical system configured to perform a procedure within the interior space of the patient, wherein the performance of the operation associated with the computer-assisted surgical system comprises: determining, based on the depth data and the force sensor data, in response to the external force applied to the surgical instrument, that the surgical instrument begins to move toward a structure located in the interior space; and in response to determining that the surgical instrument begins to move toward the structure, instructing the computer-assisted surgical system to stop the movement of the surgical instrument toward the structure.

11. The non-transitory computer-readable medium of claim 10, wherein: the depth sensor comprises a time-of-flight sensor in the imaging device; and the acquiring of the depth data representative of the depth map comprises: directing the time-of-flight sensor to acquire the depth data; and receiving the depth data from the time-of-flight sensor.

12. The non-transitory computer-readable medium of claim 11, wherein, the imaging device further comprises a visible light camera configured to acquire two-dimensional visible light images of the interior space.

13. The non-transitory computer-readable medium of claim 11, wherein: The method further includes receiving user input data indicative of selection by a user of a first two-dimensional endpoint within an image of the interior space, the first two-dimensional endpoint corresponding to a first feature within the interior space depicted by the image; and The performance of the operations associated with the computer-assisted surgical system further includes: defining, based on the depth data and the user input data, a first three-dimensional endpoint corresponding to the first feature; and determining, based on the depth data, a distance within the interior space from the first three-dimensional endpoint corresponding to the first feature to a second three-dimensional endpoint corresponding to a second feature.

14. The non-transitory computer-readable medium of claim 13, wherein: the method further includes receiving additional user input data indicative of selection of a second two-dimensional endpoint within the image, the second two-dimensional endpoint corresponding to the second feature; and the performance of the operations associated with the computer-assisted surgical system further includes defining, based on the depth data and the additional user input data, the second three-dimensional endpoint corresponding to the second feature.

15. The non-transitory computer-readable medium of claim 10, wherein: the depth sensor includes a first visible light camera of the imaging device and a second visible light camera of the imaging device; and the obtaining of the depth data representative of the depth map includes: directing the first visible light camera to obtain a first image of the interior space, directing the second visible light camera to obtain a second image of the interior space, and generating the depth map based on the first image and the second image.

16. The non-transitory computer-readable medium of claim 10, wherein: the obtaining of the depth data includes obtaining a first depth data set when a surgical instrument controlled by the computer-assisted surgical system is not in contact with tissue in the interior space of the patient and obtaining a second depth data set when the surgical instrument is in contact with the tissue; and the performance of the operations associated with the computer-assisted surgical system includes: determining, based on the force sensor data, that the surgical instrument is in contact with the tissue, and in response to determining that the surgical instrument is in contact with the tissue and based on the first depth data set and the second depth data set, determining an amount of deformation of the tissue caused by the surgical instrument being in contact with the tissue.

17. The non-transitory computer-readable medium of claim 16, wherein the method further includes determining a property of the tissue based on the amount of deformation.

Images