Method and system for providing surgical site measurements

Abstract

A system for measuring distances within a surgical site includes a camera positionable to capture 3D image data corresponding to the treatment site. Using this image data, the system determines the relative 3D positions of identified measurement points at the treatment site captured in the image and estimates or determines the distances between those measurement points. An output is generated that communicates the measured distances to a user. The measurements taken follow the 3D topography of the tissue positioned between the measurement points.

Description

[Technical Field]

[0001] This application claims the benefit of U.S. Provisional Application No. 62 / 935,585, filed November 14, 2019, and U.S. Provisional Application No. 63 / 088,409, filed October 6, 2020, each of which is incorporated herein by reference. Inventors: Kevin Andrew Hufford, Tal Nir, Lior Alpert, Gal Wiezman, Alex Maret, Mohan Nathan [Background technology]

[0002] Obtaining measurement data from a surgical site can be very useful to a surgeon or other medical practitioner.

[0003] Size measurements within the surgical field are typically estimated by the user as they view a display of a captured endoscopic image of the surgical site; the user may refer to other elements within the image that provide size cues to facilitate the estimation (e.g., known diameters or lengths of features on a surgical instrument). In more complex cases, a sterile, flexible "tape measure" may be rolled up, inserted through a trocar, stretched across the surgical field, and manipulated using laparoscopic instruments to make the necessary measurements.

[0004] Co-pending and commonly assigned U.S. application Ser. No. 17 / 035,534, entitled "Method and System for Providing Real Time Surgical Site Measurements," describes a system and method for using image processing of an endoscopic field of view to determine sizing and measurement information for a hernia defect or other region of interest within a surgical site. Summary of the Invention [Problem to be solved by the invention]

[0005] This application describes a system that provides sizing and area measurement information that is more accurate and convenient than current methods. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 is a block diagram that schematically illustrates a system according to disclosed embodiments. [Figure 2A] 1 illustrates an example of a graphical user interface (GUI) displaying an image of a surgical site. The overlay shown on the GUI display shows measurement points (marked with the overlay symbol +) between which measurements are calculated by the system, measurement lines A and B connecting pairs of measurement points, and measurement data representing the distance between the pairs of points. [Figure 2B] 2B is similar to FIG. 2A, further showing additional measurement lines. In some embodiments, a plurality of such lines can form a polygon that can provide an area measurement. [Figure 3A] FIG. 2B is similar to FIG. 2A and further illustrates the difference between geodesic measurements (identified using irregular lines and shown inset) and straight line measurements (identified by straight lines and shown non-inset). [Figure 3B] FIG. 10 shows an example of a GUI displaying an image of a surgical site where measurement points+ are identified using points on two surgical instruments within the surgical field. [Figure 4A] FIG. 10 shows an example GUI displaying an image of a surgical site and illustrating the use of a surgical tool to identify measurement points to the system. [Figure 4B] FIG. 10 shows an example GUI displaying an image of a surgical site and illustrating the use of a surgical tool to identify measurement points to the system. [Figure 4C] FIG. 10 shows an example GUI displaying an image of a surgical site and illustrating the use of a surgical tool to identify measurement points to the system. [Figure 4D] FIG. 10 shows an example GUI displaying an image of a surgical site and illustrating the use of a surgical tool to identify measurement points to the system. [Figure 5A] 4A-4D, showing measurements being taken from identified measurement points. FIG. [Figure 5B] 4A-4D, showing measurements being taken from identified measurement points. FIG. [Figure 6] 10A-10C are a series of views of a display illustrating one embodiment in which a user defines a plane along which measurements are desired. [Figure 7] 10A-10C are a series of views of a display illustrating one embodiment in which a user defines a plane along which measurements are desired. [Figure 8] 10A-10C are a series of views of a display illustrating one embodiment in which a user defines a plane along which measurements are desired. [Figure 9] 10A-10C are a series of views of a display illustrating one embodiment in which a user defines a plane along which measurements are desired. [Figure 10] 10A-10C are a series of views of a display illustrating one embodiment in which a user defines a plane along which measurements are desired. [Figure 11] 10A-10C are a series of views of a display illustrating one embodiment in which a user defines a plane along which measurements are desired. [Figure 12] 10A-10C are a series of views of a display illustrating one embodiment in which a user defines a plane along which measurements are desired. [Figure 13] 10A-10C are a series of views of a display illustrating one embodiment in which a user defines a plane along which measurements are desired. DETAILED DESCRIPTION OF THE INVENTION

[0007] This application describes a system and method for using computer vision to analyze a surgical site and measure a distance or a series of distances within the surgical site.

[0008] Referring to FIG. 1 , an exemplary system preferably includes a 3D camera 10, one or more processors 12 that receive images / video from the camera, and a display 14. The 3D camera may include a pair of cameras (a stereo rig), a structured light-based camera (such as an Intel RealSense® camera), or a 2D camera that uses other software or hardware features to determine or derive depth information. The processor(s) include at least one memory that stores instructions executable by the processor(s), such as: (i) determining the 3D positions of points on the image captured by the data; (ii) estimating or determining linear and / or geodesic distances in 3D between identified measurement points at the surgical site being imaged by the camera based on these 3D positions; and (iii) generating an output that communicates the measured distances to a user. Depending on the type of camera, the 3D information is processed either by a processor within the camera or a processor separate from the camera. The output may be in the form of a graphical overlay on a visual display displaying the measurement data (as described in connection with the drawings) and / or in other forms such as an auditory output (i.e., in the form of a graphical overlay on a visual display displaying the measurement data and / or in other forms such as an auditory output).

[0009] The measured distance can be a straight-line "ruler distance" between the measurement points, and / or a geodesic distance between the measurement points that accounts for the change in depth of a surface feature (e.g., a tissue surface) along the line between the two measurement points (i.e., a straight-line "ruler distance" between the measurement points, or a geodesic distance between the measurement points that accounts for the change in depth of a surface feature (e.g., a tissue surface) along the line between the two measurement points). Note that these measurement points are associated with locations at the appropriate depth of the tissue or other structure within the body cavity where the measurement is being made (as determined using the system, rather than floating above the tissue at a point in space).

[0010] The system further includes devices or features that allow the system to identify certain measurement points to or by the system in many different ways. As one example, a user can identify measurement points to the system using one or more user input devices 16. When user input devices are included, various different types of user input devices can be used alone or in combination. Examples include, but are not limited to, the following devices and methods. Also provided below are examples of how these devices and methods can be used to identify measurement points when the system is in a measurement point input operating mode: An eye tracking device. The system determines the location where the user is looking on the display and receives that location as an input instructing the system to set that location as a measurement point. In an identification implementation, when the system is in a measurement point input operational mode, it displays a cursor at the location on the display viewed by the user and moves the cursor as the user's gaze moves relative to the display. In this example and examples described below, a confirmatory input (described below) can be entered into the system to confirm the user's selection of the measurement point. Head tracking device or mouse-type device: When the system is in the measurement point input operation mode, the system displays a cursor on the display and moves the cursor in response to the movement of the head-worn head tracking device or mouse-type device. A touchscreen display showing real-time images captured by the camera. The user inputs desired measurement points by touching corresponding points on the displayed image. Movement of an input handle that is also used to direct movement of a component of a surgical robotic system. The input handle can be used with an actuation connection between the input handle and a robotic component that is temporarily suspended or clutched, such that the input handle moves a cursor displayed on a display. Movement of different components on input handles for robotic surgery systems such as joysticks, touchpads, trackpads, etc. Manual or robotic manipulation of a surgical instrument within the surgical field (robotic manipulation is based on the use of input from an input handle, eye tracker, or other suitable input device). For example, the instrument may have a tip or other feature (e.g., a jaw pivot, rivet, markings) that is tracked using image processing methods when the system is in instrument-as-input mode, so that the tip or other feature can function as a mouse, pointer, and / or stylus when moved within the imaging field, etc. The tracked feature may be recognized by the system or identified to the system by the user. Alternatively or additionally, graphical markings may be displayed on the instrument (as shown in FIG. 4B) or offset from the instrument (as shown in FIGS. 4A and 4C). These icons are moved by the user via movement of the surgical instrument (either manually or by a robotic manipulator that moves the instrument in response to user input). See the discussion of FIGS. 4A-5D. When a measurement point is identified to the system using a robotically operated surgical instrument, the location of the measurement point can be calculated using only 3D image data and / or using information derived from kinematic data from the robotic manipulator to which the instrument is attached (i.e., using only 3D image data, or using information derived from kinematic data from the robotic manipulator to which the instrument is attached, or both). The system can be configured or placed in a mode where measurement points are recognized on the image using computer vision. Such points can include points on surgical devices or instruments (e.g., tips or other structural features or markings), edges, or other features such as tissue structures or properties that are recognized by the system. U.S. Application No. 17 / 035,534, entitled "Method and System for Providing Real Time Surgical Site Measurements," describes techniques that can be used to identify structures or properties. -Voice input devices, switches, etc.

[0011] The above mentioned input methods can be combined in various ways: for example, the system can apply computer vision to recognize some anatomical or tissue features and then measure the distance between the identified features and a graphical icon displayed on the tip of the instrument.

[0012] The types of input devices listed are often used in combination with a second form of confirmation input device that allows the user to input or confirm the selection of a measurement point. When used with robotic systems, the confirmation input device can include switches, buttons, touchpads, trackpads, and other user inputs. Other confirmation inputs used in robotic and non-robotic situations include voice input devices, icons that the user touches on a touchscreen, foot pedal input, keyboard input, etc.

[0013] FIG. 2 shows an example of a graphical user interface (GUI) displaying an image of a surgical site. An overlay shown on the GUI display shows three measurement points marked with a "+" icon between which measurements are calculated by the system. Using input techniques of the type described herein, a user can record measurement pins / waypoints at desired locations or place measurement pins / waypoints at desired locations and instruct the system to change the location of the points between which measurements will be taken. In some embodiments or modes of operation, the system can track the tip of an instrument moving through the surgical site in real time, and measurements are taken continuously between identified points on the instrument, such as the tip of the instrument. Graphical markings can be generated and overlaid on the image of the instrument tip to provide an enhanced visual indication of the points at which measurements are taken. This is discussed in more detail with respect to FIGS. 4A-5B.

[0014] An overlay representing the measurement lines connecting pairs of measurement points is also shown along with the corresponding measurement data. Measurements can be point-to-point data, meaning the linear distance between the points or the geodesic distance across the tissue surface between the measurement points. With respect to Figure 2, which provides geodesic measurement distances, it should be noted that although some of the displayed lines appear to be of similar length on the GUI, the distances measured along those lines (and displayed numerically between those lines) can vary significantly due to variations in surface depth between the measurement points.

[0015] As mentioned above, in some cases, it is useful to provide a measurement of the actual path along the tissue rather than a straight-line measurement between two points. This may be similar to taking measurements using a rigid ruler and placing a string overlay along the surface between those two points. The "string" can snap to features captured by the image data or be smoothly adjusted. Both types of measurements are represented by the overlays shown in FIG. 3A , which shows geodesic measurements (identified using an overlay showing irregular lines and intercalative distance data) and straight-line measurements (identified using an overlay showing straight lines and non-intercalative distance data). In some embodiments, a weighted graph search along the surface is used to find the minimum path length between two points. In some embodiments, a weighting or optimization function can be used to create a smoother path between the measured points.

[0016] Referring to Figure 3A, multiple measurement points can be identified by the system, and multiple measurement lines that form a polygon can be identified by the system, which allows the system to calculate area measurements if desired.

[0017] In some embodiments, geodesic distance can be measured by projecting a line between two selected points along the view direction of the endoscope / laparoscope (i.e., endoscope or laparoscope) onto a surface and measuring the resulting path length.

[0018] When measurement data of a tracked structure (e.g., the distance between the tips of two instruments in the surgical field, or the distance between the tip of an instrument and a marked or computer-recognized feature at the surgical site) is generated, this measurement data can be dynamically updated as the instrument or other tracked structure moves at the surgical site, and the overlay position can move to track the movement of the tracked instrument / feature (i.e., the tracked instrument or tracked feature).

[0019] On some displays, it may be useful to provide an overlay that appears to be positioned over a surface / contour or series of surfaces in the endoscopic image. An example of this is the overlay showing the path of the "string" measurement in Figure 3A. This overlay is constructed using 3D data representing surface locations at the surgical site. The overlay can be at least partially transparent so as not to obscure the surgical field.

[0020] Other forms of overlays can include those in which distance measurements are conveyed using color, a series of colors, a color gradient, a gradient of the saturation of a single color, or any combination thereof. Different colors can be used to indicate different ranges of distance. For example, distances less than 1 cm can be displayed as a red bar, distances between 1 cm and 2 cm can be displayed using a yellow bar, and distances greater than 2 cm can be displayed using a green bar.

[0021] 4A-4D show some examples of how measurement points can be identified to the system. In these examples, the position of a portion of a surgical instrument is tracked by the system using computer vision, or, in the case of a robotically operated instrument, using kinematic data from computer vision and / or the robotic manipulator supporting the surgical instrument (i.e., computer vision and / or the robotic manipulator supporting the surgical instrument). In FIG. 4A, a graphical indicator is displayed on the display and positioned offset from the instrument. In FIG. 4B, the graphical indicator is displayed rather on a portion of the instrument. In either case, as the user manipulates the instrument robotically or manually at the surgical site, the graphical indicator moves with the instrument, as if the graphical indicator and the instrument were fixed or tethered to each other.

[0022] The system may be placed in a measurement mode of operation in which measurements between graphical indicators are displayed continuously and therefore continuously updated as the instrument is moved. The path along which the measurements are made may also be marked by a graphical indicator and a numerical distance displayed. In a preferred embodiment, a straight line is displayed when the system is making a "ruler" measurement (as in FIG. 5A), or a tissue topography is displayed when the system is making a "string" measurement (FIG. 5B). y) A line following the surface contour of the object is shown. If the system provides both a "ruler" measurement mode and a "string" measurement mode, the two can be displayed simultaneously, or the user can switch between the two using any of the types of input devices described above. In one specific example where the instrument is a robotically operated instrument, the user can operate the input device on a user input handle used to control the robotic manipulator, so that the user can accomplish the switch without moving their hand from the control device.

[0023] In other embodiments, at least one measurement point can be set rather than continuing to move with the instrument. FIG. 4C shows a graphical indicator offset from the instrument on the display and having a circular border and a transparent interior. In this embodiment, the surgical instrument is manipulated at the surgical site to position the graphical indicator at the location where the user wants to input a measurement point. Once positioned at the desired point, the user provides an input instructing the system to record that location as a measurement point. In an identification embodiment where the instrument is a robotically manipulated instrument, the user can manipulate an input device on the user input handle to signal the system that the location where the graphical indicator is placed is to be received as a measurement point. The system can optionally provide visual confirmation or verification of the measurement point by dropping a graphical tag at the point, as shown in FIG. 4D. Subsequent measurement points can optionally be identified in a similar manner, and the system can then be instructed to display measurements between the identified measurement points or combinations of those points. Alternatively, as the instrument is moved within the surgical site, measurements between the tag and a predetermined portion of the instrument within the surgical field (e.g., as marked in Figures 4A, 4B, or 4C) can be displayed in real time as the instrument is moved.

[0024] 6 and 7, where depth measurements are made, a user can input three points to define a plane on which the three points lie. In FIG. 6, the dimension "12.6" identifies the linear distance between the two lateral midpoints 100, 102. The dimension "19.7" is the measurement between the three points 100, 102, 104 along a plane that crosses the tissue surface and contains those points. The user can reposition one or more of the points to obtain different measurements. In FIG. 7, the midpoint 104 has been moved to a different location, thus defining a different plane along which measurements will be made.

[0025] 8-13 show a series of views of the display illustrating one embodiment in which a user can conveniently define a desired plane for measurement. Referring to FIG. 8, the user has already positioned the instrument tips at a location within the surgical site. The system engages measurement mode (preferably in response to a user input to enter measurement mode). Once in measurement mode, the system displays a line L between the instrument tips (or another portion of the instrument, or offset from those portions as described with respect to FIG. 4A). In the illustrated example, an overlay is shown on each of the instrument tips (in this case, the overlay is a circular icon or target, but can take any shape) to mark the portion of the instrument tip where the measurement is being taken. An overlay of the plane P, including the line, is also displayed. The distance between the instrument tips along the plane as it intersects the tissue surface is displayed at the bottom of the screen (1.5 cm is shown in FIG. 9). The measurement path along the tissue surface S can be highlighted as shown. To reorient the plane and thus perform measurements along different surface paths between the instrument tips, the user provides input that rotates the plane about the axis defined by line L. While various types of user input can be provided, in one specific example, a finger wheel on the user input of the robotic surgical system can be used to rotate the plane about the line. FIGS. 10-12 illustrate a sequence in which the instrument tips are not moved and the measurement points defining line L remain fixed, but plane P is rotated about line L so that it intersects the tissue surface at different locations and provides various measurement readings of the surface distance between the points. When one or both of the instrument tips are repositioned, the line changes orientation and a new plane appears, as shown in FIG. 13.

[0026] 12 and 13 show the axis defined as a straight line between the tips of the instruments, other methods can be used to define the axis, for example, the axis can be defined by a vector perpendicular to the longitudinal axis of the laparoscope.

[0027] In some embodiments, the distances determined using the methods described herein can be fed into a numerical model of the feature or structure. This numerical model can use a finite-element approach and can be updated with each image acquisition cycle or can use an approximate model of tissue behavior / properties (i.e., tissue behavior or tissue properties) that can be updated more frequently based on element properties or other predictive techniques. This model can be used for a variety of purposes, including, but not limited to, avoiding tissue damage, navigating structures, and navigating anatomical features.

[0028] All prior patents and patent applications cited herein, including those for purposes of claiming priority, are hereby incorporated by reference. The claims as originally filed are as follows: Claim 1: a camera positionable to capture image data corresponding to the treatment area; at least one processor and at least one memory; Equipped with A system for measuring distances within a surgical site, wherein the at least one memory stores instructions that enable the at least one processor to determine the relative 3D positions of several points in the treatment site captured in the image, estimate or determine the distances between identified measurement points in the treatment site, and generate an output that communicates the measured distances to the user based on the measured dimensions. Claim 2: The system of claim 1 , wherein the distance is a straight-line distance. Claim 3: The system of claim 1 , wherein the distance is a geodesic distance that follows the topography of the tissue surface between the measurement points. Claim 4: The system of claim 1 , wherein the system includes a display, and the output includes generating an overlay displaying the measured distance. Claim 5: The system of claim 1 , wherein the system includes a display, and the instructions are further executable to generate an overlay marking the measurement points. Claim 6: The system of claim 1 , wherein the instructions are further executable by the at least one processor to receive input from a user identifying the measurement point using a user input device. Claim 7: The system of claim 1 , wherein the instructions are further executable by the at least one processor to identify the measurement points using computer vision. Claim 8: 7. The system of claim 6, wherein the measurement point is a point on or a predetermined distance from a portion of a surgical instrument placed at the treatment site. Claim 9: 8. The system of claim 7, wherein the measurement point is a point on or a predetermined distance from a portion of a surgical instrument placed at the treatment site. Claim 10: 7. The system of claim 6, wherein the instructions are executable by the at least one processor to receive input from a user selecting a plane containing the measurement points and estimate or determine the distance between identified measurement points along the tissue surface where the plane intersects the tissue. Claim 11: 8. The system of claim 7, wherein the instructions are executable by the at least one processor to receive input from a user selecting a plane containing the measurement points and estimate or determine the distance between identified measurement points along the tissue surface where the plane intersects the tissue.

Claims

1. a camera positionable to capture an image corresponding to a treatment site, the treatment site including a tissue surface having a topography; a display configured to display the image together with the tissue surface and the topography in real time so that they are visible on the display; at least one processor and at least one memory; Equipped with 11. A system for measuring distances within a surgical site, wherein the at least one memory stores instructions operable to: receive an input identifying a first measurement point captured at the treatment site in the image and determine a 3D position of the first measurement point; receive an input identifying a second measurement point captured at the treatment site in the image and determine a 3D position of the second measurement point; display a graphical planar overlay over the image displayed on the display, the graphical planar overlay including the first measurement point and the second measurement point and traversing the tissue surface, the graphical planar overlay further graphically indicating a measurement path along the tissue surface, the graphical planar overlay traversing the tissue surface; estimate or determine a distance along the measurement path between the first measurement point and the second measurement point at the treatment site; and generate an output communicating the distance to a user.

2. The system of claim 1 , wherein the distance is a distance according to the topography of the tissue surface between the first measurement point and the second measurement point.

3. The system of claim 1 , wherein the output includes generating an overlay on the display that displays the distance.

4. The system of claim 1 , wherein the instructions are further executable to generate an overlay marking the first measurement point and the second measurement point.

5. 2. The system of claim 1, wherein the instructions are further executable by the at least one processor to receive input from a user identifying the first measurement point and the second measurement point using a user input device.

6. The instruction: using computer vision analysis of the image to identify at least a portion of a first surgical instrument positioned within a body cavity at the treatment site; displaying a first overlay on the display to the first surgical instrument in a known vicinity such that the first overlay moves on the display to track the movement of a predetermined portion of the first surgical instrument, wherein the 3D position of the first overlay includes the first measurement point; The system of claim 1 , wherein the at least one processor is further operable to:

7. The system of claim 1 , wherein the first measurement point is on or a predetermined distance from a portion of a first surgical instrument placed at the treatment site.

8. The system of claim 6 , wherein the first measurement point is on or a predetermined distance from a portion of a first surgical instrument placed at the treatment site.

9. The instruction: using computer vision analysis of the image to identify at least a portion of a second surgical instrument positioned at the treatment site; displaying a second overlay on the display to the second surgical instrument in a known vicinity, the second overlay moving on the display to track the movement of the second surgical instrument, wherein the 3D position of the second overlay includes the second measurement point; The system of claim 6 , wherein the at least one processor is further operable to:

10. The instruction: receiving a user input to rotate the graphical planar overlay along a line extending between the first measurement point and the second measurement point; in response to the user input rotating the graphical planar overlay, rotating the graphical planar overlay to a rotated orientation on the display; estimating or determining the distance between the first measurement point and the second measurement point along the tissue surface, wherein an overlay of the graphical plane in the rotated orientation intersects with the tissue surface; The system of claim 1 , wherein the at least one processor is further operable to:

11. The system of claim 1 , wherein the instructions are executable by the at least one processor to receive user input specifying a location of the first measurement point using a touchscreen input.

Images