System and method for measuring distance using a stereo endoscope
Through stereo endoscopic system and image matching technology, the accuracy and efficiency of distance measurement in minimally invasive surgery are solved, and measurement convenience and accuracy similar to open surgery are achieved.
Patent Information
- Application Number
- CN201980042365.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-05-03
- Filing Date
- 2019-05-02
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2039-05-02
AI Technical Summary
In minimally invasive surgery, it is difficult to accurately measure the distance between the patient's internal anatomy and the surgical instrument, especially because the position of the surgical instrument needs to be positioned through complex motor chains, resulting in inaccurate and time-consuming measurements.
Using a stereo endoscope system, images are captured from different advantageous points through the camera, the user selects 2D endpoints and matches the corresponding 3D endpoints, and the calculation system determines the distance between the feature parts to avoid relying on kinematic data of surgical instruments.
Achieves the convenience and accuracy similar to open surgery in minimally invasive surgery, reduces measurement time, improves measurement accuracy and efficiency, and allows non-surgeons to also perform distance measurements, suitable for traditional robotic surgery system.
Smart Images

Figure CN112292092B_ABST
Abstract
Description
Technical Field
[0001] This application claims priority to U.S. Provisional Patent Application No. 62 / 666,505, filed on May 3, 2018, entitled “System and Method for Measuring Distance Using a Stereoscopic Endoscope,” the contents of which are incorporated herein by reference in their entirety. Background Art
[0002] During minimally invasive surgical procedures, such as procedures utilizing a robotic surgical system, an endoscopic camera is often used to provide the surgeon with images (e.g., stereoscopic video) of the patient's internal anatomy. While viewing these images, the surgeon can manipulate one or more surgical instruments (e.g., one or more robotically manipulated surgical instruments) positioned within the patient to perform the procedure.
[0003] In some cases, it may be desirable to measure various distances associated with a patient's internal anatomical structures before, during, or after a minimally invasive surgical procedure. For example, it may be desirable to measure the size of a hernia in the patient so that a mesh patch can be sized appropriately for the hernia. As another example, it may be desirable to determine the distance of the tip of a surgical instrument from tissue within the patient.
[0004] Unfortunately, because the surgeon performing minimally invasive surgical procedures cannot directly physically approach the patient's internal anatomical structures as he or she does during conventional "open" surgical procedures, it is difficult or impossible to accurately determine such measurements to date. A method for estimating the distance associated with the patient's internal anatomical structures during minimally invasive surgical procedures must position the robot-operated surgical instruments at different locations within the patient, and estimate the distance between the instruments based on the kinetic data for the tracking of the instruments. However, positioning the instruments in this manner may be inconvenient, difficult and / or time-consuming, and the measurements produced by this conventional method may be inaccurate to some extent due to the inaccuracy introduced on a relatively long motion chain. In addition, when using such conventional techniques, the contour of the surface between the endpoints may not be taken into account. This may cause problems for the measurements (such as the above-mentioned exemplary hernia measurements), where it may be important to consider such contours. BRIEF DESCRIPTION OF THE DRAWINGS
[0005] The accompanying drawings illustrate various embodiments and are part of the specification. The embodiments shown are examples only and do not limit the scope of the present disclosure. In all drawings, the same or similar reference numerals designate the same or similar elements.
[0006] Figure 1 An exemplary robotic surgical system according to the principles described herein is shown.
[0007] Figure 2 The invention shows a method according to the principles described herein including Figure 1 An exemplary patient-side system within a robotic surgical system.
[0008] Figure 3 The invention shows a method according to the principles described herein including Figure 2 An exemplary robotic arm within the patient-side system.
[0009] Figure 4 The invention shows a method according to the principles described herein including Figure 1 An exemplary surgeon's console within a robotic surgical system.
[0010] Figure 5 An exemplary stereoscopic endoscope is shown positioned at an exemplary surgical field associated with a patient according to principles described herein.
[0011] Figure 6 An exemplary measurement system according to the principles described herein is shown.
[0012] Figure 7 Shown are exemplary images captured from a stereoscopic vantage point by a camera included within a stereoscopic endoscope, according to principles described herein.
[0013] Figure 8 An exemplary user-selected two-dimensional ("2D") endpoint specified by a user is shown, according to principles described herein.
[0014] Fig. 9 The invention shows an example of a method according to the principles described herein. Figure 8 Exemplary matching 2D endpoints of the user-selected 2D endpoint identified as corresponding to the same feature within the surgical field are shown in FIG.
[0015] Fig.10 The present invention shows a Fig. 9 An enlarged pixel-level view of certain aspects of the image shown in FIG.
[0016] Fig.11 Shown are multiple surgical instruments within a surgical field within a patient according to the principles described herein.
[0017] Fig.12 Illustrative definitions of three-dimensional ("3D") endpoints according to principles described herein are shown.
[0018] Fig.13 Shown are exemplary 3D endpoints between which one or more distances may be determined according to the principles described herein.
[0019] Fig.14An exemplary contour distance between two 3D endpoints is shown according to the principles described herein.
[0020] Fig.15 An exemplary method for measuring distance using a stereo endoscope according to the principles described herein is shown.
[0021] Fig.16 An exemplary computing system according to the principles described herein is shown. DETAILED DESCRIPTION
[0022] Systems and methods for measuring distances using a stereo endoscope are described herein. For example, in some embodiments, a measurement system may include at least one physical computing device (e.g., a processor programmed by instructions stored in a memory or the like) that can access a first image captured by a first camera from a first vantage point and a second image captured by a second camera from a second vantage point. Both the first camera and the second camera may be included in a stereo endoscope located at a surgical area associated with a patient (e.g., extending into the surgical area to be at least partially disposed within the surgical area). For example, the surgical area may be completely or partially within the patient's body at a location where a surgical procedure is to be performed, is being performed, or has been performed. In addition, the first vantage point and the second vantage point of the first camera and the second camera may be stereoscopic with each other. In other words, the vantage points may be relatively close to each other in space (e.g., at the left and right sides of the distal tip of the stereo endoscope) and may be aligned so as to point toward the same target area.
[0023] The measurement system may receive user input specifying a user-selected two-dimensional ("2D") endpoint corresponding to a feature within the surgical field. For example, the user input may be provided by a user selecting (e.g., touching, clicking, etc.) a point within the first image corresponding to the location of the feature within the first image. Based on the user-selected 2D endpoint, the measurement system may identify a matching 2D endpoint corresponding to the feature as represented in the second image. Subsequently, based on the user-selected and matched 2D endpoints, the measurement system may define a three-dimensional ("3D") endpoint corresponding to the feature within the surgical field. For example, while the user-selected and matched 2D endpoints may constitute different pixels or groups of pixels within the first image and the second image, the 3D endpoints may constitute 3D coordinates that define a specific point in space relative to a particular coordinate system (e.g., a coordinate system associated with the surgical field, a stereo endoscope, a robotic surgical system associated with the measurement system, or the like).
[0024] Once the 3D endpoints have been defined in this manner, the measurement system can determine the distance from the 3D endpoint to additional 3D endpoints corresponding to additional features within the surgical area. In some examples, the additional 3D endpoints can be user-selected 3D endpoints defined based on user-selected 2D endpoints in a manner similar to that described above. In other examples, the additional 3D endpoints can be non-user-selected 3D endpoints. For example, the additional 3D endpoints can correspond to the position of a particular surgical instrument, to the position of a stereo endoscope, to an origin defined within the surgical area or coordinate system, or to any other position within the surgical area as may serve a particular embodiment. Various ways in which the measurement system can determine distances between 3D endpoints will be described herein.
[0025] Various benefits can be provided by the systems and methods disclosed herein. For example, the systems and methods disclosed herein can facilitate point-to-point measurement of anatomical features or other points of interest within a patient's body during a minimally invasive surgical procedure, at least as conveniently and accurately as possible in an open surgical procedure in which such features can be publicly accessed. For example, the systems and methods disclosed herein can facilitate the measurement of the size of a hernia so that a mesh patch can be cut to the appropriate size (the size of a mitral valve for a mitral valve repair operation, the size of a tumor to determine how much energy may be needed to properly destroy a tumor, and / or the size of any other anatomical feature within a patient) to properly repair a hernia. Additionally or alternatively, the systems and methods disclosed herein can facilitate the measurement of the distance between the tip of a surgical instrument and tissue within a patient, facilitate the measurement of the distance between two different surgical instruments set within a surgical area of a patient, and / or facilitate the measurement of any other distance within a patient that can be known to a surgical team during a minimally invasive surgical procedure. Therefore, the systems and methods described herein can help make minimally invasive surgical procedures increasingly safe, effective, and attractive to patients and surgeons.
[0026] In addition, compared with conventional techniques for measuring distances within a surgical area associated with minimally invasive surgical procedures, the systems and methods disclosed herein can be relatively more accurate, more efficient, and / or easier to use. For example, the systems and methods disclosed herein do not require a user to specify a 3D endpoint for measurement by manually positioning the tip of a surgical instrument at a specific 3D endpoint, because the user will have to operate using conventional techniques that rely only on kinematic data to estimate the distance within the surgical area. As another example, the systems and methods described herein can facilitate relatively easy and accurate measurement of distances along the contours of anatomical structures, which has been difficult or impossible during minimally invasive surgical presentations to date. In addition, by allowing endpoints to be selected based only on 2D vision, the systems and methods described herein can enable team members other than surgeons (e.g., nurses, assistants, etc.) to specify endpoints using a screen that does not provide a stereoscopic 3D view, thereby freeing up time for the surgeon to perform other tasks specific to the surgeon.
[0027] The systems and methods disclosed herein may also advantageously utilize a minimal amount of computer processing resources, which may be beneficial in conventional systems that employ relatively outdated technology and have relatively limited computer processing resources. For example, many currently deployed robotic surgical systems may utilize relatively old computer hardware that is not designed to efficiently perform extensive image processing (e.g., generating an entire 3D depth map from each point within a stereo image captured by a stereo endoscope). Because the systems and methods do not require the generation of an entire 3D depth map in order to measure point-to-point distances, such conventional robotic surgical systems may implement the systems and methods described herein.
[0028] Various embodiments will now be described in more detail with reference to the accompanying drawings.The systems and methods described herein may provide one or more of the benefits described above and / or various additional and / or alternative benefits that will become apparent herein.
[0029] The systems and methods described herein may be operated as part of or in conjunction with a robotic surgical system.Therefore, to facilitate understanding of the systems and methods described herein for measuring distances using a stereo endoscope, an exemplary robotic surgical system will now be described.
[0030] Figure 1An exemplary robotic surgical system 100 is shown. As shown, the robotic surgical system 100 may include a patient-side system 102 (sometimes referred to as a patient-side cart), a surgeon console 104, and a vision cart 106 that are communicatively coupled to each other. The robotic surgical system 100 may be used by a surgical team to perform robotic-enabled surgical procedures on a patient 108. As shown, the surgical team may include a surgeon 110-1, an assistant 110-2, a nurse 110-3, and an anesthesiologist 110-4, all of whom are collectively referred to as "surgical team members 110." Additional or alternative surgical team members may be present during the surgical session as may serve a particular embodiment. While Figure 1 A minimally invasive surgical procedure is shown being performed, but it will be understood that the robotic surgical system 100 may similarly be used to perform open surgical procedures or other types of surgical procedures that may similarly benefit from the accuracy and convenience of the robotic surgical system 100. Additionally, it will be understood that a surgical session in which the robotic surgical system 100 may be employed may not include, for example, Figure 1 The operative phase of a surgical procedure is shown in , but may also include pre-operative, post-operative, and / or other suitable phases of a surgical procedure.
[0031] As shown, the patient-side system 102 may include a plurality of robotic arms 112 (e.g., robotic arms 112-1 to 112-4), wherein a plurality of robotically manipulated surgical instruments 114 (e.g., surgical instruments 114-1 to 114-4) may be coupled to the plurality of robotic arms 112. Each surgical instrument 114 may be implemented by any suitable surgical tool, medical tool, monitoring instrument (e.g., endoscope), diagnostic instrument, or the like that may be used for a robotic-enabled surgical procedure on a patient 108 (e.g., by being at least partially inserted into and manipulated to perform a robotic-enabled surgical procedure on the patient 108). Note that while the patient-side system 102 is depicted and described herein as a cart having a plurality of robotic arms 112 for exemplary purposes, in various other embodiments, the patient-side system 102 may include one or more carts, each having one or more robotic arms 112, the one or more robotic arms 112 being mounted on a separate structure within an operating room, such as an operating table or ceiling, and / or mounted on any other support structure(s). The patient-side system 102 will be described in more detail below.
[0032] The surgical instruments 114 can each be positioned at a surgical area associated with the patient. As used herein, in some examples, the "surgical area" associated with the patient can be disposed in its entirety within the patient and can include an area within the patient, where the surgical procedure is planned to be performed near the area within the patient, is being performed near the area within the patient, or has been performed near the area within the patient. For example, for a minimally invasive surgical procedure being performed on tissue inside the patient, the surgical area can include the space around the tissue where the surgical instruments used to perform the surgical procedure are located. In other examples, the surgical area can be at least partially disposed outside the patient. For example, the robotic surgical system 100 can be used to perform an open surgical procedure so that a portion of the surgical area (e.g., the tissue being operated on) is inside the patient, while another portion of the surgical area (e.g., the space around the tissue where one or more surgical instruments can be disposed) is outside the patient. When at least a portion of the surgical instrument is disposed within the surgical area, the surgical instrument (e.g., any surgical instrument in the surgical instrument 114) can refer to being "located" (or "located within") the surgical area.
[0033] The surgeon console 104 may be configured to facilitate control of the robotic arms 112 and surgical instruments 114 by the surgeon 110-1. For example, the surgeon console 104 may provide the surgeon 110-1 with an image (e.g., a high definition 3D image) of the surgical area associated with the patient 108 as captured by the endoscope. The surgeon 110-1 may utilize the image to perform one or more surgical procedures on the surgical instruments 114.
[0034] To facilitate control of the surgical instruments 114, the surgeon console 104 may include a set of master controls 116 (shown in close-up view 118). The master controls 116 may be manipulated by the surgeon 110-1 to control the movement of the surgical instruments 114. The master controls 116 may be configured to detect a variety of hand, wrist, and finger movements by the surgeon 110-1. In this manner, the surgeon 110-1 may intuitively perform procedures using one or more surgical instruments 114. For example, as shown in close-up view 120, the functional tips of the surgical instruments 114-1 and 114-4, respectively coupled to the robotic arms 112-1 and 112-4, may simulate the dexterity of the hands, wrists, and fingers of the surgeon 110-1 across multiple degrees of freedom of motion to perform one or more surgical procedures (e.g., incision procedures, suturing procedures, etc.).
[0035] Although the surgeon console 104 is depicted and described herein as a single unit for exemplary purposes, in various other embodiments, the surgeon console 104 may include various discrete components, such as a wired or wireless main control 116, (one or more) separate display elements (e.g., a projector or head mounted display), separate data / communication processing hardware / software, and / or any other structural or functional elements of the surgeon console 104. The surgeon console 104 will be described in more detail below.
[0036] The vision cart 106 can be configured to present visual content to surgical team members 110 who do not have access to images provided to the surgeon 110-1 at the surgeon console 104. To this end, the vision cart 106 can include a display monitor 122 that is configured to display one or more user interfaces, such as images of the surgical area (e.g., 2D images), information associated with the patient 108 and / or the surgical procedure, and / or any other visual content that can serve a particular embodiment. For example, the display monitor 122 can display an image of the surgical area with additional content (e.g., graphical content, contextual information, etc.) superimposed on the image or otherwise displayed simultaneously with the image. In some embodiments, the display monitor 122 is implemented by a touch screen display, wherein the surgical team member 110 can interact with the touch screen display (e.g., by way of touch gestures) to provide user input to the robotic surgical system 100.
[0037] The patient-side system 102, surgeon console 104, and vision cart 106 may be communicatively coupled to one another in any suitable manner. Figure 1 As shown in , the patient-side system 102, the surgeon console 104, and the vision cart 106 may be communicatively coupled via a control line 124, which may represent any wired or wireless communication link as may serve a particular implementation. To this end, the patient-side system 102, the surgeon console 104, and the vision cart 106 may each include one or more wired or wireless communication interfaces, such as one or more local area network interfaces, Wi-Fi network interfaces, cellular interfaces, etc.
[0038] The patient-side system 102, the surgeon console 104, and the vision cart 106 may each include at least one computing device configured to control, direct, and / or facilitate the operation of the robotic surgical system 100. For example, the surgeon console 104 may include a computing device configured to transmit instructions to the patient-side system 102 via one or more of the control lines 124 to control the movement of the robotic arm 112 and / or the surgical instrument 114 in accordance with manipulations performed by the surgeon 110-1 of the master control 116. In some examples, the vision cart 106 may include one or more computing devices configured to perform the primary processing operations of the robotic surgical system 100. In such a configuration, the one or more computing devices included in the vision cart 106 may control and / or coordinate operations performed by various other components of the robotic surgical system 100 (e.g., by the patient-side system 102 and / or the surgeon console 104). For example, the computing device included in the surgeon console 104 may transmit instructions to the patient-side system 102 via one or more computing devices included in the vision cart 106.
[0039] Figure 2 A perspective view of the patient-side system 102 is shown. As shown, the patient-side system 102 includes a cart column 202 supported by a base 204. In some examples, the cart column 202 can include a protective cover 206 that protects components of the balance subsystem and the brake subsystem disposed within the cart column 202 from contaminants.
[0040] The cart column 202 may support a plurality of setup arms 208 (e.g., setup arms 208-1 to 208-4) mounted thereon. Each setup arm 208 may include a plurality of links and joints that allow for manual positioning of the setup arm 208, and may each be connected to one of the robotic arms 112. Figure 2 In the example of FIG. 1 , the patient-side system 102 includes four setup arms 208 and four robotic arms 112. However, it will be appreciated that the patient-side system 102 may include any other number of setup arms 208 and robotic arms 112 as may serve a particular implementation.
[0041] The setup arm 208 can be non-robotically controllable and configured to statically hold each robotic arm 112 in a respective position as required by the person setting up or reconfiguring the patient-side system 102. The setup arm 208 can be coupled to the carriage housing 210 and manually moved and positioned during the preoperative, intraoperative, or postoperative phases of a surgical session. For example, the setup arm 208 can be moved and positioned during the preoperative phase when the robotic surgical system 100 is being prepared for and / or for a surgical procedure to be performed. In contrast, the robotic arms 112 can be automatically controlled (e.g., as described above, in response to manipulation of the master controls 116).
[0042] As shown, each robotic arm 112 can have a surgical instrument 114 coupled thereto. In some examples, three of the four robotic arms 112 can be configured to move and / or position surgical instruments 114 for manipulating patient tissue and / or other objects (e.g., suture materials, patch materials, etc.) within the surgical field. Specifically, as shown, robotic arms 112-1, 112-3, and 112-4 can be used to move and / or position surgical instruments 114-1, 114-3, and 114-4, respectively. As will be described in more detail below, the fourth robotic arm 112 (e.g., Figure 2 The robotic arm 112-2 in the example can be used to move and / or position a monitoring instrument (e.g., a stereo endoscope).
[0043] The robotic arms 112 may each include one or more displacement transducers, orientation sensors, and / or position sensors (e.g., sensors 212) for generating raw (i.e., uncorrected) kinematic information to assist in controlling and tracking the surgical instruments 114. For example, the kinematic information generated by the transducers and sensors in the patient-side system 102 may be transmitted to an instrument tracking system (e.g., a computing device included in the vision cart 106) of the robotic surgical system 100. In certain embodiments, each surgical instrument 114 may similarly include a displacement transducer, a position sensor, and / or an orientation sensor (e.g., sensors 214), each of which may provide additional kinematic information to the tracking system. The tracking system may process the kinematic information received from the sensors included on the robotic arms 112 and / or surgical instruments 114 to perform various operations, such as determining the current position of the robotic arms 112 and / or surgical instruments 114. Additionally, one or more surgical instruments 114 may include an identifier (not explicitly shown) to assist in acquiring and tracking the surgical instruments 114 as may serve a particular embodiment.
[0044] Figure 3A perspective view of an exemplary robotic arm 112 (e.g., any one of robotic arms 112-1 to 112-4) is shown. As shown, a surgical instrument 114 can be removably coupled to the robotic arm 112. Figure 3 In the example of , surgical instrument 114 is an endoscopic device (e.g., a stereo laparoscope, an arthroscope, a hysteroscope, or another type of stereoscopic or monoscopic endoscope). Alternatively, surgical instrument 114 can be a different type of imaging device (e.g., an ultrasound device, a fluoroscopy device, an MRI device, etc.), a gripping instrument (e.g., a forceps), a needle driver (e.g., a device for suturing), an energy instrument (e.g., a cauterization instrument, a laser instrument, etc.), a retractor, a clip applier, a probe holder, a cardiac stabilizer, or any other suitable instrument or tool.
[0045] In some examples, it may be desirable that the robotic arm 112 and the surgical instrument 114 coupled to the robotic arm 112 move about a single fixed center point 302 so as to constrain the motion of the center point 302. For example, the center point 302 may be located at or near the insertion point of the surgical instrument 114 into the patient 108. For example, in certain surgical sessions (e.g., surgical sessions associated with laparoscopic surgical procedures), the center point 302 may be aligned with an incision point to an internal surgical site through a trocar or cannula at the abdominal wall. As shown, the center point 302 may be located on an insertion axis 304 associated with the surgical instrument 114.
[0046] The mechanical arm 112 may include a plurality of links 306 (e.g., links 306-1 to 306-5) that are pivotally coupled in series at a plurality of joints 308 (e.g., joints 308-1 to 308-4) near respective ends of the links 306. For example, as shown, the link 306-1 is pivotally coupled to the drive mount 310 at a joint 308-1 near a first end of the link 306-1, while being pivotally coupled to the link 306-2 at a joint 308-2 near a second end of the link 306-1. The link 306-3 is pivotally coupled to the link 306-2 near a first end of the link 306-3, while being pivotally coupled to the link 306-4 at a joint 308-4 near a second end of the link 306-3. Generally, as shown, link 306-4 may be substantially parallel to insertion axis 304 of surgical instrument 114. Link 306-5 may be slidably coupled to link 306-4 to allow surgical instrument 114 to be mounted to and slide along link 306-5 as shown.
[0047] As described above, the robotic arm 112 can be configured to be mounted to the setup arm 208 (or a joint connected thereto) via the drive mount 310 so as to be supported and held in position by the setup arm 208. The drive mount 310 can be pivotally coupled to the link 306-1 and can include a first internal motor (not explicitly shown) configured to yaw the robotic arm 112 about a yaw axis of the center point 302. In a similar manner, the link 306-2 can house a second internal motor (not explicitly shown) configured to drive and pitch the linkage of the robotic arm 112 about a pitch axis of the center point 302. Similarly, the link 306-4 can include a third internal motor (not explicitly shown) configured to slide the link 306-5 and the surgical instrument 114 along the insertion axis 304. The robotic arm 112 can include a transmission system driven by one or more of these motors to control the pivoting of the link 306 about the joint 308 in any manner that can serve a particular embodiment. Thus, if surgical instrument 114 is to be moved mechanically, one or more of the motors coupled to the drive train may be turned on to move linkage 306 of robotic arm 112 .
[0048] Figure 4 104. As shown, the surgeon's console 104 may include a stereoscopic viewer 402, an arm support 404, a main control 116 ( Figure 4 406, foot pedals 408 and head sensor 410 (not shown).
[0049] In some examples, the stereoscopic viewer 402 has two displays, where a stereoscopic 3D image of the surgical area associated with the patient 108 and generated by the stereoscopic endoscope can be viewed by an operator (e.g., surgeon 110-1) during a surgical session. When using the surgeon console 104, the operator can move his or her head to align with the stereoscopic viewer 402 to view the 3D image of the surgical area. In order to ensure that the operator is viewing the surgical area when controlling the surgical instruments 114 of the patient-side system 102, the surgeon console 104 can use a head sensor 410 disposed adjacent to the stereoscopic viewer 402. Specifically, when the operator aligns his or her eyes with the binocular eyepieces of the stereoscopic viewer 402 to view the stereoscopic image of the surgical area, the operator's head can activate the head sensor 410, which can control the surgical instrument 114 through the main control 116. When the operator's head is removed from the area of the stereoscopic viewer 402, the head sensor 410 can be automatically deactivated, which can prevent the surgical instrument 114 from being controlled through the main control 116. In this manner, the position of the surgical instrument 114 may remain stationary when the robotic surgical system 100 detects that the operator is not actively engaged in attempting to control the surgical instrument 114 .
[0050] The arm supports 404 may be used to support the operator's elbows and / or forearms as the operator manipulates the main controls 116 to control the robotic arms 112 and / or surgical instruments 114. Additionally, the operator may use his or her feet to control the foot pedals 408. The foot pedals 408 may be configured to change the configuration or operating mode of the robotic surgical system 100, to generate additional control signals for controlling the surgical instruments 114, to facilitate switching control from one surgical instrument 114 to another, or to perform any other suitable operation.
[0051] Figure 5 An exemplary stereoscopic endoscope 500 is shown included within the robotic surgical system 100 and positioned at an exemplary surgical field associated with a patient. The stereoscopic endoscope 500 may be any of the surgical instruments 114 described above.
[0052] As shown, a stereo endoscope 500 may include a tube 502 having a distal tip configured to be inserted into a patient's body and a camera 504 configured to be located outside the patient. The tube 502 may be coupled to the camera 504 at a proximal end and may be rigid (e.g., Figure 5 as shown in ), joined and / or flexible.
[0053] The tube 502 may include a plurality of channels 506 (e.g., a right imaging channel 506-R, a left imaging channel 506-L, and an illumination channel 506-I) configured to conduct light between a surgical field outside the patient and the camera 504. Each channel 506 may include one or more optical fibers configured to transmit light along the tube 502, such that light generated within the camera 504 may be transmitted by the illumination channel 506-I to be output at the distal end of the tube 502, and transmitted from the distal end of the tube 502 back to the camera 504 by the imaging channels 506-R and 506-L after reflecting from the patient's anatomy and / or other objects within the surgical field. Figure 5 The arrows shown within channels 506 in are depicted to indicate the direction in which light can travel within each channel. In addition, it will be understood that tube 502 can be associated with (e.g., include) one or more lenses or other suitable optical devices (not explicitly shown) for focusing, diffusing, or otherwise processing light transmitted by channel 506 as may serve a particular embodiment. In various other embodiments, there can be additional imaging channels and / or illumination channels. In other embodiments, one or more image sensors and / or illuminators can be positioned closer to the distal end of tube 502, thereby minimizing or even eliminating the need for imaging and / or illumination channels through tube 502.
[0054] In some examples, the stereo endoscope 500 can be coupled to a robotic arm of a robotic surgical system (e.g., one of the robotic arms 112 of the robotic surgical system 100) and positioned so that the distal tip of the tube 502 is disposed within a surgical field of a patient. In this configuration, the stereo endoscope 500 can be referred to as being located at or within the surgical field, even though a portion of the stereo endoscope 500 (e.g., the camera 504 and the proximal portion of the tube 502) can be located outside the surgical field. While the stereo endoscope 500 is located at the surgical field, light reflected from the surgical field can be captured by the distal tip of the tube 502 and transmitted to the camera 504 through the imaging channels 506-R and 506-L.
[0055] The camera head 504 may include various components configured to facilitate the operation of the stereo endoscope 500. For example, as shown, the camera head 504 may include an image sensor 508 (e.g., an image sensor 508-R associated with the right imaging channel 506-R and an image sensor 508-L associated with the left imaging channel 506-L). The image sensor 508 may be implemented as any suitable image sensor, such as a charge coupled device ("CCD") image sensor, a complementary metal oxide semiconductor ("CMOS") image sensor, or the like. In addition, one or more lenses or other optical devices may be associated with the image sensor 508 (not explicitly shown). The camera head 504 may also include an illuminator 510, which is configured to generate light to travel from the camera head 504 to the surgical field via the illumination channel 506-I so as to illuminate the surgical field.
[0056] The camera head 504 may also include a camera control unit 512 disposed therein. Specifically, the camera control unit 512-R may be communicatively coupled to the image sensor 508-R, and the camera control unit 512-L may be communicatively coupled to the image sensor 508-L. The camera control units 512 may be synchronously coupled to each other via a communication link 514, and may be implemented by software and / or hardware configured to control the image sensor 508 so as to generate corresponding images 516 (i.e., an image 516-R associated with the right side and an image 516-L associated with the left side) based on the light sensed by the image sensor 508. Therefore, each corresponding combination of the imaging channel 506, the image sensor 508, the camera control unit 512, and the associated devices may be collectively referred to as a camera included in the stereo endoscope 500. For example, the stereo endoscope 500 may include two such cameras, one for the left side and one for the right side. It can be said that such a camera captures the image 516 from a vantage point at the distal end of its corresponding imaging channel 506. When generated by stereo endoscope 500, image 516 can be accessed by a measurement system and / or otherwise used in any manner described herein. For example, image 516 can be used by a measurement system to measure distances within a surgical field of a patient.
[0057] To illustrate, Figure 6 An exemplary measurement system 600 ("system 600") configured to measure distance using images generated by a stereo endoscope is shown. As shown, system 600 may include, but is not limited to, a stereo image capture device 602, a feature recognition device 604, a 3D computing device 606, and a storage device 608 selectively and communicatively coupled to each other. It will be appreciated that although devices 602 to 608 are Figure 6602 to 608 may be combined into fewer devices, such as a single device, or may be divided into more devices as may serve a particular implementation. Each of the devices 602 to 608 may be implemented by any suitable combination of hardware and / or software.
[0058] The system 600 may be associated with the robotic surgical system 100 in any suitable manner. For example, the system 600 may be implemented by the robotic surgical system 100, or included within the robotic surgical system 100. For illustration, the system 600 may be implemented by one or more computing devices included within the patient-side system 102, the surgeon console 104, and / or the vision cart 106. Additionally or alternatively, the system 600 may be implemented by one or more hardware components associated with the stereo endoscope 500, the display monitor 122 of the vision cart 106, the stereo viewer 402 of the surgeon console 104, a user interface (e.g., a touch screen, a computer mouse or other user input device, etc.), and / or any other components as may serve a particular implementation. In some examples, the system 600 may be implemented at least in part by one or more computing devices communicatively coupled to the robotic surgical system 100 but not included in the robotic surgical system 100 (e.g., one or more servers communicatively coupled to the robotic surgical system 100 via a network).
[0059] Stereoscopic image capture device 602 can be configured to access images captured by a stereoscopic endoscope located at a surgical area associated with a patient. For example, in some embodiments, stereoscopic image capture device 602 can access a first image (e.g., image 516-L) captured by a first camera included in a stereoscopic endoscope located at the surgical area from a first vantage point and a second image (e.g., image 516-R) captured by a second camera included in the stereoscopic endoscope from a second vantage point that is stereoscopic with respect to the first vantage point.
[0060] The stereoscopic image capture device 602 may access the images in any suitable manner. For example, the stereoscopic image capture device 602 may be incorporated into the stereoscopic endoscope 500 and may access the images by capturing the images from respective stereoscopic vantage points (e.g., at respective ends of the imaging channels 506-L and 506-R) using a first camera and a second camera included in the stereoscopic endoscope 500. In other embodiments, the stereoscopic image capture device 602 may not be incorporated into the stereoscopic endoscope 500, but may access the images 516 by receiving the images 516 from the stereoscopic endoscope 500.
[0061] Feature recognition device 604 may be configured to receive user input (e.g., from a user of system 600) that specifies a user-selected 2D endpoint corresponding to a feature within the surgical area as represented in the first image. Feature recognition device 604 may also be configured to identify a matching 2D endpoint corresponding to a feature as represented in the second image based on the user-selected 2D endpoint. Each of these operations will be described in more detail below.
[0062] As used herein, a "2D endpoint" may refer to a pixel or a group of contiguous pixels that is included within a 2D image and defines one of at least two endpoints, wherein a 3D measurement is performed between the at least two endpoints. For example, as used herein, a "user-selected 2D endpoint" may refer to a 2D endpoint that is manually selected by a user on a particular 2D image, and a corresponding "matched 2D endpoint" may refer to a 2D endpoint that is automatically identified as corresponding to the user-selected 2D endpoint on a different 2D image, wherein the different 2D image is stereoscopic for the particular 2D image. Examples of user-selected 2D endpoints and corresponding matched 2D endpoints will be described and illustrated in more detail below.
[0063] The 3D computing device 606 may be configured to define 3D endpoints corresponding to features within the surgical area based on the user-selected 2D endpoints and the matching 2D endpoints determined or identified by the feature recognition device 604, wherein the features within the surgical area also correspond to the user-selected 2D endpoints and the matching 2D endpoints. The 3D computing device 606 may also be configured to determine distances from the defined 3D endpoints to additional 3D endpoints corresponding to additional features within the surgical area. The additional 3D endpoints may be derived based on the additional user-selected 2D endpoints, or may be determined or selected in another suitable manner that will be described in more detail below. Each of the operations performed by the 3D computing device 606 will be described in more detail below.
[0064] As used herein, a "3D endpoint" may refer to a point or position in 3D space relative to a particular reference system, such as a global coordinate system associated with the system 600, stereo endoscope 500, or the like. As described above, a 3D endpoint may be derived based on a 2D endpoint. However, unlike a 2D endpoint that may only exist within the context of a particular image, a 3D endpoint may define three dimensions of a real point in space. Thus, while a 2D endpoint may be associated with a particular 2D image, a 3D endpoint may be associated with an anatomical feature or other feature of a real object included in a surgical field represented by the particular 2D image.
[0065] In particular embodiments, storage device 608 may store and maintain any data received, generated, managed, maintained, used, and / or transmitted by devices 602 through 606. For example, storage device 608 may store program instructions (e.g., computer code) for performing the operations described herein, image data, kinematic data, user input data, 3D position data, and / or any other data as may serve a particular embodiment.
[0066] Various examples of distance measurement operations performed by the system 600 will now be provided. It will be appreciated that the examples provided herein are illustrative only, and that the system 600 may perform additional or alternative operations as may serve a particular embodiment. It will also be appreciated that the system 600 may perform measurements at any time during a surgical session. For example, while many of the exemplary measurements described herein are described as being performed intraoperatively (i.e., during a surgical procedure), various measurements may be performed preoperatively (e.g., to determine the size of a tumor in preparation for resection) or postoperatively (e.g., to ensure that the size of a crater resected from a tumor matches the expected preoperative size).
[0067] Figure 7 Exemplary images 702 (i.e., images 702-L and 702-R) of a surgical area associated with a patient as captured from a stereoscopic vantage point by a camera included within stereo endoscope 500 are shown. For example, images 702-L and 702-R can implement images 516-L and 516-R described above, respectively. As shown, each image 702 shows a representation of a hernia 704 to be measured so that, for example, a mesh patch can be cut to the appropriate size to properly repair the hernia 704.
[0068] As shown, images 702 are relatively similar to one another. However, it will be appreciated that due to the stereoscopic nature of the vantage point from which each image 702 is captured, subtle differences also exist between images 702. Thus, when image 702-L is presented to the left eye of a user (e.g., surgeon 110-1) and image 702-R is presented to the right eye of the user, images 702 may appear to be three-dimensional.
[0069] As with image 516, one or both of images 702 may be displayed or presented by system 600 in any suitable manner and / or on any suitable display screen. For example, one or both of images 702 may be displayed on the display monitor 122 of the vision cart 106, on the stereoscopic viewer 402 on the surgeon console 104, on a monoscopic display provided by the stereo endoscope 500, and / or on any other display screen associated with the robotic surgical system 100 or system 600.
[0070] To measure distances between features displayed in image 702 (e.g., the distance between the edges of hernia 704) or between features displayed in image 702 and additional features not currently displayed in image 702, a user can interact with one or both of images 702 to specify one or more user-selected 2D endpoints corresponding to the feature(s).
[0071] To illustrate, Figure 8 An exemplary user-selected 2D endpoint 800 is shown as being specified by a user on image 702-L. Figure 8 , but it will be appreciated that the user-selected 2D endpoint 800 may alternatively be specified by the user on image 702-R. The user-selected 2D endpoint 800 may correspond to a particular feature. As used herein, a "feature" to which an endpoint (e.g., a 2D endpoint or a 3D endpoint) corresponds may refer to a particular component, point, or other feature of an anatomical structure or other object within the surgical field. For example, the feature to which the user-selected 2D endpoint 800 corresponds is a particular portion of the edge of the hernia 704 (at Figure 8 702 -R to show the feature in more detail). User-selected 2D endpoint 800 may include multiple pixels that together may depict a corresponding feature, such that the feature may be identifiable when searched within image 702 -R, as will be described in more detail below.
[0072] The system 600 may receive any suitable type of user input specifying a user-selected 2D endpoint 800, and may facilitate entry of such user input in various ways. For example, the system 600 may receive user input specifying a user-selected 2D endpoint 800 by providing the image 702-L for display on a monoscopic display screen communicatively coupled to the system 600 and receiving the user input when the user input is detected by a user interface associated with the display screen. For example, the system 600 may display the image 702-L on a display monitor 122 of the vision cart 106, which may implement a touch screen configured to receive touch input from a user (e.g., any one of the surgical team members 110). The user may provide the user input by touching a location on the touch screen corresponding to a desired feature (e.g., an edge of the hernia 704) displayed within the image 702-L.
[0073] As another example of how user input specifying a user-selected 2D endpoint 800 may be received, the system 600 may provide an image 702-L for display on each of a display screen of a stereo pair communicatively coupled to the system 600, and the system 600 may receive the user input when the user input is detected by a user interface associated with the display screens of the stereo pair. For example, the system 600 may be configured to display the same image 702 (e.g., in one example, image 702-L) on both sides of the stereo viewer 402 when requesting user input specifying a user-selected 2D endpoint 800, rather than directing image 702-L to be displayed on the left display screen of the stereo viewer 402 of the surgeon console 104 and directing image 702-R to be displayed on the right display screen of the stereo viewer 402 of the surgeon console 104 to provide the surgeon with a 3D view of the surgical area. In this manner, the system 600 may temporarily provide a 2D view of the surgical area to allow the surgeon 110-1 to move a pointer object (e.g., a cursor) to a desired point on the replicated image 702 and thereby specify the user-selected 2D endpoint 800.
[0074] The user-selected 2D endpoint 800 may be specified by the surgeon using the stereoscopic viewer in other suitable ways. For example, as one example, one side of the stereoscopic viewer may temporarily avoid displaying an image 702 (e.g., image 702-R) while the user-selected 2D endpoint 800 is specified using a pointer object presented on another image 702 (e.g., image 702-L). As another example, the two images 702-L and 702-R may be presented in a normal stereoscopic view (i.e., in 3D mode), and the pointer object may appear to be suspended above the tissue at a particular distance in front of the stereoscopic endoscope until the user-selected 2D endpoint 800 is selected, at which the cursor may appear to "drop" onto the tissue at an appropriate depth (as will be described below) that defines the corresponding 3D endpoint. Additionally, as yet another example, the images 702-L and 702-R may be presented in a normal stereoscopic view, and the user may select a point using a 3D cursor. The projection of this selected point onto one of the images (eg, image 702 -L) may then be used as the user-selected 2D endpoint 800 .
[0075] In some examples, the designation of the user-selected 2D endpoint (e.g., user-selected 2D endpoint 800) can be performed as a discrete event, such as a touch gesture, a button press, a mouse click, a button release (e.g., to end a drag action from one user-selected 2D endpoint to another user-selected 2D endpoint), or the like. For example, system 600 can receive user input by providing image 702-L for display on a display screen (e.g., stereoscopic viewer 402, display monitor 122, etc.) communicatively coupled to system 600, providing a user interface associated with the display screen and including a pointer object configured to be moved by a user to any point on the display screen, and receiving user input of a discrete user selection designating the user-selected 2D endpoint as a particular point on the display screen to which the user has moved the pointer object.
[0076] In other examples, user selection of a user-selected 2D endpoint can be performed dynamically as a pointer object (e.g., a cursor) is moved within a display screen without any additional user selection actions (i.e., without actions such as button presses, mouse clicks, etc.). In other words, in some embodiments, a number of user-selected 2D endpoints for a number of different potential measurements can be automatically selected as the user moves the pointer object across the display. For example, in this manner, a pointer object can be dynamically updated in both sides of the stereo viewer 402 so as to appear to move in 3D along the surface of the tissue.
[0077] Once a user-selected 2D endpoint 800 has been specified by the user in one of images 702 (e.g., image 702-L) in any of these or other suitable manners, system 600 can identify a matching 2D endpoint in another image (e.g., image 702-R) that corresponds to the same feature corresponding to the user-selected 2D endpoint 800 (i.e., in this example, the same portion of the edge of hernia 704).
[0078] To illustrate, Fig. 9 An exemplary matching 2D endpoint 900 is shown, which is identified as corresponding to the same feature within the surgical field as the user-selected 2D endpoint 800. As described above, the system 600 can identify the matching 2D endpoint 900 in response to a discrete event such as a touch of a screen or a button press, or automatically based on the user-selected 2D endpoint 800 (e.g., as the pointer object moves around the image 702-L) to provide the appearance of a 3D pointer object moving along the tissue as described above.
[0079] In some examples, the camera of the stereo endoscope 500 may be calibrated to be perfectly corrected (i.e., pointing in parallel directions). In such examples, the matching 2D endpoint 900 may be located within a specific range of pixel columns on a specific pixel row within the image 702-R (i.e., a pixel row corresponding to the pixel row of the image 702-L where the user-selected 2D endpoint 800 is located). In contrast, in other examples, the camera of the stereo endoscope 500 may not be corrected, but may be configured to point slightly inward toward the target area, for example, at a nominal working depth (e.g., approximately 50 mm from the distal tip of the stereo endoscope 500). In these examples, the matching 2D endpoint 900 may be located somewhere within the rectangular area 902 that spans a specific range of pixel columns and pixel rows within the image 702-R.
[0080] Pixels within a range of columns included in a particular row or a particular range of rows may define a plurality of candidate pixel blocks, one of which may be identified by the system 600 as a matching 2D endpoint 900 based on a strong match with the user-selected 2D endpoint 800. More specifically, if the user-selected endpoint 800 is implemented as a pixel block including a first number of rows and a first number of columns, the system 600 may identify the matching 2D endpoint 900 in the following manner.
[0081] First, the system 600 can identify a rectangular region 902 within the image 702-R (i.e., the region in which a matching 2D endpoint for the user-selected 2D endpoint 800 is expected). The rectangular region 902 can be identified in any suitable manner. For example, the system 600 can identify the rectangular region 902 based on the calibration of the stereo endoscope (e.g., intrinsic parameters of the camera, the target or actual alignment of the camera, etc.), the proximity of the stereo endoscope 500 to the tissue when the image 702 was captured (e.g., determined based on kinematic data or other data as described in more detail below), and / or any other suitable information as may serve a particular embodiment.
[0082] like Fig. 9 As shown in , rectangular area 902 can be identified as a relatively wide and short area including multiple candidate pixel blocks, each candidate pixel block including a first number of rows and a first number of columns. In some examples, rectangular area 902 can include a second number of rows that is larger than the first number of rows (such that rectangular area 902 is higher than the 2D endpoint 800 selected by the user) and a second number of columns that is larger than the first number of columns (such that rectangular area 902 is wider than the 2D endpoint 800 selected by the user). The second number of columns included in rectangular area 902 can be at least twice as large as the second number of rows, so that rectangular area 902 has a short and wide shape, as shown in FIG. Fig. 9 as shown in .
[0083] Once the rectangular area 902 is identified, the system 600 may analyze the candidate pixel blocks among the plurality of candidate pixel blocks included within the rectangular area 902 to assign to each analyzed candidate pixel block a corresponding similarity metric with respect to the user-selected 2D endpoint 800. Based on this analysis, the system 600 may identify as the matching 2D endpoint 900 a particular candidate pixel block that is assigned the highest similarity metric from among all similarity metrics assigned to the plurality of candidate pixel blocks.
[0084] To illustrate, Fig.10 An enlarged pixel-level view of a user-selected 2D endpoint 800 within image 702-L and a matching 2D endpoint 900 within a rectangular region 902 in image 702-R is shown. Specifically, the user-selected 2D endpoint 800 is shown as a 3×3 pixel block labeled 1 to 9, while the rectangular region 902 is shown as a 17×6 pixel block labeled 1 to 102. In the following description, pixels included within image 702-L (i.e., pixels in the user-selected 2D endpoint 800) will be referred to as pixels 1L, 2L, 3L, and so on, while pixels included within image 702-R (i.e., pixels in the rectangular region 902) will be referred to as pixels 1R, 2R, 3R, and so on. Fig.10 Each pixel in is shown as white, black, or shaded to represent different color values, grayscale values, etc. that the pixel may be associated with. However, it will be understood that Fig.10 The shading of the pixels in is intended to help describe Fig.10 , and therefore, as shown in images 702-L and 702-R, may not be aligned with the anatomical structure.
[0085] As described above, system 600 may define rectangular region 902 within image 702-R to ensure that matched 2D endpoint 900 is contained somewhere within rectangular region 902. However, analysis may be required to determine the exact location within rectangular region 902 where matched 2D endpoint 900 is located. For purposes of illustration, Fig.10 A matching 2D endpoint 900 (i.e., including pixels 30R-32R, 47R-49R, and 64R-66R) is shown as a perfect match to the user-selected 2D endpoint 800. However, it will be appreciated that using a real image including intricate details and a color palette, such a perfect match may be unlikely to be found. Instead, the system 600 may be configured to analyze each candidate pixel block included within the rectangular area 902 to identify the pixel block that is the closest match to the user-selected 2D endpoint 800. To this end, the system 600 may analyze multiple candidate pixel blocks 1002 (e.g., candidate pixel blocks 1002-1-1, 1002-1-5, 1002-2-13, and other candidate pixel blocks that are not explicitly labeled).
[0086] Each candidate pixel block 1002 in the rectangular region 902 may be referenced herein based on the row and column number of the upper left pixel as marked along the side of the rectangular region 902. For example, the candidate pixel block 1002 in the upper left corner of the rectangular region 902 is referred to as candidate pixel block 1002-1-1 due to the position of its upper left pixel (i.e., pixel 1R) at row 1 and column 1; the candidate pixel block a few columns to the right of the candidate pixel block 1002-1-1 is referred to as candidate pixel block 1002-1-5 due to the position of its upper left pixel (i.e., pixel 5R) at row 1 and column 5, and so on.
[0087] The system 600 can analyze each candidate pixel block 1002 and assign a specific similarity metric to each candidate pixel block 1002 based on how similar each candidate pixel block 1002 is to the user-selected 2D endpoint 800. For example, by analyzing the candidate pixel blocks in a sequential, parallel, or other suitable manner, the system 600 can assign a similarity metric to each candidate pixel block included in the rectangular area 902. Such similarity metrics can be calculated and assigned according to any suitable pixel comparison technique. For example, in one technique, the system 600 can base the similarity metric on the sum of the absolute differences (e.g., color value differences) between each pixel in the individual candidate pixel block and the corresponding pixel in the user-selected 2D endpoint 800. For example, the system 600 can compare the candidate pixel block 1002-1-5 with the user-selected 2D endpoint 800 by comparing pixels 1L and 5R to determine that they are the same (i.e., because they are both white), comparing pixels 2L and 6R to determine that they are very different (i.e., because one is white and the other is black), comparing pixels 4L and 22R to determine that they are somewhat different (i.e., because one is shaded and the other is black), and so on. Based on the sum of the differences between all such corresponding pixel pairs, the system 600 can assign a relatively low similarity metric to the candidate pixel block 1002-1-5 because the user-selected 2D endpoint 800 and the candidate pixel block 1002-1-5 are not particularly similar to each other (i.e., there is not a good match). However, when the system 600 performs a similar analysis on the candidate pixel block 1002-2-13 (i.e., comparing pixels 1L and 30R, pixels 2L and 31R, etc.), the system 600 may assign a relatively high similarity metric to the candidate pixel block 1002-2-13 because there is no difference between the candidate pixel block 1002-2-13 and the user-selected 2D endpoint 800, thereby constituting a good match. If such a similarity metric is as high as or higher than any other assigned similarity metric, the system 600 may identify the pixel block 1002-2-13 as the matching 2D endpoint 900 corresponding to the user-selected 2D endpoint 800. For this reason, the candidate pixel block 1002-2-13 is also marked as the matching 2D endpoint 900.
[0088] Other techniques other than the above-mentioned sum of absolute differences may also be used in certain embodiments. For example, in some embodiments, the system 600 may make the calculated similarity metric based on the sum of squared differences between each pixel 1000-R in each individual candidate pixel block 1002 and each corresponding pixel 1000-L of the pixel block of the 2D endpoint 800 selected by the user. In the same or other embodiments, the system 600 may make the calculated similarity metric based on the standardized cross-correlation between each pixel 1000-R in each individual candidate pixel block 1002 and each corresponding pixel 1000-L of the pixel block of the 2D endpoint 800 selected by the user. In other examples, the system 600 may calculate the similarity metric based on data conversion techniques (e.g., census transformation, rank transformation, etc.), screening descriptors, gradients (i.e., the sum of absolute differences between horizontal gradients and vertical gradients), or any other techniques that may serve a particular embodiment. Additionally, in some examples, a combination of two or more techniques described herein may be used. For example, in an exemplary embodiment, the system 600 may make the calculated similarity metric based on the sum of absolute differences and census transformation.
[0089] Once a particular candidate pixel block 1002 (such as candidate pixel block 1002-2-13) has been identified as the best match to the user-selected 2D endpoint 800, it is desirable to determine an objective confidence level associated with the likelihood that this candidate pixel block is an accurate and unique match. This confidence level can then be used to help qualify (or determine whether it is worth qualifying) 3D endpoints corresponding to the same feature portion. For example, if the most likely match is determined to have a relatively low likelihood of being an accurate and unique match, it is desirable that the system 600 avoids determining measurements based on the low confidence endpoint and indicates to the user that measurements cannot be performed based on the specified user-selected 2D endpoint and the user must specify another user-selected 2D endpoint or that measurements based on the low confidence endpoint are subject to relatively high errors.
[0090] The match confidence level associated with a particular match (i.e., a particular candidate pixel block identified as a matching 2D endpoint) can be determined in any suitable manner. For example, after identification of the matching 2D endpoint 900, the system can identify a reverse matching 2D endpoint corresponding to the same feature portion as represented in image 702-L based on the matching 2D endpoint 900. In other words, the system 600 can perform the same process as described above from image 702-R to image 702-L (e.g., identifying a rectangular area, assigning a similarity metric to each candidate pixel block within the rectangular area, etc.) to identify the reverse matching 2D endpoint within image 702-L (i.e., the best match for candidate pixel block 1002-2-13). Therefore, the system 600 can determine a match confidence value for the matching 2D endpoint based on the user-selected 2D endpoint and the reverse matching 2D endpoint. For example, if the reverse-matched 2D endpoint is aligned with or very close to the user-selected 2D endpoint 800, the match confidence value may be relatively high, whereas if the reverse-matched 2D endpoint is not specifically near the user-selected 2D endpoint 800, the match confidence value may be relatively low.
[0091] In addition to determining bidirectional consistency between the user-selected 2D endpoint and the matched 2D endpoint in this manner, other techniques may be used to determine a confidence level. For example, local neighborhood consistency around the user-selected 2D endpoint and the matched 2D endpoint may also indicate the likelihood of correctly identifying the matched 2D endpoint. Specifically, several points around the user-selected 2D endpoint and the matched 2D endpoint may be compared to ensure that the disparity estimate is consistent between the points.
[0092] In some examples, a match confidence value may be calculated based on the degree to which a first candidate pixel block identified as the closest match ("best match") to a user-selected 2D endpoint is a better match than a second candidate pixel block identified as the second closest match ("second best match") to the user-selected 2D endpoint. For example, if each similarity metric is normalized so that 0 represents no similarity and 1 represents perfect similarity, the confidence match value may be calculated as the difference between 1 and the quotient of the second highest similarity metric and the highest similarity metric. In this manner, if one candidate pixel block stands out as an obvious best match (i.e., because it has a higher similarity metric than the second best match), the quotient will be close to 0 and the match confidence value will be high (i.e., close to 1). Conversely, if the highest similarity metric is the same or only slightly better than the second highest similarity metric (thereby indicating that no particular match stands out as an obvious best match), the quotient will be 1 or close to 1, and the match confidence value will be low (i.e., close to 0). This match confidence value, or an analogous value calculated in another suitable manner, may be used to define a 3D endpoint (or to determine whether to define a 3D endpoint at all) in any manner described herein.
[0093] As described above, in some examples, it is undesirable to define a 3D endpoint based on a user-selected 2D endpoint and a corresponding matching 2D endpoint that does not match with a suitable high confidence level. For example, such a 3D endpoint may be inaccurate and the risk of causing an inaccurate measurement may be such that, in addition to the risk of making an inaccurate measurement, the system 600 preferably avoids making the measurement altogether. In such examples, the system 600 may indicate that a match with a suitable high confidence level could not be identified for the user, and allow the user to try again to specify the user-selected 2D endpoint for which a corresponding matching 2D endpoint was successfully identified. Additionally or alternatively, in such cases, the system 600 may be configured to automatically specify a 2D endpoint near the user-selected 2D endpoint (e.g., a 2D endpoint corresponding to a feature that can be more easily identified), and automatically retry to identify the matching 2D endpoint.
[0094] Specifically, for example, the system 600 may identify a first matching 2D endpoint based on the user-selected 2D endpoint, and may determine that the match confidence value for the first matching 2D endpoint is below a predetermined confidence threshold. Therefore, the system 600 may (e.g., based on the user-selected 2D endpoint and in response to the determination that the match confidence value is below a predetermined confidence threshold) identify a replacement 2D endpoint corresponding to the feature portion. The replacement 2D endpoint may be different from the user-selected 2D endpoint, but may be related to the user-selected 2D endpoint (e.g., immediately adjacent to the user-selected 2D endpoint). The system 600 may then identify a second matching 2D endpoint corresponding to the feature portion as represented in the second image. This identification may be based on the user-selected 2D endpoint, in the sense that it is based on the replacement 2D endpoint, which itself is identified based on the user-selected 2D endpoint. The system 600 may then define a 3D endpoint corresponding to the feature portion based on the user-selected 2D endpoint by defining the 3D endpoint based on the replacement 2D endpoint, wherein the replacement 2D endpoint is identified based on the user-selected 2D endpoint.
[0095] As discussed above, the systems and methods described herein can provide various advantages of convenience and accuracy by using a stereo endoscope rather than, for example, only kinematic data for surgical instruments to measure distances. However, in some embodiments, kinematic data can be combined with endoscopic image data to further improve the accuracy of anatomical measurement systems such as system 600. For example, system 600 can identify matching 2D endpoints by tracking the kinematics of one or more surgical instruments (e.g., one or more of surgical instruments 114, such as stereo endoscope 500) located at the surgical area. System 600 can use the tracked kinematics to identify rectangular area 902 and / or analyze candidate pixel blocks 1002 to assign similarity metrics to candidate pixel blocks 1002.
[0096] To illustrate, Fig.11 A side view 1100 of a stereoscopic endoscope 500 and surgical instruments 1102-1 through 1102-3 are shown within a surgical field including a hernia 704. The stereoscopic endoscope 500 may include any of the components described above (many of which are not explicitly shown in view 1100), and the surgical instrument 1102 may implement any of the surgical instruments described herein (e.g., surgical instrument 114).
[0097] Due to inherent errors that can be introduced and accumulated over the length of the kinematic chain, it is not possible to determine a precise location for the distal tip of the stereoscopic endoscope 500 based solely on kinematic data. However, the tracked kinematic data of the robotic arm coupled to the stereoscopic endoscope 500 can provide the system 600 with sufficient data to at least determine an approximate location of the distal tip of the stereoscopic endoscope 500. For example, based solely on the tracked kinematic data, it can be determined that the distal tip is located somewhere within the area 1104. Additionally, while the kinematic data alone may not indicate the precise location at which each surgical instrument 1102 is disposed, the tracked kinematic data of the respective robotic arm coupled to each surgical instrument 1102 can indicate a boundary 1106 beyond which the surgical instrument 1102 has not moved. However, because the surgical instrument 1102 has not been detected to have exceeded the boundary 1106, it can be assumed that the boundary 1106 is at least approximately the surface of the tissue within the surgical field, especially as the surgical session progresses, the boundary 1106 becomes increasingly clear, and more kinematic data is tracked and acquired.
[0098] By defining and tracking kinematic data representing region 1104, boundary 1106 and / or other such indications, system 100 can more accurately and successfully use the above-mentioned technology to match the 2D endpoint selected by the user to the matching 2D endpoint. For example, based solely on the relationship between region 1104 and boundary 1106 (i.e., based only on the tracked kinematic data), system 600 can approximately calculate how far the stereo endoscope 500 is from the surface of hernia 704 at any particular time. This information can be used to help determine the size of the rectangular area of the 2D endpoint to be searched for matching, to match the 2D endpoint selected by a specific user. For example, based on the distance between region 1104 and boundary 1106, system 600 can be able to reduce the size of the rectangular area without excluding any area where the 2D endpoint of the matching may be found. This can help avoid false positive matches and can improve efficiency because system 600 can have fewer candidate pixel blocks to analyze in a smaller rectangular area. Additionally or alternatively, the kinematic data represented by region 1104 and boundary 1106 can be used to otherwise eliminate false positive matches, to increase or decrease match confidence values assigned to identified matches, or to otherwise facilitate system 600 in effectively identifying the most accurate matching 2D endpoint possible for a given user-selected 2D endpoint.
[0099] In some examples, information determined based on images detected by stereo endoscope 500 can help verify or correct an estimated instrument or anatomical structure position that has been determined based on tracked kinematic data. For example, while region 1104 and / or boundary 1106 can be determined based solely on kinematic data, region 1104 can be smaller and more accurate, and boundary 1106 can be made to conform more closely to the actual tissue surface based on data derived from images captured by stereo endoscope 500. Thus, in some examples, information determined by system 600 based on stereo endoscope 500 can be fed back into the kinematic tracking system to correct and / or improve the accuracy of the raw kinematic data being tracked.
[0100] After user specification of a user-selected 2D endpoint and successful identification of a matching 2D endpoint that corresponds to the user-selected 2D endpoint in any of the above-described ways, system 600 can define a 3D endpoint based on the user-selected 2D endpoint and the matching 2D endpoint. The definition of the 3D endpoint can be performed in any suitable manner, such as by using a triangulation technique based on epipolar geometry.
[0101] Fig.12 An exemplary definition of 3D endpoints 1202 (i.e., represented by an "X" shape) corresponding to feature 1204 is shown. For example, the definition of 3D endpoints 1202 may be based on 2D endpoints 1206 (e.g., user-selected 2D endpoint 1206-L and corresponding matching 2D endpoint 1206-R). Fig.12 Various coordinates of the form “(x, y, z)” are shown to illustrate the 3D position relative to a coordinate system (e.g., a global coordinate system associated with the robotic surgical system 100, a local coordinate system associated with a stereo endoscope, or the like). For example, in (x L ,y L ,z L ) can represent the known location from which the left camera of the stereo endoscope captures the left image 1210-L, and at (x R ,y R ,z R ) can represent the known location from which the right camera of the stereo endoscope captures the right image 1210-R. P ,y P ,z P) can represent the 3D endpoint defined by system 600 based on the known positions of vantage points 1208-L and 1208-R and based on the corresponding representations of feature 1204 represented in both images 1210-L and 1210-R. Specifically, as shown, the user-selected 2D endpoint 1206-L can mark feature 1204 at the 2D coordinate (n L ,m L ), and the matched 2D endpoint 1206-R can mark the feature 1204 at the 2D coordinate (n R ,m R ).
[0102] Also identified in images 1210-L and 1210-R are respective poles e in each image 1210 that may represent the location of the relative vantage point 1208. L and e R . The relative positions of the respective epipolar points and other suitable intrinsic or extrinsic parameters defining the left and right cameras (e.g., the distance between vantage points 1208, the focus of each camera, etc.) can be determined during a calibration process or in any other suitable manner. However, based on the known positions of vantage points 1208 and their relationship to each other and the positions of 2D endpoints 1206-L and 1206-R on images 1210-L and 1210-R, system 600 can triangulate the position of 3D endpoint 1202 using conventional epipolar geometry.
[0103] Using the above operations and / or any other suitable operations, the user-selected 2D endpoint can be matched to a stereo matched 2D endpoint and triangulated to define a corresponding 3D endpoint, such as 3D endpoint 1202. Once at least two such 3D endpoints are defined, system 600 can calculate the spatial distance from one 3D endpoint to another 3D endpoint within the coordinate system of the defined 3D endpoints.
[0104] Although operations for selecting and deriving a first 3D endpoint have been described in detail above, it will be appreciated that additional 3D endpoints must also be defined in order to determine the distance from the first 3D endpoint (e.g., 3D endpoint 1202) to the additional 3D endpoints. To this end, the additional 3D endpoints may be defined in any suitable manner. For example, the system 600 may define the additional 3D endpoints in the same manner as the first 3D endpoint. That is, the system 600 may receive user input specifying an additional user-selected 2D endpoint corresponding to an additional feature portion as represented in image 702-L (e.g., from the same user who provided the user input specifying the first user-selected 2D endpoint), identify additional matching 2D endpoints corresponding to the additional feature portion as represented in image 702-R (e.g., based on the additional user-selected 2D endpoint), and define additional 3D endpoints corresponding to the additional feature portion (e.g., based on the additional user-selected 2D endpoint and the additional matching 2D endpoint). Each of these operations may be performed in any manner described herein.
[0105] In other examples, additional 3D endpoints may be selected or defined in different ways. Specifically, for example, the additional 3D endpoints may be associated with the location of a known location, such as the location of a surgical instrument, the location of an endoscope, the origin of a coordinate system, or the like. Thus, in these examples, the user may not necessarily specify additional user-selected 2D endpoints based on features presented in the 2D image. Instead, the user may indicate that the additional 3D endpoints correspond to a particular surgical instrument, endoscope, origin, or other known location in any suitable manner.
[0106] To illustrate, Fig.13 Exemplary 3D endpoints 1302 (eg, 3D endpoints 1302-1 through 1302-6) are shown between which one or more distances may be determined. Fig.13 As shown in , the first 3D endpoint 1302-1 defined based on the user-selected 2D endpoint 800 within the image 702-L can be positioned at the first edge of the hernia 704. Once the first 3D endpoint 1302-1 is defined based on the user-selected 2D endpoint 800, the distance from the 3D endpoint 1302-1 to any additional 3D endpoints represented by 3D endpoints 1302-2 to 1302-6 can be determined. For example, by determining the distance between the 3D endpoint 1302-1 and one of the 3D endpoints 1302-2 and 1302-3, the distance from one edge of the hernia 704 to another edge can be determined.
[0107] In other examples, the additional features corresponding to the additional 3D endpoints 1302 may be non-anatomical features (i.e., features included within the surgical field that are not part of the patient's anatomy). For example, 3D endpoint 1302-4 may be associated with the tip of surgical instrument 1104-1, which is not currently represented within image 702-L, but may still be within the surgical field. As another example, 3D endpoint 1302-5 may be associated with the tip of surgical instrument 1104-2, which is represented within image 702-L. As yet another example, 3D endpoint 1302-6 may be associated with the tip of stereo endoscope 500 (depicted as a dashed line surrounding image 702-L). In examples where the non-anatomical features corresponding to the 3D endpoints are not represented within image 702-L (e.g., for features corresponding to 3D endpoints 1302-4 and 1302-6), kinematic data may be used to determine the coordinates of 3D endpoint 1302. Kinematic data and / or endoscopic image data may be used to determine coordinates of non-anatomical features represented within image 702-L, such as the coordinates of 3D endpoint 1302-5.
[0108] In the example of determining the distance from an anatomical feature to a non-anatomical feature such as a surgical instrument 1104 suspended above the anatomical feature, it may be useful to measure the direct point-to-point distance between two 3D endpoints. However, in various other examples (such as examples involving 3D endpoints associated with two anatomical features), it may be desirable to measure the contour distance from one 3D endpoint to another 3D endpoint along the contour of the tissue (i.e., not directly through the tissue). For example, in the above-mentioned example of measuring the distance across the hernia 704 so that the mesh patch can be cut to the appropriate size, it may be desirable to determine the distance across the hernia 704 along the surface of the hernia 704 rather than determining the distance directly through the hernia 704.
[0109] To illustrate, Fig.14 1 shows an exemplary aspect of how the contour distance from one 3D endpoint to an additional 3D endpoint may be estimated. Specifically, as shown in side view 1400, 3D endpoints 1402-1 and 1402-2 (collectively referred to as "3D endpoints 1402") may be located at opposite edges of hernia 704. For example, 3D endpoint 1402-1 may implement 3D endpoint 1302-1, while 3D endpoint 1402-2 may implement one of 3D endpoints 1302-2 or 1302-3. Fig.14, a direct point-to-point distance 1404 is drawn between the 3D endpoints 1402, directly through the tissue of the hernia 704. While the point-to-point distance 1404 may be useful in certain contexts, it may not be an ideal distance for sizing a mesh patch to cover the hernia 704, as the mesh patch will need to cover the surface of the hernia 704. Therefore, it may be desirable to determine a distance on the surface of the hernia 704, which may be referred to herein as a "contour distance" on the tissue.
[0110] To determine this contour distance, the system 600 can automatically identify one or more 3D midpoints 1406 (e.g., 3D midpoints 1406-1 to 1406-3 and / or additional 3D midpoints not explicitly shown) on a 3D contour that connects the 3D endpoints to the additional 3D endpoints and runs along a physical surface (i.e., the outer surface of the hernia 704) on which both the 3D endpoints and the additional 3D endpoints are disposed. The system 600 can then determine, based on the 3D midpoints 1406, intermediate distances 1408 (e.g., intermediate distances 1408-1 to 1408-4) for each segment of a linear segmented route from the 3D endpoint 1402-1 to the 3D endpoint 1402-2 that passes through each adjacent 3D midpoint 1406 so as to substantially follow the 3D contour between the 3D endpoints 1402. Based on intermediate distances 1408, system 600 may calculate the silhouette distance from 3D endpoint 1402-1 to 3D endpoint 1402-2 as the sum of intermediate distances 1408. The sum of intermediate distances 1408 may provide an estimate for an accurate silhouette distance that becomes more accurate as more 3D midpoints 1406 and more intermediate distances 1408 are defined.
[0111] In some examples, the user can manually define the 3D midpoint 1406 (e.g., by selecting the 2D endpoints point by point), or can define a 2D line along which the 3D midpoint 1406 is defined. For example, a touch screen can be used to draw a line along the anatomical structure presented on the touch screen (e.g., from one side of the hernia 704 to the other side thereof) to specify the 2D endpoints selected by the user and the midpoint between them. When the midpoint is specified in this manner or in another suitable manner (e.g., automatically by the system 600), the system 600 can estimate the contour distance between the endpoints by estimating the distance along the tissue contour connecting the endpoints through the midpoint. In other examples, other types of user interfaces, such as a pointer-based interface, can be used to achieve similar results.
[0112] Fig.15 An exemplary method 1500 for measuring distance using a stereo endoscope is shown. Fig.15 Exemplary operations according to one embodiment are shown, and other embodiments may omit, add to, reorder, and / or modify Fig.15 Any of the actions shown in . Fig.15 One or more of the operations shown in may be performed by a measurement system, such as system 600, any components included herein, and / or any implementations thereof.
[0113] In operation 1502, the measurement system may access a first image. For example, the accessed first image may be captured from a first vantage point by a first camera included within a stereo endoscope located at a surgical area associated with a patient. Operation 1502 may be performed in any manner described herein.
[0114] In operation 1504, the measurement system may access a second image. For example, the accessed second image may be captured from a second vantage point by a second camera included in the stereo endoscope. The second vantage point may be stereoscopic with respect to the first vantage point, wherein the first image was captured from the first vantage point by the first camera. Operation 1504 may be performed in any manner described herein.
[0115] In operation 1506, the measurement system may receive user input specifying a user-selected 2D endpoint corresponding to a feature within the surgical area as represented in the first image. For example, the measurement system may receive the user input from a user of the measurement system, such as a member of a surgical team associated with a surgical procedure performed on a patient. Operation 1506 may be performed in any manner described herein.
[0116] In operation 1508, the measurement system may identify a matching 2D endpoint corresponding to the feature as represented in the second image. For example, the measurement system may identify the matching 2D endpoint based on determining the user-selected 2D endpoint that has been specified by the user in operation 1506. Operation 1508 may be performed in any manner described herein.
[0117] In operation 1510, the measurement system may define a 3D endpoint corresponding to a feature within the surgical field. For example, the measurement system may define the 3D endpoint based on determining the user-selected 2D endpoint specified in operation 1506 and the matching 2D endpoint identified in operation 1508. Operation 1510 may be performed in any manner described herein.
[0118] In operation 1512, the measurement system may determine distances from the 3D endpoint to additional 3D endpoints corresponding to additional features within the surgical field. Operation 1512 may be performed in any manner described herein.
[0119] In certain embodiments, one or more of the systems, components, and / or processes described herein may be implemented and / or performed by one or more appropriately configured computing devices. To this end, one or more of the systems and / or components described above may include or be implemented by any computer hardware and / or computer-implemented instructions (e.g., software) contained on at least one non-transitory computer-readable medium, at least one non-transitory computer-readable medium being configured to perform one or more of the processes described herein. In particular, the system components may be implemented on one physical computing device, or may be implemented on more than one physical computing device. Thus, the system components may include any number of computing devices, and may employ any of a variety of computer operating systems.
[0120] In some embodiments, one or more of the processes described herein may be implemented at least in part as instructions contained in a non-transitory computer-readable medium and executable by one or more computing devices. Typically, a processor (e.g., a microprocessor) receives instructions from a non-transitory computer-readable medium (e.g., a memory, etc.) and executes those instructions to perform one or more processes, including one or more of the processes described herein. Such instructions may be stored and / or transmitted using any of a variety of known computer-readable media.
[0121] Computer-readable media (also referred to as processor-readable media) include any non-temporary media that participates in providing data (e.g., instructions) that can be read by a computer (e.g., by a processor of a computer). Such media can take many forms, including, but not limited to, non-volatile media and / or volatile media. For example, non-volatile media can include optical or magnetic disks and other permanent storage. For example, volatile media can include dynamic random access memory ("DRAM") that typically constitutes main memory. For example, common forms of computer-readable media include disks, hard disks, tapes, any other magnetic media, compact disk read-only memory ("CD-ROM"), digital versatile disks ("DVD"), any other optical media, random access memory ("RAM"), programmable read-only memory ("PROM"), electrically erasable programmable read-only memory ("EPROM"), flash electrically erasable programmable read-only memory (FLASH-EEPROM), any other memory chip or cassette tape, or any other tangible medium from which a computer can read.
[0122] Fig.16 An exemplary computing device 1600 is shown that may be specifically configured to perform one or more of the processes described herein. Fig.16As shown in , computing device 1600 may include communication interface 1602, processor 1604, storage device 1606, and input / output ("I / O") module 1608 communicatively connected via communication infrastructure 1610. Although exemplary computing device 1600 is Fig.16 However, Fig.16 The components shown in the drawings are not intended to be limiting. Additional or alternative components may be used in other embodiments. Fig.16 Components of computing device 1600 are shown in .
[0123] The communication interface 1602 may be configured to communicate with one or more computing devices. Examples of the communication interface 1602 include, but are not limited to, a wired network interface (such as a network interface card), a wireless network interface (such as a wireless network interface card), a modem, an audio / video connection, and any other suitable interface.
[0124] The processor 1604 generally represents any type or form of processing unit capable of processing data or interpreting, executing and / or directing the execution of one or more of the instructions, processes and / or operations described herein. The processor 1604 may direct the execution of operations according to one or more application programs 1612 or other computer-executable instructions (such as may be stored in the storage device 1606 or other computer-readable medium).
[0125] The storage device 1606 may include one or more data storage media, devices, or configurations, and may employ any type, form, and combination of data storage media and / or devices. For example, the storage device 1606 may include, but is not limited to, a hard drive, a network drive, a flash drive, a disk, an optical disk, RAM, dynamic RAM, other non-volatile and / or volatile data storage units, or combinations or sub-combinations thereof. Electronic data, including data described herein, may be temporarily and / or permanently stored in the storage device 1606. For example, data representing one or more executable applications 1612 configured to direct the processor 1604 to perform any of the operations described herein may be stored within the storage device 1606. In some examples, the data may be arranged in one or more databases located within the storage device 1606.
[0126] I / O module 1608 may include one or more I / O modules configured to receive user input and provide user output. One or more I / O modules may be used to receive input for a single virtual reality experience. I / O module 1608 may include any hardware, firmware, software, or combination thereof that supports input capabilities and output capabilities. For example, I / O module 1608 may include hardware and / or software for capturing user input, including but not limited to a keyboard or keypad, a touch screen component (e.g., a touch screen display), a receiver (e.g., an RF or infrared receiver), a motion sensor, and / or one or more input buttons.
[0127] The I / O module 1608 may include one or more devices for presenting output to a user, including, but not limited to, a graphics engine, a display (e.g., a display screen), one or more output drivers (e.g., a display driver), one or more audio speakers, and one or more audio drivers. In some embodiments, the I / O module 1608 is configured to provide graphical data to the display for presentation to the user. The graphical data may represent one or more graphical user interfaces and / or any other graphical content as may serve a particular implementation.
[0128] In some examples, any of the facilities described herein may be implemented by or within one or more components of the computing device 1600. For example, one or more applications 1612 residing within the storage device 1606 may be configured to direct the processor 1604 to perform one or more processes or functions of the system associated with the facilities 602 to 606. Likewise, the storage facility 608 of the system 600 may be implemented by the storage device 1606 or a combination thereof.
[0129] In the foregoing description, various exemplary embodiments have been described with reference to the accompanying drawings. However, it will be apparent that various modifications and changes may be made thereto, and additional embodiments may be implemented, without departing from the scope of the invention as set forth in the appended claims. For example, certain features of one embodiment described herein may be combined with or substituted for features of another embodiment described herein. Accordingly, the description and drawings should be regarded as illustrative rather than restrictive.
Claims
1. A measurement system, comprising: At least one physical computing device, the at least one physical computing device: accessing a first image captured from a first vantage point by a first camera included within a stereoscopic endoscope located at a surgical area associated with a patient; accessing a second image captured by a second camera included in the stereoscopic endoscope from a second vantage point, the second vantage point being stereoscopic with respect to the first vantage point; providing a user interface associated with the display screen, the user interface comprising a pointer object configured to be moved by a user; receiving a user input from the user, the user input specifying a user-selected two-dimensional (2D) endpoint corresponding to a feature within the surgical field represented in the first image, wherein the user input specifying the user-selected 2D endpoint is received as a user selection of a particular point on the display screen, wherein the user has moved the pointer object to the particular point, wherein the pointer object is dynamically updated to appear to move in three dimensions (3D) along a surface of tissue; Based on the user-selected 2D endpoint, identifying a matching 2D endpoint corresponding to the feature represented in the second image; defining a 3D endpoint corresponding to the feature within the surgical area based on the user-selected 2D endpoint and the matched 2D endpoint; as well as A distance is determined from the 3D endpoint to an additional 3D endpoint corresponding to an additional feature within the surgical field.
2. The measurement system of claim 1, wherein the at least one physical computing device further: receiving user input from the user specifying an additional user-selected 2D endpoint corresponding to the additional feature represented in the first image before determining the distance, identifying, based on the additional user-selected 2D endpoints, additional matching 2D endpoints corresponding to the additional features represented in the second image, and The additional 3D endpoint corresponding to the additional feature is defined based on the additional user-selected 2D endpoint and the additional matched 2D endpoint.
3. The measurement system according to claim 1, wherein: The user-selected 2D endpoint is a block of pixels comprising a first number of rows and a first number of columns; and The at least one physical computing device identifies the matching 2D endpoint by: identifying within the second image a rectangular region including a plurality of candidate pixel blocks, each candidate pixel block including the first number of rows and the first number of columns, the rectangular region including a second number of columns and a second number of rows greater than the first number of rows, the second number of columns being greater than the first number of columns and at least twice as large as the second number of rows; analyzing the candidate pixel blocks among the plurality of candidate pixel blocks included in the rectangular area to assign to each of the candidate pixel blocks a corresponding similarity measure relative to the user-selected 2D endpoint; as well as Based on the analysis, a specific candidate pixel block is identified as the matched 2D endpoint corresponding to the feature portion, the specific candidate pixel block being assigned a highest similarity measure among the corresponding similarity measures assigned to the analyzed candidate pixel blocks among the plurality of candidate pixel blocks.
4. The measurement system of claim 3, wherein the at least one physical computing device further identifies the matching 2D endpoints by: tracking kinematics of at least one of the stereoscopic endoscope located at the surgical area and a robotically manipulated surgical instrument located at the surgical area; At least one of the identifying of the rectangular region including the plurality of candidate pixel blocks and the analyzing of the candidate pixel blocks is performed to assign the respective similarity metrics to the candidate pixel blocks based on the tracked kinematics.
5. The measurement system of claim 3, wherein the respective similarity metric assigned to each individual candidate pixel block in the plurality of candidate pixel blocks is based on at least one of: the sum of absolute differences between each pixel in the individual candidate pixel block and the corresponding pixel in the pixel block of the user-selected 2D endpoint; the sum of squared differences between each pixel in the individual candidate pixel block and the corresponding pixel of the pixel block at the user-selected 2D endpoint; a normalized cross-correlation between each pixel in the individual candidate pixel block and the corresponding pixel of the pixel block at the user-selected 2D endpoint; Census transformation; Rank transformation; as well as The sum of the absolute differences between the horizontal and vertical gradients.
6. The measurement system according to claim 1, wherein: After identification of the matching 2D endpoint corresponding to the feature represented in the second image and before definition of the 3D endpoint corresponding to the feature, the at least one physical computing device further: Based on the matched 2D endpoints, identifying inverse matched 2D endpoints corresponding to the features represented in the first image, and determining a match confidence value for the matched 2D endpoint based on the user-selected 2D endpoint and the reverse matched 2D endpoint; and The defining of the 3D endpoint corresponding to the feature is further based on the match confidence value for the matched 2D endpoint.
7. The measurement system of claim 1 , wherein the at least one physical computing device determines the distance from the 3D endpoint to the additional 3D endpoint by: automatically identifying one or more 3D midpoints on a 3D contour connecting the 3D endpoint to the additional 3D endpoint and extending along a physical surface on which both the 3D endpoint and the additional 3D endpoint are disposed; determining, based on the one or more 3D midpoints, a mid-distance of each segment of a linear segmented route from the 3D endpoint to the additional 3D endpoint, the linear segmented route passing through each adjacent 3D midpoint so as to follow the 3D contour between the 3D endpoint and the additional 3D endpoint; as well as The distance is determined to be a contour distance from the 3D endpoint to the additional 3D endpoint, the contour distance being calculated as the sum of the intermediate distances.
8. The measurement system according to claim 1, wherein: Prior to said identifying of said matching 2D endpoints corresponding to said features represented in said second image, said at least one physical computing device further: identifying low confidence matching 2D endpoints based on the user-selected 2D endpoint, determining a match confidence value for the low confidence matched 2D endpoint, determining that the match confidence value is below a predetermined confidence threshold, and identifying, based on the user-selected 2D endpoint and in response to determining that the match confidence value is below the predetermined confidence threshold, a replacement 2D endpoint corresponding to the feature, the replacement 2D endpoint being different from the user-selected 2D endpoint; identifying the matching 2D endpoint corresponding to the feature represented in the second image based on the user-selected 2D endpoint by the replacement 2D endpoint identified based on the user-selected 2D endpoint; as well as By the replacement 2D endpoint identified based on the user-selected 2D endpoint, the 3D endpoint corresponding to the feature is defined based on the user-selected 2D endpoint.
9. The measurement system of claim 1, wherein the feature and the additional feature are both anatomical features within the patient.
10. The measurement system of claim 1, wherein the feature is an anatomical feature and the additional feature is a non-anatomical feature.
11. A measurement system, comprising: A stereoscopic endoscope configured to be positioned at a surgical area associated with a patient, the stereoscopic endoscope comprising: a first camera configured to capture a first image from a first vantage point, and a second camera configured to capture a second image from a second vantage point that is stereoscopic with respect to the first vantage point; a display screen configured to display the first image; a user interface associated with the display screen and comprising a pointer object configured to be moved by a user, the user interface allowing the user to specify, from display of the first image on the display screen, a user-selected two-dimensional (2D) endpoint corresponding to a feature represented in the first image within the surgical field, wherein the user specification of the user-selected 2D endpoint is received as a user selection of a particular point on the display screen, wherein the user has moved the pointer object to the particular point, wherein the pointer object is dynamically updated so as to appear to move in three dimensions (3D) along a surface of the tissue; and at least one physical computing device communicatively coupled to the stereoscopic endoscope, the display screen, and the user interface, and configured to: identifying, based on the user-selected 2D endpoint, a matching 2D endpoint corresponding to the feature represented in the second image, defining a 3D endpoint corresponding to the feature within the surgical field based on the user-selected 2D endpoint and the matched 2D endpoint, and A distance is determined from the 3D endpoint to an additional 3D endpoint corresponding to an additional feature within the surgical field.
12. The measurement system according to claim 11, wherein: the user interface further allowing the user to specify, from the display of the first image on the display screen, additional user-selected 2D endpoints corresponding to the additional features represented in the first image; The at least one physical computing device is further configured to: identifying, based on the additional user-selected 2D endpoints, additional matching 2D endpoints corresponding to the additional features represented in the second image, and The additional 3D endpoint corresponding to the additional feature is defined based on the additional user-selected 2D endpoint and the additional matched 2D endpoint.
13. The measurement system according to claim 11, wherein: The user-selected 2D endpoint is a pixel block comprising a first number of rows and a first number of columns; and The at least one physical computing device is configured to identify the matching 2D endpoint by: tracking kinematics of at least one of the stereoscopic endoscope located at the surgical area and a robotically manipulated surgical instrument located at the surgical area; identifying within the second image and based on the tracked kinematics, a rectangular region comprising a plurality of candidate pixel blocks, each candidate pixel block comprising the first number of rows and the first number of columns, the rectangular region comprising a second number of columns and a second number of rows greater than the first number of rows, wherein the second number of columns is greater than the first number of columns and at least twice as great as the second number of rows; analyzing the candidate pixel blocks among the plurality of candidate pixel blocks included in the rectangular area to assign to each of the candidate pixel blocks a corresponding similarity measure relative to the user-selected 2D endpoint; and Based on the analysis, a specific candidate pixel block is identified as the matched 2D endpoint corresponding to the feature portion, the specific candidate pixel block being assigned a highest similarity measure among the corresponding similarity measures assigned to the analyzed candidate pixel blocks among the plurality of candidate pixel blocks.
14. A computer readable medium comprising instructions executable by a processing unit to perform a method comprising: accessing a first image captured from a first vantage point by a first camera included within a stereoscopic endoscope located at a surgical area associated with a patient; accessing a second image captured by a second camera included in the stereoscopic endoscope from a second vantage point, the second vantage point being stereoscopic with respect to the first vantage point; providing a user interface associated with the display screen, the user interface comprising a pointer object configured to be moved by a user; receiving user input from a user specifying a user-selected two-dimensional (2D) endpoint corresponding to a feature within the surgical field represented in the first image, wherein the user input specifying the user-selected 2D endpoint is received as a user selection of a particular point on the display screen, wherein the user has moved the pointer object to the particular point, wherein the pointer object is dynamically updated to appear to move in three dimensions (3D) along a surface of tissue; identifying, based on the user-selected 2D endpoint, a matching 2D endpoint corresponding to the feature represented in the second image; Defining a three-dimensional endpoint, namely a 3D endpoint, corresponding to the feature within the surgical area based on the 2D endpoint selected by the user and the matched 2D endpoint; as well as A distance is determined from the 3D endpoint to an additional 3D endpoint corresponding to an additional feature within the surgical field.
15. The computer readable medium of claim 14, the method further comprising: receiving, prior to determining the distance, user input from the user specifying an additional user-selected 2D endpoint corresponding to the additional feature represented in the first image; identifying, based on the additional user-selected 2D endpoints, additional matching 2D endpoints corresponding to the additional features represented in the second image; as well as The additional 3D endpoint corresponding to the additional feature is defined based on the additional user-selected 2D endpoint and the additional matched 2D endpoint.
16. The computer-readable medium of claim 14, wherein: The user-selected 2D endpoint is a block of pixels comprising a first number of rows and a first number of columns; and The identification of the matching 2D endpoints comprises: tracking kinematics of at least one of the stereoscopic endoscope located at the surgical area and a robotically manipulated surgical instrument located at the surgical area; identifying within the second image and based on the tracked kinematics, a rectangular region comprising a plurality of candidate pixel blocks, each candidate pixel block comprising the first number of rows and the first number of columns, the rectangular region comprising a second number of columns and a second number of rows greater than the first number of rows, wherein the second number of columns is greater than the first number of columns and at least twice as great as the second number of rows; analyzing the candidate pixel blocks among the plurality of candidate pixel blocks included in the rectangular area to assign to each of the candidate pixel blocks a corresponding similarity measure relative to the user-selected 2D endpoint; and Based on the analysis, a specific candidate pixel block is identified as the matched 2D endpoint corresponding to the feature portion, the specific candidate pixel block being assigned a highest similarity measure among the corresponding similarity measures assigned to the analyzed candidate pixel blocks among the plurality of candidate pixel blocks.
