Target anatomical feature localization

Abstract

To provide a medical system capable of facilitating the targeting of anatomical features in association with a medical procedure, such as nephroscopy or other procedure accessing the renal anatomy.SOLUTION: A medical system includes: medical instruments including a camera configured to advance to a treating site; and a control system configured to acquire a real-time video of the treating site via the camera, cause a user interface including the real-time video to be displayed in a window of the user interface, and cause one or more boundary features 1260 to be projected at a substantial center of the window in the user interface window, the one or more boundary features having a size unrelated to positions of the medical instruments.SELECTED DRAWING: Figure 12A

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This application claims priority to U.S. Provisional Patent Application No. 63 / 001,870, filed on March 30, 2020, entitled "TARGET ANATOMICAL FEATURE LOCALIZATION", the disclosure of which is hereby incorporated by reference in its entirety.

[0002] (Field of the Invention) The present invention relates to the field of medical treatment.

Background Art

[0003] Various medical procedures involve the use of one or more scopes and / or percutaneous access devices. Improper positioning or advancement of such devices can result in certain physiological and procedural complications.

Summary of the Invention

Means for Solving the Problems

[0004] Described herein are systems, devices, and methods that facilitate the identification, tracking, and targeting of various anatomical features based on certain sensor - based and / or image - based position information that can be obtained, for example, using a scope device or other medical instrument. Targeting of target anatomical features according to aspects of the present disclosure can facilitate the targeting of anatomical features in relation to medical procedures such as nephroscopy or other procedures that access renal anatomical structures.

[0005] In some implementations, the present disclosure relates to a method of localizing a target papilla. The method includes advancing a ureteroscope through at least a portion of a patient's urinary tract to a target calyx of the patient's kidney, determining a position offset between one or more position sensors associated with the ureteroscope and a target papilla of the kidney that is at least partially exposed within the target calyx, and determining a percutaneous access target based at least in part on one or more of the current positions of the one or more position sensors and the offset.

[0006] The method may further include advancing a percutaneous access device to the target calyx by targeting the percutaneous access target. In some embodiments, the method further includes contacting the target papilla with the ureteroscope, recording the position of the ureteroscope associated with the contact, retracting the ureteroscope away from the target papilla, and stationing the ureteroscope at an offset position associated with the position offset. The position offset may exhibit, for example, at least five degrees of freedom.

[0007] In some implementations, the present disclosure relates to a method of positioning a surgical instrument. The method includes advancing a medical instrument to a treatment site of a patient, the medical instrument comprising a camera, generating a real-time video of the treatment site using the camera of the medical instrument, displaying a user interface including the real-time video in a window of the user interface, and projecting an anatomical feature targeting icon at the center of the window of the user interface.

[0008] The targeting icon can include any type of form or shape, such as a crosshair, or a combination thereof. The method can further include operating a medical instrument to position the targeting icon centered over a representation of the target anatomical feature in the real-time video. For example, the method can include projecting one or more boundary features substantially centered within the window of the user interface, where the one or more boundary features have a size independent of the position of the medical instrument. In some embodiments, the method further includes operating the medical instrument to contain the representation of the target anatomical feature within the one or more boundary features. Operating the medical instrument to contain the representation of the target anatomical feature within the one or more boundary features can involve withdrawing the medical instrument away from the target anatomical feature such that the representation of the target anatomical feature shrinks within the window of the user interface. In some embodiments, the one or more boundary features have at least a partial box form.

[0009] The method can further include receiving sensor data indicative of the three-dimensional position of a percutaneous access needle within an electromagnetic field, determining, at least in part based on the sensor data, the position of the distal end of the needle relative to the camera, and displaying a needle prediction icon indicative of the position of the distal end of the needle relative to the real-time video within a window of the user interface. Some embodiments involve the presentation of one or more icons representing a predicted needle entry point into a target anatomical feature (e.g., a nipple). For example, an indicator can represent the location of the needle by displaying the entry position of the needle, providing information regarding situation awareness and / or the needle trajectory. The needle prediction / trajectory indicator can display the orientation of the needle as a line type form / shape. For example, one or both of the proximal and distal points of the needle can be predicted in a line type representation and / or associated with a line type representation.

[0010] The method may further include determining that the position of the distal end of the needle is outside the window of the user interface, and the needle prediction icon indicates the direction of the position of the distal end of the needle relative to the window. In some embodiments, the method further includes operating the medical instrument to position the needle prediction icon centered within the window of the user interface. In some embodiments, the method further includes calibrating a sensor associated with the needle within the image space of the camera. The method may include modifying the form of the needle prediction icon in response to the proximity of the distal end of the needle to the medical instrument.

[0011] The size of the needle prediction icon may be changed / modified based on the determined needle prediction / prediction accuracy. If there are significant anatomical movements that can result in errors in needle prediction, the needle prediction icon may be presented in a relatively large size to indicate a relatively large determination error with respect to the needle prediction / trajectory. In some embodiments, the form of the needle prediction icon indicates the distance of the distal end of the needle from the medical instrument.

[0012] In some implementations, the present disclosure relates to a method of targeting anatomical features. The method includes advancing an endoscope into a target anatomical lumen of a patient, the endoscope comprising a position sensor associated with the distal end of the endoscope, recording position data associated with a plurality of positions of the endoscope within the target anatomical lumen using the position sensor, estimating the surface of the target anatomical lumen based at least in part on the position data, and determining the axis of the target anatomical lumen based at least in part on the estimated surface of the target anatomical lumen.

[0013] The method may further include targeting a target anatomical lumen using a percutaneous access needle, at least in part based on a determined axis of the target anatomical lumen. For example, targeting the target anatomical lumen may involve advancing the percutaneous access needle along a path that is substantially parallel to the determined axis of the target anatomical lumen. In some embodiments, the position sensor is an electromagnetic sensor device, and recording the position data is performed using an electromagnetic field generator disposed at least in part external to the patient.

[0014] In some embodiments, estimating the surface of the target anatomical lumen may involve interpolating the position data. Determining the axis of the target anatomical lumen may involve determining a plurality of surface normal vectors associated with the estimated surface of the target anatomical lumen. For example, the method may include averaging the plurality of surface normal vectors. In some embodiments, determining the axis of the target anatomical lumen is at least in part based on one or more of a map of the target anatomical lumen and the trajectory of the endoscope.

[0015] In some implementations, the present disclosure relates to a medical system comprising an endoscope configured to access a target anatomical lumen of a patient, the endoscope having a camera and an electromagnetic position sensor associated with its distal end, a communication interface configured to receive video data from the endoscope, an electronic display device, and a control circuit communicatively coupled to the communication interface and the electronic display device. The control circuit is configured to receive real-time video data of a treatment site inside the patient from the endoscope of the communication interface, display a user interface including the real-time video within a window of the user interface on the electronic display, and display an anatomical feature targeting icon at the center of the window of the user interface. The targeting icon may include, for example, a crosshair and / or the like.

[0016] The control circuit may be further configured to display one or more boundary features approximately at the center of the window within the window of the user interface. The size of one or more boundary features relative to the representation of the target anatomical feature in the real-time video may be based on the distance of the target anatomical feature from the endoscope camera. In some embodiments, the one or more boundary features have at least a partial box shape.

[0017] The control circuit may be further configured to receive sensor data indicating the three-dimensional position of a percutaneous access needle within an electromagnetic field, determine the position of the distal end of the needle relative to the endoscope based at least in part on the sensor data, and display a needle prediction icon within the window of the user interface. The needle prediction icon may indicate the position of the distal end of the needle relative to the real-time video.

[0018] In some implementations, the present disclosure relates to a computing device comprising an endoscope interface and a control circuit including one or more processors and one or more data storage devices. The control circuit is configured to receive position data from an endoscope disposed within a target anatomical lumen of a patient, the position data indicating a plurality of positions of a position sensor associated with the distal end of the endoscope. Similar to all other descriptions herein of positions and position sensors described herein, such positions may include position and orientation aspects / information. The control circuit is further configured to estimate the surface of the target anatomical lumen based at least in part on the position data and to determine the axis of the target anatomical lumen based at least in part on the estimated surface of the target anatomical lumen.

[0019] In some embodiments, the control circuit is configured to estimate the surface of the target anatomical lumen by at least partially interpolating the position data. In some embodiments, the control circuit is configured to determine the axis of the target anatomical lumen by at least partially determining a plurality of surface normal vectors associated with the estimated surface of the target anatomical lumen.

[0020] For the purpose of summarizing the present disclosure, certain aspects, advantages, and novel features are described. It should be understood that not all such advantages may be achieved in accordance with any particular embodiment. Thus, the disclosed embodiments may be carried out in a manner that achieves or optimizes one advantage or group of advantages as taught herein, without necessarily achieving other advantages as may be taught or suggested herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Various embodiments are depicted in the accompanying drawings for purposes of illustration and should not be construed as limiting the scope of the invention. Additionally, the various features of different disclosed embodiments can be combined to form additional embodiments that are part of this disclosure. Throughout the drawings, reference numbers can be reused to indicate corresponding between reference elements.

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5-1

Figure 5-2

Figure 6-1

Figure 6-2

Figure 7

Figure 8

Figure 9

Figure 10A

Figure 10B

Figure 11

Figure 12A

Figure 12B

Figure 13

Figure 14-1

Figure 14-2

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

[0022] The headings provided herein are for convenience only and do not necessarily affect the scope or meaning of the claimed invention. Specific preferred embodiments and examples are disclosed below, but the subject matter of the invention extends beyond the specifically disclosed embodiments to other alternative embodiments and / or uses, as well as their modifications and equivalents. Accordingly, the scope of the claims that may arise from this specification is not limited by any of the specific embodiments described below. For example, in any method or process disclosed herein, the acts or operations of the method or process may be performed in any suitable order and are not necessarily limited to any particular disclosed order. Various operations may be described as a plurality of discrete operations in a manner that may be helpful in understanding a particular embodiment. However, the order of description should not be construed as implying that these operations are order-dependent. In addition, the structures, systems, and / or devices described herein may be embodied as integrated components or as separate components. For purposes of comparing various embodiments, specific aspects and advantages of these embodiments are described. Not all such aspects or advantages are necessarily achieved by any particular embodiment. Thus, for example, various embodiments may be implemented in a manner that achieves or optimizes one advantage or group of advantages as taught herein without necessarily achieving other aspects or advantages as may also be taught or suggested herein.

[0023] Certain standard anatomical terms of location are used herein, with respect to the preferred embodiments, to refer to the anatomical structures of an animal, namely a human. Certain spatially relative terms such as "lateral", "medial", "superior", "inferior", "caudal", "cephalic", "vertical", "horizontal", "upper", "bottom", and the like are used herein to describe the spatial relationship of one device / element or anatomical structure to another device / element or anatomical structure, but it is understood that these terms are used herein to facilitate the description of the positional relationship between the elements / structures as illustrated in the drawings. It is understood that the spatially relative terms are intended to encompass different orientations of the element / structure during use or operation in addition to the orientation depicted in the drawings. For example, an element / structure described as "superior" to another element / structure may represent a position that is below or beside such other element / structure relative to the subject patient or an alternative orientation of the element / structure, and vice versa.

[0024] Overview The present disclosure relates to systems, devices, and methods for localizing and targeting a patient's target anatomical features to assist in certain medical procedures. Certain aspects of the present disclosure are detailed herein in the context of renal, urological, and / or nephrological procedures such as kidney stone removal / treatment procedures, but such context is provided for convenience and clarity, and it is to be understood that the localization and targeting concepts of the anatomical features disclosed herein are applicable to any suitable medical procedure. However, as described, an explanation of the anatomical structures of the renal / urological system as well as related medical problems and procedures is presented below to assist in the explanation of the inventive concepts disclosed herein.

[0025] Kidney stone disease, also known as urolithiasis, is a relatively common medical condition that involves the formation of pieces of solid material, referred to as "kidney stones," "urinary tract stones," "renal calculi," "nephrolithiasis," or "renal lithiasis," within the urinary tract. Urinary tract stones can form and / or be found within the kidneys, ureters, and bladder (referred to as "bladder stones"). Urinary tract stones form as a result of concentrated minerals and can cause significant abdominal pain when they reach a sufficient size to obstruct the flow of urine through the ureter or urethra. Urinary tract stones can be formed from calcium, magnesium, ammonia, uric acid, cysteine, and / or other compounds.

[0026] To remove urinary stones from the bladder and ureters, a surgeon may insert a ureteroscope through the urethra into the urinary tract. Typically, the ureteroscope includes an endoscope at its distal end configured to enable visualization of the urinary tract. The ureteroscope can also include a stone extraction mechanism for capturing or fragmenting urinary tract stones. During a ureteroscopy procedure, one physician / technician can control the position of the ureteroscope while another physician / technician can control the stone extraction mechanism.

[0027] To remove relatively large stones (i.e., "kidney stones") from the kidney, a physician may use a percutaneous nephrolithotomy ("PCNL") technique that involves inserting a nephroscope through the skin (i.e., percutaneously) to break up and / or remove the stone. In some embodiments, localizing the kidney stone can be achieved using fluoroscopy to provide a target for insertion of the nephroscope. However, fluoroscopy generally increases the cost of the nephrolithotomy procedure due to the cost of the fluoroscope itself as well as the cost of the technician operating the fluoroscope. Fluoroscopy also exposes the patient to radiation over a relatively long period of time. Even with fluoroscopy, it can be difficult and unnecessarily inaccurate to make the percutaneous incision precisely to access the kidney stone. Additionally, some nephrolithotomy techniques involve a two- or three-day hospital stay. In short, certain nephrolithotomy solutions are relatively expensive and can be problematic for patients.

[0028] According to certain surgical techniques according to aspects of the present disclosure, an endoscope (e.g., a ureteroscope) can be equipped with one or more position sensors, and the position of the sensors can be used as a target for percutaneous access, such as for PCNL. For example, a ureteroscope equipped with an electromagnetic sensor and / or a percutaneous access needle equipped with an electromagnetic sensor can be used to guide percutaneous renal access for kidney stone removal and / or the like. In such a technique, a surgeon / physician can drive the ureteroscope into a target calyx of the kidney and use an electromagnetic sensor (e.g., a beacon) associated with the distal end / tip of the ureteroscope as a percutaneous access target for the needle. Generally, the effectiveness of the percutaneous axis with respect to the target calyx can at least partially depend on, for example, where the physician positions / stations the ureteroscope with respect to the position and / or leading orientation of the target calyx and / or papilla through which percutaneous access can be made to the target calyx. For some techniques in which the distal end / tip of the ureteroscope is used as a percutaneous access target, it may be desirable for the distal tip of the ureteroscope to be as close as possible to the papilla / calyx during percutaneous access / approach.

[0029] The terms "scope" and "endoscope" are used herein according to their broad and ordinary meanings and can refer to any type of elongated medical instrument having image generation, viewing, and / or capture capabilities and configured to be introduced into any type of organ, cavity, lumen, chamber, or space of the body. For example, references to a scope or endoscope herein can refer to a ureteroscope, a cystoscope, a nephroscope, a bronchoscope, an arthroscope, a colonoscope, a laparoscope, a borescope, or the like. The scope / endoscope may, in some instances, comprise a rigid or flexible tube and can be sized to pass within an outer sheath, catheter, introducer, or other lumen-type device or can be used without using such a device.

[0030] Robot-assisted percutaneous procedures can be implemented in connection with various medical procedures such as kidney stone removal procedures, and the robotic tool can enable a physician / urologist to perform endoscopic (e.g., ureteroscope) target access as well as percutaneous access / treatment. Advantageously, aspects of the present disclosure relate to real-time target tracking / localization in a medical procedure, which can be utilized by the operating physician to direct a percutaneous access instrument (e.g., a needle or other rigid tool) and / or to induce robotic instrumentation, such as by automatically adjusting the endoscopic position and / or alignment in response to such real-time target tracking information. To facilitate such functionality, embodiments of the present disclosure can advantageously provide a mechanism for targeting, tracking, and / or three-dimensional position estimation of anatomical features to assist a physician (e.g., a urologist) in achieving a relatively efficient and accurate percutaneous access for various surgical procedures such as nephroscopy. Aspects of the present disclosure are described herein for convenience in the context of ureteroscope-guided nephroscopy, but it should be understood that aspects of the invention of the present disclosure can be implemented in any suitable or desirable type of percutaneous and / or endoscopic medical procedure, whether robotic or not.

[0031] Medical system FIG. 1 illustrates an exemplary medical system 100 for performing various medical procedures according to aspects of the present disclosure. The medical system 10 can be used, for example, in ureteroscopic procedures. As mentioned and explained above, certain ureteroscopic procedures involve the investigation of ureteral abnormalities and / or the treatment / removal of kidney stones. In some embodiments, kidney stone treatment can benefit from the assistance of certain robotic technologies / devices, such as those similar to that shown in FIG. 1 and detailed below. Robotic medical solutions can provide relatively higher accuracy, better control, and / or better visual and hand coordination for certain instruments. For example, robotic-assisted percutaneous access to the kidney by some procedures can advantageously enable a urologist to perform both targeted endoscopic access and percutaneous access. However, according to some solutions, percutaneous kidney access can be troubled by certain difficulties regarding the proper alignment / positioning of the targeted ureteroscope and / or the determination of a targeted percutaneous access path that substantially coincides with the targeted infundibulum, calyx, and / or papilla. In some implementations, the present disclosure relates to systems (e.g., system 100), devices, and methods for intelligently guiding a ureteroscope and / or a percutaneous access instrument (e.g., a needle). For example, embodiments of the present disclosure relate to systems, devices, and methods that incorporate certain automatic target identification, tracking, and / or 3D physician estimation functions, which can advantageously assist a urologist or other technician in achieving an efficient and accurate percutaneous access to the kidney. Embodiments of the present disclosure are presented in the context of ureteroscopes and / or the human renal anatomy, but it should be understood that the principles disclosed herein can be implemented in any type of endoscopic procedure.

[0032] Medical system 100 includes a robotic system 10 configured to engage with and / or control a medical instrument 32 (e.g., a ureteroscope) and perform a procedure on a patient 13. The medical system 10 also includes a control system 50 having an interface with the robotic system 10, configured to provide information regarding the procedure, and / or perform various other operations. For example, the control system 50 may include one or more displays 42 configured to present specific information to assist a physician 5 and / or other technician or individual. The medical system 10 may include a table 15 configured to hold the patient 13. The system 10 may further include an electromagnetic (EM) field generator 18, which may be held by one or more of the robotic arms 12 of the robotic system 10 or may be a stand-alone device.

[0033] In some implementations, the system 10 may be used to perform a percutaneous procedure such as percutaneous nephrolithotomy (PCNL). By way of illustration, if the patient 13 has a kidney stone 80 that is too large to be removed and / or cannot pass through the urinary tract (60, 63, 65), the physician 5 can perform a procedure to remove the kidney stone 80 through a percutaneous access point / path associated with the flank / lateral side of the patient 13. In some embodiments, the physician 5 can interact with the control system 50 and / or the robotic system 10 to cause the robotic system 10 to advance and navigate a medical instrument 32 (e.g., a scope) from the urethra 65, through the bladder 60, above the ureter 63, and into the calyx of the kidney 70 where the stone 80 is located. The control system 50 can provide information associated with the medical instrument 32, such as real-time endoscopic images captured using it, via the display 42 to assist the physician 5 in navigating / controlling the medical instrument.

[0034] The renal anatomical structure is described herein for reference with respect to certain medical procedures related to aspects of the concepts of the present invention. The kidneys 70, roughly shown in their typical anatomical positions in FIG. 1, generally comprise two bean-shaped organs located on the left and right sides respectively within the retroperitoneal cavity. In adult humans, the kidneys are generally about 11 cm in height / length. The kidneys receive blood from the paired renal arteries, and the blood exits the kidneys via the paired renal veins, neither of which is shown in FIG. 1 for the sake of visual clarity. Each kidney 70 is in fluid connection with a ureter 63, which generally comprises a tube that transports urine excreted from the kidney 70 to the bladder 60.

[0035] The kidneys 70 are typically located relatively high within the abdominal cavity and are in a retroperitoneal position at a slightly oblique angle. The intra-abdominal asymmetry typically caused by the position of the liver generally causes the right kidney to be slightly lower and smaller than the left and to be positioned slightly more centrally than the left kidney (as shown in detail in FIG. 1). At the upper part of each kidney is an adrenal gland (not shown). The upper part of the kidneys is partially protected by the 11th and 12th ribs. Each kidney with its adrenal gland is generally surrounded by two layers of fat, namely, the perirenal fat present between the renal fascia and the renal capsule, and the paranephric fat above the renal fascia.

[0036] The kidneys 70 are involved in the control of the volume of various body fluid compartments, fluid osmotic pressure, acid-base balance, various electrolyte concentrations, and the removal of toxins. The kidneys 70 provide a filtration function by secreting certain substances and reabsorbing others. Examples of substances secreted into the urine are hydrogen, ammonium, potassium, and uric acid. In addition, the kidneys also perform various other functions such as hormone synthesis and others.

[0037] The concave border of the kidney 70 has a recessed area, the renal hilum 78, where the renal artery (not shown) enters the kidney 70 and the renal vein (not shown) and the ureter 63 exit. The kidney 70 is surrounded by a renal capsule 74, which is tough fibrous tissue and is itself surrounded by perirenal fat, renal fascia, and paranephric fat. The front (anterior) surface of these tissues is the peritoneum, while the back (posterior) surface is the transversalis fascia.

[0038] The functional substrate or parenchyma of the kidney 70 is divided into two main structures, namely, the outer renal cortex 77 and the inner renal medulla 87. These structures each take the form of a plurality of roughly conical renal lobes containing renal cortex that surrounds a part of the medulla called a renal pyramid 72. Between the renal pyramids 72 are projections of cortex called renal columns 73. Nephrons (not shown in detail in FIG. 1), which are the urine-producing functional structures of the kidney, span the cortex 77 and the medulla 87. The initial filtering portion of the nephron is the renal corpuscle located in the cortex, followed by renal tubules that enter from the cortex into the depths of the medullary pyramids. The medullary rays, which are part of the renal cortex, are a collection of renal tubules that drain into a single collecting duct.

[0039] The tip / apical part or papilla 79 of each pyramid discharges urine into a respective minor renal calyx 75, which discharges into a major renal calyx 76, which discharges into the renal pelvis 71 that transitions into the ureter 63. At the renal hilum 78, the ureter 63 and the renal vein exit the kidney, and the renal artery enters. Lymphatic tissue with renal hilar fat and lymph nodes surrounds these structures. The renal hilar fat is adjacent to a cavity filled with fat called the renal sinus. The renal sinus collectively contains the renal pelvis 71 and the calyces 75, 76 and separates these structures from the renal medullary tissue. The funnel-shaped / tubular anatomical structure associated with the calyces can be referred to as a funnel. That is, the funnel generally results in the terminal portion of the calyx where the papilla is exposed within the calyx.

[0040] Referring still to the medical system 10, a medical instrument (e.g., a scope) 32 can be advanced through the urinary tract and into the kidney 70. When it comes to the location of the kidney stone 80 (e.g., within the target cup 75 of the kidney 70 through which the stone 80 is accessible), the medical instrument 32 can be used to designate / tag a target location for percutaneous access to the kidney 70. To minimize damage to the kidney and / or surrounding anatomical structures, the physician 5 can designate a particular papilla 79 of the kidney 70 as the target location / anatomical feature for entering the kidney 70 using a percutaneous access instrument (e.g., a needle (not shown), see, e.g., FIG. 13). However, other target locations can also be designated or determined. When the percutaneous access instrument reaches the target location (e.g., the cup 75), the percutaneous access path utilized can be used to remove the kidney stone 80 from the patient 13. The term "percutaneous access instrument" is used herein in its broad ordinary meaning and can refer to a surgical tool or device configured to be punctured or inserted through the human skin and / or other tissue / anatomical structures such as a needle, scalpel, guidewire, etc. However, it should be understood that the percutaneous access instrument can refer to other types of medical instruments in the context of the present disclosure.

[0041] In the example of FIG. 1, the medical instrument 32 is implemented as a scope. However, the medical instrument 32 can each be implemented as any suitable type of medical instrument such as a catheter, guidewire, lithotripter, basket retrieval device, etc. In some embodiments, the medical instrument 32 is an operable device.

[0042] Scopes such as the scope 32 of the system 100 can be configured to navigate within a human anatomical structure, such as a natural orifice or within a lumen of a human anatomical structure. The scope can include, for example, a ureteroscope (e.g., for accessing the urinary tract), a laparoscope, a nephroscope (e.g., for accessing the kidney), a bronchoscope (e.g., for accessing an airway such as the bronchus), a colonoscope (e.g., for accessing the colon), an arthroscope (e.g., for accessing a joint), a cystoscope (e.g., for accessing the bladder), and the like.

[0043] Referring to FIG. 2, which shows an exemplary embodiment of the robotic system 10 of FIG. 1 according to one or more embodiments of the present disclosure, the robotic system 10 can be configured to at least partially facilitate the performance of a medical procedure. The robotic system 10 can be arranged in various ways depending on the particular procedure. The robotic system 10 can include, for example, one or more robotic arms 12 configured to engage and / or control a scope 32 (and / or a percutaneous access device, not shown) to perform one or more aspects of the procedure. As shown, each robotic arm 12 can include a plurality of arm segments 23 coupled by joints that can provide a plurality of degrees of mobility / freedom. In the example of FIG. 1, the robotic system 10 is positioned proximate to the patient's leg, and the robotic arm 12 is actuated to engage and position the scope 32 for access to an access point such as the patient 13's urethra 65. Once the robotic system 100 is properly positioned, the scope 32 can be inserted into the patient 13 using the robotic arm 12 under robotic control, manually by the physician 5, or a combination thereof.

[0044] The robotic system 10 can be coupled to any component of the medical system 100, such as a control system 50, a stand 15, an EM field generator 18, a scope 32, and / or a percutaneous access device (e.g., a needle, see FIG. 12). In some embodiments, the robotic system 10 is communicatively coupled to the control system 50. For example, the robotic system 10 can be configured to receive control signals from the control system 50 and perform operations such as positioning one or more robotic arms 12 in a particular manner, operating the scope 32, etc. In response, the robotic system 10 can control the components of the robotic system 10 using specific control circuits 202, actuators 207, and / or other components of the robotic system 10 to perform the operations. In some embodiments, the robotic system 10 is configured to receive from the scope 32 an image and / or image data representing the anatomical internal structure of the patient 13, i.e., the urinary system with respect to the particular depiction of FIG. 1, and / or transmit the image / image data to the control system 50 (which can then be displayed on a display 42 or other output device). Further, in some embodiments, the robotic system 10 is coupled to components of the medical system 10, such as the control system 50, in a manner that allows fluids, optics, power, or the like to be received therefrom. The robotic system 10 can include one or more communication interfaces 206, a power / supply interface, an electronic display 242, and / or other input / output components 210.

[0045] Referring to FIG. 1 and FIG. 3, which shows an exemplary embodiment of the control system 50 of FIG. 1 according to one or more embodiments of the present disclosure, the control system 50 can be configured to provide various functions to assist in the performance of a medical procedure. In some embodiments, the control system 50 is coupled to the robotic system 10 and operates in cooperation with the robotic system 10 to perform a medical procedure on the patient 13. For example, the control system 50 can communicate with the robotic system 10 via a wireless or wired connection (e.g., for controlling the robotic system 10 and / or the scope 32, receiving images captured by the scope 32, etc.), provide fluid to the robotic system 10 via one or more fluid channels, provide power to the robotic system 10 via one or more electrical connections, provide an optical system to the robotic system 10 via one or more optical fibers or other components, and so on. Further, in some embodiments, the control system 50 can communicate with the needle and / or the nephroscope and receive position data therefrom. Further, in some embodiments, the control system 50 can communicate with the table 15 and position the table 15 in a particular orientation or otherwise control the table 15. Further, in some embodiments, the control system 50 can communicate with the EM field generator 18 and control the generation of an EM field within the region surrounding the patient 13.

[0046] System 10 may include a specific control circuit configured to perform specific functions described herein, such as control circuit 202 of robot system 10 and / or control circuit 302 of control system 50. That is, the control circuit of system 10 may be part of robot system 10, control system 50, or both. Thus, all references to control circuits herein may refer to circuits embodied in a robot system, a control system, or any other component of a medical system, such as medical system 100 shown in FIG. 1. The term "control circuit" is used herein in its broad and ordinary sense and refers to a processor, processing circuit, processing module / unit, chip, die (e.g., a semiconductor die including one or more active and / or passive devices and / or connectivity circuits), microprocessor, microcontroller, digital signal processor, microcomputer, central processing unit, field programmable gate array, programmable logic device, state machine (e.g., a hardware state machine), logic circuit, analog circuit, digital circuit, and / or any collection of devices that operate on signals (analog and / or digital) based on hard coding and / or operating instructions of the circuit. The control circuits referred to herein may further include one or more circuit boards (e.g., printed circuit boards), conductive traces and vias, and / or mounting pads, connectors, and / or components. The control circuits referred to herein may further comprise one or more storage devices, which may be embodied in a single memory device, multiple memory devices, and / or an embedded circuit of the device. Such data storage devices may comprise read only memory, random access memory, volatile memory, non-volatile memory, static memory, dynamic memory, flash memory, cache memory, data storage registers, and / or any device for storing digital information.Note that in an embodiment where the control circuit comprises a hardware and / or software state machine, an analog circuit, a digital circuit, and / or a logic circuit, a data storage device / register that stores any associated operation instructions may be embedded within or external to a circuit that comprises a state machine, an analog circuit, a digital circuit, and / or a logic circuit.

[0047] Control circuit 202 and / or 302 may comprise a hard-coded and / or operation instruction storing and / or a computer-readable medium configured to store them, corresponding to at least some of the steps and / or functions illustrated in one or more of this figure and / or described in this specification. Such a computer-readable medium can, in some instances, be included in a manufactured article. Control circuit 202 / 302 may be maintained / disposed entirely locally or may be located at least partially remotely (e.g., communicatively coupled indirectly via a local area network and / or a wide area network).

[0048] Regarding robot system 10, at least a portion of control circuit 202 may be integrated with base 25, column 14, and / or console 16 of robot system 10 and / or another system communicatively coupled to robot system 10. Regarding control system 50, at least a portion of control circuit 302 may be integrated with console base 51 and / or display unit 42 of control system 50. It should be understood that any description of a functional control circuit or related functionality herein is implemented in either robot system 10, control system 50, or both, and / or at least partially in one or more other local or remote systems / devices.

[0049] Referring to FIG. 2, the robotic system 10 generally includes an elongated support structure 14 (also referred to as a "column"), a robotic system base 25, and a console 16 at the upper portion of the column 14. The column 14 may include one or more arm supports 17 (also referred to as "carriages") for supporting the deployment of one or more robotic arms 12 (three are shown in FIG. 2). The arm support 17 may include an individually configurable arm mount that rotates along a vertical axis to adjust the base of the robotic arm 12 for better positioning with respect to the patient. The arm support 17 also includes a column interface 19 that allows the arm support 17 to translate vertically along the column 14.

[0050] In some embodiments, the column interface 19 can be connected to the column 14 through slots such as slots 20 that are positioned on both sides of the column 14 to guide the vertical translation of the arm support 17. The slots 20 contain a vertical translation interface for positioning and holding the arm support 17 at various vertical heights relative to the robotic system base 25. The vertical translation of the arm support 17 allows the robotic system 10 to adjust the reach of the robotic arm 12 to accommodate various table heights, patient sizes, and physician preferences. Similarly, the individually configurable arm mount on the arm support 17 can allow the robotic arm base 21 of the robotic arm 12 to be angled in various configurations.

[0051] The robotic arm 12 can generally include a robotic arm base 21 and an end effector 22 separated by a series of linkages 23 connected by a series of joints 24, each joint comprising one or more independent actuators 207. Each actuator may comprise a separately controllable motor. Each separately controllable joint 24 can provide or represent an independent degree of freedom available to the robotic arm. In some embodiments, each of the arms 12 has seven joints and thus provides seven degrees of freedom, including "redundant" degrees of freedom. The redundant degrees of freedom enable the robotic arm 12 to position its respective end effector 22 at a particular position, orientation, and trajectory within the space using different joint positions and angles. This enables the system to position and orient a medical instrument from a desired point within the space, while at the same time allowing the physician to move the arm joints to a clinically advantageous position away from the patient, creating better access while avoiding collisions of the arm.

[0052] The robotic system base 25 balances the weight of the column 14, the arm support 17, and the arm 12 on the floor. Thus, the robotic system base 25 can house heavier components such as electronics, motors, power supplies, and components that selectively enable movement or immobilize the robotic system. For example, the robotic system base 25 includes wheel-shaped casters 28 that enable the robotic system to easily move around the room before a procedure. After reaching an appropriate position, the casters 28 can be immobilized using wheel locks to hold the robotic system 10 in place during the procedure.

[0053] Once positioned at the upper end of column 14, console 16 enables both a user interface for receiving user input and a display screen (or, for example, a dual-purpose device such as touch screen 26) for providing both preoperative and intraoperative data to the physician user. Potential preoperative data on touch screen 26 may include preoperative plans, navigation and mapping data derived from preoperative computerized tomography (CT) scans, and / or notes from preoperative patient interviews. Intraoperative data on the display may include optical information provided from tools, sensor and coordinate information from sensors, and vital patient statistics such as respiration, heart rate, and / or pulse. Console 16 is positioned and tiltable to enable the physician to access the console from the side of column 14 opposite arm support 17. From this position, the physician can view console 16, robotic arm 12, and the patient while operating console 16 from behind the robotic system 10. As shown, console 16 may also include a handle 27 for assisting in the operation and stabilization of robotic system 10.

[0054] Each end effector 22 of robotic arm 12 may include an instrument device manipulator (IDM) that can be attached using a mechanism changer interface (MCI). In some embodiments, the IDM can be removed and exchanged for a different type of IDM. For example, a first type of IDM can operate an endoscope while a second type of IDM can operate a laparoscope. The MCI can include connectors for transmitting pneumatic, electrical, electrical signal, and / or optical signals from robotic arm 12 to the IDM. The IDM can be configured to operate a medical instrument (e.g., a surgical tool / instrument) such as scope 32 using techniques including, for example, direct drive, harmonic drive, gear drive, belt and pulley, magnetic drive, and equivalents.

[0055] Referring to FIG. 3, the control system 50 may include various I / O components 310 configured to assist the physician 5 or others during the performance of a medical procedure. For example, the input / output (I / O) component 310 may be configured to enable user input to control the scope 32, such as navigating the scope 32 within the patient 13. In some embodiments, for example, the physician 5 can provide an input to the control system 50 and / or the robotic system 10, and in response to such input, send a control signal to the robotic system 10 to operate the scope 32. As also shown in FIG. 1, the control system 50 may include one or more display devices 42 for providing various information related to the procedure. For example, the display 42 may provide information about the scope 32. For example, the control system 50 can receive real-time images captured by the scope 32 and display the real-time images via the display 42. Additionally or alternatively, the control system 50 can receive signals (e.g., analog, digital, electrical, acoustic / acoustic wave, pneumatic, tactile, hydraulic, etc.) from medical monitors and / or sensors associated with the patient 13, and the display 42 can present information regarding the health or environment of the patient 13. Such information may include, for example, heart rate (e.g., ECG, HRV, etc.), blood pressure / blood flow velocity, muscle biosignals (e.g., EMG), body temperature, blood oxygen saturation (e.g., SpO 2 )、CO 2 , information displayed via a medical monitor, such as electroencephalogram (e.g., EEG), information regarding the environment and / or local or core body temperature, etc.

[0056] To facilitate the functions of the control system 50, the control system can include various components (sometimes referred to as "subsystems"). For example, the control system 50 can include control electronics / circuits 302, as well as one or more power supplies / interfaces 308, pneumatic devices, light sources, actuators, data storage devices 304, and / or communication interfaces 306. In some embodiments, the control system 50 includes a control circuit comprising a computer-based control system configured to store executable instructions that, when executed, cause various operations related to the functions described herein to be performed. In some embodiments, the control system 50 is movable, while in other embodiments, the control system 50 is a substantially stationary system. Although various functions and components are discussed as being implemented by the control system 50, any of such functions and / or components can be integrated into and / or implemented by other systems and / or devices, such as, for example, the robotic system 10, the platform 15, or others.

[0057] Referring further to FIG. 1, the medical system 10 can provide guidance (e.g., instrument tracking, instrument alignment information, etc.) to assist the physician during the performance of a procedure, enable the physician to perform the procedure from an ergonomic position without the need for awkward arm movements and / or positions, enable a single physician to perform a procedure using one or more medical instruments, avoid radiation exposure (e.g., related to fluoroscopy techniques), enable the procedure to be performed in a single motion setting, provide continuous suction and more efficiently remove an object (e.g., remove a kidney stone), and provide various other benefits. For example, the medical system 100 can provide guidance information to assist the physician in accessing a target anatomical feature using various medical instruments while minimizing bleeding and / or damage to anatomical structures (e.g., critical organs, blood vessels, etc.). Further, the medical system 100 can provide non-radiation-based navigation and / or localization techniques to reduce radiation exposure to the physician and patient and / or reduce the amount of equipment in the operating room. Further, the medical system 100 can provide functions that are distributed between at least the control system 50 and the robotic system 10, which may be independently movable. Such distribution of functionality and / or mobility can enable the control system 50 and / or the robotic system 10 to be positioned in a location that is optimal for a particular medical procedure, thereby maximizing the working area around the patient and / or providing an optimized location for the physician to perform the procedure.

[0058] The various components of system 100 can be communicatively coupled to each other over a network, which can include wireless and / or wired networks. Exemplary networks include one or more personal area networks (PANs), local area networks (LANs), wide area networks (WANs), Internet area networks (IANs), cellular networks, the Internet, and the like. Further, in some embodiments, the various components of system 10 can be connected via one or more support cables, tubes, or the like for data communication, fluid / gas exchange, power exchange, and the like.

[0059] Robot system 10 and / or control system 50 can include specific user control devices 246, 346 with any type of user input (and / or output) device or device interface, and / or an interface / connector therefor, such as one or more buttons, keys, joysticks, handheld controllers (e.g., video game type controllers), computer mice, track pads, track balls, control pads, and / or sensors that capture hand gestures and finger gestures (e.g., motion sensors or cameras). User control devices 246, 346 are communicatively and / or physically coupled to at least a portion of control circuits 202, 302, respectively.

[0060] In some embodiments, the user control devices 246, 346, and / or associated control circuitry are configured to enable a user to control a medical instrument, such as an instrument operable at least in part by a robotic system (e.g., an endoscope or a nephroscope), to control the pitch and yaw movement of the distal end of the instrument using the control device. For example, movement on a joystick can be mapped to yaw and pitch movement at the distal end of the scope / device. In some embodiments, the user control device is configured to provide haptic feedback to the user. For example, a joystick or other control mechanism can vibrate to indicate an invalid or potentially problematic input. In some embodiments, the control system 50 and / or the robotic system 10 can also provide visual feedback (e.g., a pop-up message) and / or audio feedback (e.g., a beep) to indicate problems associated with robotic operation.

[0061] In some embodiments, the control circuitry 202 / 302 can use a three-dimensional (3D) map of the patient's anatomical structure and / or a predetermined computer model of the patient to control a medical instrument (e.g., an endoscope). For example, the control circuitry 202 can be configured to provide a control signal to the robotic arm 12 of the robotic system 10 to operate an associated instrument and position the instrument at a target location, position, and / or orientation / alignment. For embodiments implementing 3D mapping, the position control mode may require a sufficiently accurate mapping of the patient's anatomical structure.

[0062] In some embodiments, the user can manually operate the robotic arm 12 of the robotic system 10 without using an electronic user control device. For example, during setup in the operating room, the user can move the robotic arm 12 and / or any other medical instrument to provide the desired access to the patient. The robotic system 10 can rely on force feedback and inertial control from the user to determine the appropriate configuration of the robotic arm 12 and associated instrumentation.

[0063] The display device 42 of the control system 50 can be integrated with the user control device 346, for example, as a tablet device with a touch screen that provides user input. The display device 42 can be configured to provide data and input commands to the robot system 10 using an integrated display touch control device. The display device 42 can be configured to display a graphical user interface that indicates information regarding the position and orientation of the patient and / or various instruments operating within the system, based on information provided by one or more position sensors. In some embodiments, a position sensor associated with a medical instrument (e.g., an endoscope) can be configured to generate a signal indicative of the position and transmit the signal on a wire and / or transmitter coupled to the sensor. Such connectivity components can be configured to transmit the position information to the console base 51 for its processing by the control circuit 302 and for presentation via the display device.

[0064] In the example of FIG. 3, the control system 50 is illustrated as a cart-based system that can be moved with one or more wheels 312. However, the control system 50 can be implemented as a fixed system and integrated into another system / device, etc. While specific components of the control system 50 are illustrated in FIG. 3, it should be understood that additional components not illustrated can be included in embodiments according to the present disclosure. Further, in some embodiments, some of the illustrated components can be omitted. Although the control circuit 302 is illustrated as a separate component in the figure of FIG. 3, it should be understood that any or all of the remaining components of the control system 50 can be at least partially embodied in the control circuit 302.

[0065] The location-specific component 314 can be configured to perform one or more location-specific techniques to determine and / or track the location and / or orientation of an object such as a medical instrument and / or a target anatomical feature (e.g., a nipple). For example, the location-specific component 314 can process input data (e.g., sensor data from a medical instrument, model data regarding a patient's anatomical structure, patient location data, preoperative data, robot commands, and / or kinematic data, etc.) to generate location / orientation data 320 for one or more medical instruments. The location / orientation data 320 can indicate the location and / or orientation of one or more medical instruments relative to a viewing perspective. The viewing perspective can be a viewing perspective relative to a patient's anatomical structure, a known object (e.g., an EM field generator), a coordinate system / space, a coordinate frame defined by a robotic system / cart, etc.

[0066] In some embodiments, the location-specific component 314 can process preoperative data to determine the location and / or orientation of an object. The preoperative data (which may also be referred to as "mapping data") can be generated by performing a computed tomography (CT) scan such as a low-dose CT scan. In some embodiments, the present disclosure provides a mechanism for determining / generating mapping data based on electromagnetic field position data recorded using an endoscopic electromagnetic sensor disposed within a target anatomical structure (e.g., the calyceal network of a patient's kidney). Further, in some embodiments, the location-specific component 314 can direct vision-based techniques to determine the location and / or orientation of a target anatomical feature. For example, a medical instrument can be equipped with a camera, a distance sensor (which may also be referred to as a "depth sensor"), a radar device, etc. to provide sensor data in the form of vision data.

[0067] The location-specific component 314 can direct the processing of visual data to facilitate vision-based location tracking of a medical instrument. For example, preoperative model data can be used with visual data to enable computer vision-based tracking of a medical instrument (e.g., an endoscope). Further, in some embodiments, other types of vision-based techniques can be performed to determine the position and / or orientation of an object. For example, the medical system 10 can be configured to track the movement of an image sensor (e.g., a camera or other sensor) and thus determine the medical instrument associated with the image sensor. The use of mapping data can also enhance vision-based algorithms or techniques. Further, the location-specific component 314 can direct the use of optical flow processing, another computer vision-based technique, to analyze the displacement and / or translation of image pixels within a video sequence in visual data to infer camera movement. Examples of optical flow techniques can include motion detection, object segmentation calculation, luminance, motion-compensated coding, stereoscopic parallax measurement, and the like.

[0068] In some embodiments, the location - specifying component 314 and the control circuit 302 can use electromagnetic tracking to determine the position and / or orientation of an object. For example, the location - specifying component 314 can use real - time EM tracking to determine the real - time location of a medical instrument in a coordinate system / space that can be recorded on a patient's anatomical structure, and this location can be represented by a pre - operative model or other model. In EM tracking, an EM sensor (or tracker) that includes one or more sensor coils can be embedded at one or more locations and / or orientations within a medical instrument (e.g., a scope, a needle, etc.). The EM sensor can measure the variations in the EM field created by one or more static EM field generators positioned at known locations. The location information detected by the EM sensor can be stored as EM data. The location - specifying component 314 can process the EM data to determine the position and / or orientation of an object such as a medical instrument. The EM field generator (or transmitter) can be placed near the patient (e.g., within a predetermined distance) to create a low - intensity magnetic field that the EM sensor can detect. The magnetic field can induce a small current within the sensor coils of the EM sensor, and by analyzing this, the distance and angle between the EM sensor and the EM field generator can be determined. These distances and orientations can be "recorded" during the procedure on the patient's anatomical structure (e.g., a pre - operative model) to determine the geometric transformation that aligns a single location in the coordinate system with the position in the pre - operative model of the patient's anatomical structure. Once recorded, the EM sensor (e.g., an embedded EM tracker) at one or more positions of the medical instrument (e.g., the distal tip of an endoscope, a needle, etc.) can provide a real - time indication of the position and / or orientation of the medical instrument through the patient's anatomical structure.

[0069] In some embodiments, the location identifying component 314 and the control circuit 302 can use the input data in combination. For example, the control circuit 302 can use a probabilistic approach in which confidence weights are assigned to positions / orientations determined from multiple forms of input data. By way of illustration, if the EM data is not reliable (such as in the case of EM interference), the EM data can be associated with a relatively low confidence value and can rely on other forms of input data such as visual data, robotic commands, kinematic data, etc.

[0070] The target / trajectory component 316 can be configured to determine the position of a human anatomical structure and / or a target location within a coordinate space / coordinate system. It should be understood that the "target" described in this disclosure can also refer to an anatomical region / feature such as the surface of the nipple. For example, in some embodiments, both the position of the target and the uncertainty associated with the position of the target can be captured based on sources of error such as sensor calibration. The target location can represent a point / point set within the human anatomical structure and / or a coordinate space / coordinate system. For example, the target / trajectory component 316 can identify one or more points of the target location within the coordinate system, identify the coordinates of the one or more points (e.g., the X, Y, Z coordinates of each point), and associate the coordinates with the target location. In some embodiments, the target / trajectory component 316 can use the position and / or orientation of the medical instrument to determine the position of the target location. For example, the scope can be navigated to contact the target location or be within the proximity range (e.g., be stationary in front of the target location).

[0071] The target location can represent a fixed or movable point within the human anatomical structure and / or a coordinate space / coordinate system. For example, if the nipple is initially designated as the target location and the nipple moves during the course of the procedure (e.g., due to the insertion of a medical instrument), the coordinates of the target location can be determined and updated. Here, the location of the scope (which can be within the proximity range of the nipple) can be tracked over time and used to update the coordinates of the target location. In some embodiments, the target / trajectory component 316 can estimate / determine the position of the target location (e.g., the target anatomical feature).

[0072] In some embodiments, the target trajectory and / or the trajectory of the medical device can be defined / represented with respect to one or more anatomical planes / axes. For example, the trajectory can be defined / represented as an angle with respect to the coronal / sagittal / transverse plane or another plane / axis (e.g., a 20-degree cephalic-caudal angle, a 10-degree medial-lateral angle, etc.). By way of illustration, the control system 50 can determine the posture of the medical device with respect to the EM field generator and / or the location of the target with respect to the EM field generator. The control system 50 can also determine the posture of the EM field generator with respect to the robotic system based on robotic kinematics. In some cases, the control system 50 can infer / determine that the robotic system is parallel to the bed. Based on such information, the control system 50 can determine the target trajectory and / or the trajectory of the medical device with respect to the anatomical plane, such as the angle with respect to the anatomical plane of the patient on the bed.

[0073] The target / trajectory component 316 can also be configured to determine the target trajectory of the medical device or another object. The target trajectory can represent a desired path for accessing the target location and / or anatomical features. The target trajectory can be determined based on various information such as the position of the medical device (e.g., a needle, a scope, etc.), the target location within the human anatomical structure, the position and / or orientation of the patient, the anatomical structure of the patient (e.g., the location of the organ within the patient relative to the target location). For example, the target trajectory can include a line extending from the position of the medical device and / or a location on the patient's skin to / through the position of the target location within the patient. In an example, the physician can provide an input specifying the target trajectory, such as by analyzing an image or model of the human anatomical structure and drawing a line on the image of the patient's internal anatomical structure. In some embodiments, the target / trajectory component 316 can calculate the target trajectory initially and / or update the target trajectory throughout the procedure. For example, when the target location moves during the procedure, the target trajectory can be updated due to the change in the position of the target location. In embodiments where the target location is estimated, the target trajectory can represent the estimated path to reach the target location.

[0074] The user interface component 318 can be configured to facilitate one or more user interfaces (also referred to as "one or more graphical user interfaces (GUI)"). For example, the user interface component 318 can generate user interface data 322 that represents a scope guidance and / or needle guidance interface 324 that includes one or more visualizations to indicate the orientation and / or position of a medical device. The user interface component 318 can use position / orientation data 320 regarding one or more medical devices, information regarding a target location, and / or information regarding a target trajectory to indicate one or more visual features / icons that indicate a target scope position, a needle trajectory, and / or the like, as represented by the interface data 322. Further, such icons / features can be shown relatively and / or overlaid on a scope vision / image data / representation. For example, the needle guidance feature and / or the scope guidance feature can be shown overlaid on a scope image window to provide an extended scope image view / window. The user interface component 318 can provide the user interface data 322 or other data to one or more displays 42 and / or another display for displaying its representation.

[0075] One or more communication interfaces 306 can be configured to communicate with one or more devices / sensors / systems, such as via a wireless and / or wired network connection. Networks according to embodiments of the present disclosure can include a local area network (LAN), a wide area network (WAN) (e.g., the Internet), a personal area network (PAN), a body area network (BAN), and the like. In some embodiments, one or two or more communication interfaces 1206 can implement wireless technologies such as Bluetooth, Wi-Fi, near field communication (NFC), and the like.

[0076] Although not shown in FIG. 3, the control system 50 can include and / or control other components such as one or more pumps, flow meters, valve controllers, and / or fluid access components to provide controlled perfusion and / or aspiration capabilities to medical instruments (e.g., scopes), devices that can be deployed through the medical instruments, and the like. In some embodiments, the perfusion and aspiration capabilities can be delivered directly to the medical instrument through separate cables. In some embodiments, the control system 50 can be coupled to medical instruments such as the robotic system 10, the table 15, and / or the scope 32 and / or a needle or other percutaneous access instrument (not shown) via one or more cables or connections (not shown).

[0077] Figure 4 illustrates a ureteroscope 440 disposed in a portion of a patient's urinary system according to one or more embodiments of the present disclosure. As referenced above, ureteroscopic procedures can be implemented to investigate and / or treat abnormalities within a human ureter. For example, ureteroscopic procedures can be implemented to treat and / or remove kidney stones. Such procedures can be performed at least partially manually and / or can be performed using robotic technology at least in part, such as the robotic system 10 shown in FIG. 1. For example, the use of robotic devices and / or systems for certain endoscopic procedures can provide relatively high accuracy, control, and / or coordination as compared to strictly manual procedures. In some embodiments, the scope 440 includes a working channel 444 for deploying medical instruments (e.g., lithotripters, basket devices, forceps, etc.), irrigation, and / or suction to an operating area at the distal end of the scope.

[0078] The scope 440 can be articulable with respect to at least the distal portion of the scope so that the scope can be maneuvered within a human anatomical structure. In some embodiments, the scope 440 is configured to be articulated with five degrees of freedom, including, for example, XYZ coordinate movement and pitch and yaw. In some embodiments, a needle sensor provides six degrees of freedom, including X, Y, and Z longitudinal coordinate positions and pitch, roll, and yaw. The position sensors of the scope 440 can similarly have a similar degree of freedom with respect to the position information they generate / provide. The figure illustrates multiple degrees of movement of the scope 440 according to some embodiments. As shown in FIG. 4, the tip 442 of the scope 440 can be oriented with zero deflection with respect to its longitudinal axis 406 (also referred to as the "roll axis").

[0079] To capture images at different orientations of the tip 442, the robotic system can be configured to deflect the tip 442 on the positive yaw axis 402, negative yaw axis 403, positive pitch axis 404, negative pitch axis 405, or roll axis 406. The tip 442 or the body 445 of the scope 442 can be extended or translated along the longitudinal axis 406, x-axis 408, or y-axis 409. The scope 440 can include a reference structure (not shown) for calibrating the position of the scope. For example, the robotic system can measure the deflection of the scope 440 relative to the reference structure. The reference structure can be located, for example, on the proximal end of the endoscope 440 and can include a key, slot, or flange. The reference structure can be coupled to a first drive mechanism for initial calibration and to a second drive mechanism for performing a surgical procedure.

[0080] For robotic implementation, the robotic arm of the robotic system can be configured to operate the scope 440 using an elongate movement member. The elongate movement member can include one or more pull wires (e.g., pull or push wires), cables, fibers, and / or a flexible shaft. For example, the robotic arm can be configured to actuate a plurality of pull wires (not shown) coupled to the scope 440 to deflect the tip 442 of the scope 440. The pull wires can include any suitable or desirable material such as metals and non-metallic materials such as stainless steel, Kevlar, tungsten, carbon fiber, and the like. In some embodiments, the scope 440 is configured to exhibit non-linear behavior in response to the force applied by the elongate movement member. The non-linear behavior can be based on the stiffness and compressibility of the scope and the variability of slack or stiffness between different elongate movement members.

[0081] The scope (e.g., endoscope / ureteroscope) 440 can comprise a tubular and flexible medical instrument configured to be inserted into a patient's anatomical structure to capture an image of the anatomical structure. In some embodiments, the scope 440 can include an imaging device 448 such as an optical camera, and can accommodate wires and / or optical fibers for transmitting signals to / from the distal end 442 of the optical assembly and the scope 440.

[0082] The camera / imaging device 448 can be used to capture an image of an internal anatomical space such as a target calyx / papilla of the kidney 70. The scope 440 can be further configured to accommodate an optical fiber for conveying light from a proximally located light source such as a light emitting diode to the distal end 442 of the scope. The distal end 442 of the scope 440 can include a port for a light source to illuminate the anatomical space when using the camera / imaging device. In some embodiments, the scope 440 is configured to be controlled by a robotic system 10 similar to the robotic systems shown in FIGS. 1 and 2 in one or more respects. The imaging device may comprise an optical fiber, a fiber array, and / or a lens. The optical components move with the tip of the scope 440 such that movement of the tip of the scope results in a change in the image captured by the imaging device 448.

[0083] For percutaneous nephrolithotomy (PCNL) procedures, access is made through the patient's skin and intervening tissue into the target calyx. Generally, the preferred access to the calyces of the kidney is through the soft tissue papillary structure, and access through such tissue can generally be associated with a reduced risk of bleeding and / or other complications. When a needle is inserted through the papillary structure, in addition to there being no bleeding, such a path can provide complete access to the interconnected internal channels (e.g., calyces) of the kidney.

[0084] PCNL represents a relatively effective method for treating large kidney stones, but many physicians choose other procedures, partly due to the difficulty of accurately targeting the target papilla / calyx. More specifically, performing PCNL involves using a needle to gain percutaneous access to the target calyx of the kidney through the patient's flank. This step can be considered extremely important for the ultimate success of the procedure, as the physician must select a needle path to the kidney that does not traverse surrounding organs and that allows a rigid instrument to reach and treat the urinary tract stone. If the physician cannot effectively do so, there is a risk of causing visceral or pleural injury or of being unable to completely treat the stone. Due to these challenges, the learning curve associated with obtaining percutaneous needle access to perform PCNL in a suitable patient position (e.g., modified supine position) is very high.

[0085] In some procedures, the physician studies the patient's preoperative computed tomography (CT) images to determine the location of the urinary tract stone, the location of surrounding organs and bone structures, and to examine the calyx morphology. Using this knowledge, the physician can mentally generate a preoperative plan for the percutaneous needle path. Typically, the physician must identify the posterior calyx to be punctured to accommodate the rigid instrument. Specifically, the posterior calyx generally provides a relatively straight emission into the renal pelvis. The physician must attempt to insert the needle into the kidney through the papilla to avoid damaging the renal vasculature and causing bleeding. During the procedure, the physician relies on fluoroscopy or ultrasound in some procedures to guide the alignment and insertion of the needle into the target calyx. However, the resolution and interpretation difficulties associated with such imaging techniques can pose a relatively high degree of difficulty in adequately performing the needle puncture. Accordingly, embodiments of the present disclosure that provide improved tracking and visualization of target anatomical features such as papillae and calyces can improve surgical outcomes and attract the interest of some physicians more than other PCNL methodologies.

[0086] In some embodiments, the medical instrument (e.g., a scope) 440 includes a sensor configured to generate sensor position data and / or transmit it to another device. The sensor position data can indicate the position and / or orientation of the medical instrument 440 (e.g., its distal end 442), and / or can be used to determine / estimate the position / orientation of the medical instrument. For example, the sensor (which may also be referred to as a "position sensor") can include an electromagnetic (EM) sensor with a coil of conductive material, or other forms / embodiments of an antenna.

[0087] FIG. 4 shows an EM field generator 415 configured to broadcast an EM field 90 detected by an EM sensor on a medical instrument. The magnetic field 90 can induce a small current in the coil of the EM position sensor that can be analyzed to determine the distance and / or angle / orientation between the EM sensor and the EM field generator 415. Further, the medical instrument / scope 440 can include other types of sensors such as shape sensing fibers, accelerometers, gyroscopes, satellite-based positioning sensors (e.g., global positioning system (GPS) sensors), high-frequency transceivers, etc. In some embodiments, the sensors on the medical instrument can provide sensor data to a control system, which is then used to determine the position and / or orientation of the medical instrument. In some embodiments, the position sensor is positioned on the distal end 442 of the medical instrument 440, while in other embodiments, the sensor is positioned at another location on the medical instrument. The ureteroscope can be driven to a position proximate to the target papilla.

[0088] In some implementations, as described in further detail below, the distal end of the ureteroscope 440 can be advanced to contact a target anatomical feature (e.g., a papilla). When the position sensor associated with the distal end of the scope 440 is in contact with and / or proximate to the target anatomical feature, the position of the distal end of the scope 440 can be recorded as a target percutaneous access position such that a percutaneous access instrument (e.g., a needle) can be directed to access the target cup through the papilla.

[0089] Certain embodiments of the present disclosure advantageously automate and assist a physician through a process for obtaining percutaneous access to a target anatomical feature. For example, electromagnetic positioning and scope imaging can be used together to guide the insertion of a needle into a patient. Such a solution can enable a novice physician to obtain access to the kidney, for example, in a modified supine position and to perform PCNL.

[0090] Certain embodiments of the present disclosure involve position sensor-guided percutaneous access to a target treatment site, such as a target location within the kidney. For example, when a scope 440 is equipped with one or more electromagnetic sensors and a nephroscope access needle further includes one or more electromagnetic sensors and such sensors are exposed to an electromagnetic field 90 created by an electric field generator 415, the associated system control circuitry can be configured to detect and track their locations. In some embodiments, the tip of the ureteroscope 440 acts as a guiding beacon while the user is inserting the percutaneous access needle. Such a solution can enable the user to hit the target from various approaches, thereby eliminating the need to rely on fluoroscopy or ultrasound imaging.

[0091] In some embodiments, a control system (not shown in FIG. 4) associated with the scope 440 is configured to implement a localization / positioning technique to determine and / or track the location / position of medical devices such as the scope 440 and / or a percutaneous access needle (not shown). In some examples, as described above, the EM field generator 415 is configured to provide an EM field 90 within the patient's environment. The scope 440 and / or the percutaneous access needle may include an EM sensor configured to detect an EM signal and transmit sensor data regarding the detected EM signal to the control system. The control system can analyze the sensor data to determine the position and / or orientation of the scope 440 (e.g., the distance and / or angle / orientation between the EM sensor and the EM field generator 415). Alternatively or additionally, in some examples, the control system can use other techniques to determine the position and / or orientation of the scope 440. For example, the scope 440 (and / or the needle) may include shape sensing fibers, accelerometers, gyroscopes, accelerometers, satellite-based positioning sensors (e.g., a Global Positioning System (GPS)), radio frequency transceivers, and the like. The control system can receive sensor data from the scope 440 and determine its position and / or orientation. In some embodiments, the control system can track the position and / or orientation of the scope 440 in real time with respect to the patient's coordinate system and / or anatomical structures.

[0092] The scope 440 can be controllable in any suitable or desirable manner, either based on user input or automatically. The control devices 411, 412 provide examples that can be used to receive user input. In some embodiments, the control device of the scope 440 is located on the proximal handle of the scope, which may make it relatively difficult to grip in some procedural postures / positions as the orientation of the ureteroscope changes. In some embodiments, the scope 440 is controlled using a two-handed controller, such as the image 412. The controllers 411, 412 are shown as handheld controllers, but user input can be received using any type of I / O device, such as a touch screen / pad, mouse, keyboard, microphone, etc.

[0093] FIG. 5 (represented in parts 5-1 and 5-2) is a flow diagram illustrating a process 500 for accessing a target cup or other organ of a patient according to one or more embodiments of the present disclosure. FIG. 6 (represented in parts 6-1 and 6-2) shows specific images corresponding to various blocks, states, and / or operations related to the process of FIG. 5 according to one or more embodiments. The process 500 may involve percutaneous access to the kidney 70 for the removal of kidney stones (e.g., PCNL). Such percutaneous access may be desirable for the removal of stones that are too large to be removed via ureteroscopy or for which ureteroscopic removal is impractical or undesirable. For example, a stone may be larger than 2 cm in diameter, while a particular ureteroscope has a working channel through which stones or fragments having a diameter of about 1.2 mm can be removed. Fragmenting a stone into smaller pieces for removal via ureteroscopy is successful in many cases, but studies have shown that the remaining stone fragments are often a source of new stone formation and require future similar treatments. Although the processes described herein are described in the context of ureteroscopy, they can be applied to any other type of surgical procedure that utilizes a position sensor (e.g., an electromagnetic field sensor) and / or a camera to track target anatomical features such as papillae or urinary tract stones.

[0094] In block 502, process 500 involves accessing the kidney through the patient's ureter using the ureteroscope 540 as described above. Specifically, the operation of block 502 may involve advancing the scope 540 through the ureter 63, past the renal pelvis 71, and into the region within or near one or more calyces.

[0095] In block 504, process 500 involves locating the kidney stone 580 in the patient's kidney for treatment using an image capture device (e.g., a camera) associated with the distal end 542 of the endoscope. For example, the kidney stone 580 may be removed at least partially for the purposes of process 500.

[0096] In block 506, process 500 involves identifying the target papilla 579 that is exposed within the target calyx 575 through which access to the kidney stone 580 can be achieved. Identifying the target papilla 579 can be important for creating a workable tract through which access to the kidney stone 580 can be achieved via percutaneous access. For example, it may be necessary to determine an appropriate angle for access by a relatively rigid nephroscope in such a way as to access a calyx (e.g., a minor calyx 575) through which the kidney stone 580 can be reached. In some implementations, it may be desirable or necessary to reach the kidney stone 580 through a posterior calyx in order to provide a sufficiently straight access to the ureteropelvic junction 71. Generally, the target minor calyx can be considered a relatively small target. For example, such a calyx can have a diameter of about 5 - 8 mm. Therefore, precise targeting can be important for effectively removing kidney stones.

[0097] The path through which needle / nephroscope access to the target calyx 575 is achieved should preferably be as straight as possible to avoid hitting the blood vessels surrounding the renal pyramid 576 associated with the papilla 579 through which the needle / nephroscope can be positioned. Further, the location of various important anatomical structures of the patient may require navigation through a restricted window of the patient's tissue / anatomical structure. For example, the lower pole calyx located below the 12th rib may provide a suitable access to avoid the pulmonary pleura. Further, this access path may preferably be medial to the posterior axillary line (e.g., about 1 cm below and 1 cm medial to the tip of the 12th rib) to avoid the colon and / or paravertebral muscles. In addition, this access path may preferably avoid coming in close proximity to the rib and may avoid the intercostal nerve. Further, in some cases, by targeting the entry within the axial central region of the calyx 575, the main artery and / or other blood vessels can be avoided.

[0098] In block 508, process 500 involves tagging / recording the location of the exposed papilla 579 within the target calyx 579 through which the desired access is to be achieved. For example, the location information / data may be represented / identifiable in a three-dimensional space such as an electromagnetic field space or a robotic space (e.g., a coordinate frame).

[0099] To record the position of the papilla 579, the scope 540 can be advanced until it physically touches / comes into contact with the target papilla 579, as indicated by the advanced scope tip 543. In relation to this, such a contact position can be identified as the target position by the scope 540 and / or the operator and / or otherwise indicated. In some implementations, an electromagnetic beacon or other sensor device associated with the distal end / tip 542 of the ureteroscope indicates the target position, whereby the target position within the electromagnetic field space can be recorded. After touching / coming into contact with the papilla 579 and recording the position, the end 542 of the scope can be retracted, and the depth of such retraction can be measured in a certain manner. In some implementations, the operator can generally be notified that the distal end 543 of the scope 540 is in contact with the papilla 579 by monitoring the camera image generated when the contact occurs, which may become occluded / darkened. In some implementations, a user input device (e.g., a pendant type) can be used to notify the system of contact with the target anatomical feature.

[0100] In block 510, process 500 involves percutaneously introducing a medical instrument 550, such as a needle, into the patient. For example, in some implementations, such access can be performed through the patient's flank. In block 512, process 500 involves directing the percutaneously advanced medical instrument 550 towards the target position and ultimately traversing the target papilla 579 to access the target cup 575 through it.

[0101] In some embodiments, visual confirmation of the entry of the tip of the needle 550 into the target cup 575 can be provided by the camera of the scope 540. For example, the scope 540 can be retracted from the target position as described above, thereby providing a field of view that includes the papilla 579 within the cup 575 such that the tip of the needle 550 can be seen as it protrudes through the surface of the papilla 579.

[0102] Once the target location is recorded, a medical device (e.g., needle 550) inserted percutaneously can be directed toward the recorded position. However, if such a recorded position is static, anatomical movement occurring after the recording of the target position can result in a target position that does not accurately reflect the real-time position associated with the target anatomical feature to which access is desired through it. For example, the act of inserting the needle 550 into a patient can move and / or distort certain anatomical structures around the target organ (e.g., kidney 70) and / or the target organ itself in a certain manner, thereby causing the target anatomical feature (e.g., papilla 579) to assume a position / shape different from that at the time when the target access position was recorded. With respect to kidney procedures, the ureteroscope 540 can be fixed at the position of the renal pelvis 71 such that deformation and / or movement of the kidney 70 with respect to the ureteroscope can result in such target position disruption. Thus, the papilla 579 may not be accurately tracked when anatomical movement is introduced into the system.

[0103] When access to the cup 575 with the needle is made, a device with a larger diameter can be exchanged for the needle 550 to provide a larger port for stone removal. In some implementations, the needle 550 includes a stylet and a cannula. The stylet can be removed with the needle tip advanced into the cup 575, leaving the cannula to form an open port to the location of the kidney stone. A guidewire can be placed through and used with the cannula to perform the remainder of the process of removing the stone 580. For example, the guidewire can be used to pass a deflated balloon or dilator along the wire. The balloon or dilator can be expanded to create a port large enough to directly introduce a hollow suction tube, such as a nephrostomy tube, into the cup 575. At this point, either a nephroscope or any one of several other instruments can be introduced into the suction tube to assist in removing the stone 580. For example, a lithotripter, laser, ultrasound, basket, grasper, drainage tube, etc. can be used to remove the stone or its fragments, and / or a drainage tube, such as a nephrostomy catheter, can be deployed under the suction tube to reduce the intrarenal pressure. Any combination of such functions can be embodied in a nephroscope (not shown) and / or a ureteroscope 540.

[0104] Target localization Various aspects of the present disclosure relate to systems, devices, and methods for localizing a target (e.g., a target anatomical feature) associated with a medical procedure. Specifically, target localization according to the present disclosure may involve recording / tagging the location of a target anatomical feature (e.g., a papilla) using an endoscope (e.g., a ureteroscope), determining the position of the target anatomical feature based on the position of the endoscope not in physical contact with the target anatomical feature, determining / recording a position offset / translation between the target anatomical feature and the endoscope (e.g., an associated position sensor) for the purpose of determining the position of the target anatomical feature, and / or dynamically updating the target position associated with the target anatomical feature based on electromagnetic sensor and / or camera data associated with the ureteroscope, among other steps and / or functions. As described, static position markers may be aligned / recorded to identify the target position associated with the target anatomical feature / landmark. In some embodiments, the present disclosure provides systems, devices, and methods for guiding and / or automating an endoscope and / or a percutaneous access instrument based at least in part on static position markers, considering a particular target localization technique. Target localization according to embodiments of the present disclosure can be applied to any type of robotic endoscopy procedure.

[0105] FIG. 7 is a flow diagram illustrating a process 700 for localizing a target anatomical feature according to one or more embodiments. Generally, target localization may be performed to localize the position of a target anatomical feature (e.g., a papilla) relative to a ureteroscope. The target position may be recorded / stored with respect to an electromagnetic field generator / space, a robotic coordinate frame, and / or an anatomical coordinate frame defined, for example, by kidney mapping. In block 710, process 700 involves advancing a medical instrument, such as a scope (e.g., a ureteroscope), to a treatment site, such as a lumen or chamber disposed at least in part within the target organ. For example, the operation of block 710 may involve advancing the medical instrument to a target calyx of a patient's kidney.

[0106] As described above, robot - endoscopic - guided percutaneous access according to aspects of the present disclosure can induce / determine a percutaneous access path for accessing a target anatomical feature / site using target localization techniques with respect to the target anatomical feature. For example, a position - tracking mechanism / sensor associated with the distal end of a medical instrument (e.g., a scope) and / or a percutaneous access instrument (e.g., a needle) can be implemented to guide a physician / technician when aligning the percutaneous access instrument with a treatment site (e.g., a target cup). Accurate real - time target localization / tracking, as enabled by aspects of the present disclosure, can enable a relatively precise single - puncture access to the treatment site.

[0107] In block 720, process 700 involves determining the position of a target anatomical feature. For example, determining the position of a target anatomical feature can be performed in any suitable or desirable way, such as using at least a partially contact - based positioning sub - process 722, or at least a partially image - based positioning sub - process 721, which are described below in connection with blocks 724 and 723, respectively.

[0108] Regarding a particular contact - based positioning process, in block 722, process 700 involves contacting the target anatomical feature of the treatment site with the distal end of a medical instrument. For example, the medical instrument can be equipped with a sensor device, such as an electromagnetic sensor / beacon, that can indicate the position of the distal end of the medical instrument. Thus, when the distal end of the medical instrument is in contact with and / or adjacent to the target anatomical feature, such position measurements can be relied upon to indicate the current position of the target anatomical feature. Contact - based positioning may not be necessary when an image - processing approach is implemented to provide the 3D location / position of the target. For example, in block 723, process 700 involves determining the position of the target anatomical feature using image data input from an endoscopic camera.

[0109] Process 700 may involve advancing a percutaneous access device, such as a needle or the like, along / over the access path in the direction of the target anatomical feature while tracking / locating the target anatomical feature over the course of the procedure, and proceeds to subprocess 730. In some implementations, electromagnetic (EM) position sensing technology is used to track / locate the target anatomical feature. For example, as described above, the target anatomical feature (e.g., the nipple) may be contacted at the distal end portion of the scope at one or more locations / regions, and the local position and orientation of the target feature (e.g., the funnel axis) may be determined based thereon relative to the position of the scope. In some embodiments, as will be described in further detail below with respect to FIGS. 15-17, mapping of the target site (e.g., the target cup / nipple and associated funnel) may be generated based on a plurality of recorded positions from the EM sensor data.

[0110] Subprocess 730 may be implemented in various ways. For example, as shown as subprocess 732, live direct instrument (e.g., scope) targeting / tracking may be implemented to provide motion tracking of the target anatomical feature. For example, throughout the associated procedure, the distal end of the medical instrument may be maintained in contact with the target anatomical feature such that the position sensor data indicated by the medical instrument can provide the accurate real-time location of the target anatomical feature (block 734). Thus, as shown in block 736, the live position of the medical instrument may be targeted to provide the desired percutaneous access path. However, when the distal end of the medical instrument is in proximity / contact with the target anatomical feature, real-time visualization of the target anatomical feature may not be possible or may not be sufficiently clear due to occlusion of the target anatomical feature by the feature itself within the field of view of the camera. That is, the camera associated with the local instrument may be sufficiently blocked or hidden by the mass of the target anatomical feature, thereby preventing the physician / user from having a visual confirmation of the penetration of the target anatomical feature by the percutaneous access device (e.g., the needle).

[0111] Instead, sub-process 731 is shown for tracking the target anatomical feature while still maintaining a clear visual of the target anatomical feature during approach of the percutaneous access device. Sub-process 731 involves identifying the target anatomical feature using a determined positional offset / translation between the position of the scope and the position of the target anatomical feature, and determining the live / current position of the target anatomical feature by applying the offset / translation to the current position of the scope.

[0112] In block 733, sub-process 731 may involve recording the determined position of the contact location of the target feature related to contact with the target anatomical feature performed in relation to the operation of block 720 above. As an example, the user may provide an input for notifying a related control / medical system of the feature contact location of the target anatomical feature by tagging / recording the position of the exposed surface of the target anatomical feature (e.g., the papillary surface exposed within the target cup) in a certain manner. Such tagging may be implemented through the provision of user input in a certain manner or may be substantially automatic based on sensed tissue contact or the like. The position data may be obtained in the volatile and / or non-volatile data storage of a particular control circuit as shown and described herein.

[0113] After determining the location / position of the target anatomical feature, the scope may be retracted and / or dwelled as shown in block 735 to provide visualization facing the target anatomical features (e.g., papillae). Such dwelling may be carried out with the aid of specific scope guidance features / overlays presented in or near the camera view interface window, which will be detailed below with respect to FIGS. 9 - 13.

[0114] Rather than providing live motion tracking while maintaining the medical device (e.g., a scope) in contact with / close to the target anatomical feature as in sub-process 732, sub-process 731 may involve determining the position of the target anatomical feature based on a determined positional offset / translation between the position / orientation of the stationary scope and the position / orientation of the target anatomical feature. When parking the scope, the scope may be retracted away from the target anatomical feature by a distance (e.g., in the proximal direction), thereby enabling the medical device to clearly capture the target anatomical feature within the field of view of its associated camera. For example, in some implementations, the physician / user may notify the system in a certain manner when the medical device is parked at a distance away from the target anatomical feature by a desired distance.

[0115] For clarity, note that sub-processes 731, 732 represent alternative embodiments of sub-process 730. That is, process 700 may generally involve either implementation, but not both sub-process 732 and sub-process 731.

[0116] In some cases, it can be assumed that the offset / translation between the nipple and the scope is generally maintained for a long period of time so that the scope remains inside the target cup. If there is no relative movement of the target anatomical feature with respect to the scope position sensor, the target position can be continuously updated based on the determined current scope position. Position data (e.g., EM data) collected in relation to the retraction / redirection of the scope can be used to determine the offset / translation of the location / orientation of the nipple with respect to the scope. For example, according to one use case, the retraction / positioning of the scope can be about 5 mm in front of the nipple and 2 mm to the left. Such a position offset can be used to determine the position of the target relative to the current position of the scope. The translation / offset information can further incorporate orientation information, which can be made possible in any suitable or desirable manner. If there is relative movement between the target anatomical feature and the scope, the reliability of the determined offset / translation can be lost. In some implementations, relative motion compensation can be performed to compensate for and / or adjust the offset / translation when the relative position / orientation between the scope and the target anatomical feature changes.

[0117] Sub - process 731 may or may not include / accompany the contact 724 and retraction 735 steps, where the user physically contacts the location of the target nipple and retracts the scope to show the nipple within the scope's field of view. For example, when image - based tagging 721 is performed in relation to block 720, it may not be necessary to physically contact the target anatomical feature to determine its position / location. Rather, the position / location can be determined using a target identification mechanism based on image data captured / generated by one or more cameras of the scope / instrument. For example, in some embodiments, the target is identified and tracked using multiple frames of image / visual and / or position (e.g., EM) data. An example of such target positioning is described below in relation to FIG. 18. In some implementations, as described below in relation to FIG. 18, the target position can be estimated / determined with respect to three - dimensional space by viewing the target anatomical feature (e.g., the nipple) from two clearly different positions and / or alignments.

[0118] In block 739, sub-process 731 involves targeting the tracked location of the target anatomical feature using a percutaneous access device. For example, the center of gravity of an identified nipple shape or morphology within the real-time image of the treatment site can be used at the target position for the percutaneous access device. In block 740, process 700 involves piercing the target anatomical feature either without visually confirming the target anatomical feature in relation to sub-process 732 or with visual confirmation by sub-process 731, depending on the particular implementation of process 700.

[0119] The various position sensors used in connection with embodiments of the present disclosure, such as determining / recording the feature contact position in block 733 or targeting the live instrument position in block 736, can be any type of position sensor. As an example, such a sensor can be an electromagnetic (EM) sensor / probe. With respect to the scope, the position sensor can be attached proximal to its tip or integrated. Alternatively, the sensor can comprise a coil connected to an electrical wire extending the length of the scope, which is connected to an external control circuit configured to interpret electrical signals generated in the coil and passed along the wire. Examples of types of position sensor devices that can be implemented in connection with embodiments of the present disclosure include, but are not limited to, accelerometers, gyroscopes, magnetometers, fiber optic shape sensing (e.g., via Bragg gratings, Rayleigh scattering, interferometry, or related techniques). Depending on the implementation, alignment to a separate form of the patient image, such as a CT scan, may or may not be necessary to provide a reference frame for localizing urinary tract stones within the patient.

[0120] Regarding EM type sensors such as coils or other antennas, such sensor devices can be configured to detect changes in the EM field as the EM sensor moves within the field (e.g., within the kidney). Thus, certain embodiments are implemented using one or more EM generators configured to emit an EM field that is picked up / affected by the EM sensor. The EM generators can be modulated in any suitable or desirable manner such that when their emitted EM fields are captured / affected by the EM sensor and processed by a suitable control circuit, signals from different EM generators are separable to provide an additional dimension / degree of freedom of position information. The EM generators can be time or frequency modulated and can use orthogonal modulation such that each signal is completely separable from each other signal, even though they may potentially overlap in time. Further, separate EM generators can be oriented relative to each other in Cartesian space at non-zero non-orthogonal angles such that a change in the orientation of the EM sensor causes the EM sensor to receive at least some signal from at least one of the EM generators at any instant.

[0121] Referring further to recording the feature contact position at block 733 of FIG. 7, EM position data can be recorded on an image of the patient captured using a different technique other than EM, such as a CT scan, to establish a reference frame / space for the EM data (or any mechanism is used to capture the data of the alignment sensor). In addition to the scope, the percutaneous access needle may include one or more position / alignment sensors such as an EM sensor. The position / alignment data received from the needle EM sensor can be received and processed in the same manner as the scope position data as described above. It should be understood that the various processes described herein can be implemented fully or partially manually and / or fully or partially using robots.

[0122] The processes disclosed herein may be implemented in connection with procedures other than cholecystolithiasis removal, lung (pulmonary / trans-thoracic) tumor biopsies, and kidney stone removal procedures such as these. Generally, any type of percutaneous procedure may be performed by using an endoscope configured to capture image data for feature identification and tracking using neural network processing according to embodiments of the present disclosure. Additional examples include gastric surgery, esophageal and lung surgery, and the like. Further, the objects to be removed need not necessarily be urinary tract stones, and they may be any objects such as foreign bodies or objects generated within the human body.

[0123] Process 700 may be implemented to localize a target anatomical feature, at least in part based on determining an offset / translation of the scope from the target anatomical feature, so as to be implemented in connection with any of the embodiments disclosed herein. An electromagnetic sensor incorporated into the distal end of a ureteroscope may have any suitable or desirable form and / or configuration, such as one or more conductor coils, rings, cylinders, and / or the like, and local distortions in the broadcast electromagnetic field generated by such conductor elements may provide information regarding its location.

[0124] An electromagnetic positioning system, such as an electromagnetic field generator and one or more electromagnetic sensors / beacons, may be used to track the current position of the papilla and facilitate real-time targeting of the papilla with a percutaneous access instrument (e.g., a needle). For example, the targeted position of the papilla may be updated in real time based on electromagnetic sensor data, such as real-time electromagnetic sensor data associated with one or more sensors / beacons associated with the distal end of the endoscope. In some implementations, the real-time tracking location of the target anatomical feature may be determined based on the position and / or orientation of the scope, even in the absence of real-time visual confirmation and / or other image data associated with the scope and the target anatomical feature.

[0125] Translation of the Endoscope and the Target Feature Position Figure 8 shows a scope device 840 disposed within a target cup 812 for target localization according to one or more embodiments. A particular process may be implemented to determine and / or maintain a known offset P between a recorded papilla contact position 801 and a current position of a distal end 847 of the endoscope 840. offset The position sensor / beacon of the scope 840 may be configured to provide sensor data indicative of 5 or 6 degrees-of-freedom (DOF) with respect to the position of the scope 840. For example, a coil or other type of sensor device may have a cylindrical shape, or any other shape that allows three-dimensional position data as well as yaw and / or pitch data. In some embodiments, as in embodiments that include a 5DOF sensor, the position sensor / beacon does not provide roll information. In some embodiments, multiple 5DOF sensors may be used / arranged in relative axial angles with respect to each other, and the composite data provided / generated based on such position sensors / beacons may define a plane that can be used to construct a 6DOF that provides scope roll information.

[0126] In some implementations, a breath hold may be performed on the patient during at least a portion of the scope offset determination / maintenance process, thereby eliminating the need to account for anatomical movements associated with the lung cycle and allowing such an operation to be performed. For example, the patient may be the subject of a breath hold at least during the tagging and retraction portions of the process. In some implementations, any anatomical movement experienced after the offset determination may be considered to similarly affect the stationary endoscope and the target anatomical feature (e.g., the papilla), but the determined translation P offset may not need to be updated in real time, and thus the transformation / translation between the two positions may be considered to be independent of anatomical movement and / or other factors and substantially constant.

[0127] The relative transformation P between the stationary scope end 847 and the previously recorded papilla contact position 801 offsetThe determination can be made using precise electromagnetic position sensor data or can be made using the image processing described herein. For example, calibration of the scope camera with respect to the electromagnetic field space can enable a visual determination of the distance and / or positional change between the contact position 801 and the retracted position 847.

[0128] The retraction path 808 between the scope position 841 and the scope position 840 may or may not be linear. In some implementations, such as implementations related to the lower pole target cup / papilla, the retraction path may be at least partially arcuate. Thus, the translation P offset can be determined not only with respect to the straight-line distance, but can incorporate the orientation of the scope and / or other position-related parameters. Thus, in some implementations, the translation P offset can be regarded as a six-degree-of-freedom translation / transformation. Such a translation determination at least partially accounts for cases where the target cup 812 and / or the associated funnel may have a central axis / center of gravity that does not necessarily coincide with the retraction path taken by the scope 840. Thus, a translation with respect to six or more degrees of freedom may be desirable to generate a mapping translation / transformation that accurately represents the position offset P offset between the position 801 and the position 847.

[0129] In some implementations, certain image data can be collected and used to identify target anatomical features. For example, the systems, devices, and methods of the present disclosure can provide for the identification of target anatomical features within real-time endoscopic images, and the identification of the target anatomical features within the image can prompt a specific response action. For example, a control circuit communicatively coupled to a robotic endoscopy and / or percutaneous access device can be configured to track the movement of the target feature and take measures such as articulating one or more portions of the endoscope (e.g., the distal end 847), or adjusting the target position data. For example, the control circuit can be configured to articulate the endoscope so as to center the target position / point at or near the center of the field of view of the endoscope camera and / or the image field, and / or maintain a desired positional offset (e.g., P offset ) between the scope and the target anatomical feature.

[0130] By utilizing robot-assisted percutaneous access, a physician may be able to perform operative target access and treatment. Further, percutaneous access can be further assisted by utilizing automatic target identification and tracking according to aspects of the present disclosure described in more detail below, which can be relied upon to accurately maintain the target position for percutaneous access guidance. Percutaneous access guided by scope-corresponding target tracking according to aspects of the present disclosure can be relatively less skill-intensive. In some implementations, a single operator or robotic system can perform the process. Further, the need for fluoroscopy can be eliminated.

[0131] Scope Offset / Dwell Induction As described above, the features of the invention of the present disclosure can be used for endoscopic / ureteroscopic-based targeting for percutaneous renal access, and a ureteroscope equipped with an electromagnetic beacon / sensor and / or a percutaneous access instrument (e.g., a needle) equipped with an electromagnetic beacon / sensor are used to obtain percutaneous renal access into the target cup through the target papilla. Regarding the scope targeting technique, the effectiveness of percutaneous access to the target location / site, such as access to the target cup where the scope is stationed, can depend on where the physician / technician stations the distal end of the scope relative to the target cup / papilla. Specifically, as described according to an embodiment of the present disclosure, the stationing of the distal end of the endoscope / ureteroscope can be related to the position and / or the leading orientation of the distal end of the ureteroscope relative to the target papilla where percutaneous access can be performed. To provide further context for the various scope stationing features disclosed herein, a specific surgical system and environment are described below with respect to endoscope stationing.

[0132] FIG. 9 illustrates a robotic medical system 900 arranged to facilitate navigation of a scope within a patient according to one or more embodiments. For example, a physician 5 can connect a scope 92 to a robotic arm 12(c) of a robotic system 10 and / or position the scope 92 at least partially within a medical instrument (e.g., a catheter / sheath) and / or within a patient 13. The scope 92 can be connected to the robotic arm 112(c) at any point in time, such as before or during the procedure (e.g., after positioning the robotic system 10). The physician 5 can then interact with a control system 50, such as an I / O device 46, to navigate the scope 92 within the patient 13. For example, the physician 5 can provide an input via the I / O device 46 to control the robotic arm 112(c) to navigate the scope 92 through the urethra 65, bladder 60, ureter 63 to the kidney 70.

[0133] As shown, the control system 50 can assist the physician 5 during control of the scope 92 by presenting, via the display 42, a screen 910 that includes one or more scope guidance interfaces 950 for viewing a real-time camera image / view 952 captured by the scope 92. The physician 5 can navigate the scope 92 to identify, for example, the location of kidney stones, target anatomical features, and / or the like. In some embodiments, the control system 50 can be configured to perform certain location identification techniques to determine the position and / or orientation of the scope 92, which can also be viewed by the physician 5 through the display 142 and assist during control of the scope 92. Further, in some embodiments, other types of information, such as an x-ray image of the internal anatomical structure of the patient 13, can be presented through the display 42 to assist the physician 5 during control of the scope 92.

[0134] The physician 5 can use the control device 46 to drive the scope 92 to find / identify a targeted kidney stone 908 or other artifact for removal / treatment. The physician 5 can further drive the scope to localize the target papilla and occupy a desired dwell position. Such driving of the scope can be at least partially guided by the scope dwell guidance feature 954 of the scope guidance interface 950, which is described in detail below.

[0135] Figures 10A and 10B show renal anatomical structures with a ureteroscope stationed at various positions according to one or more embodiments. FIG. 10A shows a ureteroscope 1040 disposed within the ureter 63, renal pelvis 71, and / or calyx (e.g., major calyx and / or minor calyx) of the kidney 1010. For example, an operator / physician can drive the scope 1040 into the calyx 1012 and use an electromagnetic beacon associated with the distal end / tip of the scope 1040 as a target to which a percutaneous access device (e.g., a needle) can be directed. In some embodiments, the scope 1040 is used to record the position of the target papilla 1079, and then the scope 1040 is retracted to some extent and stationed at a distance from the papilla 1079 to provide a desired position of the scope for visualizing percutaneous access using the camera of the scope 1040. In some implementations, the target position targeted by a percutaneous access device (not shown) can be determined based on a known offset distance, orientation, and / or position of the scope 1040 relative to the target papilla 1079 or other anatomical features. Thus, the functions disclosed herein can facilitate proper / desired stationing of a scope device that can be implemented and relied upon for dynamic targeting purposes as described herein.

[0136] Regarding percutaneous access to the calyx, such as may be performed to reach / treat a kidney stone, access through the renal pyramid 1057 / papilla 1079 may be necessary or desirable to access the target calyx 1012 without excessive bleeding. Further, access through the papilla 1079 can provide complete access to the calyx network of the kidney 1010.

[0137] The target cup 1012 surrounds the papilla 1079 (i.e., the renal pyramid, shown in the dashed form in FIGS. 10A and 10B for clarity), through which appropriate percutaneous access to the target cup 1012 can be obtained. Generally, one or more points at or near the distal tip / end of the scope 1040 can be used as targets for percutaneous access. For example, as described above, the distal tip / end of the scope 1040 can have one or more electromagnetic sensors or beacons associated therewith to determine its position / orientation within the electromagnetic field. The figure of FIG. 10A shows three different exemplary dwell positions (1042, 1044, 1046) of the distal end of the scope 1040. FIG. 10A further shows respective coaxial trajectories (1002, 1004, 1006) associated with each of the dwell positions of the scope. Such trajectories can be determined based on the derived position and / or orientation / alignment information associated with each of the dwell positions of the scope and can represent possible paths along which percutaneous renal access can be induced / achieved.

[0138] FIG. 10A shows a dwell position 1046 where the distal end of the scope 1040 is generally aligned with the central axis 1006 of the target cup 1012 and / or the associated funnel. FIG. 10A further shows another dwell position 1042 where the distal end of the scope 1040 is positioned at a distance d2 unnecessarily far from the papilla 1079 and / or is not aligned with the central axis of the target cup 1012 and / or the associated funnel. For example, as shown as trajectory 1002, the dwell position 1042 can generally generate a target trajectory 1002 that does not center on the papilla 1079 and / or the associated renal complex. FIG. 10 further shows another dwell position 1044 that is not aligned with the axis of the target cup 1012, the papilla 1079, and / or the associated funnel.

[0139] Figure 10B shows the minimum percutaneous access tube paths corresponding to each of the illustrated exemplary scope dwell positions. Specifically, FIG. 10 shows the minimum tube path 1001 associated with the dwell position 1042, where the scope 1040 is dwelled unnecessarily far from and / or not aligned with the target papilla 1079. FIG. 10B further shows the minimum tube path 1005 associated with the misaligned dwell position 1044. The access path 1003 associated with the dwell position 1046 can generally be aligned with the central region of the cup 1012. Based on the access paths shown in FIGS. 10A and 10B, the dwell position 1046 can be considered a suitable or desirable dwell position for visualizing and / or accessing the cup 1012.

[0140] Certain embodiments of the present disclosure advantageously provide various mechanisms and means for estimating or determining the position of the target cup / papilla, as well as certain visualization mechanisms / means, to facilitate guiding the operator to correctly place the ureteroscope in the target cup at a position, alignment, and / or orientation effective for percutaneous access targeting. Embodiments of the present disclosure advantageously provide certain visual guidance / support features and / or functions to facilitate effective placement of an endoscope at a target site, such as placement of the scope in or near a target cup within a patient's kidney.

[0141] Figures 11, 12A, 12B, 14-1, and 14-2 show certain scope guidance features that can be used to guide an operator / physician in positioning / docking an endoscope at a desired location with respect to a target papilla. Such features can be generated and / or displayed in any suitable or desired manner or form. For example, one or more markers / icons can be generated and / or displayed in or near a camera view presented on a display device associated with a robotic control system, and such markers / icons indicate to the physician / operator how to manipulate or activate the endoscope to position the distal end of the endoscope at a suitable or most effective docking position. The scope docking guidance features can, for example, instruct the user / operator to align the ureteroscope with the target papilla such that the position of the distal end of the ureteroscope can be substantially targeted using a percutaneous access device (e.g., a needle).

[0142] Figure 11 shows an exemplary image 1101 from a camera associated with the distal end of an endoscope, such as a ureteroscope, according to one or more embodiments of the present disclosure. Image 1101 can be presented on a scope guidance interface as shown in FIG. 9 and described above. Image 1101 can represent a window or a portion of a window (e.g., a sub-window) configured to display a camera view that includes at least a portion of the field of view of a camera associated with the endoscope.

[0143] In some embodiments, the camera view interface / window 1101 may display one or more icons or features 1150 on top of it that indicate the target position of the target anatomical feature within the camera view 1101. The target icon / feature 1150 can help guide the operator regarding proper alignment with the target anatomical feature (e.g., the papilla) at the distal end of the endoscope. In some embodiments, the target anatomical feature alignment guidance icon / feature 1150 can be generated and / or displayed such that the icon / feature 1150 remains in approximately the same position within the window 1101 even as the camera image displayed in the window 1101 moves or changes as the scope is moved. That is, the operator of the scope can manipulate / actuate the distal end of the scope, thereby changing the camera's field of view and / or the representative anatomical structures captured within that field of view. As the camera's field of view 1101 changes, the icon / feature 1150 can remain in the same representation, overlaid at or near the center of the window / image 1101, as shown in FIG. 11. Thus, the operator can move the camera's field of view, such as by changing the position of the scope, to selectively position the icon / feature 1150 over the target anatomical feature of interest, such as the papilla or other anatomical feature.

[0144] The integrated icon / feature 1150 may instruct the operator to position the field of view of the camera so as to position and / or center the icon / feature 1150 over the target anatomical feature 1116 (e.g., nipple). For example, as shown in image 1103 representing the changed field of view of the camera associated with the scope after movement of the scope by the operator, the icon / feature 1150 may be used to guide the operator to center the icon / feature 1150 over the target nipple 1116, while in the initial field of view 1101, the target nipple 1116 is generally not centered within the field of view of the camera or with respect to the targeting icon / feature 1150. To change the field of view to center the icon / feature 1150 as shown in image 1103, the distal end of the endoscope and / or the associated camera, as represented by icon 1140, may be panned left, for example, as shown in image 1103 compared to image 1101, to bring the target nipple / feature 1116 into the field of view of the camera.

[0145] The icon / feature 1150 may have any suitable or desirable shape, form, configuration, and / or number of visual features. For example, some embodiments, such as the embodiment shown in FIG. 11, include a cross-shaped feature or other feature oriented on two or more axes that indicates a center point defined by the intersection of extrapolated lines. The illustrated icon / feature 1150 may additionally or alternatively include one or more circular forms, as illustrated, the central axis of which represents the center point of the icon / feature 1150. Generally, the icon / feature 1150 indicates that the operator should generally align the center point of the icon / feature 1152 with the center (e.g., volumetric and / or geometric center) of the target anatomical feature 1116.

[0146] The icon / feature 1150 shown in FIG. 11 can facilitate proper alignment of the endoscope with respect to the target anatomical feature. In addition to (or as an alternative to) alignment guidance, embodiments of the present disclosure relate to endoscope docking guidance features that facilitate proper positioning of the endoscope with respect to its distance / depth from the target anatomical feature (e.g., the papilla 1116). In some embodiments, certain icons / features are generated and / or displayed over the camera view, and the relative size of this icon / feature compared to the size of the representation of the target anatomical feature within the camera view indicates whether the scope is positioned at the appropriate distance / depth and / or how the scope should be moved to reach the target anatomical feature at the appropriate distance / depth. Further, in some embodiments, the depth positioning icon / feature can further include an icon / feature that also indicates alignment guidance.

[0147] FIGS. 12A and 12B show a configuration of a scope camera view / window including a target boundary feature 1260 according to one or more embodiments, and the target boundary feature 1260 guides the operator with respect to the appropriate depth / offset position of the scope. FIGS. 12A and 12B show exemplary images 1201a, 1201b from a camera associated with the distal end of an endoscope, such as a ureteroscope, according to one or more embodiments of the present disclosure. The images 1201a, 1201b are shown in FIG. 9 and can be presented on the scope guidance interface as described above. The images 1201a, 1201b can represent a window or a portion of a window (e.g., a sub-window) configured to display a camera view including at least a portion of the field of view of the camera associated with the endoscope.

[0148] In some embodiments, the camera view interfaces / windows 1201a, 1201b may display one or more icons or features 1260 indicating the target depth of the target anatomical feature 1216 overlaid thereon within the camera views 1201a, 1201b. The offset icon / feature 1260 can include, for example, one or more brackets, bounding boxes, and / or other shapes or features that represent containing the image content displayed between or within such icons / features. The bounding box or other similar feature may be displayed at or near the center of the camera's field of view and / or the window showing the camera view, and such feature can indicate or instruct the operator to maintain all or at least a portion (e.g., at least the central portion) of the target anatomical feature within the bounding box form / feature so that the endoscope does not dwell in a position too close to the target anatomical feature 1216. For example, in some embodiments, the bounding box feature 1260 can indicate or instruct the operator that the operator should maintain at least a majority of the area of the anatomical feature 1216 represented in the camera image within the boundaries defined by and / or indicated by the offset guidance feature 1260 with respect to one or more axes (e.g., horizontally and / or vertically). In addition to reducing the likelihood that the endoscope will dwell in a position too close to the target anatomical feature portion 1216, by instructing the operator to substantially fill the boundary region of the offset feature / icon 1260 with the target anatomical feature, or by instructing the operator to maintain the camera in a position such that most of the area surrounded by the offset feature / icon 1260 is filled / covered by a representative image of the target anatomical feature (e.g., the papilla), the offset feature / icon 1260 will be able to reduce the likelihood that the endoscope will dwell in a position too far from the target anatomical feature 1216.

[0149] Regarding the images 1201a, 1201b, and 1203 shown in FIGS. 12A and 12B, the images 1201a, 1201b show an offset feature / icon 1260 that is configured and / or located substantially at the center of the windows 1201a, 1201b. As shown, in image 1201a, the target anatomical feature (e.g., nipple) 1216 does not fully fill the extent of the offset future 1260. The subsequent image 1203 can result from an endoscopic operation that brings the endoscope and / or associated camera closer to the target anatomical feature 1216, and thus the resulting image 1203 shows the target anatomical feature 1216 that substantially fills the extent of the offset feature / icon 1260.

[0150] Regarding FIG. 12B, the image 1201b shows an offset feature / icon 1260 where the representation of the target anatomical feature 1216 extends substantially outside the extent of the offset feature 1260 because the endoscopic camera is too close to the target anatomical feature 1216, and thus the target anatomical feature 1216 may obstruct the visibility of the target anatomical feature 1216 when it is deformed as a result of advancement inside and / or through it by a percutaneous access device (e.g., a needle).

[0151] The subsequent image 1203 can result from an endoscopic operation that pulls the endoscope away from the target anatomical feature 1216, and thus most of the target anatomical feature 1216, or its associated portion, is within the extent of the offset feature / icon 1260 with respect to one or both dimensions / axes of the image 1204.

[0152] In certain embodiments, as shown in FIGS. 12A and 12B, the boundary box 1260 is displayed at or near the center of the view windows (1201a, 1201b, 1203), thereby instructing / commanding the user to maintain the view representation of the target anatomical feature 1216 (e.g., nipple) entirely or at least partially (e.g., more than half) within the boundaries of the boundary box 1260, thereby advantageously reducing the event and / or the possibility that the scope dwells at a position too far from the scope position 1042 as in the image 1201 and / or as shown in FIGS. 10A and 10B, or at a position too close to the scope position 1044 as in the image 1201 of FIG. 12B and / or as shown in FIGS. 10A and 10B.

[0153] The boundary box feature 1260 can advantageously facilitate the standardization of how different users / physicians operate the scope for different sizes and different patient target anatomical features, thereby achieving a preferred position of the scope and reducing the dependence on the skill and / or experience of the physician and / or the specific anatomical structure of the patient. In some embodiments, the scope management system includes a device configured to at least partially perform an automatic positioning of the scope to position and / or maintain the target anatomical feature at least partially within the boundaries of the boundary box feature. Thus, anatomical movements that result in a relative displacement of the target anatomical feature with respect to the camera of the scope can be automatically compensated for by using image processing functions and / or other means or mechanisms for automatically determining the relative position of the target anatomical feature with respect to the boundary box feature.

[0154] In some embodiments, the size and / or other characteristics of the boundary box feature 1260 may at least partially depend on the specific anatomical specifications of the patient. For example, a scope positioned within a relatively small cup may desirably be positioned relatively close to the target papilla to reduce the risk of sliding / movement of the scope. For example, a guidance feature (e.g., a boundary box feature) overlaid / displayed on the endoscopic view window may have a fixed size (e.g., pixel size). This may enable the user to be induced to dwell closer for a smaller cup. That is, it may be necessary to dwell the scope relatively close to enclose a relatively smaller papilla within the boundary, offset guidance icon. When the cup is small, the scope may tend to deviate from the cup, and thus closer dwelling is desirable. Accordingly, even when the size of the target cup / papilla is unknown, by implementing a fixed-size target / boundary icon / feature and instructing the user to dwell the scope such that the icon / feature is substantially (e.g., 50%, 60%, 75%, 80%, 90%, or other percentage) filled with the target anatomical feature / region, an adaptive dwelling distance may be enabled for different cups. It should be understood that such an example is one of many possible examples where the characteristics of the boundary box feature are generated and / or presented at least partially depending on the specific determined characteristics of the patient's anatomical structure and / or parameters manually entered by the technician. The various offset guidance features described above with respect to FIGS. 12A and 12B can advantageously suppress / limit potential errors in percutaneous access targeting by controlling the distance and / or position range of the scope relative to the target anatomical feature.

[0155] FIG. 13 illustrates a robotic medical system arranged to facilitate percutaneous access to a patient's renal anatomy according to one or more embodiments. As shown in FIG. 13, a physician 5 can perform percutaneous access by positioning a needle 17 for insertion into a target location. In some embodiments, the physician 5 can use his or her best judgment based on knowledge of the patient 13's anatomy, previous experience performing the procedure, analysis of CT / X-ray images, or other pre-operative information of the patient 13 to place the needle 17 in the patient 13 at the incision site. Further, in some embodiments, a control system 50 can provide information regarding where to place the needle 17 in the patient 13. The physician 5 can attempt to avoid important anatomical structures of the patient 13 such as the lungs, pleura, colon, paravertebral muscles, ribs, intercostal nerves, etc. In some examples, the control system 50 can use CT / X-ray / ultrasonic images to provide information regarding where to place the needle 17 in the patient 13.

[0156] The control system 50 can include a control circuit configured to determine a target trajectory 902 for inserting the needle 17 to assist the physician 5 in reaching a target location (i.e., the nipple 914). The target trajectory 902 can represent a desired path for accessing the target location. The target trajectory 902 can be determined based on the position of one or more medical instruments (e.g., the needle 17, the scope 92, etc.), the target location within the patient's anatomy, the position and / or orientation of the patient 13, and the patient's anatomy (e.g., the location of the organs within the patient relative to the target location). In some implementations, the target trajectory 902 represents a straight line passing through the points of the nipple 914 and the needle 17. The trajectory can generally coincide with the axis of the funnel associated with the target cup. However, the target trajectory 502 can take other forms such as a curved path and / or can be otherwise defined. In some examples, the needle 17 is implemented as at least a partially flexible needle and / or a needle with a tilted tip. The control system 50 can be configured to provide information for guiding the user / physician 5 when advancing the needle, such as to compensate for a deviation of the needle trajectory or to keep the user on the target trajectory.

[0157] The example of FIG. 13 illustrates a target trajectory 902 that extends coaxially through the papilla 914, but the target trajectory 902 can have other positions, angles, and / or configurations. For example, the target trajectory can be implemented with a lower pole access point such as passing through a papilla located below the kidney stone 908 shown in FIG. 13, and can have a non-coaxial angle through the papilla, which can be used to avoid the hip joint. In some implementations, the minimum cannula path / trajectory is taken when accessing the target cup / papilla.

[0158] As described above, the control system 50 can assist the physician 5 during control of the scope 92 by presenting, via the display 42, a screen 910 that includes one or more scope guidance interfaces 950 for viewing the real-time camera image / view 952 captured by the scope 92. As shown in FIG. 13, during percutaneous access, the scope 92 can be stationed within the target cup and provide visibility of the target papilla 914. As the needle 17 advances in the direction of the target papilla 914, the adjustment of the scope 92 can be guided, at least in part, by one or more needle trajectory alignment / position guidance features 956. This will be described in detail below. Such features 956 can facilitate positioning of the scope 92 by including an icon in the field of view 952 of the scope camera that indicates the predicted representation of the needle 17 within the target cup (e.g., the predicted needle tip position in the scope camera image window). For example, if the predicted needle entry point into the target cup and / or through the target papilla 914 is outside the field of view 952 of the scope camera, the interface feature 956 can notify the physician 5 of the direction in which the scope 92 can be moved to bring the predicted needle entry point within the camera view 952.

[0159] When the needle 17 reaches the target location, the physician 5 can insert another medical instrument such as a catheter, a vacuum, a ureteroscope, etc. into the path created by the needle 17 and / or over the needle 17 and / or dilator (not shown) disposed in the access path. The physician 5 can use other medical instruments and / or the scope 92 to fragment and remove fragments of the kidney stone from the kidney 70.

[0160] In some embodiments, the position of the medical device can be represented by a point / point set, and / or the orientation of the medical device can be represented as an angle / offset relative to an axis / plane. For example, the position of the medical device can be represented by the coordinates of a point / point set within a coordinate system (e.g., one or more X, Y, Z coordinates), and / or the orientation of the medical device can be represented as an angle relative to the axis / plane of the coordinate system (e.g., an angle relative to the X-axis / plane, Y-axis / plane, and / or Z-axis / plane). Here, a change in the orientation of the medical device can correspond to a change in the angle of the medical device relative to the axis / plane. Further, in some embodiments, the orientation of the medical device is represented by yaw, pitch, and / or roll information.

[0161] In some embodiments, the trajectory refers to the pose. For example, the trajectory of a medical device can refer to the pose of the medical device, including / suggesting both the position and orientation of the medical device. Similarly, a target trajectory can refer to a target pose including / suggesting both the position and orientation of a desired path. However, in other embodiments, the trajectory refers to either the orientation or the position.

[0162] In some embodiments, the ureteroscope 92 and the needle or other percutaneous access device 17 have a common positioning coordinate system. This can be, for example, an implementation where an electromagnetic field generator 18 is utilized and each of the scope 92 and the needle 17 has a respective associated electromagnetic sensor / beacon that can be used to determine the position of each of the scope 92 and the needle 17 within a common electromagnetic field space. For example, prior to a medical procedure involving the use of an endoscope and / or a percutaneous access needle, the sensor / beacon associated with the distal end of the scope can be calibrated with the scope's camera to determine the conversion / relationship between the visual image space of the camera and the position within the electromagnetic field associated with the image / features depicted and / or otherwise appearing in the image area of the camera. Such calibration can enable the determination and / or prediction of position information based on the position, size, and / or configuration / orientation of features / object representations within the image space (e.g., the field of view) of the camera. Thus, when the position of the percutaneous needle is determined and / or predicted relative to the position of the camera and / or the associated sensor / beacon as indicated by the associated electromagnetic sensor / beacon, such position can be determined / generated and / or displayed on the field of view image displayed on the scope camera view interface 952, at least in part based on the calibration of the camera as described above. For example, as will be described in more detail below, camera calibration can rely on overlaying / presenting a needle trajectory icon / feature in the camera view window, and such icon / feature indicates the predicted / predicted position within / by the field of view where the percutaneous needle is expected to appear / predictably appear as the needle approaches the target.

[0163] The scope guidance interface 950 can be configured to show certain needle trajectory alignment / position guidance features 956 that can be presented in relation to (e.g., overlaid or adjacent to) the scope camera view / window 952. For example, the needle trajectory alignment / position guidance features 956 can be displayed as part of the interface 950 and include one or more icons or features indicating the position relative to the camera view 952 where a percutaneous access device (e.g., a needle) is expected to enter the field of view of the camera and / or pierce the target anatomical feature / nipple. Such a needle trajectory alignment / position guidance mechanism can facilitate the operator of the scope in maintaining the predicted entry point of the percutaneous access device within the field of view 952 of the camera. The features 956 indicating the predicted position of the percutaneous access device can have any suitable or desirable shape, form, and / or representation, similar to the other interface icons / features disclosed herein.

[0164] Figures 14-1 and 14-2 illustrate the configuration of a scope camera view / window including one or more needle trajectory features according to one or more embodiments. Figure 14-1 shows an exemplary image 1401 from a camera associated with the distal end of an endoscope, such as a ureteroscope, according to one or more embodiments of the present disclosure. The image 1401 can be presented to the needle trajectory scope guidance interface as shown in Figure 13 and described above. The image 1401 can represent a window or a portion of a window (e.g., a sub-window) configured to display a camera view including at least a portion of the field of view of a camera associated with the endoscope (e.g., a ureteroscope).

[0165] The position of the target anatomical feature (e.g., nipple) 1416 is tracked / located and can be used as a target for percutaneous access to other regions (e.g., cup) where the cell duct or endoscope is disposed / stationed. For example, in some implementations, the position of the target anatomical feature 1416 (e.g., nipple) can be determined at least in part based on one or more of the size, shape, and / or configuration / orientation of the target anatomical feature 1416 within the image / field of view of the camera, the recorded contact position of the nipple, which is by an implementation involving contacting the nipple with the distal end of the endoscope and recording position information related to the endoscope in relation to such contact, and can provide position information based on the calibration of the camera. For embodiments relying on camera calibration information, the relative size, shape, and / or orientation / configuration of the target anatomical feature, compared to its visual characteristics at a later point in time after the scope has been retracted / pulled out to some extent from contact with the target anatomical feature 1416, can provide information indicating the current position of the target anatomical feature 1416. Calibration of the camera image and the electromagnetic field space can be achieved and / or performed in any suitable or desirable manner. For example, in some implementations, a checkerboard pattern type, or other object / pattern of known shape, size, and / or configuration can be used to determine the position of the object representation within the camera image.

[0166] In some implementations, the determination of the translation between the three-dimensional electromagnetic space positioning and the position / configuration in the camera space can be utilized to generate / display needle trajectory icons / features / features in camera image windows such as the camera image windows 1401, 1403 of FIG. 14. For example, the current position and / or predicted position of the tip of a percutaneous needle (not shown) can be visualized / represented in the camera windows 1401, 1403 to provide the operator with directions regarding the position where the scope camera will capture the needle within its field of view when the needle punctures the anatomical feature 1416 and / or the surrounding anatomical structure. By presenting the needle trajectory alignment / position guidance feature / icon 956, the success and / or failure of the percutaneous needle access / puncture can be confirmed, facilitating access to the target anatomical site.

[0167] The representation of icons / features (e.g., 1470, 1475, 1480) indicating the position of the needle can be useful when the percutaneous needle passes through / through the target anatomical feature / tissue and the target anatomical feature / tissue is at least partially deformed, and in some cases, can cause visual interference. That is, as the needle advances closer, the visibility of the target anatomical feature can become more uncertain. Therefore, it can be difficult to determine the success of targeting with a percutaneous needle without using the needle trajectory features according to the aspects of the present disclosure.

[0168] Image 1401 shows an exemplary image and associated needle trajectory / position features 1470, 1475 in the context where the predicted needle position and / or the current needle position is outside the camera field of view 1401. In such a situation, the icons / features 1470, 1475 can be generated and / or displayed so as to indicate the direction and / or positioned with respect to the field of view 1401, where the predicted or current position of the needle is relative to the current field of view of image 1401. For example, an arrow or other icon 1470 can be displayed in a region outside the window 1401 indicating the direction / position outside the window 1401 related to the needle trajectory. Alternatively or additionally, one or more icons / features 1475 can be superimposed on image 1401 so as to indicate the direction of the field of view of image 1401 and / or the position outside the field of view of image 1401, and the needle position is predicted when the size of the field of view of image 1401 is large.

[0169] Image 1403 shows the camera view after a specific articulation / movement of the distal end of the scope and / or associated camera. For example, according to the illustrated needle trajectory (e.g., 1470 or 1475) in image 1401, the camera represented by icon 1440 can be panned in the direction of the illustrated needle trajectory to bring the predicted position of the needle into the camera's field of view. Image 1403 can further bring the needle trajectory icon / feature 1480 further into the camera's field of view and / or generally in the central direction thereof, reflecting a slight upward pitch of the scope / camera 1440. The icon / feature 1480 can represent or correspond to the position of the needle tip or can correspond to the predicted entry point.

[0170] In some embodiments, the needle trajectory icon 1480 can be configured to vary with respect to size, color, shape, and / or one or more other features / characteristics thereof, and can represent distance, alignment, and / or other positional characteristics of the tip of the needle. In some embodiments, as the needle tip approaches the scope / camera, the icon 1480 grows (or shrinks). In some embodiments, the icon 1480 can grow when the needle is relatively far away to represent the error range of the predicted needle tip position, while as the needle tip approaches the scope / camera, the certainty regarding the specific position where the needle will appear when piercing the target anatomical structure can increase, and thus, the area of the icon 1480 can shrink to reflect the area where the needle can appear. Generally, some embodiments can include an icon having specific visual characteristics that represent the three-dimensional alignment, orientation, shape, and / or position of the needle as represented in the two-dimensional image space of the camera. For example, other icons such as a conical icon or a shape / feature having a vertex or other directionality can be generated and / or presented to indicate depth perception with respect to the predicted or determined position of the needle. In some embodiments, specific shading features are incorporated into the needle trajectory icon / feature (e.g., 1470, 1475, and / or 1480) to indicate additional positioning, alignment, and / or orientation information regarding the needle.

[0171] The needle alignment / position guidance feature 956 can include a needle trajectory icon as represented in the image 1401 and / or 1403, and can further include a specific visual representation / interface that provides the forward direction of the percutaneous needle and directs the needle towards the tracked target anatomical feature. For example, one or more needle viewpoint interface features can be provided to indicate the alignment, orientation, and / or positioning of the needle and guide its advancement, while additional scope viewpoint interfaces and associated needle trajectory icons / features can be presented to provide guidance indicating additional confirmation and / or the positioning of the needle with respect to the scope's camera.

[0172] The size of the needle prediction icon 1480 can be changed / modified based on the determined needle prediction / prediction accuracy. If there are significant anatomical movements that can result in errors in needle prediction, the needle prediction icon may be presented in a relatively large size, indicating a relatively large, determined error with respect to the needle prediction / trajectory.

[0173] FIG. 14-2 shows an image 1405 that includes a needle trajectory prediction feature 1483 visually predicted from the needle prediction icon 1480. The feature 1483 may have specific visual characteristics that indicate a three-dimensional prediction of the needle along the path along which the movement of the needle 1475 is predicted. Image 1407 shows a prediction along the path indicated by the needle 1475 piercing the anatomical feature 1416 and the prediction feature 1483. That is, in image 1405, the needle tip is inside / behind the anatomical feature 1416, and image 1407 shows the needle tip 1475 after piercing the anatomical feature 1416 and entering the treatment site. Image 1405 includes both the tip location icon 1480 and an overlay 1483 on the needle shaft to indicate the needle posture. Image 1407 shows both an orientation indicator of the location where the tip is present and an overlay on the needle shaft to indicate the three-dimensional perception of the needle posture.

[0174] Estimation of the cup / funnel axis for nipple localization To provide nipple offset trajectory information for nipple localization / tracking, an estimation of the cup and / or funnel axis may be performed in connection with any of the embodiments disclosed herein. The axis of the cup and / or associated funnel can be estimated / determined in various ways. For example, cup mapping can be performed as a step of determining the axis of the cup / funnel, as detailed below.

[0175] For the purposes of percutaneous access needle targeting and / or target localization according to embodiments of the present disclosure, kidney mapping can be used to localize target papillae. In some implementations, the surface normal associated with the target cup can be estimated / determined using a kidney map that can be generated using any of the methods disclosed herein, and this surface normal can then be used to determine the cup / funnel axis. In some implementations, a percutaneous access path to the target anatomical site can be determined and / or aligned with the orientation of the target papilla / cup, which in some cases can generally coincide with the central axis of the associated funnel. In some implementations, the axis of the funnel associated with the target cup can be determined and / or estimated using the trajectory of the distal end of the endoscope leading to the target cup / papilla. For example, the path of the scope can be linearized in some way to generate an axis path / estimate. Even when the endoscope is not aligned with the target papilla / cup, aspects of the present disclosure provide a mechanism for determining the orientation of the target cup / papilla, which can be used as a target trajectory for percutaneous access and / or for target localization.

[0176] FIG. 15 shows a scope device 1550 disposed within a target renal anatomical structure for calyx mapping according to one or more embodiments. Referring to FIG. 15, mapping of the calyces of the kidney can involve advancing and / or manipulating the ureteroscope 1550 within the calyx network. For example, the ureteroscope 1550 can be advanced to the end points of one or more calyces (e.g., at each papilla) and / or against the walls or other boundaries of the calyces, and perform periodic or sporadic position tagging / recording 1501-1504 to generate a skeletal mapping of the cross-sectional area of the calyx. For example, the operator of the ureteroscope 1540 can be instructed to articulate the scope to tag a plurality of points 1501-1504 inside the target calyx. Such points 1501-1504 can be used to construct the shape of the calyx, and the center of the constructed calyx can be used as a target for percutaneous access and / or can be traced when the current scope position is translated by an offset distance / position to localize the papilla. In some implementations, the scope 1550 can articulate in an arcing motion, as shown, to trace the contour of the cross-sectional area / to cover the cross-sectional area.

[0177] In some implementations, calyx mapping can be used to determine the axis and / or orientation of the infundibulum. For example, substantially all of the calyx network of the kidney, or a subset of the calyx network, can be mapped. In some implementations, only the target calyx and / or associated infundibulum are mapped to determine the orientation / axis of the calyx and / or infundibulum. Localization of the papilla targeting trajectory and / or papilla offset can be determined based at least in part on the determined calyx / infundibulum axis and / or other local topologies associated with the target calyx / papilla. For example, as detailed above with respect to FIG. 8, localization of the target papilla is based on a papilla-scope offset translation (e.g., P offset) can be achieved at least in part by performing and applying a translation / offset to the current position of the endoscope (e.g., within an electromagnetic field space). The determination of the scope-papilla offset / translation can be based at least in part on the axis of the cup and / or the associated funnel. That is, the determination of the trajectory pointed to by the target cup can be used to determine the offset of the target papilla from the distal end of the scope. For example, the determined offset of the scope can be added to the current position of the scope in the direction of the orientation / axis of the cup or funnel to determine the target position. In some cases, the target cup can be oriented in generally the same direction / orientation as the associated funnel. However, with respect to fused cups, which may be more common generally at the lower pole of the kidney and / or the cup network, the orientation of the cup can be relied upon regardless of the funnel axis.

[0178] While driving the scope towards the target papilla / cup, a plurality of positions associated with the distal end of the scope can be tagged / recorded along the path to provide a scatter plot type mapping of at least a portion of the cup network (e.g., the target cup and / or the associated funnel). For example, FIG. 16 shows a map 1600 of a cup network including a plurality of cups (both major and minor cups) communicating from the ureter 1663. The various cup axes 1615 shown in FIG. 16 can be determined based on the recorded scope positions along the movement path of the scope (not shown). Specifically, the individual illustrated circles (e.g., 1610) can represent the tagged / recorded positions of the scope. In some implementations, Gaussian interpolation can be performed along the reported trajectory or a portion thereof within the target cup 1612 and / or the associated funnel 1613, and such interpolation is used to generate a surface estimate for at least a portion of the cup network.

[0179] In some embodiments, a surface and / or volume estimation algorithm / mechanism may be implemented to determine the center / axis of the target cup and / or associated funnel. FIG. 17 shows a generated three-dimensional surface map 1700 generated in any suitable manner, such as using Gaussian interpolation. For example, when the cup and / or funnel surface is determined / generated, the cup / funnel axis 1715 may be determined based at least in part on the estimated surface. In some embodiments, a plurality of surface normals 1701 may be determined for various regions of the surface 1705, and the surface normals 1701 are vectors / trajectories that are substantially orthogonal to the surface 1705 at the origin associated with each surface normal 1701. The internal axis of the funnel / cup may be determined based at least in part on the averaging of surface normals extending around and / or on both sides of the morphology / structure 1700. For example, the surface normal vectors may be summed to determine a central axis for each portion of the cup network 1700. Such an axis can be extrapolated to provide the cup trajectory 1715. In some embodiments, the determination of the cup trajectory may be based at least in part on the summation of surface normal vectors and / or the determination of the cup / funnel axis.

[0180] In some embodiments, as shown in FIG. 15, circular motion may occur within the scope to generate a sufficient position plot of the target cup from which a cup map is formed. The orientation 1715 of the cup may be estimated from the cup map 1700, and such a trajectory 1715 can provide a path along which the offset of the scope / papilla can be predicted / determined. This may make it possible to more reliably update the target position when the tip of the scope changes relative to the cup, funnel, and / or papilla axis due to tissue deformation during needle insertion. Depending on the particular procedure, it may not be necessary to map the entire cup network of the kidney for the purpose of percutaneous access targeting. For example, in some implementations, only the cup associated with the target papilla may be mapped. In some implementations, multiple cups may be mapped to provide increased spatial information / mapping.

[0181] Embodiments of the present disclosure can eliminate the need to acquire one or more fluoroscopic images of the collection system at different angles to determine the location and orientation of a target papilla / cup, as compared to solutions that perform fluoroscopy to find the position of a target cup for targeting purposes. Further, embodiments of the present disclosure can advantageously provide targeting of target anatomical features without the need for retrograde injection of a contrast agent or air to visualize the target anatomical features. Thus, embodiments of the present disclosure can be performed without the need for conventional cystoscopy and / or ureteral catheterization. As compared to solutions that perform ultrasound examinations for the purpose of determining the determination of a target cup for targeting purposes, ultrasound examinations can show only two-dimensional images of the collection system and / or can provide relatively limited visualization of the ureter to the scope and / or percutaneous access needle.

[0182] Vision-based Target Localization In some implementations, a vision-based process, such as a process performed while the scope is being driven towards the target cup / papilla, can be performed to determine the direction / axis of the funnel / cup. For example, multiple camera images and EM-based position data regarding the endoscope can be obtained in a clear position / orientation relative to the target papilla, and the three-dimensional (3D) location of the papilla with respect to the scope can be determined based on such data.

[0183] The localization of a target anatomical feature (e.g., a target nipple) can be achieved using any suitable image processing mechanism / function. For example, the control circuitry of a medical system can receive image data from a scope camera and perform certain image processing processes / functions therein to identify the target anatomical feature. In some implementations, at least two separate images of the target anatomical feature are processed to track its location. For example, as shown in FIG. 18, an image 1801 including a target anatomical feature 1806 can be captured from a first viewpoint / position 1807 of a scope camera, and a second image 1802 can be captured by the scope camera either before or after the capture of the first image 1801, and the second image 1802 is captured from a different viewpoint / position 1808 of the scope camera. In some implementations, a Structure-from-Motion technique can be implemented to determine the three-dimensional position of the target anatomical feature 1806 based at least in part on the images 1801, 1802. The images 1801, 1802 can represent masked or otherwise analyzed / processed feature depiction images, and the target anatomical feature 1806 is defined as shown in FIG. 18. Any computer vision algorithm or process can be implemented in connection with the localization and targeting processes disclosed herein. For example, such an image processing process can be a relatively sophisticated contour detection process, a black box machine learning process, and / or the like.

[0184] Three-dimensional (3D) position estimation for the purpose of localizing a target anatomical feature according to aspects of the present disclosure can be performed according to any suitable or desirable technique or mechanism. For example, in some implementations, the distance between an endoscope camera and a target anatomical feature can be estimated based on the representative size of the anatomical feature 1806 in the image.

[0185] In some embodiments, information regarding the movement of the scope and / or the angle of anatomical features can be used to determine the 3D position. For example, electromagnetic sensors / beacons within an electromagnetic field / space can provide such movement angle information. By combining electromagnetic sensor data with image data, using the mapping between the distance from the target anatomical feature and the size of the target anatomical feature in the resulting image captured after such movement of the distance, the depth / distance of the feature in subsequent images can be estimated. In some embodiments, when contacting a target anatomical feature (e.g., the nipple), retracting the scope away from such feature, and stationing the scope at a position that provides a desired field of view, the movement distance can be recorded using, for example, electromagnetic sensor data. Further, subsequent images can provide information regarding how large anatomical features appear in such images, and thus, the relationship / mapping between feature size and distance can be determined and used for future positioning. The camera calibration described herein can be implemented for such purposes. In some implementations, machine learning can be utilized to classify images and determine position information based on the size of features within such images.

[0186] In some embodiments, certain sensors associated with a medical instrument (e.g., a scope) can be utilized to obtain the 3D location of a target. For example, a structured light sensor and / or a time-of-flight sensor can be used in the determination of 3D positioning. According to some embodiments, a geometric translation approach can be implemented to detect the 3D position of a target anatomical feature. For example, similar to certain other embodiments of the present disclosure, potential images 1801, 1802 associated with distinct time stamps can be captured. In connection with such images, rotational translation information regarding cameras (1807, 1808), which can be determined based on sensor information from any suitable or desirable sensor or device, can be used to triangulate and / or determine the position of such images in 3D space, thereby providing information indicative of the 3D location of a target anatomical feature 1806 in 3D space. The rotational translation information can be associated with the camera and / or the scope and can be based on robotic actuator movement and / or position sensor information from, for example, an electromagnetic beacon device that indicates the position of the camera in the electromagnetic field space.

[0187] Taking into account the intrinsic and extrinsic parameters (principal point, focal length, and distortion coefficients, relative motion) of the cameras (1807, 1808), the 3D location of the target anatomical feature 1806 can be calculated based at least in part on the tracked target two-dimensional (2D) locations on the images 1801, 1802. For the intrinsic parameters, the camera principal point and focal length can be considered. Additional data that can be considered can include the radial and tangential distortion coefficients. Based on sensor measurements (e.g., robot base and / or EM based), extrinsic parameters can also be obtained, including the rotation R and translation T of the scope between the locations where the two images were taken. For convenience, K can be represented as a matrix containing the intrinsic parameters, and H can be represented as a 4×4 matrix containing the extrinsic rotation and translation between the camera position (C t ) of the first image and the camera position (C t+1 ) of the second image.

[0188] For Ct, the 3D to 2D projection relationship is x tcan be expressed as KX, where X is the 3D coordinates w.r.t. C t and x t are the 2D coordinates (the detected centroid of the target) on the image t. Here, K is a 3×4 matrix that can be expressed as follows:

[0189] [Number] K (n) is the nth row in K.

[0190] Similarly, for C t+1 it is x t+1 = K’X,

[0191] [Number] and it is as follows.

[0192] x t and KX are parallel vectors, so x t × KX = 0, and similarly, x t+1 × K’X = 0. Here, "×" is the cross product operator. Therefore,

[0193] [Number] it is as follows.

[0194] The above can generate i(v t K (3) X - K (2) X) - j(u t K (3) X - K (1) X) + k(u t K (2) X - v t K (1) X) = 0, where u t and v t are the 2D coordinates of x t . Therefore, v t K (3) X - K(2) X = 0 u t K (3) X - K (1) X = 0 u t K (2) X - v t K (1) X = 0

[0195] Here, since the third equation is a linear combination of the first two, only the first two equations may be required. Similarly, for C t+1 two equations can be obtained as follows. v t+1 K’ (3) X - K’ (2) X = 0 u t+1 K’ (3) X - K’ (1) X = 0

[0196] C t and C t+1 After stacking the equations of, the following can be generated. AX = 0 where A is a 4×4 matrix.

[0197]

Number

[0198] Since the elements in A are known (detected 2D coordinates, intrinsic and extrinsic parameters), X can be calculated by performing a singular value decomposition (SVD) on A. A = UΣV T The last column of V is the solution of X.

[0199] Accordingly, in view of the foregoing disclosure, the various inventive concepts disclosed herein can be utilized to perform automatic target localization, such as automatic target detection, target tracking, and / or three-dimensional position estimation. In some embodiments, aspects of the present disclosure advantageously enable tracking of target anatomical features without the need for physical contact with the target anatomical features, which can facilitate improved ergonomics in the use of ureteroscopes.

[0200] In some implementations, structured light and / or other non-contact light sensing mechanisms, such as optical coherence tomography or other interferometry, can be used and may be used to determine depth / offset information. Such techniques can advantageously provide 3D papilla / cup position information. However, structured devices can be relatively large and can increase the profile of the ureteroscope with which they are configured. In some embodiments, the ureteroscope includes a Time-of-Flight camera configured to admit light and receive its reflection, and the time between light emission and reception can be used to determine distance information within the kidney.

[0201] Distance / Angle Measurement Tool FIG. 19 shows a distal end of an endoscope 1940, such as a ureteroscope, equipped with an offset distance measurement tool 1970 according to one or more aspects of the present disclosure. For example, the offset distance measurement tool 1970 can be a ruler-type distance measurement tool and / or an angle measurement tool and can be configured to extend from the working channel 1944 of the scope 1940. The extension / protrusion of the measurement tool 1970 from the distal end of the scope 1940 provides a mechanism for estimating the distance from the scope to the papilla for the purpose of localizing the papilla when it is predicted that the measurement tool 1970 will contact and / or be sufficiently far from the target papilla.

[0202] In some embodiments, the measurement tool 1970 includes certain visual markings associated with the distal and / or proximal portions (not shown) of the measurement tool 1970. Such markings may indicate the amount of the distance / length L of the measurement tool 1970 that protrudes beyond the distal end of the camera 1948 and / or the scope 1940. Although a marking 1971 is shown having any suitable or desirable form at the distal end of the tool 1970, in some embodiments, the marking is not present at the distal end of the tool 1970, but rather is associated with a portion of the tool proximal to the illustrated portion of the scope 1940. For example, the marking may be associated with a portion of the tool 1970 that is proximally exposed to the scope 1940, and thus, the relative movement of the tool 1970 proximal to the scope 1940 can indicate the extent to which the tool 1970 protrudes from the distal end of the scope 1940. In some embodiments, the marking can be manually made in place at the proximal and / or distal portions of the measurement tool 1970 before extension from the distal end of the scope 1940 and / or after contact of the tool 1970 with the target anatomical feature.

[0203] In some implementations, the electromagnetic sensor can be disposed on, and / or otherwise attached or coupled to, a target anatomical feature (e.g., a nipple). For example, an electromagnetic visible wire can be induced through the scope 1940 (e.g., within the working channel 1944) and can be at least partially embedded in the target anatomical feature, such that its electromagnetic sensor / beacon feature can maintain contact with, or proximity to, the target anatomical feature through the targeting / localization process, thereby providing an accurate real-time target for percutaneous access. In some embodiments, the electromagnetic sensor / beacon device can be coupled to the target anatomical feature using an adhesive or other attachment means. In some embodiments, a conductive adhesive can be applied to the target anatomical feature, and the conductive adhesive itself functions as a visible electromagnetic material within the electromagnetic field space. In some embodiments, a balloon-type device can be at least partially implanted / embedded within the target anatomical feature (e.g., a nipple), and the balloon can be configured to expand within the target anatomical feature, thereby securing itself to the feature. Such balloon-type electromagnetic sensors / beacons, or other expandable mechanical electromagnetic sensor / beacon devices, can advantageously provide a relatively large target for a percutaneous access instrument (e.g., a needle) to the target. Embodiments of electromagnetic sensor / beacon devices configured to be at least partially coupled to and / or embedded within the target anatomical feature can be permitted / likely to move along with the movement and / or deformation of the target anatomical feature, thereby providing a relatively robust localization / targeting solution with respect to the deformation and / or movement of the target anatomical feature.

[0204] Additional embodiments Depending on the embodiment, any particular act, event, or function of any of the processes or algorithms described herein can be performed in a different order, can be added, merged, or completely excluded. Thus, in a particular embodiment, not all of the described acts or events are necessary for the practice of the process.

[0205] In particular, hypothetical language used in this specification such as "can", "could", "might", "may", "e.g.", and equivalents is intended in its ordinary sense, unless specifically described otherwise or understood otherwise within the context in which it is used, and generally is not intended to convey that a particular embodiment includes a particular feature, element, and / or step while other embodiments do not. Thus, such hypothetical language generally does not imply that a feature, element, and / or step is required in any way for one or more embodiments, or that one or more embodiments necessarily include logic for determining whether these features, elements, and / or steps are included in or implemented in any particular embodiment, whether or not there is author input or prompting. Terms such as "comprising", "including", "having", and equivalents are used in their ordinary sense and are used inclusively in a non-limiting manner, and do not exclude additional elements, features, acts, operations, etc. Also, the term "or", when used, for example, to connect a list of elements, is used in its inclusive sense (and not in its exclusive sense) so as to mean one, some, or all of the listed elements. Unless specifically described otherwise, connective language such as the phrase "at least one of X, Y, and Z" is understood in the context in which it is generally used to convey that an item, term, element, etc. can be any of X, Y, or Z. Thus, such connective language generally does not imply that a particular embodiment requires the presence of at least one of each of X, at least one of each of Y, and at least one of each of Z.

[0206] In the above description of the embodiments, it should be understood that various features are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of rationalizing the present disclosure and assisting in the understanding of one or more of the various aspects of the invention. However, the methods of the present disclosure should not be construed as reflecting an intention that any claim requires more features than are explicitly recited in that claim. Further, any component, feature, or step illustrated and / or described in a particular embodiment herein may be applied to or used in conjunction with any other embodiment. Further, no component, feature, step, or group of components, features, or steps is necessary or essential for each embodiment. Accordingly, it is intended that the scope of the invention as disclosed herein and claimed below not be limited by the above specific embodiments, but rather be determined only by a fair reading of the following claims.

[0207] It should be understood that specific ordinal terms (e.g., "first" or "second") may be provided to facilitate reference and do not necessarily imply physical characteristics or ordering. Thus, as used herein, ordinal terms (e.g., "first," "second," "third," etc.) used to modify elements such as structures, components, operations, etc. do not necessarily indicate a priority or order of an element with respect to any other element, but rather generally serve to distinguish an element (apart from the use of the ordinal term) from another element having a similar or identical name. Further, as used herein, indefinite articles ("a" and "an") may indicate "one or more" rather than "one." Further, an operation performed "based on" a condition or event may also be performed based on one or more other conditions or events not explicitly recited.

[0208] Unless otherwise defined, all terms (including technical and scientific terms) used herein shall have the same meaning as commonly understood by one of ordinary skill in the art to which the exemplary embodiments belong. Terms such as those defined in commonly used dictionaries shall be interpreted as having a meaning consistent with their meaning in the context of the relevant art, and it should be further understood that they should not be interpreted in an idealized or overly formal sense unless explicitly defined as such herein.

[0209] Spatially relative terms such as "outer", "inner", "upper", "lower", "beneath", "above", "vertical", "horizontal", and similar terms may be used herein to facilitate description of the relationship between one element or component and another element or component as illustrated in the drawings. It should be understood that spatially relative terms are intended to encompass different orientations of the device during use or operation in addition to the orientation depicted in the drawings. For example, if the device shown in the drawings is inverted, a device positioned "beneath" or "below" another device may be positioned "above" the other device. Thus, the exemplary term "beneath" can include both lower and upper positions. The device may also be oriented in other directions, and thus, spatially relative terms may be interpreted differently depending on the orientation.

[0210] Unless otherwise specified, comparative and / or quantitative terms such as "less", "more", "greater", etc. are intended to encompass the concept of equality. For example, "less" can mean not only "less" in a strict mathematical sense but also "less than or equal to".

[0211] 〔Embodiments〕 (1) A method for identifying a target papilla, comprising: advancing a ureteroscope through at least a portion of a patient's urinary tract to a target calyx of the patient's kidney; Determining a position offset between one or more position sensors associated with the ureteroscope and the target papilla of the kidney that is at least partially exposed within the target cup; Determining a percutaneous access target based at least in part on one or more of the current positions of the one or more position sensors and the offset, a method comprising. (2) The method according to embodiment 1, further comprising advancing a percutaneous access instrument to the target cup by targeting the percutaneous access target. (3) Contacting the target papilla with the ureteroscope; Recording the position of the ureteroscope associated with the contacting; Retracting the ureteroscope away from the target papilla; The method according to embodiment 1, further comprising stationing the ureteroscope at an offset position associated with the position offset. (4) The method according to embodiment 1, wherein the position offset exhibits at least five degrees of freedom. (5) A method of positioning a surgical instrument, comprising: Advancing a medical instrument to a treatment site of a patient, the medical instrument comprising a camera; Using the camera of the medical instrument to generate a real-time video of the treatment site; Displaying a user interface including the real-time video in a window of the user interface; Projecting an anatomical feature targeting icon at the center of the window of the user interface.

[0212] (6) The method according to embodiment 5, wherein the targeting icon includes a crosshair. (7) The method according to embodiment 5, further comprising operating the medical instrument to position the targeting icon centered over a representation of the target anatomical feature of the real-time video. (8) Further including projecting one or more boundary features at substantially the center of the window within the user interface window, wherein the one or more boundary features have a size independent of the position of the medical device, the method according to embodiment 7. (9) Further including operating the medical device to fit the representation of the target anatomical feature within the one or more boundary features, the method according to embodiment 8. (10) Operating the medical device to fit the representation of the target anatomical feature within the one or more boundary features involves moving the medical device backward away from the target anatomical feature such that the representation of the target anatomical feature shrinks within the window of the user interface, the method according to embodiment 9.

[0213] (11) The one or more boundary features have at least a partial box form, the method according to embodiment 8. (12) Receiving sensor data indicating the three-dimensional position of a percutaneous access needle within an electromagnetic field, Determining the position of the distal end of the needle relative to the camera based at least in part on the sensor data, Further including displaying a needle prediction icon within the window of the user interface indicating the position of the distal end of the needle relative to the real-time video, the method according to embodiment 5. (13) Further including determining that the position of the distal end of the needle is outside the window of the user interface, wherein the needle prediction icon indicates the direction of the position of the distal end of the needle relative to the window, the method according to embodiment 12. (14) Further including operating the medical device to center the needle prediction icon within the window of the user interface, the method according to embodiment 12. (15) Further including calibrating a sensor associated with the needle within the image space of the camera, the method according to embodiment 12.

[0214] (16) The method according to embodiment 12, further comprising modifying a form of the needle prediction icon in response to an approach of the distal end of the needle to the medical device. (17) The method according to embodiment 12, wherein the form of the needle prediction icon indicates a distance of the distal end of the needle from the medical device. (18) A method of targeting an anatomical feature, advancing an endoscope into a target anatomical lumen of a patient, the endoscope comprising a position sensor associated with a distal end portion of the endoscope, using the position sensor to record position data associated with a plurality of positions of the endoscope within the target anatomical lumen, estimating a surface of the target anatomical lumen based at least in part on the position data, determining an axis of the target anatomical lumen based at least in part on the estimated surface of the target anatomical lumen, the method comprising. (19) The method according to embodiment 18, further comprising targeting the target anatomical lumen using a percutaneous access needle based at least in part on the determined axis of the target anatomical lumen. (20) Targeting the target anatomical lumen involves advancing a percutaneous access needle along a path that is substantially parallel to the determined axis of the target anatomical lumen, the method according to embodiment 19.

[0215] (21) The position sensor is an electromagnetic sensor device, Recording the position data is performed using an electromagnetic field generator disposed at least partially external to the patient, the method according to embodiment 18. (22) Estimating the surface of the target anatomical lumen involves interpolating the position data, the method according to embodiment 18. (23) Determining the axis of the target anatomical lumen involves determining a plurality of surface normal vectors associated with the estimated surface of the target anatomical lumen, the method according to embodiment 18. The method according to embodiment 23, further comprising averaging the plurality of surface normal vectors. (25) The method according to embodiment 18, wherein determining the axis of the target anatomical lumen is at least partially based on one or more of a map of the target anatomical lumen and a trajectory of the endoscope.

[0216] (26) A medical system, An endoscope configured to access a target anatomical lumen of a patient, the endoscope having a camera and an electromagnetic position sensor associated with its distal end, the endoscope, A communication interface configured to receive video data from the endoscope, An electronic display device, A control circuit communicatively coupled to the communication interface and the electronic display device, and Receiving real-time video data of a treatment site inside the patient from the endoscope of the communication interface, Causing the electronic display to display the user interface including the real-time video within a window of the user interface, A control circuit configured to display an anatomical feature targeting icon at the center of the window of the user interface, a medical system comprising. (27) The medical system according to embodiment 26, wherein the targeting icon includes a crosshair. (28) The medical system according to embodiment 26, wherein the control circuit is further configured to display one or more boundary features substantially at the center of the window within the window of the user interface. (29) The medical system according to embodiment 28, wherein a size of the one or more boundary features relative to a representation of a target anatomical feature of the real-time video is based on a distance of the target anatomical feature from the camera of the endoscope. (30) The medical system according to embodiment 29, wherein the one or more boundary features have at least a partial box shape.

[0217] (31) The control circuit receives sensor data indicating the three-dimensional position of a transcutaneous access needle within an electromagnetic field, determines the position of the distal end of the needle relative to the endoscope based at least in part on the sensor data, and is further configured to display a needle prediction icon within the window of the user interface, wherein the needle prediction icon indicates the position of the distal end of the needle relative to the real-time video, the medical system according to embodiment 26. (32) A computing device, comprising an endoscope interface, a control circuit including one or more processors and one or more data storage devices, receives position data indicating a plurality of positions of a position sensor associated with a distal end portion of an endoscope from the endoscope disposed within a target anatomical lumen of a patient, estimates a surface of the target anatomical lumen based at least in part on the position data, and a control circuit configured to determine an axis of the target anatomical lumen based at least in part on the estimated surface of the target anatomical lumen. (33) The control circuit of the computing device according to embodiment 32 is configured to estimate the surface of the target anatomical lumen by at least partially interpolating the position data. (34) The control circuit of the computing device according to embodiment 32 is configured to determine the axis of the target anatomical lumen by at least partially determining a plurality of surface normal vectors associated with the estimated surface of the target anatomical lumen.

Claims

1. The medical system a medical instrument configured to be advanced to a treatment site, the medical instrument including a camera; 1. A control system comprising: obtaining real-time video of the treatment area via the camera; displaying a user interface including the real-time video in a window of the user interface; a control system configured to project one or more boundary features within the window of the user interface approximately at the center of the window, the one or more boundary features having a size independent of a position of the medical instrument.

2. The medical system of claim 1 , wherein the control system is further configured to project an anatomical feature targeting icon onto the center of the window of the user interface, the anatomical feature targeting icon including a crosshair.

3. The medical system a medical instrument configured to be advanced to a treatment site, the medical instrument including a camera; a percutaneous access needle configured to access the treatment site via a percutaneous access pathway separate from the medical device; 1. A control system comprising: obtaining real-time video of the treatment area via the camera; displaying a user interface including the real-time video in a window of the user interface; receiving sensor data indicative of a three-dimensional position of the percutaneous access needle within an electromagnetic field; determining a position of the distal end of the percutaneous access needle relative to the camera based at least in part on the sensor data; a control system configured to display within the window of the user interface a needle prediction icon indicating a position of the distal end of the percutaneous access needle relative to the real-time video.

4. 4. The medical system of claim 3, wherein the control system is further configured to modify a configuration of the needle prediction icon in response to a change in a relative position between the distal end of the percutaneous access needle and the medical instrument.

5. 1. A method of positioning a surgical instrument, comprising: advancing a medical instrument to a treatment site, the medical instrument comprising a camera; generating a real-time video of the treatment site using the camera of the medical instrument; displaying a user interface including the real-time video in a window of the user interface; projecting one or more boundary features within the window of the user interface approximately at a center of the window; The one or more boundary features have a size that is independent of a position of the medical device.

6. The method of claim 5 , further comprising projecting an anatomical feature targeting icon onto the center of the window of the user interface, the anatomical feature targeting icon including a crosshair.

7. 7. The method of claim 6, further comprising manipulating the medical instrument to center the anatomical feature targeting icon over a representation of a target anatomical feature in the real-time video.

8. The method of claim 5 , further comprising manipulating the medical instrument to cause the representation of a target anatomical feature in the real-time video to fall within the one or more boundary features.

9. The method of claim 5 , wherein the one or more boundary features have at least a partial box form.

10. 1. A method of positioning a surgical instrument, comprising: advancing a medical instrument to a treatment site, the medical instrument comprising a camera; generating a real-time video of the treatment site using the camera of the medical instrument; displaying a user interface including the real-time video in a window of the user interface; receiving sensor data indicative of a three-dimensional position of a percutaneous access needle within an electromagnetic field, the percutaneous access needle being positioned to access the treatment site via a percutaneous access pathway separate from the medical instrument; determining a position of the distal end of the percutaneous access needle relative to the camera based at least in part on the sensor data; displaying within the window of the user interface a needle prediction icon indicating a position of the distal end of the percutaneous access needle relative to the real-time video.

11. 11. The method of claim 10, further comprising determining that the location of the distal end of the percutaneous access needle is outside the window of the user interface, and wherein the needle prediction icon indicates a direction of the location of the distal end of the percutaneous access needle relative to the window.

12. The method of claim 10 , further comprising manipulating the medical instrument to center the needle prediction icon within the window of the user interface.

13. The method of claim 10, further comprising calibrating a sensor associated with the percutaneous access needle in an image space of the camera.

14. The method of claim 10, further comprising modifying a configuration of the needle prediction icon in response to a change in relative position between the distal end of the percutaneous access needle and the medical instrument.

15. The method of claim 10 , wherein the configuration of the needle prediction icon indicates a distance of the distal end of the percutaneous access needle from the medical instrument.

Images