Localization of target anatomical features

Abstract

The process of locating a target papilla of a renal anatomical structure involves advancing a ureteroscope through at least a portion of the patient's urinary tract to a target calyx of the patient's kidney, determining a position offset between one or more position sensors associated with the ureteroscope and a target papilla of the kidney at least partially exposed within the target calyx, and determining a percutaneous access target based at least in part on one or more of the current positions and offsets of the one or more position sensors.

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Provisional Patent Application No. 63 / 001,870, filed March 30, 2020, and entitled “TARGET ANATOMICAL FEATURE LOCALIZATION,” the disclosure of which is incorporated herein by reference in its entirety.

[0002] FIELD OF THEINVENTION The present invention relates to the field of medical procedures. [Background technology]

[0003] Various medical procedures involve the use of one or more scopes and / or percutaneous access instruments. Improper positioning or advancement of such devices can result in certain physiological and procedural complications. Summary of the Invention [Means for solving the problem]

[0004] Described herein are systems, devices, and methods that facilitate identification, tracking, and targeting of various anatomical features based on specific sensor-based and / or image-based location information, which may be obtained, for example, using a scope device or other medical instrument. Localization of target anatomical features according to aspects of the present disclosure can facilitate targeting of anatomical features in connection with medical procedures, such as, for example, nephroscopy or other procedures that access the renal anatomy.

[0005] In some implementations, the present disclosure relates to a method of locating a target papilla, the method including advancing a ureteroscope through at least a portion of a patient's urinary tract to a target calyx of a kidney of the patient, determining a position offset between one or more position sensors associated with the ureteroscope and a target papilla of the kidney at least partially exposed within the target calyx, and determining a percutaneous access target based at least in part on one or more of the current positions and offsets of the one or more position sensors.

[0006] The method may further include advancing a percutaneous access instrument to the target cup by targeting the percutaneous access target. In some embodiments, the method further includes contacting the target papilla with a ureteroscope, recording a position of the ureteroscope associated with the contact, retracting the ureteroscope away from the target papilla, and parking the ureteroscope at an offset position associated with the position offset. The position offset may, for example, represent at least five degrees of freedom.

[0007] In some implementations, the present disclosure relates to a method of positioning a surgical instrument, the method including advancing a medical instrument to a treatment site on a patient, the medical instrument including a camera, generating a real-time video of the treatment site using the medical instrument's camera, displaying a user interface including the real-time video in a window of the user interface, and projecting an anatomical feature targeting icon at the center of the user interface window.

[0008] The targeting icon may include any type of form or shape, such as a crosshair, or a combination thereof. The method may further include manipulating the medical instrument to center the targeting icon over a representation of the target anatomical feature of the real-time video. For example, the method may include projecting one or more boundary features within a window of the user interface at approximately the center of the window, the one or more boundary features having a size independent of the position of the medical instrument. In some embodiments, the method further includes manipulating the medical instrument to fit the representation of the target anatomical feature within the one or more boundary features. Manipulating the medical instrument to fit the representation of the target anatomical feature within the one or more boundary features may involve retracting the medical instrument away from the target anatomical feature such that the representation of the target anatomical feature shrinks within the window of the user interface. In some embodiments, the one or more boundary features have at least a partial box form.

[0009] The method may further include receiving sensor data indicative of a three-dimensional position of a percutaneous access needle within the electromagnetic field, determining a position of a distal end of the needle relative to the camera based at least in part on the sensor data, and displaying a needle prediction icon in a window of the user interface indicating the position of the distal end of the needle relative to the real-time video. Some embodiments involve presenting one or more icons representing a predicted needle entry point into the target anatomical feature (e.g., the nipple). For example, the indicator may represent the needle location by displaying the needle entry position to provide situational awareness and / or information regarding the needle trajectory. The needle prediction / trajectory indicator may display the needle orientation as a line type morphology / shape. For example, one or both of the proximal and distal points of the needle may be predicted and / or coupled with the line type representation.

[0010] The method may further include determining that a location of the distal end of the needle is outside a window of the user interface, the needle prediction icon indicating a direction of the location of the distal end of the needle relative to the window. In some embodiments, the method further includes manipulating the medical instrument to center the needle prediction icon within the window of the user interface. In some embodiments, the method further includes calibrating a sensor associated with the needle within an image space of the camera. The method may include modifying a configuration of the needle prediction icon in response to an approach of the distal end of the needle to the medical instrument.

[0011] The size of the needle prediction icon may be changed / modified based on the determined needle prediction / prediction accuracy. If there is significant anatomical motion that may result in an error in the needle prediction, the needle prediction icon may be presented in a larger size to indicate a larger determined error for the needle prediction / trajectory. In some embodiments, the form of the needle prediction icon indicates the distance of the distal end of the needle from the medical instrument.

[0012] In some implementations, the present disclosure relates to a method of targeting an anatomical feature, the method including advancing an endoscope into a target anatomical lumen of a patient, the endoscope comprising a position sensor associated with a distal end of the endoscope, recording position data associated with a plurality of positions of the endoscope within the target anatomical lumen using the position sensor, estimating a surface of the target anatomical lumen based at least in part on the position data, and determining an axis of the target anatomical lumen based at least in part on the estimated surface of the target anatomical lumen.

[0013] The method may further include targeting the target anatomical lumen with a percutaneous access needle based at least in part on the determined axis of the target anatomical lumen. For example, targeting the target anatomical lumen may involve advancing the percutaneous access needle along a path that is substantially parallel to the determined axis of the target anatomical lumen. In some embodiments, the position sensor is an electromagnetic sensor device, and recording the position data is performed using an electromagnetic field generator disposed at least in part external to the patient.

[0014] In some embodiments, estimating a surface of the target anatomical lumen may involve interpolating the position data. Determining an axis of the target anatomical lumen may involve determining a plurality of surface normal vectors associated with the estimated surface of the target anatomical lumen. For example, the method may include averaging the plurality of surface normal vectors. In some embodiments, determining the axis of the target anatomical lumen is based at least in part on one or more of a map of the target anatomical lumen and a trajectory of the endoscope.

[0015] In some implementations, the present disclosure relates to a medical system including an endoscope configured to access a target anatomical lumen of a patient, the endoscope having a camera and an electromagnetic position sensor associated with a distal end thereof, a communication interface configured to receive video data from the endoscope, an electronic display device, and control circuitry communicatively coupled to the communication interface and the electronic display device. The control circuitry is configured to receive real-time video data of an internal treatment site of the patient from the endoscope of the communication interface, cause the electronic display to display a user interface including the real-time video in a window of the user interface, and cause an anatomical feature targeting icon to be displayed in the center of the window of the user interface. The targeting icon may include, for example, a crosshair and / or the like.

[0016] The control circuitry may be further configured to cause the one or more boundary features to be displayed within the window of the user interface approximately at the center of the window. A size of the one or more boundary features relative to the representation of the target anatomical feature in the real-time video may be based on a distance of the target anatomical feature from a camera of the endoscope. In some embodiments, the one or more boundary features have at least a partial box form.

[0017] The control circuitry may be further configured to receive sensor data indicative of a three-dimensional position of the percutaneous access needle within the electromagnetic field, determine a position of a distal end of the needle relative to the endoscope based at least in part on the sensor data, and cause a needle prediction icon to be displayed in a window of the user interface. The needle prediction icon may indicate the position of the distal end of the needle relative to the real-time video.

[0018] In some implementations, the present disclosure relates to a computing device comprising an endoscope interface and a control circuit including one or more processors and one or more data storage devices. The control circuit is configured to receive position data from an endoscope disposed within a target anatomical lumen of a patient, the position data indicating a plurality of positions of a position sensor relative to a distal end of the endoscope. As with all other descriptions herein of positions and position sensors described herein, such positions may include position and orientation aspects / information. The control circuit is further configured to estimate a surface of the target anatomical lumen based at least in part on the position data, and determine an axis of the target anatomical lumen based at least in part on the estimated surface of the target anatomical lumen.

[0019] In some embodiments, the control circuitry is configured to estimate a surface of the target anatomical lumen at least in part by interpolating the position data, hi some embodiments, the control circuitry is configured to determine an axis of the target anatomical lumen at least in part by determining a plurality of surface normal vectors associated with the estimated surface of the target anatomical lumen.

[0020] For purposes of summarizing the disclosure, certain aspects, advantages, and novel features have been described. It is to be understood that not necessarily all such advantages may be achieved in accordance with any particular embodiment. Thus, the disclosed embodiments may be practiced in a manner that achieves or optimizes one advantage or group of advantages as taught herein, without necessarily achieving other advantages as may be taught or suggested herein. [Brief description of the drawings]

[0021] Various embodiments are depicted in the accompanying drawings for illustrative purposes and should not be construed as limiting the scope of the present invention in any way. In addition, various features of different disclosed embodiments can be combined to form additional embodiments that are part of the present disclosure. Throughout the drawings, reference numerals may be reused to indicate correspondence between referenced elements. [Figure 1] 1 illustrates an embodiment of a robotic medical system, in accordance with one or more embodiments. [Diagram 2] 2 illustrates an example robotic system that may be implemented in the medical system of FIG. 1 in accordance with one or more embodiments. [Diagram 3] 2 illustrates an example control system that may be implemented in the medical system of FIG. 1 in accordance with one or more embodiments. [Figure 4] 1 illustrates a ureteroscope disposed in a portion of a patient's urinary system, in accordance with one or more embodiments. [Figure 5-1] A flow diagram illustrating a process for performing guided percutaneous nephrolithotomy, according to one or more embodiments, is presented. [Figure 5-2] A flow diagram illustrating a process for performing guided percutaneous nephrolithotomy, according to one or more embodiments, is presented. [Figure 6-1] 5-1 and 5-2, respectively, in accordance with one or more embodiments. [Figure 6-2]5-1 and 5-2, respectively, in accordance with one or more embodiments. [Figure 7] FIG. 1 is a flow diagram illustrating a process for locating a target anatomical feature, in accordance with one or more embodiments. [Figure 8] 1 illustrates a scope device disposed within a target cup for target location, according to one or more embodiments. [Figure 9] 1 illustrates a robotic medical system positioned to facilitate navigation of a scope within a patient, in accordance with one or more embodiments. [Figure 10A] 1 illustrates the renal anatomy with a ureteroscope parked in various positions, according to one or more embodiments. [Figure 10B] 1 illustrates the renal anatomy with a ureteroscope parked in various positions, according to one or more embodiments. [Figure 11] 1 illustrates a scope camera view / window configuration including a scope targeting feature in accordance with one or more embodiments. [Figure 12A] 1 illustrates a configuration of a scope camera view / window including a target boundary feature, in accordance with one or more embodiments. [Figure 12B] 1 illustrates a configuration of a scope camera view / window including a target boundary feature, in accordance with one or more embodiments. [Figure 13] 1 illustrates a robotic medical system positioned to facilitate percutaneous access of a patient's renal anatomy, according to one or more embodiments. [Figure 14-1] 1 illustrates a scope camera view / window configuration including one or more needle trajectory features, according to one or more embodiments. [Figure 14-2] 1 illustrates a scope camera view / window configuration including one or more needle trajectory features, according to one or more embodiments. [Figure 15] 1 illustrates a scoping device disposed within a target renal anatomy for calyx mapping, according to one or more embodiments. [Figure 16]1 illustrates an image for an image-based target localization system in accordance with one or more embodiments. [Figure 17] 13 illustrates a cup map formed from recorded scope positions in accordance with one or more embodiments. [Figure 18] 1 illustrates a cup surface map including surface normals and cup trajectory indicators in accordance with one or more embodiments. [Figure 19] 1 illustrates a distal portion of an endoscope including a distance measurement tool in accordance with one or more embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] The headings provided herein are merely for convenience and do not necessarily affect the scope or meaning of the claimed invention. Although certain preferred embodiments and examples are disclosed below, the subject matter of the invention extends beyond the specifically disclosed embodiments to other alternative embodiments and / or applications, as well as to modifications and equivalents thereof. Thus, the scope of claims that may arise from this specification is not limited by any of the specific embodiments described below. For example, in any method or process disclosed herein, the acts or operations of the method or process may be performed in any suitable order and are not necessarily limited to any particular disclosed order. Various operations may be described as multiple separate operations in a manner that may be helpful in understanding a particular embodiment. However, the order of description should not be construed to imply that these operations are order dependent. In addition, the structures, systems, and / or devices described herein may be embodied as integrated components or as separate components. For purposes of comparing various embodiments, certain aspects and advantages of these embodiments are described. Not necessarily all such aspects or advantages are achieved by any particular embodiment. Thus, for example, various embodiments may be implemented in a manner that achieves or optimizes one advantage or group of advantages as taught herein, without necessarily achieving other aspects or advantages that may also be taught or suggested herein.

[0023] Certain standard anatomical terms of location are used herein to refer to animal, i.e., human, anatomical structures with respect to preferred embodiments. Certain spatially relative terms, such as "outer," "inner," "superior," "lower," "below," "upper," "vertical," "horizontal," "top," "bottom," and similar terms, are used herein to describe the spatial relationship of one device / element or anatomical structure to another device / element or anatomical structure, with it being understood that these terms are used herein for ease of description to describe the positional relationships between the elements / structures as illustrated in the drawings. It should be understood that the spatially relative terms are intended to encompass different orientations of the elements / structures during use or operation in addition to the orientation depicted in the drawings. For example, an element / structure described as "above" another element / structure may represent a position that is below or beside such other element / structure relative to the subject patient or alternative orientations of the element / structure, and vice versa.

[0024] Overview The present disclosure relates to systems, devices, and methods for locating and targeting target anatomical features of a patient to aid in certain medical procedures. Although certain aspects of the present disclosure are described herein in detail in the context of renal, urological, and / or nephrological procedures, such as kidney stone removal / treatment procedures, it is understood that such context is provided for convenience and clarity, and that the anatomical feature locating and targeting concepts disclosed herein are applicable to any suitable medical procedures. However, as noted, a description of the anatomical structures of the renal / urinary system and related medical problems and procedures is presented below to aid in the explanation of the inventive concepts disclosed herein.

[0025] Kidney stone disease, also known as urolithiasis, is a relatively common medical condition that involves the formation of solid pieces of material in the urinary tract, referred to as "kidney stones," "urinary stones," "nephrolithiasis," "nephrolithiasis," or "nephrolithiasis." Urinary stones can form and / or be found in the kidneys, ureters, and bladder (referred to as "bladder stones"). Urinary stones form as a result of concentrated minerals and can cause significant abdominal pain if they reach a size sufficient to block urine flow through the ureter or urethra. Urinary stones can be formed from calcium, magnesium, ammonia, uric acid, cysteine, and / or other compounds.

[0026] To remove urinary stones from the bladder and ureters, a surgeon may insert a ureteroscope through the urethra and into the urinary tract. Typically, the ureteroscope includes an endoscope at its distal end configured to allow visualization of the urinary tract. The ureteroscope may also include a stone extraction mechanism for capturing or breaking up urinary stones. During a ureteroscopy procedure, one physician / technologist may control the position of the ureteroscope, while another physician / technologist may control the stone extraction mechanism.

[0027] To remove relatively large stones (i.e., "kidney stones") from the kidney, physicians may use percutaneous nephrolithotomy ("PCNL") techniques, which involve inserting a nephroscope through the skin (i.e., percutaneously) to break up and / or remove the stone. In some embodiments, locating the kidney stone may be accomplished using fluoroscopy to provide a target for the insertion of the nephroscope. However, fluoroscopy generally increases the cost of the nephrolithotomy procedure due to the cost of the fluoroscope itself, as well as the cost of the technician to operate the fluoroscope. Fluoroscopy also exposes the patient to radiation for a relatively long period of time. Even with fluoroscopy, accurately making the percutaneous incisions to access the kidney stone can be difficult and unnecessarily inaccurate. Furthermore, some nephrolithotomy techniques involve a two- or three-day hospital stay. In short, certain nephrolithotomy solutions can be relatively expensive and problematic for patients.

[0028] According to certain surgical procedures according to aspects of the present disclosure, an endoscope (e.g., a ureteroscope) can be equipped with one or more position sensors, and the location of the sensor is used as a target for percutaneous access, such as for PCNL. For example, a ureteroscope equipped with an electromagnetic sensor and / or a percutaneous access needle equipped with an electromagnetic sensor can be used to guide percutaneous renal access for kidney stone removal and / or the like. In such a procedure, a surgeon / physician can drive the ureteroscope to a target calyx of the kidney and use an electromagnetic sensor (e.g., a beacon) associated with the distal / tip of the ureteroscope as a percutaneous access target for the needle. In general, the effectiveness of the percutaneous shaft relative to the target calyx can depend at least in part on where the physician positions / parks the ureteroscope, for example, relative to the location and / or leading orientation of the target calyx and / or papilla through which percutaneous access can be made to the target calyx. For some procedures where the distal end / tip of the ureteroscope is used as the percutaneous access target, it may be desirable for the distal tip of the ureteroscope to be as close to the papilla / calyx as possible during percutaneous access / approach.

[0029] The terms "scope" and "endoscope" are used herein according to their broad and ordinary meaning and may refer to any type of elongated medical instrument having imaging, viewing, and / or capturing capabilities and configured to be introduced into any type of organ, cavity, lumen, chamber, or space in the body. For example, references to a scope or endoscope herein may refer to a ureteroscope, cystoscope, nephroscope, bronchoscope, arthroscope, colonoscope, laparoscope, borescope, or the like. A scope / endoscope may, in some cases, comprise a rigid or flexible tube and may be sized to be passed within an outer sheath, catheter, introducer, or other luminal device, or may be used without such a device.

[0030] Robot-assisted percutaneous procedures can be implemented in connection with various medical procedures, such as kidney stone removal procedures, where the robotic tools can enable the physician / urologist to perform endoscopic (e.g., ureteroscope) target access as well as percutaneous access / treatment. Advantageously, aspects of the present disclosure relate to real-time target tracking / localization in medical procedures, which can be utilized by the operating physician to direct the percutaneous access instrument (e.g., needle or other rigid tool) and / or to guide the robotic instrumentation, such as by automatically adjusting the endoscope position and / or alignment in response to such real-time target tracking information. To facilitate such functionality, embodiments of the present disclosure can advantageously provide mechanisms for target localization, tracking, and / or three-dimensional localization of anatomical features to assist physicians (e.g., urologists) in achieving relatively efficient and accurate percutaneous access for various surgical procedures, such as nephroscopy. Although aspects of the present disclosure are described herein for convenience in the context of ureteroscopically guided nephroscopy, it should be understood that the inventive aspects of the present disclosure may be implemented in any suitable or desirable type of percutaneous and / or endoscopic medical procedure, whether robotic or not.

[0031] Healthcare System FIG. 1 illustrates an exemplary medical system 100 for performing various medical procedures according to aspects of the present disclosure. The medical system 10 may be used, for example, for ureteroscopic procedures. As mentioned and described above, certain ureteroscopic procedures involve the investigation of ureteral abnormalities and / or the treatment / removal of kidney stones. In some embodiments, kidney stone treatment may benefit from the assistance of certain robotic techniques / devices, such as those that may be similar to those shown in FIG. 1 and detailed below. Robotic medical solutions may provide relatively higher precision, better control, and / or better eye-hand coordination for certain instruments. For example, robotically assisted percutaneous access to the kidney according to some procedures may advantageously enable a urologist to perform both targeted endoscopic access and percutaneous access. However, according to some solutions, percutaneous kidney access may suffer from certain difficulties with regard to proper alignment / positioning of the targeted ureteroscope and / or determining a targeted percutaneous access path that is substantially coincident with the targeted infundibulum, calyx, and / or papilla. In some implementations, the present disclosure relates to systems (e.g., system 100), devices, and methods for intelligently guiding a ureteroscope and / or a percutaneous access instrument (e.g., a needle). For example, embodiments of the present disclosure relate to systems, devices, and methods incorporating certain automatic target localization, tracking, and / or 3D physician estimation features that may advantageously assist a urologist or other technician in achieving efficient and accurate percutaneous access to the kidney. Although embodiments of the present disclosure are presented in the context of a ureteroscope and / or human renal anatomy, it should be understood that the principles disclosed herein may be implemented in any type of endoscopic procedure.

[0032] The medical system 100 includes a robotic system 10 configured to engage and / or control a medical instrument 32 (e.g., a ureteroscope) and perform a procedure on a patient 13. The medical system 10 also includes a control system 50 configured to interface with the robotic system 10, provide information regarding the procedure, and / or perform various other operations. For example, the control system 50 may include one or more displays 42 configured to present certain information to assist the physician 5 and / or other technicians or individuals. The medical system 10 may include a platform 15 configured to hold the patient 13. The system 10 may further include an electromagnetic (EM) field generator 18, which may be held by one or more of the robotic arms 12 of the robotic system 10 or may be a stand-alone device.

[0033] In some implementations, the system 10 may be used to perform a percutaneous procedure such as a percutaneous nephrolithotomy (PCNL). Illustratively, if the patient 13 has a kidney stone 80 that is too large to remove / pass through the urinary tract (60, 63, 65), the physician 5 may perform a procedure to remove the kidney stone 80 through a percutaneous access point / pathway associated with the flank / side of the patient 13. In some embodiments, the physician 5 may interact with the control system 50 and / or the robotic system 10 to cause the robotic system 10 to advance and navigate / control the medical instrument 32 (e.g., a scope) from the urethra 65, through the bladder 60, up the ureter 63, and into the calyx of the kidney 70 where the stone 80 is located. The control system 50 may provide information related to the medical instrument 32 via the display 42, such as real-time endoscopic images captured therewith, to assist the physician 5 in navigating / controlling the medical instrument.

[0034] Renal anatomy is described herein for reference with respect to certain medical procedures related to aspects of the inventive concept. The kidneys 70, shown generally in a typical anatomical location in FIG. 1, generally include two bean-shaped organs located on the left and right sides, respectively, in the retroperitoneal space. In an adult human, the kidneys generally are about 11 cm in height / length. The kidneys receive blood from paired renal arteries, and blood leaves the kidneys via paired renal veins, neither of which are shown in FIG. 1 for visual clarity. Each kidney 70 is fluidly connected to a ureter 63, which generally comprises a tube that carries urine excreted from the kidney 70 to the bladder 60.

[0035] The kidneys 70 typically sit relatively high in the abdominal cavity, in a retroperitoneal position at a slight oblique angle. Intraperitoneal asymmetries, generally caused by the position of the liver, typically cause the right kidney to be slightly lower and smaller than the left, and slightly more centrally located than the left kidney (as shown in detail in FIG. 1). Above each kidney is an adrenal gland (not shown). The upper portion of the kidney is partially protected by the 11th and 12th ribs. Each kidney with its adrenal gland is generally surrounded by two layers of fat: perirenal fat, which resides between the renal fascia and the renal capsule, and pararenal fat, which resides above the renal fascia.

[0036] The kidneys 70 are responsible for controlling the volumes of various body fluid compartments, fluid osmolality, acid-base balance, various electrolyte concentrations, and the removal of toxins. The kidneys 70 provide a filtration function by secreting certain substances and reabsorbing others. Examples of substances secreted in urine are hydrogen, ammonium, potassium, and uric acid. In addition, the kidneys also perform various other functions, such as hormone synthesis and others.

[0037] The recessed area on the concave border of kidney 70 is the renal hilum 78, where the renal artery (not shown) enters kidney 70 and where the renal vein (not shown) and ureter 63 exit. Kidney 70 is surrounded by a tough fibrous tissue, the renal capsule 74, which is itself surrounded by perirenal fat, renal fascia, and pararenal fat. The anterior (front) surface of these tissues is the peritoneum, while the posterior (back) surface is the transversalis fascia.

[0038] The functional matrix or parenchyma of the kidney 70 is divided into two major structures: the outer renal cortex 77 and the inner renal medulla 87. These structures each take the form of multiple roughly conical renal lobes that contain a renal cortex surrounding a portion of the medulla called the renal pyramids 72. Between the renal pyramids 72 are processes of the cortex called the renal columns 73. The urine-producing functional structure of the kidney, the nephron (not shown in detail in FIG. 1 ), spans the cortex 77 and medulla 87. The initial filtering portion of the nephron is the renal corpuscle, located in the cortex, followed by the renal duct, which enters the cortex deep into the medullary pyramid. The medullary ray, which is part of the renal cortex, is a collection of tubules that drain into a single collecting duct.

[0039] The tip / apex or papilla 79 of each pyramid drains urine into a respective minor calyx 75, which drains into a major calyx 76, which drains into the renal pelvis 71 where it passes into the ureter 63. At the renal hilum 78, the ureter 63 and renal vein exit the kidney and the renal artery enters. Hilar fat and lymphatic tissue with lymph nodes surround these structures. The hilar fat is adjacent to a fat-filled cavity called the renal sinus. The renal sinus collectively contains the renal pelvis 71 and calyxes 75, 76 and separates these structures from the renal medullary tissue. The funnel-shaped / tubular anatomical structure associated with the calyx may be referred to as the infundibulum; that is, the infundibulum generally provides the terminal portion of the calyx where the papilla is exposed within the calyx.

[0040] With further reference to the medical system 10, a medical instrument (e.g., a scope) 32 may be advanced through the urinary tract into the kidney 70. Upon reaching the site of the kidney stone 80 (e.g., within a target calyx 75 of the kidney 70 through which the stone 80 is accessible), the medical instrument 32 may be used to designate / tag a target location for percutaneous access to the kidney 70. To minimize damage to the kidney and / or surrounding anatomical structures, the physician 5 may designate a particular papilla 79 of the kidney 70 as a target location / anatomical feature for entering the kidney 70 with a percutaneous access instrument (e.g., a needle (not shown), see, e.g., FIG. 13 ). However, other target locations may also be designated or determined. Once the percutaneous access instrument reaches the target location (e.g., calyx 75), the utilized percutaneous access path may be used to extract the kidney stone 80 from the patient 13. The term "percutaneous access instrument" is used herein according to its broad ordinary meaning and may refer to a surgical tool or device configured to be punctured or inserted through human skin and / or other tissue / anatomical structures, such as a needle, scalpel, guidewire, etc. However, it should be understood that a percutaneous access instrument may refer to other types of medical instruments in the context of this disclosure.

[0041] 1, the medical instruments 32 are implemented as scopes. However, each of the medical instruments 32 may be implemented as any suitable type of medical instrument, such as a catheter, a guidewire, a lithotriptor, a basket retrieval device, etc. In some embodiments, the medical instruments 32 are steerable devices.

[0042] A scope, such as scope 32 of system 100, may be configured to navigate within the human anatomy, such as within an orifice or lumen of the human anatomy. Scopes may include, for example, a ureteroscope (e.g., for accessing the urinary tract), a laparoscope, a nephroscope (e.g., for accessing the kidney), a bronchoscope (e.g., for accessing the airways such as the bronchi), a colonoscope (e.g., for accessing the colon), an arthroscope (e.g., for accessing a joint), a cystoscope (e.g., for accessing the bladder), etc.

[0043] With reference to FIG. 1 and FIG. 2, which illustrates an exemplary embodiment of the robotic system 10 of FIG. 1, in accordance with one or more embodiments of the present disclosure, the robotic system 10 may be configured to at least partially facilitate the performance of a medical procedure. The robotic system 10 may be positioned in a variety of ways depending on the particular procedure. The robotic system 10 may include, for example, one or more robotic arms 12 configured to engage and / or control a scope 32 (and / or a percutaneous access instrument, not shown) to perform one or more aspects of the procedure. As shown, each robotic arm 12 may include multiple arm segments 23 coupled to joints that may provide multiple degrees of movement / freedom. In the example of FIG. 1, the robotic system 10 is positioned adjacent to a patient's leg, and the robotic arm 12 is actuated to engage and position the scope 32 for access to an access point, such as the urethra 65 of the patient 13. Once the robotic system 100 is properly positioned, the scope 32 may be inserted into the patient 13 robotically using the robotic arm 12, manually by the physician 5, or a combination thereof.

[0044] The robotic system 10 may be coupled to any component of the medical system 100, such as the control system 50, the platform 15, the EM field generator 18, the scope 32, and / or the percutaneous access instrument (e.g., needle, see FIG. 12). In some embodiments, the robotic system 10 is communicatively coupled to the control system 50. For example, the robotic system 10 may be configured to receive control signals from the control system 50 to perform actions, such as positioning one or more robotic arms 12 in a particular manner, manipulating the scope 32, etc. In response, the robotic system 10 may use certain control circuits 202, actuators 207, and / or other components of the robotic system 10 to control the components of the robotic system 10 to perform actions. In some embodiments, the robotic system 10 is configured to receive images and / or image data from the scope 32 representative of the internal anatomical structure of the patient 13, i.e., the urinary system with respect to the particular depiction of FIG. 1, and / or transmit the image / image data to the control system 50, which may then be displayed on the display 42 or other output device. Additionally, in some embodiments, the robotic system 10 is coupled to components of the medical system 10, such as the control system 50, in a manner that allows fluids, optics, power, or the like to be received therefrom. The robotic system 10 may include one or more communications interfaces 206, power sources / interfaces, electronic displays 242, and / or other input / output components 210.

[0045] With reference to FIG. 1 and FIG. 3 , which illustrates an exemplary embodiment of the control system 50 of FIG. 1 in accordance with one or more embodiments of the present disclosure, the control system 50 can be configured to provide various functions to assist in performing a medical procedure. In some embodiments, the control system 50 can be coupled to the robotic system 10 and operate in cooperation with the robotic system 10 to perform a medical procedure on the patient 13. For example, the control system 50 can communicate with the robotic system 10 via a wireless or wired connection (e.g., to control the robotic system 10 and / or the scope 32, receive images captured by the scope 32, etc.), provide fluid to the robotic system 10 via one or more fluid channels, provide power to the robotic system 10 via one or more electrical connections, provide optics to the robotic system 10 via one or more optical fibers or other components, etc. Additionally, in some embodiments, the control system 50 can communicate with the needle and / or nephroscope to receive position data therefrom. Additionally, in some embodiments, the control system 50 can communicate with the table 15 to position the table 15 in a particular orientation or otherwise control the table 15. Additionally, in some embodiments, the control system 50 may be in communication with the EM field generator 18 to control the generation of the EM field in the area surrounding the patient 13 .

[0046] System 10 may include certain control circuits configured to perform certain functions described herein, such as control circuit 202 of robotic system 10 and / or control circuit 302 of control system 50. That is, control circuitry of system 10 may be part of robotic system 10, control system 50, or both. Thus, all references to control circuits herein may refer to circuitry embodied in a robotic system, a control system, or any other component of a medical system, such as medical system 100 shown in FIG. 1. The term "control circuitry" is used herein according to its broad and ordinary meaning and may refer to any collection of processors, processing circuits, processing modules / units, chips, dies (e.g., semiconductor dies including one or more active and / or passive devices and / or connectivity circuits), microprocessors, microcontrollers, digital signal processors, microcomputers, central processing units, field programmable gate arrays, programmable logic devices, state machines (e.g., hardware state machines), logic circuits, analog circuits, digital circuits, and / or any devices that manipulate signals (analog and / or digital) based on hard-coded and / or operational instructions in the circuits. The control circuitry referred to herein may further include one or more circuit boards (e.g., printed circuit boards), conductive traces and vias, and / or mounting pads, connectors, and / or components. The control circuitry referred to herein may further include one or more storage devices, which may be embodied in a single memory device, multiple memory devices, and / or embedded circuitry of a device. Such data storage devices may include read-only memory, random access memory, volatile memory, non-volatile memory, static memory, dynamic memory, flash memory, cache memory, data storage registers, and / or any device that stores digital information.It should be noted that in embodiments in which the control circuitry comprises hardware and / or software state machines, analog circuits, digital circuits, and / or logic circuits, the data storage devices / registers that store any associated operational instructions may be embedded within or external to the circuitry that comprises the state machines, analog circuits, digital circuits, and / or logic circuits.

[0047] The control circuitry 202 and / or 302 may comprise computer readable media that stores and / or is configured to store hard-coded and / or operational instructions corresponding to at least some of the steps and / or functions illustrated in one or more of the present figures and / or described herein. Such computer readable media may, in some cases, be included in an article of manufacture. The control circuitry 202 / 302 may be maintained / disposed entirely locally or may be at least partially remotely located (e.g., indirectly communicatively coupled via a local area network and / or wide area network).

[0048] With respect to robotic system 10, at least a portion of the control circuitry 202 may be integrated with the base 25, column 14, and / or console 16 of robotic system 10, and / or another system communicatively coupled to robotic system 10. With respect to control system 50, at least a portion of the control circuitry 302 may be integrated with the console base 51 and / or display unit 42 of control system 50. Any description of functional control circuitry or related functionality herein should be understood to be embodied either in robotic system 10, control system 50, or both, and / or at least partially in one or more other local or remote systems / devices.

[0049] With reference to FIG. 2, the robotic system 10 generally includes an elongated support structure 14 (also referred to as a "column"), a robotic system base 25, and a console 16 at the top of the column 14. The column 14 may include one or more arm supports 17 (also referred to as a "carriage") to support the deployment of one or more robotic arms 12 (three are shown in FIG. 2). The arm supports 17 may include individually configurable arm mounts that rotate along a vertical axis to adjust the base of the robotic arms 12 for better positioning relative to the patient. The arm supports 17 also include a column interface 19 that allows the arm supports 17 to translate vertically along the column 14.

[0050] In some embodiments, the column interface 19 can connect to the column 14 through slots, such as slots 20, positioned on either side of the column 14 to guide the vertical translation of the arm support 17. The slots 20 contain vertical translation interfaces to position and hold the arm support 17 at various vertical heights relative to the robotic system base 25. The vertical translation of the arm support 17 allows the robotic system 10 to adjust the reach of the robotic arm 12 to meet various table heights, patient sizes, and physician preferences. Similarly, an individually configurable arm mount on the arm support 17 can allow the robotic arm base 21 of the robotic arm 12 to be angled in various configurations.

[0051] The robotic arm 12 may generally comprise a robotic arm base 21 and an end effector 22 separated by a series of linkages 23 connected by a series of joints 24, each joint comprising one or more independent actuators 207. Each actuator may comprise an independently controllable motor. Each independently controllable joint 24 may provide or represent an independent degree of freedom available to the robotic arm. In some embodiments, each of the arms 12 has seven joints, thus providing seven degrees of freedom, including "redundant" degrees of freedom. The redundant degrees of freedom allow the robotic arms 12 to position their respective end effectors 22 at specific positions, orientations, and trajectories in space using different joint positions and joint angles. This allows the system to position and orient the medical instrument from a desired point in space, while at the same time allowing the physician to move the arm joints to a clinically advantageous position away from the patient to create better access while avoiding arm collisions.

[0052] The robotic system base 25 balances the weight of the column 14, arm support 17, and arm 12 on the floor. Thus, the robotic system base 25 can house heavier components such as electronics, motors, power supplies, and components that selectively enable movement or immobilize the robotic system. For example, the robotic system base 25 includes casters 28 in the form of wheels that allow the robotic system to be easily moved around a room prior to a procedure. After reaching the proper position, the casters 28 can be immobilized using wheel locks to hold the robotic system 10 in place during the procedure.

[0053] When positioned at the top of the column 14, the console 16 allows for both a user interface for receiving user input and a display screen (or dual-purpose device, such as, for example, a touch screen 26) for providing both pre-operative and intra-operative data to the physician user. Potential pre-operative data on the touch screen 26 may include pre-operative planning, navigation and mapping data derived from a pre-operative computerized tomography (CT) scan, and / or notes from a pre-operative patient interview. Intra-operative data on the display may include optical information provided by tools, sensor and coordinate information from sensors, and vital patient statistics such as respiration, heart rate, and / or pulse. The console 16 may be positioned and tilted to allow the physician to access the console from the side of the column 14 opposite the arm support 17. From this position, the physician may view the console 16, the robotic arm 12, and the patient while operating the console 16 from behind the robotic system 10. As shown, the console 16 may also include a handle 27 to aid in the manipulation and stabilization of the robotic system 10.

[0054] Each end effector 22 of the robotic arm 12 may include an instrument device manipulator (IDM), which may be attached using a mechanism changer interface (MCI). In some embodiments, the IDM may be removed and replaced with a different type of IDM, e.g., a first type of IDM may operate an endoscope while a second type of IDM may operate a laparoscope. The MCI may include connectors for communicating air pressure, power, electrical signals, and / or optical signals from the robotic arm 12 to the IDM. The IDM may be configured to operate a medical instrument (e.g., a surgical tool / instrument), such as the scope 32, using techniques including, for example, direct drive, harmonic drive, gear drive, belt and pulley, magnetic drive, and the like.

[0055] 3, the control system 50 may include various I / O components 310 configured to assist the physician 5 or others in performing a medical procedure. For example, the input / output (I / O) components 310 may be configured to allow user input to control the scope 32, such as to navigate the scope 32 within the patient 13. In some embodiments, for example, the physician 5 may provide input to the control system 50 and / or the robotic system 10, and in response to such input, control signals may be sent to the robotic system 10 to operate the scope 32. As also shown in FIG. 1, the control system 50 may include one or more display devices 42 to provide various information regarding the procedure. For example, the display 42 may provide information regarding the scope 32. For example, the control system 50 may receive real-time images captured by the scope 32 and display the real-time images via the display 42. Additionally or alternatively, the control system 50 may receive signals (e.g., analog, digital, electrical, acoustic / sonic, pneumatic, tactile, hydraulic, etc.) from medical monitors and / or sensors associated with the patient 13, and the display 42 may present information regarding the health or environment of the patient 13. Such information may include, for example, heart rate (e.g., ECG, HRV, etc.), blood pressure / blood velocity, muscle biosignals (e.g., EMG), body temperature, blood oxygen saturation (e.g., SpO 2 ), CO 2 , may include information displayed via a medical monitor, such as information regarding brain waves (e.g., EEG), environmental and / or local or core body temperature.

[0056] To facilitate the functions of the control system 50, the control system may include various components (sometimes referred to as "subsystems"). For example, the control system 50 may include control electronics / circuitry 302, as well as one or more power sources / interfaces 308, pneumatic devices, light sources, actuators, data storage devices 304, and / or communication interfaces 306. In some embodiments, the control system 50 includes control circuitry that comprises a computer-based control system configured to store executable instructions that, when executed, cause the control system 50 to perform various operations associated with the functions described herein. In some embodiments, the control system 50 is mobile, while in other embodiments, the control system 50 is a substantially stationary system. Although various functions and components are discussed as being implemented by the control system 50, any of such functions and / or components may be integrated with and / or performed by other systems and / or devices, such as, for example, the robotic system 10, the platform 15, or others.

[0057] 1, the medical system 10 can provide various benefits such as providing guidance (e.g., instrument tracking, instrument alignment information, etc.) to assist the physician in performing the procedure, allowing the physician to perform the procedure from an ergonomic position without requiring awkward arm movements and / or positions, allowing one physician to perform the procedure with one or more medical instruments, avoiding radiation exposure (e.g., associated with fluoroscopy techniques), allowing the procedure to be performed in a single motion setting, providing continuous suction to more efficiently remove objects (e.g., removing kidney stones), etc. For example, the medical system 100 can provide guidance information to assist the physician in accessing target anatomical features using various medical instruments while minimizing bleeding and / or damage to anatomical structures (e.g., critical organs, blood vessels, etc.). Additionally, the medical system 100 can provide non-radiation based navigation and / or localization techniques to reduce the physician and patient's exposure to radiation and / or reduce the amount of equipment in the operating room. Additionally, the medical system 100 can provide functionality distributed between at least the control system 50 and the robotic system 10, which may be independently mobile. Such distribution of functionality and / or mobility can allow the control system 50 and / or robotic system 10 to be placed in a location that is optimal for a particular medical procedure, thereby maximizing the working area around the patient and / or providing an optimized location for the physician to perform the procedure.

[0058] The various components of system 100 may be communicatively coupled to one another over a network, which may include wireless and / or wired networks. Exemplary networks include one or more personal area networks (PANs), local area networks (LANs), wide area networks (WANs), Internet area networks (IANs), cellular networks, the Internet, etc. Additionally, in some embodiments, the various components of system 10 may be connected for data communication, fluid / gas exchange, power exchange, etc., via one or more supporting cables, conduits, or the like.

[0059] The robotic system 10 and / or control system 50 include specific user controls 246, 346, which may comprise any type of user input (and / or output) device or device interface, such as one or more buttons, keys, joysticks, handheld controllers (e.g., video game type controllers), computer mice, track pads, track balls, control pads, and / or sensors (e.g., motion sensors or cameras) that capture hand and finger gestures, and / or interfaces / connectors therefor. The user controls 246, 346 are communicatively and / or physically coupled to at least a portion of the control circuits 202, 302, respectively.

[0060] In some embodiments, the user controls 246, 346 and / or associated control circuitry are configured to allow a user to control a medical instrument, such as an instrument operable at least in part by a robotic system (e.g., an endoscope or nephroscope), using the controls to control pitch and yaw movements of the distal end of the instrument. For example, movements on a joystick may be mapped to yaw and pitch movements at the distal end of the scope / device. In some embodiments, the user controls are configured to provide haptic feedback to the user. For example, the joystick or other control mechanism may vibrate to indicate invalid or potentially problematic inputs. In some embodiments, the control system 50 and / or robotic system 10 may also provide visual feedback (e.g., a pop-up message) and / or audio feedback (e.g., a beep) to indicate a problem associated with the robotic operation.

[0061] In some embodiments, the control circuitry 202 / 302 may control a medical instrument (e.g., an endoscope) using a three-dimensional (3D) map of the patient's anatomy and / or a predetermined computer model of the patient. For example, the control circuitry 202 may be configured to provide control signals to the robotic arm 12 of the robotic system 10 to manipulate and position an associated instrument at a target location, position, and / or orientation / alignment. For embodiments implementing 3D mapping, the position control mode may require sufficiently accurate mapping of the patient's anatomy.

[0062] In some embodiments, a user can manually manipulate the robotic arm 12 of the robotic system 10 without the use of electronic user controls. For example, during a surgical room setup, a user may move the robotic arm 12 and / or any other medical instruments to provide desired access to a patient. The robotic system 10 may rely on force feedback and inertial control from the user to determine the appropriate configuration of the robotic arm 12 and associated instrumentation.

[0063] The display device 42 of the control system 50 may be integrated with the user controls 346, for example as a tablet device with a touch screen that provides user input. The display device 42 may be configured to provide data and input commands to the robotic system 10 using the integrated display touch controls. The display device 42 may be configured to display a graphical user interface that shows information regarding the position and orientation of the patient and / or various instruments operating within the system based on information provided by one or more position sensors. In some embodiments, a position sensor associated with a medical instrument (e.g., an endoscope) may be configured to generate a signal indicative of the position and transmit the same over a wire and / or transmitter coupled to the sensor. Such connectivity components may be configured to transmit the position information to the console base 51 for processing thereof by the control circuitry 302 and for presentation via the display device.

[0064] In the example of FIG. 3, the control system 50 is illustrated as a cart-based system that is mobile with one or more wheels 312. However, the control system 50 may be implemented as a stationary system, integrated into another system / device, etc. Although certain components of the control system 50 are illustrated in FIG. 3, it should be understood that additional components not illustrated may be included in embodiments according to the present disclosure. Furthermore, in some embodiments, some of the illustrated components may be omitted. Although the control circuit 302 is illustrated as a separate component in the diagram of FIG. 3, it should be understood that any or all of the remaining components of the control system 50 may be at least partially embodied in the control circuit 302.

[0065] The localization component 314 may be configured to perform one or more localization techniques to determine and / or track the position and / or orientation of an object, such as a medical instrument, and / or a target anatomical feature (e.g., a nipple). For example, the localization component 314 may process input data (e.g., sensor data from a medical instrument, model data related to a patient's anatomy, patient position data, pre-operative data, robotic commands, and / or kinematic data, etc.) to generate position / orientation data 320 of one or more medical instruments. The position / orientation data 320 may indicate the location and / or orientation of one or more medical instruments relative to a viewpoint. The viewpoint may be relative to a patient's anatomy, a known object (e.g., an EM field generator), a coordinate system / space, a coordinate frame defined by a robotic system / cart, etc.

[0066] In some embodiments, the localization component 314 can process pre-operative data to determine the location and / or orientation of the object. The pre-operative data (sometimes referred to as "mapping data") can be generated by performing a computed tomography (CT) scan, such as a low-dose CT scan. In some embodiments, the present disclosure provides a mechanism for determining / generating mapping data based on electromagnetic field location data recorded using an endoscopic electromagnetic sensor disposed within a target anatomical structure (e.g., the calyx of a patient's kidney). Furthermore, in some embodiments, the localization component 314 can direct vision-based techniques to determine the location and / or orientation of the target anatomical feature. For example, the medical instrument can include a camera, a distance sensor (sometimes referred to as a "depth sensor"), a radar device, etc., to provide sensor data in the form of visual data.

[0067] The localization component 314 can direct the processing of the visual data to facilitate vision-based location tracking of the medical instrument. For example, pre-operative model data can be used with the visual data to enable computer vision-based tracking of the medical instrument (e.g., endoscope). Additionally, in some embodiments, other types of vision-based techniques can be implemented to determine the position and / or orientation of an object. For example, the medical system 10 can be configured to track to determine the motion of an image sensor (e.g., a camera or other sensor) and thus the medical instrument associated with the image sensor. The use of mapping data can also enhance vision-based algorithms or techniques. Additionally, the localization component 314 can direct the use of optical flow processing, another computer vision-based technique, to analyze the displacement and / or translation of image pixels in a video sequence in the visual data to infer camera movement. Examples of optical flow techniques can include motion detection, object segmentation calculations, luminance, motion compensated coding, stereo disparity measurements, etc.

[0068] In some embodiments, the localization component 314 and the control circuitry 302 can use electromagnetic tracking to determine the location and / or orientation of an object. For example, the localization component 314 can use real-time EM tracking to determine the real-time location of the medical instrument in a coordinate system / space, which may be registered to the patient's anatomy, and the location may be represented by a pre-operative or other model. In EM tracking, an EM sensor (or tracker) including one or more sensor coils may be embedded in one or more locations and / or orientations within the medical instrument (e.g., a scope, needle, etc.). The EM sensor can measure variations in an EM field created by one or more static EM field generators positioned at known locations. The location information detected by the EM sensor can be stored as EM data. The localization component 314 can process the EM data to determine the location and / or orientation of an object, such as a medical instrument. The EM field generator (or transmitter) can be placed near the patient (e.g., within a predetermined distance) to create a low-intensity magnetic field that the EM sensor can detect. The magnetic field can induce small currents in the sensor coils of the EM sensor, which can be analyzed to determine the distance and angle between the EM sensor and the EM field generator. These distances and orientations can be "recorded" intraoperatively on the patient's anatomy (e.g., a pre-operative model) to determine a geometric transformation that aligns a single location in a coordinate system with a location in the pre-operative model of the patient's anatomy. Once recorded, an EM sensor (e.g., an embedded EM tracker) at one or more locations on the medical instrument (e.g., the distal tip of an endoscope, a needle, etc.) can provide a real-time indication of the position and / or orientation of the medical instrument through the patient's anatomy.

[0069] In some embodiments, the localization component 314 and the control circuitry 302 can use a combination of input data. For example, the control circuitry 302 can use a probabilistic approach in which confidence weights are assigned to positions / orientations determined from multiple forms of input data. To illustrate, if the EM data is unreliable (such as in the presence of EM interference), the EM data may be associated with a relatively low confidence value and other forms of input data, such as visual data, robotic commands, kinematic data, etc., can be relied upon.

[0070] The target / trajectory component 316 may be configured to determine the position of the target location within a human anatomy and / or coordinate space / system. It should be understood that a "target" as described in this disclosure may also be a reference to an anatomical region / feature, such as the surface of the nipple. For example, in some embodiments, both the location of the target and the uncertainty associated with the location of the target may be captured based on sources of error, such as sensor calibration. The target location may represent a point / set of points within the human anatomy and / or coordinate space / system. For example, the target / trajectory component 316 may identify one or more points of the target location within the coordinate system, identify coordinates of the one or more points (e.g., X, Y, Z coordinates for each point), and associate the coordinates with the target location. In some embodiments, the target / trajectory component 316 may use the position and / or orientation of the medical instrument to determine the position of the target location. For example, the scope may be navigated to touch or be within close range of the target location (e.g., resting in front of the target location).

[0071] The target location may represent a fixed or movable point within a human anatomy and / or coordinate space / system. For example, if the nipple is initially designated as the target location, the coordinates of the target location may be determined and updated as the procedure progresses and the nipple moves (e.g., due to insertion of a medical instrument). Here, the location of the scope (which may be within close range of the nipple) may be tracked over time and used to update the coordinates of the target location. In some embodiments, the target / trajectory component 316 may estimate / determine the position of the target location (e.g., target anatomical feature).

[0072] In some embodiments, the target trajectory and / or the medical instrument trajectory may be defined / expressed with respect to one or more anatomical planes / axes. For example, the trajectory may be defined / expressed as an angle relative to the coronal / sagittal / transverse plane or another plane / axis (e.g., 20 degree cranial-caudal angle, 10 degree medial-lateral angle, etc.). To illustrate, the control system 50 may determine the orientation of the medical instrument relative to the EM field generator and / or the location of the target relative to the EM field generator. The control system 50 may also determine the orientation of the EM field generator relative to the robotic system based on the robot kinematics. In some cases, the control system 50 may infer / determine that the robotic system is parallel to the bed. Based on such information, the control system 50 may determine the target trajectory and / or the medical instrument trajectory with respect to an anatomical plane, such as an angle with respect to an anatomical plane of the patient on the bed.

[0073] The target / trajectory component 316 may also be configured to determine a target trajectory of the medical instrument or another object. The target trajectory may represent a desired path to access a target location and / or anatomical feature. The target trajectory may be determined based on various information such as the location of the medical instrument (e.g., needle, scope, etc.), the target location within the human anatomy, the patient's position and / or orientation, the patient's anatomy (e.g., the location of an organ within the patient relative to the target location), etc. For example, the target trajectory may include a line extending from the location of the medical instrument and / or a location on the patient's skin to / through the location of the target location within the patient. In an example, a physician may provide input specifying the target trajectory, such as by analyzing an image or model of the human anatomy and drawing a line on an image of the patient's internal anatomy. In some embodiments, the target / trajectory component 316 may calculate the target trajectory initially and / or update the target trajectory throughout the procedure. For example, as the target location moves during the procedure, the target trajectory may be updated due to changes in the location of the target location. In an example where the target location is estimated, the target trajectory may represent an estimated path to reach the target location.

[0074] The user interface component 318 may be configured to facilitate one or more user interfaces (also referred to as one or more graphical user interfaces (GUIs)). For example, the user interface component 318 may generate user interface data 322 representing a scope guidance and / or needle guidance interface 324 including one or more visualizations to indicate the orientation and / or position of the medical instrument. The user interface component 318 may use position / orientation data 320 relating to one or more medical instruments, information regarding the target location, and / or information regarding the target trajectory to display one or more visual features / icons indicative of the target scope position, needle trajectory, and / or the like, as represented by the interface data 322. Furthermore, such icons / features may be shown / represented relative to and / or overlaid on the scope vision / image data / representation. For example, the needle guidance feature and / or the scope guidance feature may be overlaid on the scope image window to provide an expanded scope image view / window. The user interface component 318 can provide user interface data 322 or other data to one or more displays 42 and / or another display for displaying a representation thereof.

[0075] The one or more communication interfaces 306 may be configured to communicate with one or more devices / sensors / systems, such as via wireless and / or wired network connections. A network according to an embodiment of the present disclosure may include a local area network (LAN), a wide area network (WAN) (e.g., the Internet), a personal area network (PAN), a body area network (BAN), etc. In some embodiments, the one or more communication interfaces 1206 may implement wireless technologies such as Bluetooth, Wi-Fi, near field communication (NFC), etc.

[0076] 3, the control system 50 may include and / or control other components, such as one or more pumps, flow meters, valve controls, and / or fluid access components, to provide controlled irrigation and / or aspiration capabilities to a medical instrument (e.g., a scope), a device that may be deployed through the medical instrument, and the like. In some embodiments, irrigation and aspiration capabilities may be delivered directly to the medical instrument through separate cables. In some embodiments, the control system 50 may be coupled to the robotic system 10, the stage 15, and / or a medical instrument, such as a scope 32 and / or a needle or other percutaneous access instrument (not shown), via one or more cables or connections (not shown).

[0077] FIG. 4 illustrates a ureteroscope 440 disposed in a portion of a patient's urinary system, according to one or more embodiments of the present disclosure. As referenced above, ureteroscopic procedures can be implemented to investigate and / or treat abnormalities within a human ureter. For example, ureteroscopic procedures can be implemented to treat and / or remove kidney stones. Such procedures can be performed at least in part manually and / or at least in part using robotic technology, such as the robotic system 10 shown in FIG. 1. For example, the use of robotic devices and / or systems for certain endoscopic procedures can provide relatively greater precision, control, and / or coordination compared to strictly manual procedures. In some embodiments, the scope 440 includes a working channel 444 for deploying medical instruments (e.g., lithotriptors, basket devices, forceps, etc.), irrigation, and / or suction to a working area at the distal end of the scope.

[0078] The scope 440 may be articulatable, such as with respect to at least a distal portion of the scope, so that the scope may be maneuvered within the human anatomy. In some embodiments, the scope 440 is configured to be articulated with five degrees of freedom, including, for example, XYZ coordinate movement, as well as pitch and yaw. In some embodiments, the needle sensor provides six degrees of freedom, including X, Y, and Z ordinate positions, as well as pitch, roll, and yaw. The position sensors of the scope 440 may similarly have similar degrees of freedom with respect to the position information they generate / provide. The figures illustrate multiple degrees of motion of the scope 440, according to some embodiments. As shown in FIG. 4, the tip 442 of the scope 440 may be oriented with zero deflection relative to its longitudinal axis 406 (also referred to as the "roll axis").

[0079] To capture images at different orientations of the tip 442, the robotic system may be configured to deflect the tip 442 on the positive yaw axis 402, the negative yaw axis 403, the positive pitch axis 404, the negative pitch axis 405, or the roll axis 406. The tip 442 or the body 445 of the scope 442 may be extended or translated in the longitudinal axis 406, the x-axis 408, or the y-axis 409. The scope 440 may include a reference structure (not shown) for calibrating the position of the scope. For example, the robotic system may measure the deflection of the scope 440 relative to the reference structure. The reference structure may be located, for example, on the proximal end of the endoscope 440 and may include a key, a slot, or a flange. The reference structure may be coupled to a first drive mechanism for initial calibration and coupled to a second drive mechanism to perform the surgical procedure.

[0080] For robotic implementations, a robot arm of a robotic system may be configured / configurable to manipulate the scope 440 using an elongated motion member. The elongated motion member may include one or more pull wires (e.g., pull or push wires), cables, fibers, and / or a flexible shaft. For example, the robot arm may be configured to actuate multiple pull wires (not shown) coupled to the scope 440 to deflect the tip 442 of the scope 440. The pull wires may include any suitable or desired material, such as metallic and non-metallic materials, such as stainless steel, Kevlar, tungsten, carbon fiber, and the like. In some embodiments, the scope 440 is configured to exhibit non-linear behavior in response to forces applied by the elongated motion members. The non-linear behavior may be based on the stiffness and compressibility of the scope, as well as the slack or stiffness variability between different elongated motion members.

[0081] The scope (e.g., endoscope / ureteroscope) 440 may comprise a tubular, flexible medical instrument configured to be inserted into a patient's anatomy to capture images of the anatomy. In some embodiments, the scope 440 may house an optical assembly that may include an imaging device 448, such as an optical camera, and wires and / or optical fibers for transmitting signals to / from a distal end 442 of the scope 440.

[0082] A camera / imaging device 448 can be used to capture images of an internal anatomical space, such as a target cupule / papilla of the kidney 70. The scope 440 may be further configured to accommodate an optical fiber for carrying light from a proximally located light source, such as a light emitting diode, to the distal end 442 of the scope. The distal end 442 of the scope 440 may include a port for a light source to illuminate the anatomical space when the camera / imaging device is used. In some embodiments, the scope 440 is configured to be controlled by a robotic system similar in one or more respects to the robotic system 10 shown in FIGS. 1 and 2. The imaging device may comprise an optical fiber, a fiber array, and / or a lens. The optical components move with the tip of the scope 440 such that movement of the tip of the scope results in a change in the image captured by the imaging device 448.

[0083] For percutaneous nephrolithotomy (PCNL) procedures, access is made through the patient's skin and intervening tissue into the target calyx. Generally, the preferred access to the renal calyx is through the soft tissue papillary structures, and access through such tissues may generally be associated with a reduced risk of bleeding and / or other complications. In addition to the absence of bleeding when the needle is inserted through the papillary structures, such a route may provide complete access to the interconnected internal channels (e.g., calyx) of the kidney.

[0084] Although PCNL represents a relatively effective method for treating large kidney stones, many physicians opt for other procedures due in part to the difficulty of accurately targeting the target papilla / calyx. More specifically, performing PCNL involves using a needle to gain percutaneous access to the target calyx of the kidney through the patient's flank. This step can be considered crucial to the ultimate success of the procedure, as the physician must select a needle path to the kidney that does not traverse surrounding organs and allows a rigid instrument to reach and treat the urinary stone. If the physician is unable to do so effectively, there is a risk of causing visceral or pleural damage or failing to completely treat the stone. Due to these challenges, the learning curve associated with gaining percutaneous needle access to perform PCNL in a suitable patient position (e.g., modified supine) is very high.

[0085] In some procedures, the physician studies the patient's preoperative computed tomography (CT) images to determine the location of the urinary stone, the location of the surrounding organs and bone structures, and to examine the morphometry of the calyx. With this knowledge, the physician can mentally generate a preoperative plan for the percutaneous needle path. Typically, the physician must identify the posterior calyx to puncture to accommodate the rigid instrument. Specifically, the posterior calyx generally provides a relatively straight launch into the renal pelvis. The physician must attempt to insert the needle into the kidney through the papilla to avoid damaging the renal vasculature and causing bleeding. Intraoperatively, the physician relies on fluoroscopy or ultrasound in some procedures to guide the alignment and insertion of the needle into the target calyx. However, the resolution and interpretation difficulties associated with such imaging techniques can result in a relatively high degree of difficulty in adequately performing the needle puncture. Thus, embodiments of the present disclosure that provide improved tracking and visualization of target anatomical features such as the papilla and calyx may improve surgical outcomes and attract more physician interest than other PCNL methodologies.

[0086] In some embodiments, the medical instrument (e.g., scope) 440 includes a sensor configured to generate and / or transmit sensor position data to another device. The sensor position data may indicate a position and / or orientation of the medical instrument 440 (e.g., its distal end 442) and / or may be used to determine / estimate the position / orientation of the medical instrument. For example, the sensor (sometimes referred to as a "position sensor") may include an electromagnetic (EM) sensor with a coil of conductive material or other form / embodiment of an antenna.

[0087] 4 shows an EM field generator 415 configured to broadcast an EM field 90 that is detected by an EM sensor on the medical instrument. The magnetic field 90 can induce a small current in the coil of the EM position sensor that can be analyzed to determine the distance and / or angle / orientation between the EM sensor and the EM field generator 415. Additionally, the medical instrument / scope 440 can include other types of sensors, such as shape-sensing fibers, accelerometers, gyroscopes, satellite-based positioning sensors (e.g., global positioning system (GPS) sensors), radio frequency transceivers, etc. In some embodiments, the sensors on the medical instrument can provide sensor data to a control system, which is then used to determine the position and / or orientation of the medical instrument. In some embodiments, the position sensor is positioned on the distal end 442 of the medical instrument 440, while in other embodiments, the sensor is positioned elsewhere on the medical instrument. The ureteroscope can be driven to a position proximate to the target papilla.

[0088] In some implementations, as described in further detail below, the distal end of the ureteroscope 440 may be advanced to contact a target anatomical feature (e.g., the papilla). When a position sensor associated with the distal end of the scope 440 is in contact with and / or proximity to the target anatomical feature, the position of the distal end of the scope 440 may be recorded as a target percutaneous access position at which a percutaneous access instrument (e.g., a needle) may be directed to pass through the papilla to access the target cupule.

[0089] Certain embodiments of the present disclosure advantageously help automate and guide a physician through the process for gaining percutaneous access to a target anatomical feature. For example, electromagnetic positioning and scope imaging can be used together to guide the insertion of a needle into a patient. Such a solution can enable a non-skilled physician to gain access into a kidney in, for example, a modified supine position and perform PCNL.

[0090] Certain embodiments of the present disclosure involve position sensor guided percutaneous access to a target treatment site, such as a target location within a kidney. For example, if the scope 440 is fitted with one or more electromagnetic sensors and the nephroscope access needle further includes one or more electromagnetic sensors, and such sensors are exposed to the electromagnetic field 90 created by the electric field generator 415, the associated system control circuitry can be configured to detect and track their location. In some embodiments, the tip of the ureteroscope 440 acts as a guidance beacon while the user is inserting the percutaneous access needle. Such a solution can allow the user to hit the target from a variety of approaches, thereby eliminating the need to rely on fluoroscopy or ultrasound imaging.

[0091] In some embodiments, a control system (not shown in FIG. 4 ) associated with the scope 440 is configured to perform localization / positioning techniques to determine and / or track the location / position of the scope 440 and / or a medical instrument, such as a percutaneous access needle (not shown). In some examples, as described above, the EM field generator 415 is configured to provide an EM field 90 in the patient's environment. The scope 440 and / or the percutaneous access needle may include an EM sensor configured to detect EM signals and transmit sensor data related to the detected EM signals to the control system. The control system may analyze the sensor data to determine the position and / or orientation of the scope 440 (e.g., the distance and / or angle / orientation between the EM sensor and the EM field generator 415). Alternatively or additionally, in some examples, the control system may use other techniques to determine the position and / or orientation of the scope 440. For example, the scope 440 (and / or needle) may include shape-sensing fibers, accelerometers, gyroscopes, accelerometers, satellite-based positioning sensors (e.g., Global Positioning System (GPS)), radio frequency transceivers, etc. The control system can receive sensor data from the scope 440 and determine its position and / or orientation. In some embodiments, the control system can track the position and / or orientation of the scope 440 in real time relative to the patient's coordinate system and / or anatomy.

[0092] The scope 440 may be controllable in any suitable or desirable manner, either based on user input or automatically. The controls 411, 412 provide examples that may be used to receive user input. In some embodiments, the controls for the scope 440 are located on the proximal handle of the scope, which may be relatively difficult to grasp in some procedural postures / positions as the orientation of the ureteroscope changes. In some embodiments, the scope 440 is controlled using two-handed controllers, as in image 412. Although the controllers 411, 412 are shown as handheld controllers, user input may be received using any type of I / O device, such as a touch screen / pad, mouse, keyboard, microphone, etc.

[0093] FIG. 5 (represented in portions 5-1 and 5-2) is a flow diagram illustrating a process 500 for accessing a target cupule or other organ of a patient, according to one or more embodiments of the present disclosure. FIG. 6 (represented in portions 6-1 and 6-2) shows certain images corresponding to various blocks, states, and / or operations associated with the process of FIG. 5, according to one or more embodiments. The process 500 may involve percutaneous access to the kidney 70 for removal of kidney stones (e.g., PCNL). Such percutaneous access may be desirable for extraction of stones that are large enough that removal via a ureteroscope is impractical or undesirable. For example, stones may be larger than 2 cm in diameter, while certain ureteroscopes have a working channel through which stones or fragments having a diameter of about 1.2 mm may be removed. Although breaking up stones into smaller fragments for removal via ureteroscopy works in many cases, studies have shown that the remaining stone fragments are often the source of new stone formation, requiring similar treatment in the future. The process described herein is described in the context of ureteroscopy, but may be applied to any other type of surgical procedure that utilizes a position sensor (e.g., an electromagnetic field sensor) and / or a camera to track a target anatomical feature, such as a papilla or urinary stone.

[0094] At block 502, the process 500 involves accessing the kidney through the patient's ureter using a ureteroscope 540, as described above. Specifically, the acts of block 502 may involve advancing the scope 540 through the ureter 63, past the renal pelvis 71, and into one or more calyxes or nearby areas.

[0095] At block 504, the process 500 involves locating a kidney stone 580 for which the patient is to receive treatment using an image capture device (e.g., a camera) associated with the distal end 542 of the endoscope. For example, the kidney stone 580 may be extracted, at least in part, as an objective of the process 500.

[0096] At block 506, the process 500 entails identifying a target papilla 579 exposed within the target calyx 575, through which access to the kidney stone 580 may be achieved. Identifying the target papilla 579 may be important to create a workable tract through which access to the kidney stone 580 may be achieved via percutaneous access. For example, it may be necessary to determine an angle that is appropriate for access with a relatively rigid nephroscope in such a way as to access a calyx (e.g., minor calyx 575) through which the kidney stone 580 may be reached. In some implementations, it may be desirable or necessary to reach the kidney stone 580 through the posterior calyx to provide sufficiently straight access to the ureteropelvic junction 71. In general, the target minor calyx may be considered a relatively small target. For example, such a calyx may be approximately 5-8 mm in diameter. Thus, precise targeting may be important to effectively extract the kidney stone.

[0097] The path through which needle / nephroscope access to the target cup 575 is achieved may advantageously be as straight as possible to avoid hitting the blood vessels surrounding the renal pyramid 576 associated with the papilla 579 through which the needle / nephroscope may be positioned. Furthermore, the location of various critical patient anatomical structures may require navigation through a constrained window of the patient's tissue / anatomy. For example, the inferior pole cup below the 12th rib may provide suitable access to avoid the pulmonary pleura. Furthermore, this access path may advantageously be medial to the posterior axillary line (e.g., approximately 1 cm below and 1 cm medial to the tip of the 12th rib) to avoid the colon and / or paraspinal muscles. Additionally, this access path may advantageously avoid being within the proximity of the ribs and avoid the intercostal nerves. Furthermore, by targeting the entry within the area of ​​the axial center of the cup 575, major arteries and / or other blood vessels may be avoided in some cases.

[0098] At block 508, the process 500 entails tagging / recording the location of the exposed nipple 579 within the target cup 579 through which the desired access is to be achieved. For example, the location information / data may be represented / identifiable in a three-dimensional space, such as electromagnetic field space or robot space (e.g., coordinate frame).

[0099] To record the position of the nipple 579, the scope 540 may be advanced until it physically touches / contacts the target nipple 579, as indicated by the advanced scope tip 543, and in association therewith, such contact location may be identified and / or otherwise indicated by the scope 540 and / or the operator as a target location. In some implementations, an electromagnetic beacon or other sensor device associated with the distal end / tip 542 of the ureteroscope may indicate the target location, thereby recording the target location in the electromagnetic field space. After contacting / touching the nipple 579 and recording the location, the end 542 of the scope may be retracted, and the depth of such retraction may be measured in some manner. In some implementations, the operator may generally be in an occluded / blacked out state when contact is made, and may be notified that the distal end 543 of the scope 540 is contacting the nipple 579 by monitoring the camera image generated thereby. In some implementations, a user input device (e.g., pendant type) may be used to inform the system that contact is being made with the target anatomical feature.

[0100] At block 510, the process 500 involves percutaneously introducing a medical instrument 550, such as a needle, into a patient. For example, such access may occur via the patient's flank in some implementations. At block 512, the process 500 involves directing the percutaneously advanced medical instrument 550 towards the target location, ultimately traversing the target papilla 579 and therethrough to access the target cupule 575.

[0101] In some embodiments, visual confirmation of the entry of the tip of the needle 550 into the target cup 575 may be provided by a camera on the scope 540. For example, the scope 540 may be retracted from the target location, as described above, thereby providing a field of view that includes the nipple 579 within the cup 575 as the tip of the needle 550 is seen as it protrudes through the surface of the nipple 579.

[0102] Once a target location is recorded, a percutaneously inserted medical instrument (e.g., needle 550) may be directed toward the recorded location. However, if such a recorded location is static, anatomical motion occurring after recording of the target location may result in a target location that does not accurately reflect the real-time location associated with the target anatomical feature through which access is desired. For example, the act of inserting needle 550 into a patient may cause certain anatomical structures surrounding the target organ (e.g., kidney 70) and / or the target organ itself to move and / or become distorted / unnatural in some manner, thereby causing the target anatomical feature (e.g., papilla 579) to assume a different position / shape than at the time the target access location was recorded. For renal procedures, the ureteroscope 540 may be fixed in position at the renal pelvis 71 such that deformation and / or motion of the kidney 70 relative to the ureteroscope may result in such target location damage. Thus, the papilla 579 may not be tracked accurately once anatomical motion is introduced into the system.

[0103] Once needle access to the cupule 575 has been achieved, a larger diameter device may be exchanged for the needle 550 to provide a larger port for stone removal. In some implementations, the needle 550 comprises a stylet and a cannula. The stylet may be removed with the needle tip advanced into the cupule 575, leaving the cannula to create an open port to the location of the kidney stone. A guidewire may be placed through the cannula and used to perform the remainder of the process of removing the stone 580. For example, the guidewire may be used to pass a deflated balloon or dilator over the wire. The balloon or dilator may be expanded to create a port large enough to introduce a hollow suction tube, such as a nephrostomy tube, directly into the cupule 575. At this point, a nephroscope or any one of several other instruments may be introduced into the suction tube to aid in the removal of the stone 580. For example, lithotriptors, lasers, ultrasound, baskets, graspers, drainage tubes, etc. may be used to remove the stone or its fragments, and / or a drainage tube, such as a nephrostomy catheter, may be deployed under the suction tube to reduce intrarenal pressure. Any combination of such functionality may be embodied in the nephroscope (not shown) and / or ureteroscope 540.

[0104] Target location Various aspects of the present disclosure relate to systems, devices, and methods for localizing targets (e.g., target anatomical features) associated with medical procedures. In particular, localizing targets according to the present disclosure may involve various steps and / or functions such as recording / tagging the location of the target anatomical feature (e.g., papilla) using an endoscope (e.g., ureteroscope), determining / recording a position offset / translation between the target anatomical feature and the endoscope (e.g., a position sensor associated therewith) for the purpose of determining the location of the target anatomical feature based on the position of the endoscope not in physical contact with the target anatomical feature, and / or dynamically updating the target location associated with the target anatomical feature based on electromagnetic sensor and / or camera data associated with the ureteroscope. As described, static position markers may be registered / recorded to identify target locations associated with the target anatomical features / landmarks. In some embodiments, the present disclosure provides systems, devices, and methods for guiding and / or automating endoscopes and / or percutaneous access instruments based at least in part on static position markers in view of certain target localization techniques. Target localization according to embodiments of the present disclosure can be applied to any type of robotic endoscopy procedure.

[0105] FIG. 7 is a flow diagram illustrating a process 700 for localizing a target anatomical feature, according to one or more embodiments. In general, target localization may be performed to locate a target anatomical feature (e.g., a papilla) relative to a ureteroscope. The target location may be recorded / stored with respect to an electromagnetic field generator / space, a robotic coordinate frame, and / or an anatomical coordinate frame defined, for example, by renal mapping. In block 710, the process 700 involves advancing a medical instrument, such as a scope (e.g., a ureteroscope), to a treatment site, such as a lumen or chamber disposed at least partially within a target organ. For example, the operation of block 710 may involve advancing a medical instrument to a target calyx of a kidney of a patient.

[0106] As described above, robotic endoscope-guided percutaneous access according to aspects of the present disclosure can use target localization techniques relative to a target anatomical feature to guide / determine a percutaneous access path to access the target anatomical feature / site. For example, position tracking mechanisms / sensors associated with the distal end of the medical instrument (e.g., scope) and / or percutaneous access instrument (e.g., needle) can be implemented to guide the physician / technologist in aligning the percutaneous access instrument with the treatment site (e.g., target cup). Accurate real-time target localization / tracking, as enabled by aspects of the present disclosure, can enable relatively precise single-puncture access to the treatment site.

[0107] At block 720, process 700 involves determining the location of the target anatomical feature. For example, determining the location of the target anatomical feature may be performed in any suitable or desirable manner, such as using at least partially a contact-based location determination sub-process 722 or at least partially an image-based location determination sub-process 721, described below in connection with blocks 724 and 723, respectively.

[0108] With respect to a particular contact-based location process, in block 722, the process 700 involves contacting a target anatomical feature of the treatment site with the distal end of a medical instrument. For example, the medical instrument may include a sensor device, such as an electromagnetic sensor / beacon, that may indicate the location of the distal end of the medical instrument, and thus, when the distal end of the medical instrument is placed against and / or adjacent to the target anatomical feature, such location measurements may be relied upon as indicating the current location of the target anatomical feature. Contact-based location may not be necessary when an image processing approach is implemented to provide a 3D location / position of the target. For example, in block 723, the process 700 involves determining the location of the target anatomical feature using image data input from an endoscopic camera.

[0109] Process 700 proceeds to sub-process 730, which may involve tracking / locating the target anatomical feature over the course of the procedure while advancing a percutaneous access instrument, such as a needle or equivalent, on / along the access path in the direction of the target anatomical feature. In some implementations, electromagnetic (EM) position sensing techniques are used to track / locate the target anatomical feature. For example, as described above, the target anatomical feature (e.g., papilla) may be contacted with a distal end portion of the scope at one or more locations / regions, and the local location and orientation (e.g., infundibulum axis) of the target feature may be determined based thereon relative to the position of the scope. In some embodiments, as described in further detail below with respect to FIGS. 15-17, a mapping of the target site (e.g., target cupule / papilla and associated infundibulum) may be generated based on multiple recorded locations from the EM sensor data.

[0110] Sub-process 730 may be implemented in a variety of ways. For example, as shown as sub-process 732, live direct instrument (e.g., scope) targeting / tracking may be implemented to provide motion tracking of the target anatomical feature. For example, throughout the relevant surgical period, the distal end of the medical instrument may be maintained in contact with the target anatomical feature such that position sensor data exhibited by the medical instrument may provide a real-time accurate location of the target anatomical feature (block 734). Thus, as shown in block 736, the live position of the medical instrument may be targeted to provide a desired percutaneous access path. However, when the distal end of the medical instrument is in close proximity / contact with the target anatomical feature, real-time visualization of the target anatomical feature may not be possible or may not be clear enough due to occlusion of the target anatomical feature by the feature itself within the camera's field of view. That is, the camera associated with the local instrument may be sufficiently blocked or obscured by the mass of the target anatomical feature, thereby preventing the physician / user from having visual confirmation of penetration of the target anatomical feature by the percutaneous access instrument (e.g., needle).

[0111] An alternative sub-process 731 is shown for tracking the target anatomical feature while still maintaining a clear view of the target anatomical feature during approach of the percutaneous access instrument. Sub-process 731 involves locating the target anatomical feature using a determined position offset / translation between the position of the scope and the position of the target anatomical feature, and determining the live / current position of the target anatomical feature by applying the offset / translation to the current position of the scope.

[0112] In block 733, sub-process 731 may involve recording the determined location of the contact location of the target feature associated with contact with the target anatomical feature performed in connection with the operations of block 720 above. As an example, a user may provide input to inform an associated control / medical system of the feature contact location of the target anatomical feature by tagging / recording in some manner the location of an exposed surface of the target anatomical feature (e.g., the exposed papillary surface in the target cupule). Such tagging may be implemented through the provision of user input in some manner or may be substantially automatic based on perceived tissue contact or the like. The location data may be available in volatile and / or non-volatile data storage of a particular control circuit as shown and described herein.

[0113] After determining the location / position of the target anatomical feature, the scope may be retracted and / or parked to face the target anatomical feature (e.g., the nipple) and provide visualization thereof, as shown in block 735. Such parking may be performed with the aid of certain scope guidance features / overlays presented in or near the camera view interface window, as described in more detail below with respect to Figures 9-13.

[0114] Rather than maintaining a medical instrument (e.g., scope) in contact / proximity with the target anatomical feature as in sub-process 732 to provide live motion tracking, sub-process 731 may involve determining the position of the target anatomical feature based on a determined position offset / translation between the position / orientation of the parked scope and the position / orientation of the target anatomical feature. When parking the scope, the scope may be retracted a distance (e.g., proximally) away from the target anatomical feature to enable the medical instrument to clearly capture the target anatomical feature within the field of view of its associated camera. For example, in some implementations, the physician / user may notify the system in some manner when the medical instrument is parked a desired distance away from the target anatomical feature.

[0115] For clarity, it should be noted that sub-processes 731, 732 represent alternative embodiments of sub-process 730. That is, process 700 may generally involve the implementation of either sub-process 732 or sub-process 731, but not both.

[0116] In some cases, it may be assumed that the offset / translation between the nipple and the scope is generally maintained for a long period of time in order for the scope to remain inside the target cup. In the absence of relative movement of the target anatomical feature with respect to the scope position sensor, the target position may be continuously updated based on the determined current scope position. The position data (e.g., EM data) collected in association with the retraction / reorientation of the scope may be used to determine the offset / translation of the nipple location / orientation with respect to the scope. For example, according to one use case, the retraction / positioning of the scope may be approximately 5 mm in front of the nipple and 2 mm to the left. Such a position offset may be used to determine the position of the target with respect to the current position of the scope. The translation / offset information may further incorporate orientation information, which may be possible in any suitable or desirable manner. If there is a relative motion between the target anatomical feature and the scope, the determined offset / translation may become unreliable. In some implementations, relative motion compensation may be implemented to compensate and / or adjust the offset / translation when the relative position / orientation between the scope and the target anatomical feature changes.

[0117] Sub-process 731 may or may not include / involve contact 724 and retract 735 steps, where the user physically contacts the location of the target nipple and retracts the scope to present the nipple within the field of view of the scope. For example, if image-based tagging 721 is performed in connection with block 720, it may not be necessary to physically contact the target anatomical feature to determine its position / location. Rather, the position / location may be determined using a target identification mechanism based on image data captured / generated by one or more cameras of the scope / instrument. For example, in some embodiments, the target is identified and tracked using multiple frames of image / visual and / or position (e.g., EM) data. An example of such target position determination is described below in connection with FIG. 18. In some implementations, the target position may be estimated / determined with respect to three-dimensional space by viewing the target anatomical feature (e.g., nipple) from two distinctly different positions and / or alignments, as described below in connection with FIG. 18.

[0118] At block 739, sub-process 731 involves targeting the tracked location of the target anatomical feature with a percutaneous access instrument. For example, the centroid of an identified nipple shape or morphology within a real-time image of the treatment site may be used in the target location for the percutaneous access instrument. At block 740, process 700 involves puncturing the target anatomical feature either without visually identifying the target anatomical feature in connection with sub-process 732 or with visual identification by sub-process 731, depending on the particular implementation of process 700.

[0119] The various position sensors used in connection with the embodiments of the present disclosure, such as to determine / record feature contact positions in block 733 or to target live instrument positions in block 736, may be any type of position sensor. As an example, such a sensor may be an electromagnetic (EM) sensor / probe. With respect to a scope, the position sensor may be mounted proximal to or integrated with its tip. Alternatively, the sensor may comprise a coil connected to an electrical wire running the length of the scope, which is connected to an external control circuit configured to interpret an electrical signal generated in the coil and passed down the wire. Examples of types of position sensor devices that may be implemented in connection with the embodiments of the present disclosure include, but are not limited to, accelerometers, gyroscopes, magnetometers, fiber optic shape sensing (e.g., via Bragg grating, Rayleigh scattering, interferometry, or related techniques), and the like. Depending on the implementation, registration to a separate form of patient image, such as a CT scan, may or may not be necessary to provide a frame of reference for locating urinary stones within the patient.

[0120] With respect to EM-type sensors such as coils or other antennas, such sensor devices can be configured to detect changes in the EM field as the EM sensor moves within the field (e.g., within the kidney). Thus, certain embodiments are implemented using one or more EM generators configured to emit an EM field that is picked up / affected by the EM sensor. The EM generators can be modulated in any suitable or desirable manner such that when their emitted EM fields are captured / affected by the EM sensor and processed by appropriate control circuitry, the signals from the different EM generators are separable to provide an additional dimension / degree of freedom of positional information. The EM generators can be modulated in time or frequency and may use orthogonal modulation such that each signal is completely separable from each other signal, despite possibly overlapping in time. Furthermore, the separate EM generators can be oriented relative to one another in Cartesian space at non-zero, non-orthogonal angles such that a change in the orientation of the EM sensor causes the EM sensor to receive at least some signal from at least one of the EM generators at any given moment.

[0121] With further reference to recording feature contact location in block 733 of FIG. 7, EM position data may be recorded on images of the patient captured using a different technique other than EM, such as a CT scan, to establish a frame of reference / space for the EM data (or any mechanism may be used to capture alignment sensor data). In addition to the scope, the percutaneous access needle may include one or more position / alignment sensors, such as an EM sensor. Position / alignment data received from the needle EM sensor may be received and processed similarly to the scope position data, as described above. It should be understood that the various processes described herein may be performed fully or partially manually and / or fully or partially using a robot.

[0122] The processes disclosed herein may be implemented in connection with procedures other than kidney stone removal procedures, such as gallbladder stone removal, pulmonary (pulmonary / transthoracic) tumor biopsy, and others. In general, any type of percutaneous procedure may be performed by using an endoscope configured to capture image data for feature identification and tracking using neural network processing according to embodiments of the present disclosure. Additional examples include gastric surgery, esophageal and pulmonary surgery, and the like. Furthermore, the objects to be removed do not necessarily have to be urinary stones, they may be any object, such as a foreign body or object generated within the human body.

[0123] Process 700, as may be implemented in connection with any of the embodiments disclosed herein, may be implemented to locate a target anatomical feature based at least in part on a determination of the offset / translation of the scope from the target anatomical feature. The electromagnetic sensor incorporated in the distal end of the ureteroscope may have any suitable or desirable form and / or configuration, such as one or more conductive coils, rings, cylinders, and / or the like, and local distortions in the broadcast electromagnetic field caused by such conductive elements may provide information regarding its location.

[0124] An electromagnetic positioning system, such as an electromagnetic field generator and one or more electromagnetic sensors / beacons, can be used to track the current location of the nipple to facilitate real-time targeting of the nipple with a percutaneous access instrument (e.g., a needle). For example, the targeting location of the nipple can be updated in real-time based on electromagnetic sensor data, such as real-time electromagnetic sensor data associated with one or more sensors / beacons associated with the distal end of the endoscope. In some implementations, a real-time tracking location of the target anatomical feature can be determined based on the position and / or orientation of the scope, even in the absence of real-time visual confirmation and / or other image data associated with the scope and the target anatomical feature.

[0125] Translation of endoscope and target feature location 8 illustrates a scope device 840 disposed within a target cup 812 for target localization, in accordance with one or more embodiments. The localization process involves determining a known offset P between the recorded nipple contact position 801 and the current position of the distal end 847 of the endoscope 840. offset The position sensor / beacon of the scope 840 may be configured to provide sensor data indicative of 5 or 6 degrees-of-freedom (DOF) for the position of the scope 840. For example, the coil or other type of sensor device may have a cylindrical shape or any other shape that allows for three-dimensional position data as well as yaw and / or pitch data. In some embodiments, the position sensor / beacon does not provide roll information, as in embodiments including a 5DOF sensor. In some embodiments, multiple 5DOF sensors may be used / combined and arranged at axial angles relative to one another, and the composite data provided / generated based on such position sensor / beacons may define a plane that can be used to construct a 6DOF that provides scope roll information.

[0126] In some implementations, a breath-hold may be performed on the patient during at least a portion of the scope offset determination / maintenance process, allowing such operations to be performed without the need to consider anatomical motion associated with the pulmonary cycle. For example, the patient may be subject to a breath-hold during at least the tagging and retraction portions of the process. In some implementations, the determined translation P may be adjusted to accommodate the offset determination, even though it is considered that any anatomical motion experienced after the offset determination may similarly affect the parked endoscope and the target anatomical feature (e.g., the nipple). offset may not need to be updated in real time, and thus the transformation / translation between the two positions may be considered to be substantially constant, independent of anatomical motion and / or other factors.

[0127] The relative translation P between the parked scope tip 847 and the previously recorded nipple contact position 801 offsetDetermination of can be determined using exact electromagnetic position sensor data, or may be determined using image processing as described herein. For example, calibration of a scope camera to the electromagnetic field space may allow visual determination of the distance and / or position change between the contact location 801 and the retracted position 847.

[0128] The path of retraction 808 between the scope position 841 and the scope position 840 may or may not be linear. In some implementations, such as those relating to a lower pole target cup / papilla, the path of retraction may be at least partially arcuate. Thus, the translation P offset may be determined not just with respect to linear distance, but may incorporate the orientation of the scope and / or other position-related parameters. Thus, in some implementations, the translation P offset may be considered a six degree of freedom translation / transformation. Such a translation determination at least partially accounts for cases where the target cup 812 and / or associated funnel may have a central axis / center of gravity that does not necessarily align with the retraction path taken by the scope 840. Thus, translations with respect to six or more degrees of freedom may be considered to be a six degree of freedom translation / transformation. offset It may be desirable to generate a mapping translation / transformation that represents

[0129] In some implementations, certain image data can be collected and used to identify a target anatomical feature. For example, the systems, devices, and methods of the present disclosure can provide for identification of a target anatomical feature in a real-time endoscopic image, where identification of the target anatomical feature in the image can prompt a certain responsive action. For example, a control circuit communicatively coupled to the robotic endoscopy and / or percutaneous access device can be configured to track movement of the target feature and take action, such as articulating one or more portions of the endoscope (e.g., the distal end 847) or adjusting the target position data. For example, the control circuit can be configured to center the target location / point at or near the center of the field of view of the endoscopic camera interface and / or image field, and / or to adjust a desired position offset (e.g., P) between the scope and the target anatomical feature. offset ) to articulate the endoscope.

[0130] By utilizing robot-assisted percutaneous access, a physician may be able to perform operational target access and treatment. Additionally, percutaneous access may be further assisted utilizing automatic target identification and tracking according to aspects of the present disclosure described in more detail below, which may be relied upon to accurately maintain target location for percutaneous access guidance. Percutaneous access guided by scope-enabled target tracking according to aspects of the present disclosure may be relatively low skill-intensive. In some implementations, a single operator or robotic system may perform the process. Additionally, the need for fluoroscopy may be eliminated.

[0131] Scope offset / parking guidance As described above, the inventive features of the present disclosure may be used in endoscope / ureteroscope-based targeting for percutaneous renal access, where an electromagnetic beacon / sensor equipped ureteroscope and / or an electromagnetic beacon / sensor equipped percutaneous access instrument (e.g., needle) is used to obtain percutaneous renal access through a target papilla into a target calyx. With respect to a scope targeting procedure, the effectiveness of percutaneous access to a target location / site, such as access to a target calyx where a scope is parked, may depend on where a physician / technologist parks the distal end of the scope relative to the target calyx / papilla. Specifically, as described in accordance with embodiments of the present disclosure, the parking of the distal end of the endoscope / ureteroscope may relate to the position and / or leading orientation of the distal end of the ureteroscope relative to a target papilla where percutaneous access can be achieved. To provide further context to the various scope parking features disclosed herein, specific surgical systems and environments are described below for endoscope parking.

[0132] FIG. 9 illustrates a robotic medical system 900 arranged to facilitate navigation of a scope within a patient, according to one or more embodiments. For example, a physician 5 can connect a scope 92 to a robotic arm 12(c) of a robotic system 10 and / or position the scope 92 at least partially within a medical instrument (e.g., catheter / sheath) and / or patient 13. The scope 92 can be connected to the robotic arm 112(c) at any time, such as before or during the procedure (e.g., after positioning the robotic system 10). The physician 5 can then interact with a control system 50, such as an I / O device 46, to navigate the scope 92 within the patient 13. For example, the physician 5 can provide input via the I / O device 46 to control the robotic arm 112(c) to navigate the scope 92 through the urethra 65, bladder 60, ureter 63, and to the kidney 70.

[0133] As shown, the control system 50 may present a screen 910 including one or more scope navigation interfaces 950 via the display 42 for viewing a real-time camera image / view 952 captured by the scope 92 to assist the physician 5 in controlling the scope 92. The physician 5 may navigate the scope 92 to locate, for example, kidney stones, target anatomical features, and / or the like. In some embodiments, the control system 50 may be configured to perform certain localization techniques to determine the position and / or orientation of the scope 92, which may also be viewed by the physician 5 via the display 142 to assist in controlling the scope 92. Additionally, in some embodiments, other types of information may be presented via the display 42 to assist the physician 5 in controlling the scope 92, such as x-ray images of the internal anatomy of the patient 13.

[0134] Physician 5 can use control device 46 to drive scope 92 to find / identify a kidney stone 908 or other artifact targeted for removal / treatment. Physician 5 can further drive the scope to locate the target papilla and assume a desired parking position. Such scope drive can be at least partially guided by scope parking guidance features 954 of scope guidance interface 950, which are described in more detail below.

[0135] 10A and 10B show the renal anatomy with a ureteroscope parked in various positions, according to one or more embodiments. FIG. 10A shows a ureteroscope 1040 disposed within the ureter 63, renal pelvis 71, and / or calyx (e.g., greater and / or lesser calyx) of a kidney 1010. For example, an operator / physician may drive the scope 1040 into the calyx 1012 and use an electromagnetic beacon associated with the distal end / tip of the scope 1040 as a target to which a percutaneous access instrument (e.g., needle) may be directed. In some embodiments, the scope 1040 is used to record the location of the target papilla 1079, after which the scope 1040 is retracted some distance and parked away from the papilla 1079 to provide the desired position of the scope for visualizing the percutaneous access using the camera of the scope 1040. In some implementations, the target location targeted by a percutaneous access instrument (not shown) may be determined based on a known offset distance, orientation, and / or position of the scope 1040 relative to the target papilla 1079 or other anatomical feature. Thus, the features disclosed herein may facilitate proper / desired parking of the scope device, as described herein, which may be implemented and relied upon for dynamic targeting purposes.

[0136] With regard to percutaneous access of the calyx, such as may be performed to reach / treat kidney stones, access through the renal pyramid 1057 / papilla 1079 may be necessary or desirable to access the target calyx 1012 without excessive bleeding. Additionally, access through the papilla 1079 may provide complete access to the calyx rete of the kidney 1010.

[0137] The target cup 1012 surrounds the papilla 1079 (i.e., the renal pyramid, shown in dashed form in Figs. 10A and 10B for clarity), through which suitable percutaneous access to the target cup 1012 may be obtained. In general, one or more points at or near the distal tip / end of the scope 1040 may be used as a target for percutaneous access. For example, as discussed above, the distal tip / end of the scope 1040 may have one or more electromagnetic sensors or beacons associated therewith to determine its position / orientation within the electromagnetic field. The diagram of Fig. 10A shows three different possible exemplary parking positions (1042, 1044, 1046) of the distal end of the scope 1040. Fig. 10A further shows the respective coaxial trajectories (1002, 1004, 1006) associated with each of the parking positions of the scope. Such a trajectory may be determined based on derived position and / or orientation / alignment information associated with each of the scope's parking positions and may represent possible paths along which percutaneous renal access may be guided / achieved.

[0138] FIG. 10A shows a parking position 1046 in which the distal end of the scope 1040 is generally aligned with the central axis 1006 of the target cupule 1012 and / or associated funnel. FIG. 10A further shows another parking position 1042 in which the distal end of the scope 1040 is positioned at an unnecessarily distant distance d2 from the papilla 1079 and / or is not aligned with the central axis of the target cupule 1012 and / or associated funnel. For example, as shown as trajectory 1002, the parking position 1042 may generate a target trajectory 1002 that is generally not centered on the papilla 1079 and / or associated renal complex. FIG. 10 further shows another parking position 1044 that is not aligned with the axis of the target cupule 1012, papilla 1079, and / or associated funnel.

[0139] FIG 10B illustrates the minimum percutaneous access duct path corresponding to each of the illustrated exemplary scope parking positions. Specifically, FIG 10 illustrates a minimum duct path 1001 associated with parking position 1042, where the scope 1040 is parked unnecessarily far from the target papilla 1079 and / or is not aligned with its center. FIG 10B further illustrates a minimum duct path 1005 associated with a misaligned parking position 1044. The access path 1003 associated with parking position 1046 may generally be aligned with a central region of the cup 1012. Based on the access paths illustrated in FIG 10A and FIG 10B, the parking position 1046 may be considered a preferred or desirable parking position for viewing and / or accessing the cup 1012.

[0140] Certain embodiments of the present disclosure advantageously provide various mechanisms and means for estimating or determining the location of the target calyx / papilla, as well as certain visualization mechanisms / means, to facilitate guiding the operator to properly place the ureteroscope in the target calyx in a position, alignment, and / or orientation effective for percutaneous access targeting.Embodiments of the present disclosure advantageously provide certain visual guidance / assistance features and / or functions to facilitate effective placement of an endoscope at a target site, such as placement of a ureteroscope at or near a target calyx within a patient's kidney.

[0141] 11, 12A, 12B, 14-1, and 14-2 show certain scope guidance features that can be used to guide the operator / physician in parking / positioning the endoscope in a desired position relative to the target papilla. Such features can be generated and / or displayed in any suitable or desired manner or form. For example, one or more markers / icons can be generated and / or displayed in or near a camera view presented on a display device associated with the robotic control system, which markers / icons instruct the physician / operator as to how to manipulate or actuate the endoscope to position the distal end of the endoscope in a suitable or most effective parking position. The scope parking guidance features can, for example, instruct the user / operator to align the ureteroscope with the target papilla so that the position of the distal end of the ureteroscope can be virtually targeted using a percutaneous access instrument (e.g., a needle).

[0142] 11 illustrates an example image 1101 from a camera associated with a distal end of an endoscope, such as a ureteroscope, in accordance with one or more embodiments of the present disclosure. Image 1101 may be presented in a scope navigation interface as shown in FIG 9 and described above. Image 1101 may represent a window or window portion (e.g., a sub-window) configured to display a camera view that includes at least a portion of a field of view of a camera associated with the endoscope.

[0143] In some embodiments, the camera view interface / window 1101 may display one or more icons or features 1150 superimposed thereon that indicate a target location of a target anatomical feature within the camera view 1101. The target icon / feature 1150 may help guide the operator regarding proper alignment of the distal end of the endoscope with the target anatomical feature (e.g., the nipple). In some embodiments, the target anatomical feature alignment guidance icon / feature 1150 may be generated and / or displayed such that the icon / feature 1150 remains in approximately the same position within the window 1101 as the camera image displayed in the window 1101 moves or changes as the scope moves. That is, the scope operator may manipulate / actuate the distal end of the scope, thereby changing the camera's field of view and / or the representative anatomical structure captured in that field of view. As the camera's field of view 1101 changes, the icon / feature 1150 may remain a representation superimposed at or near the center of the window / image 1101, as shown in FIG. 11 . Thus, the operator may be able to move the camera's field of view, such as by changing the position of the scope, to selectively position the icon / feature 1150 over a target anatomical feature of interest, such as the nipple or other anatomical feature.

[0144] The matching icon / feature 1150 may instruct the operator to position the field of view of the camera to locate and / or center the icon / feature 1150 over the anatomical feature of interest 1116 (e.g., the nipple). For example, as shown in image 1103, which represents a changed field of view of the camera associated with the scope after the operator moves the scope, the icon / feature 1150 may be used to guide the operator to center the icon / feature 1150 over the target nipple 1116, whereas in the initial field of view 1101, the target nipple 1116 is generally not centered within the field of view of the camera or with respect to the targeted icon / feature 1150. To change the field of view to center the icon / feature 1150 as shown in image 1103, the distal end of the endoscope and / or its associated camera may be panned to the left, for example, to bring the target nipple / feature 1116 into the field of view of the camera, as shown in image 1103 compared to image 1101.

[0145] The icon / feature 1150 may have any suitable or desirable shape, form, configuration, and / or number of visual features. For example, some embodiments, such as the embodiment shown in FIG. 11, include a cross-shaped feature indicating a center point defined by the intersection of extrapolated lines or other features oriented on two or more axes. The illustrated icon / feature 1150 may additionally or alternatively include one or more circular forms, as shown, whose central axis represents the center point of the icon / feature 1150. Generally, the icon / feature 1150 indicates that the operator should generally align the center point of the icon / feature 1152 with the center (e.g., volume and / or geometric center) of the target anatomical feature 1116.

[0146] The icon / feature 1150 shown in FIG. 11 can facilitate proper alignment of the endoscope with respect to the target anatomical feature. In addition to (or as an alternative to) alignment guidance, embodiments of the present disclosure relate to endoscope parking guidance features that facilitate proper positioning of the endoscope with respect to its distance / depth from the target anatomical feature (e.g., the nipple 1116). In some embodiments, a particular icon / feature is generated and / or displayed over the camera view, configured such that its relative size compared to the size of the representation of the target anatomical feature in the camera view indicates whether the scope is positioned at the proper distance / depth and / or how the scope should be moved to reach the proper distance / depth from the target anatomical feature. Furthermore, in some embodiments, the depth positioning icon / feature can further include an icon / feature that also indicates alignment guidance.

[0147] 12A and 12B show a scope camera view / window configuration including a target boundary feature 1260 that guides an operator as to the appropriate depth / offset position of the scope, according to one or more embodiments. FIGURES 12A and 12B show example images 1201a, 1201b from a camera associated with a distal end of an endoscope, such as a ureteroscope, according to one or more embodiments of the present disclosure. The images 1201a, 1201b may be presented in a scope navigation interface, as shown in FIGURE 9 and described above. The images 1201a, 1201b may represent a window or a portion of a window (e.g., a sub-window) configured to display a camera view that includes at least a portion of the field of view of a camera associated with the endoscope.

[0148] In some embodiments, the camera view interface / window 1201a, 1201b may display one or more icons or features 1260 superimposed thereon that indicate a target depth of the target anatomical feature 1216 within the camera view 1201a, 1201b. The offset icon / feature 1260 may include, for example, one or more brackets, bounding boxes, and / or other shapes or features that represent the containment of image content displayed between or within such icons / features. The bounding box or other similar feature may be displayed at or near the center of the camera's field of view and / or window showing the camera view, such feature may indicate or instruct the operator to keep all or at least a portion (e.g., at least a central portion) of the target anatomical feature within the bounding box shape / feature to avoid parking the endoscope in a position too close to the target anatomical feature 1216. For example, in some embodiments, the bounding box feature 1260 may instruct and / or indicate to the operator that the operator should maintain and / or contain at least a majority of the area of ​​the anatomical feature 1216 represented in the camera image within a boundary defined and / or indicated by the offset guidance feature 1260 relative to one or more axes (e.g., horizontally and / or vertically). In addition to reducing the likelihood of the endoscope being parked too close to the target anatomical feature 1216, the offset feature / icon 1260 could reduce the likelihood of the endoscope being parked too far from the target anatomical feature 1216 by instructing the operator to substantially fill the bounding area of ​​the offset feature / icon 1260 with the target anatomical feature or by instructing the operator to maintain the camera in a position such that a majority of the area enclosed by the offset feature / icon 1260 is filled / covered with a representative image of the target anatomical feature (e.g., the nipple).

[0149] 12A and 12B, images 1201a, 1201b, and 1203 show an offset feature / icon 1260 configured and / or located approximately in the center of window 1201a, 1201b. As shown, in image 1201a, the target anatomical feature (e.g., nipple) 1216 does not fully fill the extent of offset window 1260. A subsequent image 1203, as represented by image 1240a, may result from a manipulation of the endoscope that brings the endoscope and / or associated camera closer to the target anatomical feature 1216, such that the resulting image 1203 shows the target anatomical feature 1216 that approximately fills the extent of offset feature / icon 1260.

[0150] With respect to FIG. 12B, image 1201b shows an offset feature / icon 1260 in which the endoscopic camera is too close to the target anatomical feature 1216 such that the representation of the target anatomical feature 1216 extends substantially outside the range of the offset feature 1260, and therefore may obstruct visibility of the target anatomical feature 1216, such as when the target anatomical feature 1216 is deformed as a result of advancement therein / through a percutaneous access instrument (e.g., a needle).

[0151] A subsequent image 1203 may result from manipulating the endoscope away from the target anatomical feature 1216, as represented by image 1240b, such that the target anatomical feature 1216, or a majority of a relevant portion thereof, is within the range of the offset feature / icon 1260 relative to one or both dimensions / axes of image 1204.

[0152] In certain embodiments, as shown in FIGS. 12A and 12B, a bounding box 1260 may be displayed at or near the center of the view window (1201a, 1201b, 1203) to instruct / instruct the user to maintain the view representation of the target anatomical feature 1216 (e.g., the nipple) entirely or at least partially (e.g., more than half) within the bounding box 1260, which may advantageously reduce the occurrence and / or likelihood of the scope being parked either too far away, as in image 1201 and / or with respect to scope position 1042 shown in FIGS. 10A and 10B, or too close, as in image 1201 in FIG. 12B and / or with respect to scope position 1044 shown in FIGS. 10A and 10B.

[0153] The bounding box features 1260 can advantageously facilitate standardization of how different users / physicians operate the scope with respect to target anatomical features of different sizes and different patients, thereby achieving a preferred position of the scope and reducing dependency on the skill and / or experience of the physician and / or the specific anatomical structure of the patient. In some embodiments, the scope management system comprises a device configured to at least partially perform automatic positioning of the scope to position and / or maintain the target anatomical feature at least partially within the bounding box features. Thus, anatomical movements resulting in a relative displacement of the target anatomical feature with respect to the camera of the scope can be automatically compensated for using image processing functions and / or other means or mechanisms for automatically determining the relative position of the target anatomical feature with respect to the bounding box features.

[0154] In some embodiments, the size and / or other characteristics of the bounding box feature 1260 may depend at least in part on the particular anatomical specifications of the patient. For example, a scope positioned in a relatively small cup may be desirable to be positioned relatively close to the target papilla to reduce the risk of scope slippage / movement. For example, the guidance feature (e.g., bounding box feature) superimposed / displayed on the endoscope view window may have a fixed size (e.g., pixel size). This may allow the user to be guided to park closer for smaller cups. That is, to fit the relatively smaller papilla within the bounds, offset guidance icon, it may be necessary to park the scope relatively close. If the cup is small, the scope may tend to move out of the cup, and therefore closer parking is desirable. Thus, even if the size of the target cup / papilla is unknown, adaptive parking distances may be possible for different cups by implementing a fixed size target / boundary icon / feature and instructing the user to park the scope so that the icon / feature is substantially (e.g., 50%, 60%, 75%, 80%, 90%, or other percentage) filled with the target anatomical feature / region. It should be appreciated that such an example is one of many possible examples in which the characteristics of the bounding box feature are generated and / or presented based at least in part on certain determined characteristics of the patient's anatomy and / or parameters manually entered by the technician. The various offset guidance features described above with respect to Figures 12A and 12B can advantageously limit / limit potential errors in percutaneous access targeting by controlling the range of distances and / or positions of the scope relative to the target anatomical feature.

[0155] FIG. 13 illustrates a robotic medical system arranged to facilitate percutaneous access of a patient's renal anatomy, according to one or more embodiments. As shown in FIG. 13, a physician 5 can perform percutaneous renal access by positioning a needle 17 for insertion into a target location. In some embodiments, the physician 5 can place the needle 17 into the patient 13 at the incision site using his or her best judgment, such as based on knowledge of the patient's 13 anatomy, experience performing previous procedures, analysis of CT / X-ray images, or other pre-operative information of the patient 13. Additionally, in some embodiments, the control system 50 can provide information regarding where to place the needle 17 into the patient 13. The physician 5 can attempt to avoid important anatomical structures of the patient 13, such as the lungs, pleura, colon, paraspinal muscles, ribs, intercostal nerves, etc. In some examples, the control system 50 can provide information regarding where to place the needle 17 into the patient 13 using CT / X-ray / ultrasound images.

[0156] The control system 50 may include control circuitry configured to determine a target trajectory 902 for inserting the needle 17 to assist the physician 5 in reaching the target location (i.e., the papilla 914). The target trajectory 902 may represent a desired path to access the target location. The target trajectory 902 may be determined based on the position of one or more medical instruments (e.g., the needle 17, the scope 92, etc.), the target location within the patient's anatomy, the position and / or orientation of the patient 13, and the patient's anatomy (e.g., the location of an organ within the patient relative to the target location). In some implementations, the target trajectory 902 represents a straight line passing through the papilla 914 and the point of the needle 17. The trajectory may generally coincide with the axis of the funnel associated with the target cupule. However, the target trajectory 502 may take other forms, such as a curved path, and / or may be defined differently. In some examples, the needle 17 is implemented with an at least partially flexible needle and / or a beveled tip needle. The control system 50 may be configured to provide information to guide the user / physician 5 in advancing the needle, such as to compensate for deviations in the needle trajectory or to keep the user on a target trajectory.

[0157] 13 illustrates a target trajectory 902 extending coaxially through the papilla 914, the target trajectory 902 may have another location, angle, and / or configuration. For example, the target trajectory may be implemented with a lower pole access point, such as through the papilla located below the kidney stone 908 shown in FIG. 13, and have a non-coaxial angle through the papilla, which may be used to avoid the hip joint. In some implementations, the smallest duct path / trajectory is taken in accessing the target cupule / papilla.

[0158] As mentioned above, the control system 50 may present, via the display 42, a screen 910 including one or more scope guidance interfaces 950 for viewing a real-time camera image / view 952 captured by the scope 92 to assist the physician 5 in controlling the scope 92. As shown in FIG. 13, during percutaneous access, the scope 92 may be parked within the target cup to provide visibility of the target papilla 914. As the needle 17 advances toward the target papilla 914, adjustments of the scope 92 may be guided, at least in part, by one or more needle trajectory alignment / position guidance features 956, which are described in more detail below. Such features 956 may facilitate positioning of the scope 92 in a manner that includes within the scope camera's field of view 952 an icon indicative of a predicted representation of the needle 17 within the target cup (e.g., a predicted needle tip position in the scope camera image window). For example, if the predicted needle entry point into the target cup and / or through the target papilla 914 is outside the field of view 952 of the scope camera, the interface feature 956 can inform the physician 5 in a direction in which the scope 92 can be moved to bring the predicted needle entry point within the camera view 952.

[0159] Once the needle 17 reaches the target location, the physician 5 can insert another medical instrument, such as a catheter, vacuum, nephroscope, etc., into the pathway created by the needle 17 and / or over the needle 17 and / or dilators (not shown) disposed in the access pathway. The physician 5 can use other medical instruments and / or the scope 92 to fragment and remove kidney stone fragments from the kidney 70.

[0160] In some embodiments, the position of the medical instrument may be represented by a point / set of points and / or the orientation of the medical instrument may be represented as an angle / offset relative to an axis / plane. For example, the position of the medical instrument may be represented by the coordinates of a point / set of points in a coordinate system (e.g., one or more X, Y, Z coordinates) and / or the orientation of the medical instrument may be represented by an angle relative to an axis / plane of the coordinate system (e.g., an angle relative to the X axis / plane, the Y axis / plane, and / or the Z axis / plane). Here, a change in the orientation of the medical instrument may correspond to a change in the angle of the medical instrument relative to an axis / plane. Furthermore, in some embodiments, the orientation of the medical instrument is represented by yaw, pitch, and / or roll information.

[0161] In some embodiments, the trajectory refers to a pose. For example, the trajectory of a medical instrument can refer to a pose of the medical instrument, which includes / suggests both the position and orientation of the medical instrument. Similarly, the target trajectory can refer to a target pose, which includes / suggests both the position and orientation of the desired path. However, in other embodiments, the trajectory refers to either an orientation or a position.

[0162] In some embodiments, the ureteroscope 92 and the needle or other percutaneous access instrument 17 have a common positioning coordinate system, such as in implementations where an electromagnetic field generator 18 is utilized and the scope 92 and needle 17 each have an associated respective electromagnetic sensor / beacon that can be used to determine the respective positions of the scope 92 and needle 17 within the common electromagnetic field space. For example, prior to a medical procedure involving the use of an endoscope and / or a percutaneous access needle, the sensor / beacon associated with the distal end of the scope may be calibrated with the scope's camera to determine a transformation / relationship between the camera's visual image space and positions within the electromagnetic field associated with images / features that are depicted and / or otherwise appear in the camera's image field. Such calibration may allow for determining and / or predicting position information based on the position, size, and / or configuration / or orientation of feature / object representations within the camera's image space (e.g., field of view). Thus, where the position of the percutaneous needle is determined and / or predicted relative to the position of the camera and / or associated sensor / beacon, as indicated by its associated electromagnetic sensor / beacon, such position may be determined / generated and / or displayed on the field of view image displayed in the scope camera view interface 952 based at least in part on the calibration of the camera, as described above. For example, as described in more detail below, the camera calibration may rely on overlaying / presenting a needle trajectory icon / feature in the camera view window, such icon / feature indicating the predicted / projected position within / by the field of view where the percutaneous needle is predicted / expected to appear once the needle approaches the target.

[0163] The scope guidance interface 950 may be configured to indicate certain needle trajectory alignment / position guidance features 956 that may be presented in association with (e.g., superimposed or adjacent to) the scope camera view / window 952. For example, the needle trajectory alignment / position guidance features 956 may include one or more icons or features that are displayed as part of the interface 950 and indicate a location relative to the camera view 952 where a percutaneous access device (e.g., a needle) is expected to enter the camera's field of view and / or pierce the target anatomical feature / papilla. Such needle trajectory alignment / position guidance mechanisms may facilitate the scope operator to maintain the predicted entry point of the percutaneous access device within the camera's field of view 952. The features 956 indicating the predicted location of the percutaneous access device, as well as other interface icons / features disclosed herein, may have any suitable or desirable shape, form, and / or representation.

[0164] 14-1 and 14-2 illustrate a scope camera view / window configuration including one or more needle trajectory features, according to one or more embodiments. FIG. 14-1 shows an example image 1401 from a camera associated with a distal end of an endoscope, such as a ureteroscope, according to one or more embodiments of the present disclosure. Image 1401 may be presented in a needle trajectory scope guidance interface, as shown in FIG. 13 and described above. Image 1401 may represent a window or a portion of a window (e.g., a sub-window) configured to display a camera view including at least a portion of the field of view of a camera associated with an endoscope (e.g., a ureteroscope).

[0165] The position of the target anatomical feature (e.g., papilla) 1416 may be tracked / located and used as a target for percutaneous access to the chamber conduit or other area (e.g., cupule) where the endoscope is located / parked. For example, in some implementations, the position of the target anatomical feature 1416 (e.g., papilla) may be determined based at least in part on one or more of the recorded contact position of the papilla by implementations involving contacting the papilla with the distal end of the endoscope and recording position information related to the endoscope in association with such contact, as well as the size, shape, and / or configuration / or orientation of the target anatomical feature 1416 within the image / field of view of the camera, which may provide position information based on the calibration of the camera. For embodiments relying on camera calibration information, the relative size, shape, and / or orientation / configuration of the target anatomical feature compared to its visual characteristics at a later time point after the scope has been retracted / withdrawn to some extent from contact with the target anatomical feature 1416 may provide information indicative of the current location of the target anatomical feature 1416. Calibration of the camera image and electromagnetic field space may be accomplished and / or performed in any suitable or desirable manner, for example, in some implementations, a checkerboard or other object / pattern of known shape, size, and / or configuration may be used to determine the location of an object representation within the camera image.

[0166] In some implementations, the determination of the translation between the three-dimensional electromagnetic spatial positioning and the position / configuration in the camera space can be utilized to generate / display a needle trajectory icon / feature / characteristics in a camera image window, such as the camera image windows 1401, 1403 of FIG. 14. For example, the current and / or predicted position of the tip of a percutaneous needle (not shown) can be visualized / represented in the camera windows 1401, 1403 to provide an operator with orientation as to where the scope camera will capture the needle within its field of view as the needle penetrates the anatomical feature 1416 and / or surrounding anatomical structures. Needle trajectory alignment / position guidance features / icons 956 can be presented to confirm the success and / or failure of the percutaneous needle access / piercing to facilitate accessing the target anatomical site.

[0167] Representation of needle location icons / features (e.g., 1470, 1475, 1480) may be helpful in situations where the target anatomical feature / tissue may at least partially deform as the percutaneous needle passes through / through the target anatomical feature / tissue, potentially resulting in visual obstruction. That is, the closer the needle advances, the more uncertain the visibility of the target anatomical feature may be. Thus, confirmation of successful targeting by the percutaneous needle may be difficult to determine without the use of needle trajectory features according to aspects of the present disclosure.

[0168] Image 1401 shows example images and associated needle trajectory / position features 1470, 1475 in the context where the predicted and / or current needle position is outside the camera field of view 1401. In such a situation, icons / features 1470, 1475 may be generated and / or displayed and / or positioned relative to the field of view 1401 to indicate the direction, predicted or current position of the needle relative to the current field of view of image 1401. For example, an arrow or other icon 1470 may be displayed in an area outside the window 1401 indicating a direction / position outside the window 1401 related to the needle trajectory. Alternatively or additionally, one or more icons / features 1475 may be superimposed on image 1401 to indicate the direction of the field of view of image 1401 and / or the position outside the field of view of image 1401 where the needle position would be predicted if the size of the field of view of image 1401 were large.

[0169] Image 1403 shows a camera view after a particular articulation / movement of the distal end of the scope and / or associated camera. For example, according to an indicated needle trajectory (e.g., 1470 or 1475) illustrated in image 1401, the camera represented by icon 1440 may be panned in the direction of the indicated needle trajectory to bring the predicted location of the needle into the camera's field of view. Image 1403 may further reflect a slight upward pitch of the scope / camera 1440 to bring a needle trajectory icon / feature 1480 further into the camera's field of view and / or generally toward its center. Icon / feature 1480 may represent or correspond to the location of the needle tip or may correspond to the predicted entry point.

[0170] In some embodiments, the needle trajectory icon 1480 can be configured to vary with respect to size, color, shape, and / or one or more other features / characteristics thereof to represent the distance, alignment, and / or other position-related characteristics of the needle tip. In some embodiments, the icon 1480 gets larger (or smaller) as the needle tip gets closer to the scope / camera. In some embodiments, the icon 1480 can get larger as the needle gets relatively farther away to represent the margin of error of the predicted needle tip location, while as the needle tip gets closer to the scope / camera, there can be more certainty as to the particular location where the needle will appear when puncturing the target anatomy, and thus the area of ​​the icon 1480 can get smaller to reflect the area where the needle may appear. In general, some embodiments can include an icon having certain visual characteristics that represent the three-dimensional alignment, orientation, shape, and / or location of the needle as represented in the two-dimensional image space of the camera. For example, other icons such as cone icons or apex or other directional shapes / features may be generated and / or presented to indicate depth perception relative to the predicted or determined location of the needle. In some embodiments, certain shading features are incorporated into the needle trajectory icons / features (e.g., 1470, 1475, and / or 1480) to indicate additional positioning, alignment, and / or orientation information regarding the needle.

[0171] Needle alignment / position guidance features 956 may include a needle trajectory icon as depicted in images 1401 and / or 1403, and may further include a particular visual representation / interface that provides a direction of advancement of the percutaneous needle to direct the needle towards the tracked target anatomical feature. For example, one or more needle viewpoint interface features may be provided to indicate alignment, orientation, and / or positioning of the needle and guide its advancement, while additional scope viewpoint interface and associated needle trajectory icon / features may be presented to provide additional confirmation and / or guidance indicating the positioning of the needle relative to the scope's camera.

[0172] The size of the needle prediction icon 1480 may be changed / modified based on the determined needle prediction / prediction accuracy. If there is significant anatomical motion that may result in an error in the needle prediction, the needle prediction icon may be presented in a larger size to indicate a larger determined error to the needle prediction / trajectory.

[0173] FIG. 14-2 shows an image 1405 including a needle trajectory prediction feature 1483 that visually predicts from a needle prediction icon 1480. The feature 1483 may have certain visual characteristics that indicate a three-dimensional prediction of the needle along the path that the needle 1475 is predicted to travel. Image 1407 shows the needle 1475 penetrating an anatomical feature 1416 and the prediction along the path indicated by the prediction feature 1483. That is, in image 1405, the needle tip is inside / behind the anatomical feature 1416, and image 1407 shows the needle tip 1475 after penetrating the anatomical feature 1416 and entering the treatment site. Image 1405 includes both a tip location icon 1480 and an overlay 1483 on the needle shaft to indicate the needle pose. Image 1407 shows both an orientation indicator of where the tip is located as well as an overlay on the needle shaft to indicate a three-dimensional perception of the needle pose.

[0174] Estimation of the cupule / infundibulum axis for nipple localization In order to provide nipple offset trajectory information for nipple localization / tracking, an estimation of the cupule and / or funnel axis may be performed in association with any of the embodiments disclosed herein. The axis of the cupule and / or associated funnel may be estimated / determined in a variety of ways. For example, cupule mapping may be performed as a step to determine the cupule / funnel axis, as described in more detail below.

[0175] For purposes of percutaneous access needle targeting and / or target localization according to embodiments of the present disclosure, renal mapping can be used to localize the target papilla. In some implementations, a renal map, which may be generated using any of the methods disclosed herein, can be used to estimate / determine a surface normal associated with the target cupule, which can then be used to determine the cupule / infundibulum axis. In some implementations, a percutaneous access path to the target anatomical site can be determined and / or aligned with the orientation of the target cupule / cupule, which may in some cases be approximately coincident with the central axis of the associated funnel. In some implementations, the trajectory of the distal end of the endoscope leading to the target cupule / cupule can be used to determine and / or estimate the axis of the infundibulum associated with the target cupule. For example, the path of the scope may be linearized in some manner to generate an axis path / estimate. Even if the endoscope is not aligned with the target cupule / cupule, aspects of the present disclosure provide a mechanism for determining the orientation of the target cupule / cupule, which can be used as a target trajectory for percutaneous access and / or for target localization.

[0176] FIG. 15 illustrates a scope device 1550 disposed within a target renal anatomy for calyx mapping, according to one or more embodiments. With reference to FIG. 15, mapping of the renal calyx may involve advancing and / or maneuvering a ureteroscope 1550 within the calyx plexus. For example, the ureteroscope 1550 may be advanced to one or more calyx end points (e.g., at the respective papilla) and / or against the calyx wall or other boundary, and periodic or sporadic position tagging / recording 1501-1504 may be performed to generate skeletal mapping of a transverse region of the calyx. For example, the operator of the ureteroscope 1540 may be instructed to articulate the scope to tag multiple points 1501-1504 on the inside of the target calyx. Such points 1501-1504 may be used to construct the shape of the calyx, and the constructed calyx center may be used as a target for percutaneous access and / or may be followed when translating the current scope position by an offset distance / position to locate the papilla. In some implementations, the scope 1550 may be articulated in an arcing motion to outline / cover the traversal area, as shown.

[0177] In some implementations, mapping of the calyx may be used to determine the axis and / or orientation of the infundibulum. For example, substantially all of the calyx network of the kidney, or a subset of the calyx network, may be mapped. In some implementations, only the target calyx and / or associated infundibulum may be mapped to determine the orientation / axis of the calyx and / or infundibulum. The localization of the nipple targeting trajectory and / or nipple offset may be determined based at least in part on the determined calyx / infundibulum axis and / or other local topology associated with the target calyx / papilla. For example, as detailed above with respect to FIG. 8, the localization of the target nipple may be determined based on the nipple-scope offset translation (e.g., P offset) and applying a translation / offset (e.g., in electromagnetic field space) to the current position of the endoscope. The determination of the scope-papilla offset / translation may be based at least in part on the axis of the calyx and / or associated infundibulum. That is, a determination of the trajectory in which the target calyx is pointing may be used to determine the offset of the target papilla from the distal end of the scope. For example, the determined scope offset may be added to the current position of the scope in the direction of the calyx or infundibulum orientation / axis to determine the target position. In some cases, the target calyx may be oriented in generally the same direction / orientation as the associated infundibulum. However, for fused calyxes, which may generally be more common in the lower pole of the kidney and / or calyx plexus, the calyx orientation may be relied upon without regard to the infundibulum axis.

[0178] While driving the scope to the target papilla / calyx, multiple positions associated with the distal end of the scope can be tagged / recorded along the path to provide a scatter plot type mapping of at least a portion of the calyx network (e.g., the target calyx and / or associated funnel). For example, FIG. 16 shows a map 1600 of the calyx network including multiple calyxes (both major and minor calyxes) leading from the ureter 1663. The various calyx axes 1615 shown in FIG. 16 can be determined based on recorded scope positions along the path of travel of the scope (not shown). In particular, each illustrated circle (e.g., 1610) can represent a tagged / recorded position of the scope. In some implementations, Gaussian interpolation can be performed along the reported trajectory or portions thereof within the target calyx 1612 and / or associated funnel 1613, and such interpolation is used to generate a surface estimate for at least a portion of the calyx network.

[0179] In some implementations, a surface and / or volume estimation algorithm / mechanism may be implemented to determine the center / axis of the target cup and / or associated funnel. FIG. 17 illustrates a generated three-dimensional surface map 1700 generated in any suitable manner, such as using Gaussian interpolation. For example, once the cup and / or funnel surface is determined / generated, the cup / funnel axis 1715 may be determined based at least in part on the estimated surface. In some embodiments, multiple surface normals 1701 may be determined for various regions of the surface 1705, where the surface normals 1701 are vectors / trajectories that are substantially orthogonal to the surface 1705 at the origin associated with each surface normal 1701. The internal axis of the funnel / cup may be determined based at least in part on an averaging of surface normals that extend around and / or on both sides of the form / structure 1700. For example, the surface normal vectors may be summed to determine a central axis for each portion of the cup network 1700. Such an axis may be extrapolated to provide the cup trajectory 1715. In some embodiments, the determination of the cup trajectory may be based at least in part on the sum of surface normal vectors and / or the determination of the cup / funnel axis.

[0180] In some embodiments, as shown in FIG. 15, a circular motion may occur within the scope to generate a sufficient location plot of the target calyx, from which a calyx map is formed. The calyx orientation 1715 may be estimated from the calyx map 1700, and such a trajectory 1715 may provide a path along which the scope / papilla offset may be predicted / determined. This may allow for more reliable updating of the target location when the scope tip changes relative to the calyx, infundibulum, and / or papilla axis due to tissue deformation during needle insertion. Depending on the particular procedure, it may not be necessary to map the entire calyx network of the kidney for percutaneous access targeting purposes. For example, in some implementations, only the calyx associated with the target papilla may be mapped. In some implementations, multiple calyxes may be mapped to provide increased spatial information / mapping.

[0181] Compared to a solution that performs fluoroscopy to find the location of the target cupule for targeting purposes, the embodiments of the present disclosure may eliminate the need to acquire one or more fluoroscopic images of the collection system at different angles to determine the location and orientation of the target papilla / cupule. Furthermore, the embodiments of the present disclosure may advantageously provide targeting of the target anatomical feature without requiring retrograde injection of contrast or air to visualize the target anatomical feature. Thus, the embodiments of the present disclosure may be performed without requiring traditional cystoscopy and / or ureteral catheterization. Compared to a solution that performs an ultrasound examination for the purpose of determining the location of the target cupule for targeting purposes, the ultrasound examination may show only a two-dimensional image of the collection system and / or provide a relatively limited visualization of the ureter to the scope and / or percutaneous access needle.

[0182] Vision-Based Target Localization In some implementations, a vision-based process, such as a process performed while the scope is driven to the target cup / papilla, can be performed to determine the direction / axis of the funnel / cup. For example, multiple camera images and EM-based position data for the endoscope can be obtained at a well-defined position / orientation relative to the target cup / papilla, and the three-dimensional (3D) location of the cup relative to the scope can be determined based on such data.

[0183] Localization of the target anatomical feature (e.g., the target papilla) may be accomplished using any suitable image processing mechanism / function. For example, the medical system's control circuitry may receive image data from the scope camera and perform certain image processing processes / functions thereon to identify the target anatomical feature. In some implementations, at least two separate images of the target anatomical feature are processed to track its location. For example, as shown in FIG. 18, an image 1801 including the target anatomical feature 1806 may be captured from a first viewpoint / position 1807 of the scope camera, and a second image 1802 may be captured by the scope camera either before or after the capture of the first image 1801, and the second image 1802 is captured from a different viewpoint / position 1808 of the scope camera. In some implementations, a Structure-from-Motion technique may be implemented to determine a three-dimensional location of the target anatomical feature 1806 based at least in part on the images 1801, 1802. Images 1801, 1802 may represent masked or otherwise analyzed / processed characterization images in which a target anatomical feature 1806 is defined as shown in Figure 18. Any computer vision algorithm or process may be implemented in connection with the localization and targeting processes disclosed herein. For example, such image processing processes may be relatively sophisticated contour detection processes, black box machine learning processes, and / or the like.

[0184] Three-dimensional (3D) position estimation for purposes of locating a target anatomical feature, according to aspects of the present disclosure, may be performed according to any suitable or desirable technique or mechanism. For example, in some embodiments, the distance between the endoscopic camera and the target anatomical feature may be estimated based on a representative size of the anatomical feature 1806 in the image.

[0185] In some embodiments, information regarding the angle of movement of the scope and / or anatomical features may be used to determine 3D location. For example, electromagnetic sensors / beacons in an electromagnetic field / space may provide such movement angle information. By combining the electromagnetic sensor data with image data, a mapping between distance from a target anatomical feature and the size of the target anatomical feature in a resulting image captured after movement of such distance may be used to estimate the depth / distance of the feature in subsequent images. In some embodiments, upon contacting a target anatomical feature (e.g., nipple), retracting the scope away from such feature, and parking the scope in a position that provides a desired field of view, the distance traveled may be recorded, for example, using the electromagnetic sensor data. Furthermore, subsequent images may provide information related to how large an anatomical feature appears in such images, and thus a relationship / mapping between feature size and distance may be determined and used for future location determination. Camera calibration as described herein may be implemented for such purposes. In some implementations, machine learning may be utilized to classify images and determine location information based on the size of features in such images.

[0186] In some embodiments, certain sensors associated with the medical instrument (e.g., scope) can be utilized to obtain the 3D location of the target. For example, structured illumination sensors and / or time-of-flight sensors can be used in determining the 3D positioning. According to some embodiments, a geometric translation approach can be implemented to detect the 3D location of the target anatomical feature. For example, potential images 1801, 1802 associated with distinct timestamps can be captured, as in certain other embodiments of the present disclosure. In association with such images, rotational translation information regarding the cameras (1807, 1808), which can be determined based on sensor information from any suitable or desired sensor or device, can be used to triangulate and / or determine the position of such images in 3D space, thereby providing information indicative of the 3D location of the target anatomical feature 1806 in 3D space. The rotational translation information can be based on robotic actuator movement and / or position sensor information, such as from an electromagnetic beacon device, associated with the camera and / or scope and indicative of the position of the camera in electromagnetic space.

[0187] Considering the intrinsic and extrinsic parameters of the cameras (1807, 1808) (principal point, focal length and distortion coefficients, relative motion), the 3D location of the target anatomical feature 1806 can be calculated based at least in part on the tracked target two-dimensional (2D) location on the images 1801, 1802. For the intrinsic parameters, the camera principal point and focal length may be considered. Additional data that may be considered may include radial and tangential distortion coefficients. Based on sensor measurements (e.g., robotic and / or EM based), extrinsic parameters may also be obtained, including the rotation R and translation T of the scope between where the two images were taken. For convenience, K may be expressed as a matrix containing the intrinsic parameters, and H may be the matrix containing the camera position (C) of the first image. t ) and the camera position of the second image (C t+1 ) can be represented as a 4x4 matrix containing the extrinsic rotations and translations between

[0188] For Ct, the 3D and 2D projection relationship is x t=KX, where X is the 3D coordinate wrt. C t and x t is the 2D coordinate on image t (detected centroid of the target), where K is a 3×4 matrix that can be expressed as:

[0189]

number

[0190] Similarly, C t+1 About x t+1 = K'X,

[0191]

number

[0192] x t and KX are parallel vectors, so x t ×KX=0, and similarly, x t+1 ×K'X=0, where "×" is the cross product operator. Therefore,

[0193]

number

[0194] The above is i(v t K (3) XK (2) X)-j(u t K (3) XK (1) X)+k(u t K (2) Xv t K (1) X)=0, where u t and v t x t are the 2D coordinates of . Therefore, v t K (3) XK(2) X=0 u t K (3) XK (1) X=0 u t K (2) Xv t K (1) X=0

[0195] Here, only the first two equations may be needed since the third equation is a linear combination of the first two. Similarly, C t+1 For , we can obtain the following two equations: v t+1 K' (3) X-K' (2) X=0 u t+1 K' (3) X-K' (1) X=0

[0196] C t and C t+1 After stacking the equations, the following can be generated: AX=0 where A is a 4×4 matrix.

[0197]

number

[0198] Since the elements in A are known (detected 2D coordinates, intrinsic and extrinsic parameters), X can be computed by performing a singular value decomposition (SVD) on A. A=UΣV T The last column of V is the solution for X.

[0199] Thus, in view of the foregoing disclosure, various inventive concepts disclosed herein may be utilized to perform automatic target detection, target tracking, and / or automatic target localization, such as three-dimensional position estimation. In some embodiments, aspects of the present disclosure advantageously enable target anatomical feature tracking without requiring physical contact with the target anatomical feature, which may facilitate improved ergonomics of ureteroscope use.

[0200] In some implementations, structured light and / or other non-contact light sensing mechanisms such as optical coherence tomography or other interferometry may be employed and used to determine depth / offset information. Such techniques may advantageously provide 3D papilla / cupule position information. However, structured-like devices may be relatively large and may increase the profile of the ureteroscope configured therewith. In some embodiments, the ureteroscope includes a Time-of-Flight camera configured to admit fight light and receive its reflection, and the time between emission and reception of light may be used to determine distance information within the kidney.

[0201] Distance / Angle Measurement Tool 19 illustrates a distal end of an endoscope 1940, such as a ureteroscope, equipped with an offset distance measuring tool 1970, according to one or more embodiments of the present disclosure. For example, the offset distance measuring tool 1970 may be a ruler-type distance measuring tool and / or an angle measuring tool, and may be configured to extend from a working channel 1944 of the scope 1940. The extension / protrusion of the measuring tool 1970 from the distal end of the scope 1940 may provide a mechanism for estimating the distance from the scope to the nipple for purposes of nipple localization when the measuring tool 1970 is predicted to be sufficiently far away to contact and / or be in close proximity to the target nipple.

[0202] In some embodiments, the measurement tool 1970 includes certain visual markings associated with a distal and / or proximal portion (not shown) of the measurement tool 1970. Such markings may indicate the amount of distance / length L of the measurement tool 1970 that protrudes beyond the distal end of the camera 1948 and / or scope 1940. While a marking 1971 is shown at the distal end of the tool 1970 that may have any suitable or desirable form, in some embodiments the markings are not present at the distal end of the tool 1970, but rather are associated with a portion of the tool that is proximal to the depicted portion of the scope 1940. For example, the markings may be associated with a portion of the tool 1970 that may be exposed proximally to the scope 1940, such that relative movement of the tool 1970 proximal to the scope 1940 can indicate the extent to which the tool 1970 protrudes from the distal end of the scope 1940. In some embodiments, the markings may be made manually at the proximal and / or distal portions of the measurement tool 1970 prior to extension from the distal end of the scope 1940 and / or after contact of the tool 1970 with the target anatomical feature.

[0203] In some implementations, the electromagnetic sensor may be disposed in and / or otherwise attached or coupled to the target anatomical feature (e.g., nipple). For example, an electromagnetic visible wire may be guided through the scope 1940 (e.g., within the working channel 1944) and at least partially embedded in the target anatomical feature, such that the electromagnetic sensor / beacon feature may maintain contact with or proximity to the target anatomical feature throughout the targeting / localization process, thereby providing accurate real-time targeting for percutaneous access. In some embodiments, the electromagnetic sensor / beacon device may be coupled to the target anatomical feature using an adhesive or other attachment means. In some embodiments, a conductive adhesive may be applied to the target anatomical feature, the conductive adhesive itself acting as a visible electromagnetic material in the electromagnetic field space. In some embodiments, a balloon-type device may be at least partially implanted / embedded in the target anatomical feature (e.g., nipple), and the balloon may be configured to expand within the target anatomical feature, thereby anchoring itself to the feature. Such balloon-type electromagnetic sensor / beacons, or other expandable mechanical electromagnetic sensor / beacon devices, may advantageously provide a relatively large target for a percutaneous access instrument (e.g., needle) to a target. Embodiments of the electromagnetic sensor / beacon device configured to be at least partially coupled to and / or embedded in a target anatomical feature may be permitted / prone to move in conjunction with movement and / or deformation of the target anatomical feature, thereby providing a relatively robust localization / targeting solution with respect to deformation and / or movement of the target anatomical feature.

[0204] Additional Embodiments Depending on the embodiment, certain acts, events, or functions of any of the processes or algorithms described herein may be performed in a different order, may be added, merged, or omitted entirely, and thus, in a particular embodiment, not all described acts or events may be required to practice a process.

[0205] In particular, hypothetical language used herein, such as "can," "could," "might," "may," "eg," and the like, is intended in its ordinary sense unless specifically stated otherwise or understood otherwise within the context in which it is used, and is intended to convey, generally, that certain embodiments include certain features, elements, and / or steps, while other embodiments do not. Thus, such hypothetical language is not intended to imply that features, elements, and / or steps are generally required in any way for one or more embodiments, or that one or more embodiments necessarily include logic for determining whether those features, elements, and / or steps are included or performed in any particular embodiment, with or without author input or prompting. Terms such as "comprising," "including," "having," and the like, are used in their ordinary sense and are used inclusively in a non-limiting manner and do not exclude additional elements, features, acts, operations, etc. Also, when the term "or" is used, for example, to connect a list of elements, the term "or" is used in its inclusive sense (and not its exclusive sense) to mean one, some, or all of the listed elements. Unless specifically stated otherwise, connective language such as the phrase "at least one of X, Y, and Z" is understood in the context as it is commonly used to convey that an item, term, element, etc. can be either X, Y, or Z. Thus, such connective language is generally not intended to imply that a particular embodiment requires that at least one of X, at least one of Y, and at least one of Z, respectively, be present.

[0206] In the above description of the embodiments, it should be understood that various features are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the method of disclosure should not be interpreted as reflecting an intention that any claim requires more features than are expressly recited in that claim. Moreover, any component, feature, or step illustrated and / or described in a particular embodiment herein may be applied to or used with any other embodiment. Moreover, no component, feature, step, or group of components, features, or steps is necessary or essential for each embodiment. Thus, it is intended that the scope of the invention disclosed herein and hereinafter claimed should not be limited by the specific embodiments described above, but should be determined solely by a fair reading of the following claims.

[0207] It should be understood that certain ordinal terms (e.g., "first" or "second") may be provided for ease of reference and do not necessarily imply physical characteristics or ordering. Thus, as used herein, ordinal terms (e.g., "first," "second," "third," etc.) used to modify elements, such as structures, components, operations, etc., do not necessarily indicate a priority or order of the element relative to any other elements, but rather may generally distinguish an element from another element having a similar or identical name (apart from the use of the ordinal term). Furthermore, as used herein, the indefinite articles ("a" and "an") may indicate "one or more" rather than "one." Furthermore, an operation performed "based on" a condition or event may also be performed based on one or more other conditions or events not expressly recited.

[0208] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the exemplary embodiments belong. It is further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning consistent with their meaning in the context of the relevant art, and should not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0209] The spatially relative terms "outer", "inner", "upper", "lower", "below", "upper", "vertical", "horizontal", and similar terms may be used herein for ease of description to describe the relationship between one element or component and another element or component as illustrated in the drawings. It should be understood that the spatially relative terms are intended to encompass different orientations of the device during use or operation in addition to the orientation depicted in the drawings. For example, if the device shown in the drawings is inverted, a device positioned "below" or "under" another device may be positioned "above" the other device. Thus, the exemplary term "lower" may include both lower and upper positions. The device may also be oriented in other directions, and thus the spatially relative terms may be interpreted differently depending on the orientation.

[0210] Unless otherwise specified, comparative and / or quantitative terms such as "less," "more," "greater than," and the like are intended to encompass the notion of equality. For example, "less" can mean not only "less than" in the strict mathematical sense, but also "less than or equal to."

[0211] [Embodiment] (1) A method for locating a target nipple, comprising: advancing a ureteroscope through at least a portion of a patient's urinary tract to a target calyx of a kidney of the patient; determining a position offset between one or more position sensors associated with the ureteroscope and a target papilla of the kidney that is at least partially exposed within the target calyx; and determining a percutaneous access target based at least in part on one or more of the current positions of the one or more position sensors and the offset. (2) The method of embodiment 1, further comprising advancing a percutaneous access device to the target cup by targeting the percutaneous access target. (3) contacting the target papilla with the ureteroscope; recording a position of the ureteroscope relative to said contacting; retracting the ureteroscope away from the target papilla; 2. The method of claim 1, further comprising: parking the ureteroscope at an offset position related to the position offset. (4) The method of claim 1, wherein the position offset represents at least five degrees of freedom. (5) A method for positioning a surgical instrument, comprising: advancing a medical instrument to a treatment site on a patient, the medical instrument comprising a camera; generating a real-time video of the treatment site using the camera of the medical instrument; displaying a user interface including the real-time video in a window of the user interface; and projecting an anatomical feature targeting icon at the center of the window of the user interface.

[0212] (6) The method of embodiment 5, wherein the targeting icon includes a crosshair. (7) The method of embodiment 5, further comprising manipulating the medical instrument to center the targeting icon over the representation of the target anatomical feature in the real-time video. (8) The method of claim 7, further comprising projecting one or more boundary features within the window of the user interface approximately at the center of the window, the one or more boundary features having a size independent of a position of the medical instrument. (9) The method of embodiment 8, further comprising manipulating the medical instrument to bring the representation of the target anatomical feature within the one or more boundary features. (10) The method of embodiment 9, wherein manipulating the medical instrument to bring the representation of the target anatomical feature within the one or more boundary features involves retracting the medical instrument away from the target anatomical feature such that the representation of the target anatomical feature shrinks within the window of the user interface.

[0213] (11) The method of claim 8, wherein the one or more boundary features have at least a partial box shape. (12) receiving sensor data indicative of a three-dimensional position of a percutaneous access needle within an electromagnetic field; determining a position of the distal end of the needle relative to the camera based at least in part on the sensor data; 6. The method of claim 5, further comprising: displaying a needle prediction icon within the window of the user interface indicating a position of the distal end of the needle relative to the real-time video. (13) The method of claim 12, further comprising determining that the position of the distal end of the needle is outside the window of the user interface, and the needle prediction icon indicates a direction of the position of the distal end of the needle relative to the window. (14) The method of claim 12, further comprising manipulating the medical instrument to center the needle prediction icon within the window of the user interface. (15) The method of claim 12, further comprising calibrating a sensor associated with the needle within the image space of the camera.

[0214] (16) The method of claim 12, further comprising modifying a configuration of the needle prediction icon in response to approach of the distal end of the needle to the medical instrument. (17) The method of claim 12, wherein the configuration of the needle prediction icon indicates the distance of the distal end of the needle from the medical instrument. (18) A method for targeting an anatomical feature, comprising: advancing an endoscope into a target anatomical lumen of a patient, the endoscope comprising a position sensor associated with a distal end of the endoscope; recording position data associated with a plurality of positions of the endoscope within the target anatomical lumen using the position sensor; estimating a surface of the target anatomical lumen based at least in part on the position data; and determining an axis of the target anatomical lumen based at least in part on the estimated surface of the target anatomical lumen. (19) The method of embodiment 18, further comprising targeting the target anatomical lumen with a percutaneous access needle based at least in part on the determined axis of the target anatomical lumen. (20) The method of embodiment 19, wherein targeting the target anatomical lumen involves advancing a percutaneous access needle along a path that is substantially parallel to the determined axis of the target anatomical lumen.

[0215] (21) The position sensor is an electromagnetic sensor device; 20. The method of claim 18, wherein recording the position data is performed using an electromagnetic field generator disposed at least in part external to the patient. (22) The method of embodiment 18, wherein estimating the surface of the target anatomical lumen involves interpolating the position data. (23) The method of embodiment 18, wherein determining the axis of the target anatomical lumen involves determining a plurality of surface normal vectors associated with the estimated surface of the target anatomical lumen. (24) The method of embodiment 23, further comprising averaging the multiple surface normal vectors. (25) The method of embodiment 18, wherein determining the axis of the target anatomical lumen is based at least in part on one or more of a map of the target anatomical lumen and a trajectory of the endoscope.

[0216] (26) A health care system, comprising: an endoscope configured to access a target anatomical lumen of a patient, the endoscope having a camera and an electromagnetic position sensor associated with a distal end thereof; a communications interface configured to receive video data from the endoscope; an electronic display device; a control circuit communicatively coupled to the communication interface and to the electronic display device; and receiving real-time video data of an internal treatment site of the patient from the endoscope of the communication interface; displaying the user interface on the electronic display, the user interface including the real-time video within a window of the user interface; A control circuit configured to cause an anatomical feature targeting icon to be displayed in the center of the window of the user interface. (27) The medical system of embodiment 26, wherein the targeting icon includes a crosshair. (28) The medical system of embodiment 26, wherein the control circuitry is further configured to cause one or more boundary features to be displayed within the window of the user interface approximately at a center of the window. (29) The medical system of embodiment 28, wherein the size of the one or more boundary features relative to the representation of the target anatomical feature in the real-time video is based on the distance of the target anatomical feature from the camera of the endoscope. (30) The medical system of embodiment 29, wherein the one or more boundary features have at least a partial box shape.

[0217] (31) The control circuit receiving sensor data indicative of a three-dimensional position of a percutaneous access needle within an electromagnetic field; determining a position of the distal end of the needle relative to the endoscope based at least in part on the sensor data; further configured to display a needle prediction icon within the window of the user interface; 27. The medical system of claim 26, wherein the needle prediction icon indicates the position of the distal end of the needle relative to the real-time video. (32) A computing device comprising: An endoscope interface; A control circuit including one or more processors and one or more data storage devices, receiving position data from an endoscope disposed within a target anatomical lumen of a patient, the position data indicating a plurality of positions of a position sensor associated with a distal end of the endoscope; estimating a surface of the target anatomical lumen based at least in part on the position data; and a control circuit configured to determine an axis of the target anatomical lumen based at least in part on the estimated surface of the target anatomical lumen. (33) The computing device of embodiment 32, wherein the control circuitry is configured to estimate the surface of the target anatomical lumen by at least partially interpolating the position data. (34) The computing device of claim 32, wherein the control circuitry is configured to determine the axis of the target anatomical lumen by at least partially determining a plurality of surface normal vectors associated with the estimated surface of the target anatomical lumen.

Claims

1. A method for locating a target nipple, implemented by a control system, comprising: the control system determining a position offset between one or more position sensors associated with a ureteroscope advanced through at least a portion of the urinary tract to a target calyx of a kidney and the target papilla of the kidney at least partially exposed within the target calyx; and the control system determining a percutaneous access target based at least in part on one or more of the current positions of the one or more position sensors or the position offset.

2. The control system determines a position of the ureteroscope relative to contacting the target papilla; and The method performed by the control system of claim 1, further comprising: in response to the control system retracting the ureteroscope away from the target papilla, determining that the ureteroscope is parked at an offset position associated with the position offset.

3. A method performed by the control system of claim 1, wherein the control system determines the position offset based on the position and orientation of the target papilla and the position and orientation of the ureteroscope.

4. A computer-readable medium that, when executed by one or more processors of a control system, causes the control system to: determining a position offset between one or more position sensors associated with a ureteroscope advanced through at least a portion of the urinary tract to a target calyx of the kidney and a target papilla of the kidney at least partially exposed within the target calyx; determining a percutaneous access target based at least in part on one or more of the current positions of the one or more position sensors or the position offset; A computer-readable medium storing instructions for executing the method.

5. Executing the instructions further comprises: determining a position of the ureteroscope relative to contacting the target papilla; 5. The computer readable medium of claim 4, further comprising: determining that the ureteroscope is parked at an offset position associated with the position offset in response to retracting the ureteroscope away from the target papilla.

6. The computer-readable medium of claim 4, wherein the control system determines the position offset based on the position and orientation of the target papilla and the position and orientation of the ureteroscope.

Images